Skip to content

Commit 8cb1245

Browse files
grandchildricegrandchildrice
committed
docs: add preliminary advisory selection analysis
1 parent c81eecb commit 8cb1245

7 files changed

Lines changed: 736 additions & 2 deletions

File tree

‎docs/paper/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,9 @@ The working thesis is:
1313

1414
1. [`snapshot_audit.md`](snapshot_audit.md) freezes the current dataset and
1515
resolves definition and documentation mismatches before hypothesis testing.
16-
2. Advisory-selection bias: compare advisory-linked and non-advisory records
17-
across protocol area, root cause, attack path, client, language, and layer.
16+
2. [`advisory_bias_preliminary.md`](advisory_bias_preliminary.md) runs the
17+
first advisory-selection analysis and identifies dependency/tooling
18+
contamination that must be reviewed before interpreting protocol labels.
1819
3. CWE comparison: test how much protocol context adds beyond generic weakness
1920
labels when explaining network impact.
2021
4. Prior-work comparison: replicate and extend MineBlockVuln (ESEC/FSE 2022)

‎docs/paper/advisory_bias_preliminary.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# RQ1 preliminary: what gets an advisory identifier?
2+
3+
**Question:** Are advisory-linked records merely a smaller sample, or a
4+
distributionally different sample of Ethereum-client security work?
5+
6+
**Status:** exploratory result; the manual scope review is not complete.
7+
8+
The analysis uses the broad definition established in
9+
[`snapshot_audit.md`](snapshot_audit.md): a row is advisory-linked when a
10+
case-insensitive CVE, GHSA, or RustSec identifier occurs in any provenance
11+
field. Results are generated in:
12+
13+
- [`tables/advisory_prevalence.csv`](tables/advisory_prevalence.csv)
14+
- [`tables/advisory_bias_global.csv`](tables/advisory_bias_global.csv)
15+
- [`tables/advisory_bias_categories.csv`](tables/advisory_bias_categories.csv)
16+
- [`tables/advisory_review_queue.csv`](tables/advisory_review_queue.csv)
17+
18+
## 1. Prevalence depends on the evidence population
19+
20+
| Population | Rows | Advisory-ID rows | Share |
21+
|---|---:|---:|---:|
22+
| A_authoritative only | 235 | 144 | 61.3% |
23+
| A_authoritative ∪ B_corroborated | 1,808 | 152 | 8.4% |
24+
| All tiers | 2,225 | 172 | 7.7% |
25+
26+
The A-only percentage is descriptive but circular: an advisory identifier is
27+
one of the signals used to assign A_authoritative. The defensible populations
28+
for selection analysis are A∪B and all tiers.
29+
30+
## 2. Raw associations are large, but not yet protocol findings
31+
32+
For each category with at least 20 rows, the script computes:
33+
34+
- advisory prevalence;
35+
- a one-category-versus-rest odds ratio with Haldane–Anscombe correction;
36+
- a Wald 95% confidence interval;
37+
- a two-sided normal-approximation p-value;
38+
- Benjamini–Hochberg correction within each population/dimension.
39+
40+
Global association is summarized with Pearson chi-square and uncorrected
41+
Cramér's V.
42+
43+
For all tiers, protocol label has the largest raw association with advisory
44+
presence (V=0.380), followed by attack path (V=0.189), root cause (V=0.182),
45+
client (V=0.162), language (V=0.106), and layer (V=0.032). The A∪B results are
46+
similar: label V=0.388, attack path V=0.212, and root cause V=0.205.
47+
48+
Selected all-tier associations are:
49+
50+
| Dimension | Category | Advisory share | Odds ratio | 95% CI |
51+
|---|---|---:|---:|---:|
52+
| label | crypto | 40.0% | 8.73 | 4.58–16.65 |
53+
| label | p2p-interface | 27.6% | 6.03 | 4.15–8.75 |
54+
| label | build-ci | 26.2% | 4.89 | 3.06–7.80 |
55+
| label | p2p | 23.2% | 3.98 | 2.34–6.79 |
56+
| label | state-trie | 1.4% | 0.18 | 0.06–0.53 |
57+
| root cause | crypto_misuse | 38.7% | 8.13 | 3.92–16.83 |
58+
| root cause | race_condition | 1.4% | 0.18 | 0.06–0.52 |
59+
| attack path | malformed_input | 11.9% | 2.68 | 1.94–3.70 |
60+
| attack path | crafted_state | 1.9% | 0.21 | 0.10–0.42 |
61+
| attack path | malicious_attestation | 1.1% | 0.16 | 0.05–0.56 |
62+
63+
All listed categories remain below q=0.05 in the exploratory within-dimension
64+
tests.
65+
66+
## 3. The first validity gate fails: product scope is mixed
67+
68+
The 172 advisory-linked rows do not form one coherent population. They mix:
69+
70+
1. direct vulnerabilities in Ethereum-client code;
71+
2. upstream dependency CVEs and RustSec advisories;
72+
3. documentation, CI, test, and developer-tool dependency updates.
73+
74+
A conservative automated triage of the 172 rows currently suggests:
75+
76+
| Suggested scope | Rows | Share |
77+
|---|---:|---:|
78+
| dependency_or_tooling | 117 | 68.0% |
79+
| client_implementation | 46 | 26.7% |
80+
| needs_manual_review | 9 | 5.2% |
81+
82+
These are review suggestions, not final labels. The checked-in review queue
83+
contains blank `reviewed_scope` and `review_notes` columns for human validation.
84+
85+
This mixture materially affects the raw result. For example, many dependency
86+
updates in documentation or JavaScript tooling carry a protocol-looking
87+
`p2p-interface` label. The raw p2p-interface odds ratio therefore cannot be
88+
interpreted as evidence that protocol P2P bugs are more likely to receive a CVE.
89+
Similarly, language and client associations partly reflect dependency managers
90+
and disclosure practices rather than vulnerability incidence.
91+
92+
**Observation.** In the current corpus, the public advisory-identifier view is
93+
dominated by software-supply-chain and tooling work, while the non-advisory view
94+
contains most historical client-fix candidates.
95+
96+
**Hypothesis.** After scope validation, direct client advisories will still
97+
overrepresent remotely triggered crash/DoS classes and underrepresent
98+
state-dependent, concurrency, and consensus-edge-case fixes.
99+
100+
**Do not claim yet.** The raw odds ratios above do not establish that CVE
101+
assignment is caused by protocol area, root cause, client, or language.
102+
103+
## 4. Revised comparison design
104+
105+
The publication analysis should compare three groups rather than binary
106+
“CVE versus no CVE”:
107+
108+
1. **Direct client advisory** — a CVE/GHSA describing the Ethereum client
109+
implementation itself.
110+
2. **Upstream dependency/tooling advisory** — a library, build, docs, CI, or
111+
test dependency.
112+
3. **Non-advisory client fix** — a historical client-code fix with no public
113+
advisory identifier.
114+
115+
After manual review:
116+
117+
- rerun odds ratios and effect sizes using the reviewed scope;
118+
- exclude dependency/tooling rows from protocol-area inference;
119+
- add client fixed effects so disclosure-policy differences do not masquerade
120+
as bug-type differences;
121+
- report A∪B as the primary population and all tiers as sensitivity;
122+
- keep A-only descriptive because it is selected partly by advisory evidence.
123+
124+
## 5. Research contribution exposed by the failed naive analysis
125+
126+
The contamination is not merely a cleaning nuisance. It supports a sharper
127+
question:
128+
129+
> Does the CVE ecosystem around Ethereum repositories describe protocol risk,
130+
> or mostly the inherited software supply chain?
131+
132+
If validated, the paper can show two distinct blind spots:
133+
134+
- **scope bias:** public identifiers disproportionately describe dependencies
135+
rather than Ethereum protocol implementation failures;
136+
- **type bias:** within direct client vulnerabilities, public advisories may
137+
favor easily described remote DoS over silent determinism and state-machine
138+
edge cases.
139+
140+
This distinction is stronger and more reproducible than treating every CVE
141+
mention in an Ethereum repository as an Ethereum-client vulnerability.

0 commit comments

Comments
 (0)