Skip to content

EVAL.md

Latest commit

 

History

History
65 lines (55 loc) · 2.55 KB

EVAL.md

File metadata and controls

65 lines (55 loc) · 2.55 KB
Raw

Eval results

Generated by npm run eval (KB-only+baseline mode) on 2026-06-22. Measured, not asserted — see server/eval/run.ts and server/eval/README.md.

Reading the numbers. Covered-vendor accuracy can be optimistic when labels and the KB share sources, so the bias-resistant signals are: held-out (never-in-KB) accuracy, the baseline comparison (does the KB beat a naive floor / a no-KB model?), the confusion matrix, and calibration. Label-independence discipline is documented in eval/README.md.

  • Cases: 17 vendors · 102 control labels
  • Overall accuracy: 82/102 = 80.4%
  • Held-out (never-in-KB) accuracy: 17/36 = 47.2% across 6 vendors — the generalization number (no KB curation for these).
  • KB coverage: 53/102 = 52.0% (controls with a non-unknown prediction)

Baseline comparison

How much does the KB / the model actually add over a naive floor? (Same cases for all rows.)

Predictor Accuracy (all) Accuracy (held-out) Coverage
naive (always-unknown) 29.4% 47.2% 0.0%
KB-only (deterministic) 80.4% 47.2% 52.0%
KB-verified-only (human facts) 47.1% 47.2% 17.6%

Per-control accuracy

Control Correct Accuracy
sso 11/17 64.7%
ulm 10/17 58.8%
entitlements 12/17 70.6%
riskSignals 17/17 100.0%
logout 17/17 100.0%
tokenRevocation 15/17 88.2%

Per-verdict precision / recall

Verdict Precision Recall Support
supported 100.0% 68.6% 35
partial 96.4% 75.0% 36
unsupported 100.0% 100.0% 1
unknown 61.2% 100.0% 30

Confusion matrix

expected ↓ \ predicted → supported partial unsupported unknown
supported 24 1 0 10
partial 0 27 0 9
unsupported 0 0 1 0
unknown 0 0 0 30

Confidence calibration

Does predicted confidence track actual accuracy? (Well-calibrated ⇒ accuracy ≈ avg confidence per bucket.)

Confidence bucket n Accuracy Avg confidence
0.00–0.50 32 100.0% 0.38
0.50–0.70 10 90.0% 0.53
0.70–0.85 17 100.0% 0.75
0.85–1.00 7 100.0% 0.88

Recent runs (trend)

Date Mode Accuracy Coverage
2026-06-21 KB-only 89.7% 67.9%
2026-06-22 KB-only+baseline 80.4% 52.0%
2026-06-22 KB-only+baseline 80.4% 52.0%
2026-06-22 KB-only+baseline 80.4% 52.0%