Commit 36daf3f
committed
docs(packs/atr): correct the benchmark figures and their caveats
The three numbers in this pack's README and in pack.yaml's description were
written when the pack was first contributed and were never revisited. Upstream
has since re-measured, and two of the three are wrong in a direction that
overstates the pack.
garak, in-the-wild jailbreaks 97.2% -> 92.5% recall (650 samples, ATR 3.5.11)
PINT 99.7% precision on 850 samples, stated in a way
that reads as a run of Lakera's PINT benchmark
SKILL.md 100% recall, stated without the lane it was
measured in
The PINT line is the one worth fixing carefully. Lakera's official PINT corpus
is private and roughly five times larger; upstream's is a self-built corpus in
the same format, and 226 of its 272 detections come from one rule
(ATR-2026-00001). Presenting it as a PINT result implies a third-party
evaluation that did not happen.
The SKILL.md 100% holds for this pack, which loads the full static ruleset, but
not for an ATR enforce-lane deployment: that lane loads only maturity:stable and
all 38 skill-scanning rules are maturity:test, so it detects 0 of 32 malicious
samples. Worth stating so a reader does not carry the number somewhere it is
false.
OWASP Agentic Top 10 10/10 is a taxonomy mapping rather than a detection
benchmark, so it moves out of the Benchmarks line.
Rather than replace one set of hand-copied numbers with another, the README now
links the upstream benchmark table, which is version-pinned and regenerated from
measurement files, and reproduces only the rows that apply here. Upstream added
a citation check on 2026-08-04 that fails when a quoted figure has no
measurement file behind it, which is what let this drift in the first place.1 parent 48f5934 commit 36daf3f
2 files changed
Lines changed: 28 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
8 | 34 | | |
9 | 35 | | |
10 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
0 commit comments