Skip to content

Commit 36daf3f

Browse files
committed
docs(packs/atr): correct the benchmark figures and their caveats
The three numbers in this pack's README and in pack.yaml's description were written when the pack was first contributed and were never revisited. Upstream has since re-measured, and two of the three are wrong in a direction that overstates the pack. garak, in-the-wild jailbreaks 97.2% -> 92.5% recall (650 samples, ATR 3.5.11) PINT 99.7% precision on 850 samples, stated in a way that reads as a run of Lakera's PINT benchmark SKILL.md 100% recall, stated without the lane it was measured in The PINT line is the one worth fixing carefully. Lakera's official PINT corpus is private and roughly five times larger; upstream's is a self-built corpus in the same format, and 226 of its 272 detections come from one rule (ATR-2026-00001). Presenting it as a PINT result implies a third-party evaluation that did not happen. The SKILL.md 100% holds for this pack, which loads the full static ruleset, but not for an ATR enforce-lane deployment: that lane loads only maturity:stable and all 38 skill-scanning rules are maturity:test, so it detects 0 of 32 malicious samples. Worth stating so a reader does not carry the number somewhere it is false. OWASP Agentic Top 10 10/10 is a taxonomy mapping rather than a detection benchmark, so it moves out of the Benchmarks line. Rather than replace one set of hand-copied numbers with another, the README now links the upstream benchmark table, which is version-pinned and regenerated from measurement files, and reproduces only the rows that apply here. Upstream added a citation check on 2026-08-04 that fails when a quoted figure has no measurement file behind it, which is what let this drift in the first place.
1 parent 48f5934 commit 36daf3f

2 files changed

Lines changed: 28 additions & 2 deletions

File tree

skill_scanner/data/packs/atr/README.md

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,33 @@
44
**ATR Version:** 3.5.6
55
**Rules:** 712 signatures across 10 signature files
66
**License:** MIT
7-
**Benchmarks:** 97.2% recall on NVIDIA garak (650 in-the-wild jailbreaks) · 100% recall on 498 labeled SKILL.md samples · 99.7% precision on 850 PINT adversarial samples · OWASP Agentic Top 10: 10/10
7+
**Benchmarks:** see the [ATR benchmark table](https://github.com/Agent-Threat-Rule/agent-threat-rules#benchmarks),
8+
which is version-pinned and regenerated from measurement files rather than
9+
hand-copied. Selected rows, measured on ATR 3.5.11:
10+
11+
| Corpus | Samples | Recall | Precision |
12+
|---|---:|---:|---:|
13+
| NVIDIA garak, in-the-wild jailbreaks | 650 | 92.5% | 100% |
14+
| NVIDIA garak, all 23 probe families | 3,475 | 57.2% | 100% |
15+
| Labeled SKILL.md, internal | 498 | 100% | 97% |
16+
| PINT-format (deepset + Lakera Gandalf) | 850 | 60.3% | 100% |
17+
18+
Three caveats that belong next to those numbers.
19+
20+
The PINT-format row is **not** a run of Lakera's official PINT benchmark. That
21+
corpus is private and roughly five times larger. This is a self-built corpus in
22+
the same format, and 226 of its 272 detections come from a single rule
23+
(`ATR-2026-00001`), so it measures one rule family rather than overall coverage.
24+
25+
The SKILL.md row is measured with every maturity loaded. ATR's enforce lane
26+
loads only `maturity: stable`, and all 38 skill-scanning rules are currently
27+
`maturity: test` — so an enforce-lane deployment detects 0 of the 32 malicious
28+
samples. This pack loads the full static ruleset, which is the configuration the
29+
100% figure describes.
30+
31+
OWASP Agentic Top 10 coverage (10/10 categories, 1,179 mappings) is a taxonomy
32+
mapping, not a detection benchmark, and is listed separately upstream for that
33+
reason.
834

935
## Overview
1036

skill_scanner/data/packs/atr/pack.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ pack:
99
name: ATR Agent Threat Rules
1010
version: 3.5.6
1111
description: >-
12-
Open-source AI agent security detection rules from the Agent Threat Rules (ATR) project. 712 supported static rules across 10 attack categories: prompt injection, agent manipulation, skill compromise, context exfiltration, tool poisoning, privilege escalation, model abuse, excessive autonomy, data poisoning, and model security. Benchmarks: 97.2% recall on NVIDIA garak 650 in-the-wild jailbreaks, 100% recall on 498 labeled SKILL.md samples, 99.7% precision on 850 PINT adversarial samples. OWASP Agentic Top 10: 10/10. MIT license.
12+
Open-source AI agent security detection rules from the Agent Threat Rules (ATR) project. 712 supported static rules across 10 attack categories: prompt injection, agent manipulation, skill compromise, context exfiltration, tool poisoning, privilege escalation, model abuse, excessive autonomy, data poisoning, and model security. Benchmark results are published upstream in the ATR benchmark table, version-pinned per measurement; see the pack README for the figures that apply to this ruleset and their caveats. MIT license.
1313
author: ATR Community
1414
license: MIT
1515
source_url: https://github.com/Agent-Threat-Rule/agent-threat-rules

0 commit comments

Comments
 (0)