Add AgentThreatBench: OWASP Agentic Top 10 security evaluation suite (UK AISI)

[vgudur-dev]

Summary

AgentThreatBench is the first evaluation suite that operationalizes the OWASP Top 10 for Agentic Applications (2026) into executable tasks. It was merged into UKGovernmentBEIS/inspect_evals — the official evaluation suite of the UK AI Safety Institute (AISI).

What it measures

Three tasks targeting distinct agentic attack surfaces that are directly relevant to GPT-4o and o3 agent deployments:

Task	OWASP ID	Attack
Memory Poison	ASI06	Adversarial entries in agent memory/RAG store
Autonomy Hijack	ASI01	Indirect prompt injection via email tool output
Data Exfiltration	ASI01	Indirect injection → PII leak via send_message tool

Why this belongs in openai/evals

• Measures agentic security — a gap in the current openai/evals suite
• Dual-metric scoring (utility + security) measures whether safety improvements come at the cost of capability
• Maintained by UK AISI — credible, independent benchmark
• Directly relevant to OpenAI's Preparedness Framework and safety commitments

Proposal

Add AgentThreatBench as a reference eval in the openai/evals registry, or reference it in the evals documentation as a recommended agentic security benchmark.

Benchmark docs: https://ukgovernmentbeis.github.io/inspect_evals/evals/safeguards/agent_threat_bench/
Source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/agent_threat_bench
OWASP reference: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

[vgudur-dev]

Hi @etr2460 @kwhinnery-openai @maxb-openai — tagging you as the active maintainer(s) here to ask if you'd be willing to take a look at this request.

Quick context: openai/evals is the canonical registry for LLM evaluation tasks — AgentThreatBench is already part of the UK AI Safety Institute's official inspect_evals suite and provides 10 executable tasks covering the OWASP Agentic Top 10.

The ask: adding AgentThreatBench as a registered eval task, or a reference in the docs for users evaluating agentic security

AgentThreatBench is already merged into the UK AI Safety Institute's official inspect_evals suite and is installable via pip install inspect_evals. The benchmark covers 10 executable tasks aligned to the OWASP Top 10 for Agentic Applications 2025.

Happy to provide a prototype integration, a code snippet, or a draft docs PR — whatever would be most useful. Would love to know if this is something the team would consider.