Skip to content

Docs/cybergym cleanups #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sklinglernv merged 4 commits into from
Jul 31, 2026
Merged

Docs/cybergym cleanups #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
23c3f2e
docs: cosmetics
nv-pberard Jul 30, 2026
995bc20
docs: clarified "non-deterministic" checks
nv-pberard Jul 30, 2026
9ae46ed
docs: clarified that the model brings cybersecurity domain knowledge
nv-pberard Jul 30, 2026
0991bc3
docs: naming capitalization
nv-pberard Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 20 additions & 23 deletions examples/cybergym/Technical_Report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# NOOA CyberGym

**Contact:** TODO: Shall we add this?
<!-- **Contact:** TODO -->

## 1. Overview

This submission evaluates an agent built on [**NVIDIA-labs Object-Oriented Agents (NOOA)**](https://github.com/NVIDIA-NeMo/labs-OO-Agents) on the **CyberGym Level 1** benchmark ([cybergym.io](https://www.cybergym.io/cybergym/)), where the agent gets a vulnerability description plus the pre-patch codebase and must produce a proof-of-concept input that crashes the pre-patch binary but not the patched one.

Our CyberGym agent runs as a CodeAct agent with a shell and a todo manager, surveying the mounted source to locate the vulnerable function and author a minimal PoC. No cybersecurity domain knowledge or benchmark-specific hints are supplied.
Our CyberGym agent runs as a CodeAct agent with a shell and a todo manager, surveying the mounted source to locate the vulnerable function and author a minimal PoC. No cybersecurity domain knowledge or benchmark-specific hints are supplied beyond what the base model already brings from pretraining.

The underlying model is **OpenAI GPT-5.5** with reasoning effort set to `xhigh`.

Expand All @@ -20,20 +20,18 @@ NVIDIA-labs Object-Oriented Agents (NOOA) is a model-agnostic, open-source Pytho

The design unifies six model-facing ideas: typed input/output, pass by reference to live Python objects, code as action, programmable orchestration loops, explicit typed object state, and model-callable harness APIs.

Code: [NVIDIA-labs Object-Oriented Agents (NOOA)](https://github.com/NVIDIA-NeMo/labs-OO-Agents).
Paper: [NVIDIA-labs OO Agents: Native Python Object-Oriented Agents](https://arxiv.org/abs/2607.20709).
* Code: [NVIDIA-labs Object-Oriented Agents (NOOA)](https://github.com/NVIDIA-NeMo/labs-OO-Agents).
* Paper: [NVIDIA-labs OO Agents: Native Python Object-Oriented Agents](https://arxiv.org/abs/2607.20709).

### 2.2 NOOA CyberGym Agent

The NOOA CyberGym agent runs inside each trial container as a CodeAct agent that has full access to a Python runtime and is equipped with two additional tools: a shell (file search, source inspection, and command execution over the mounted codebase) and a todo manager for tracking multi-step work. On each task it reads the vulnerability description, surveys the mounted pre-patch source and build setup, identifies the vulnerable function and the input shape that reaches it, and authors a minimal, deterministic proof-of-concept, which it submits through the CyberGym submission interface.

A deterministic scoring layer wraps the agent and keeps the scoring logic out of the agent's context. A submission method sends the authored PoC, replays it against the sanitizer-instrumented vulnerable binary, and returns a typed outcome (crash, ambiguous crash, no crash, or timeout) rather than raw tool output. Before a submission is accepted, a lightweight single-turn judge confirms that the model's summary still targets the specific vulnerability class described in the task; a mismatch is fed back as structured feedback and the agent retries. Accepted PoCs are re-submitted a few times and are only kept if they crash reliably, rejecting non-deterministic crashes that would not survive server-side differential verification. A soft timeout well inside the harness limit returns the best crashing PoC found so far if the loop has not already converged.
A deterministic scoring layer wraps the agent and keeps the scoring logic out of the agent's context. A submission method sends the authored PoC, replays it against the sanitizer-instrumented vulnerable binary, and returns a typed outcome (crash, ambiguous crash, no crash, or timeout) rather than raw tool output. Before a submission is accepted, a lightweight single-turn judge confirms that the model's summary still targets the specific vulnerability class described in the task. On a mismatch, structured feedback is returned to the agent and it retries. Accepted PoCs are re-submitted three times and are only kept if they crash in at least two of the three replays, rejecting non-deterministic crashes that would not survive server-side differential verification. A soft timeout well inside the harness limit returns the best crashing PoC found so far if the loop has not already converged.

No cybersecurity domain knowledge, exploit templates, or benchmark-specific hints are supplied to the agent; the workflow above is generic vulnerability validation. Performance is therefore attributable to the agent architecture rather than to task-specific steering.
No cybersecurity domain knowledge, exploit templates, or benchmark-specific hints are supplied to the agent beyond what the base model already brings from pretraining; the workflow above is generic vulnerability validation. Performance is therefore attributable to the agent architecture and the underlying model rather than to task-specific steering.

<!-- TODO: Add GitHub link -->

Code: [`nooa_cybergym`](nooa_cybergym/main.py)
* Code: [NOOA CyberGym](nooa_cybergym/main.py)

## 3. Methodology

Expand All @@ -48,11 +46,10 @@ In the primary *Level 1* setting, agents receive a vulnerability description and
### 3.2 Agent Configuration

* **Agent framework**: NVIDIA-labs Object-Oriented Agents (NOOA)
* **Model**: OpenAI GPT-5.5 (`openai/azure/openai/gpt-5.5`)
* **Model**: OpenAI GPT-5.5
* **Reasoning effort**: `xhigh`
* **Tools**: Python runtime with shell + todo manager
* **Soft timeout**: 13,920 s (~3.87 h), returns best crashing PoC found so far
<!-- TODO: Double check soft timeout -->

### 3.3 Access to Vulnerable vs. Patched Builds

Expand Down Expand Up @@ -102,18 +99,18 @@ The token, cost, and timing figures below are per-trial averages over the valid

Top 9 published results on the CyberGym Level 1 leaderboard (one trial, sorted by success rate, as reported on [cybergym.io](https://www.cybergym.io/cybergym/), retrieved 2026-07-28).

| # | Submission | Model(s) | Score | Date | Source |
|----|----------------------------|---------------------------------------------|-----------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1 | Wiz Atlas | GPT-5.5, Claude Opus 4.6 | 90.9% | 2026-07-27 | [Wiz](https://www.wiz.io/blog/atlas-ai-vulnerability-researcher) |
| 2 | Crystalline | Claude Opus 4.6 | 89.6% | 2026-06-08 | [Independent researcher](https://github.com/synchopate/cybergym-logos) |
| 3 | MDASH | GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6 | 88.4% | 2026-05-12 | [Microsoft](https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/) |
| 4 | **NOOA CyberGym** | **GPT-5.5** | **86.8%** | 2026-07-28 | This work |
| 5 | Sangfor AI | GLM-5.2 | 86.3% | 2026-07-21 | [Sangfor AI](https://github.com/Sangfor-AI/cybergym-submission-sangfor-ai) |
| 6 | GPT-5.5-Cyber | GPT-5.5-Cyber (OpenAI Agent) | 85.6% | 2026-06-22 | [OpenAI](https://openai.com/index/daybreak-securing-the-world/) |
| 7 | Xuanwu Atuin AI | GLM-5.2 | 84.8% | 2026-07-22 | [Tencent Xuanwu Lab](https://xlab.tencent.com/en/2026/07/17/xuanwu-atuin-cybergym-glm52/) |
| 8 | Claude Mythos Preview | Claude Mythos Preview (Anthropic Agent) | 83.1% | 2026-04-07 | [Anthropic](https://www.anthropic.com/claude-mythos-preview-system-card) |
| 9 | GPT-5.5 | GPT-5.5 (OpenAI Agent) | 81.8% | 2026-04-23 | [OpenAI](https://openai.com/index/introducing-gpt-5-5) |
| 10 | GPT-5.4 | GPT-5.4 (OpenAI Agent) | 79.0% | 2026-04-23 | [OpenAI](https://openai.com/index/introducing-gpt-5-5) |
| # | Submission | Model(s) | Score | Date | Source |
|----|----------------------------|---------------------------------------------|-----------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1 | Wiz Atlas | GPT-5.5, Claude Opus 4.6 | 90.9% | 2026-07-27 | [Wiz](https://www.wiz.io/blog/atlas-ai-vulnerability-researcher) |
| 2 | Crystalline | Claude Opus 4.6 | 89.6% | 2026-06-08 | [Independent researcher](https://github.com/synchopate/cybergym-logos) |
| 3 | MDASH | GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6 | 88.4% | 2026-05-12 | [Microsoft](https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/) |
| 4 | **NOOA CyberGym** | **GPT-5.5** | **86.8%** | **2026-07-28** | **This work** |
| 5 | Sangfor AI | GLM-5.2 | 86.3% | 2026-07-21 | [Sangfor AI](https://github.com/Sangfor-AI/cybergym-submission-sangfor-ai) |
| 6 | GPT-5.5-Cyber | GPT-5.5-Cyber (OpenAI Agent) | 85.6% | 2026-06-22 | [OpenAI](https://openai.com/index/daybreak-securing-the-world/) |
| 7 | Xuanwu Atuin AI | GLM-5.2 | 84.8% | 2026-07-22 | [Tencent Xuanwu Lab](https://xlab.tencent.com/en/2026/07/17/xuanwu-atuin-cybergym-glm52/) |
| 8 | Claude Mythos Preview | Claude Mythos Preview (Anthropic Agent) | 83.1% | 2026-04-07 | [Anthropic](https://www.anthropic.com/claude-mythos-preview-system-card) |
| 9 | GPT-5.5 | GPT-5.5 (OpenAI Agent) | 81.8% | 2026-04-23 | [OpenAI](https://openai.com/index/introducing-gpt-5-5) |
| 10 | GPT-5.4 | GPT-5.4 (OpenAI Agent) | 79.0% | 2026-04-23 | [OpenAI](https://openai.com/index/introducing-gpt-5-5) |

## 5. Artifacts

Expand Down
2 changes: 1 addition & 1 deletion examples/cybergym/nooa_cybergym/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""CyberGym-native nooa agent package."""
"""CyberGym-native NOOA agent package."""

__all__ = ["__version__"]

Expand Down
2 changes: 1 addition & 1 deletion examples/cybergym/nooa_cybergym/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Native CyberGym runner for the nooa CyberGym agent."""
"""Native CyberGym runner for the NOOA CyberGym agent."""

from __future__ import annotations

Expand Down
Loading