Summary
When running DevOps Bench evaluations, evaluated agents with file system tools (or search capabilities) can scan workspace paths, discover task definitions (tasks/**/task.yaml), and read golden outputs directly.
Scope
- Execute agent subprocesses inside isolated ephemeral directories without mounting the
tasks/ directory.
- Prevent direct file discovery of golden manifests and evaluation keys.
Ref: Internal bug b/552484058 (b/533473646)
Summary
When running DevOps Bench evaluations, evaluated agents with file system tools (or search capabilities) can scan workspace paths, discover task definitions (
tasks/**/task.yaml), and read golden outputs directly.Scope
tasks/directory.Ref: Internal bug b/552484058 (b/533473646)