Everything you need to understand, run, and extend devops-bench. New here? Start with Getting started, then skim the architecture and glossary.
- Getting started — set up your dev environment, understand how evals run, and meet the skills in this repo.
- Run evals — single runs and parallel matrices.
- Add a task — author a new benchmark task.
- Add a model provider — wire up a new LLM backend.
- Add an agent harness — plug in a new agent under test.
- Architecture — the eval lifecycle and how the components fit together.
- Glossary — the vocabulary, plus an annotated map of the codebase.
- Model providers — the LLMs that power agents and judges, and how to configure them.
- Infrastructure — deployers and cloud providers, and how to configure infra for a task.
- Agents — the agent harnesses under test and their capabilities.
- Metrics — the scoring framework and how to read results.
- Known issues — a recovery router for eval failures, plus the current known hacks.
- Roadmap — current initiatives and how to get involved.
- Agent skills — operational skills and shared references for coding agents working on this repo.