Skip to content

Commit fedc303

Browse files
nripankadas07nripankadas07
committed
Strengthen PatchGym flagship positioning
1 parent 711b717 commit fedc303

5 files changed

Lines changed: 162 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
# Changelog
22

3-
## 0.1.0 - Unreleased
3+
## 0.1.1 - Unreleased
4+
5+
- Added a benchmark page for reviewer-oriented flagship context.
6+
- Expanded comparisons into a table against benchmarks, repo-to-prompt tools,
7+
coding agents, and test runners.
8+
- Added a 90-second walkthrough script for a recorded demo.
9+
10+
## 0.1.0 - 2026-05-25
411

512
- Initial public PatchGym implementation.
613
- Added local demo, task mining, hidden-test/oracle verification, agent runs,

‎README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,22 @@ agent 'bash .../examples/custom_agent/agent.sh' solved 1/1 task(s)
4040
PatchGym demo complete
4141
```
4242

43+
## Flagship Evidence
44+
45+
PatchGym is the flagship because it is a concrete evaluation loop, not just a
46+
demo wrapper. It mines real Git history, proves task validity with hidden tests
47+
and an oracle patch, runs an agent command in a fresh workspace, and writes
48+
auditable reports.
49+
50+
Reviewer-oriented entry points:
51+
52+
- [Benchmark page](docs/benchmark-page.md): what PatchGym measures, what it does
53+
not claim, and how the local benchmark invariant works.
54+
- [Comparisons](docs/comparisons.md): a table against public benchmarks,
55+
repo-to-prompt tools, coding agents, and plain test runners.
56+
- [90-second walkthrough script](docs/walkthrough-90-seconds.md): the short
57+
recorded-demo script for explaining the project quickly and honestly.
58+
4359
## How It Works
4460

4561
For each selected historical commit, PatchGym splits the change into:

‎docs/benchmark-page.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# PatchGym Benchmark Page
2+
3+
PatchGym answers one narrow question:
4+
5+
```text
6+
Can a coding agent fix tasks mined from this repository's own Git history,
7+
under this repository's tests and constraints?
8+
```
9+
10+
It is not a hosted leaderboard and it does not claim broad model superiority.
11+
It is a local benchmark harness for maintainers who want repeatable evidence
12+
before trusting agents on their own code.
13+
14+
## What It Measures
15+
16+
PatchGym measures whether an agent can edit a base repository snapshot so that
17+
hidden tests and the repository validation command pass.
18+
19+
For every mined task, the benchmark invariant is:
20+
21+
```text
22+
base commit + hidden tests fails
23+
base commit + hidden tests + oracle patch passes
24+
```
25+
26+
Only tasks that satisfy that invariant are useful. The invariant keeps the
27+
benchmark grounded: the base must fail, the known historical fix must pass, and
28+
the agent must produce a patch that survives the same hidden test boundary.
29+
30+
## What A Run Produces
31+
32+
A run produces local artifacts that can be inspected without a hosted service:
33+
34+
- task metadata;
35+
- hidden test patch;
36+
- oracle solution patch;
37+
- agent patch;
38+
- validation command result;
39+
- stdout and stderr captures;
40+
- JSON, Markdown, and HTML reports.
41+
42+
## Comparison Table
43+
44+
| Tool Type | Primary Job | Shared Public Score | Local Repo History | Hidden-Test Task Generation | Agent Execution |
45+
|---|---|---:|---:|---:|---:|
46+
| SWE-bench-style benchmarks | Compare agents on a shared public task set | Yes | No | Curated externally | Usually external harness |
47+
| Repo-to-prompt tools | Package repository context for a model | No | Yes | No | No |
48+
| Coding agents | Edit code in a workspace | No | Yes | No | Yes |
49+
| Plain test runners | Check a current codebase | No | Yes | No | No |
50+
| PatchGym | Build and run local coding-agent benchmark tasks | No | Yes | Yes | Yes |
51+
52+
PatchGym is intentionally smaller than public benchmark infrastructure. Its
53+
value is that it lets a maintainer evaluate agents against their own repository
54+
history with ordinary Git patches and local tests.
55+
56+
## Reproducible Demo
57+
58+
From a clean checkout:
59+
60+
```bash
61+
python3 -m venv .venv
62+
source .venv/bin/activate
63+
python -m pip install --upgrade pip
64+
python -m pip install -e ".[dev]"
65+
bash scripts/demo.sh
66+
```
67+
68+
Expected shape:
69+
70+
```text
71+
mined 1 task(s)
72+
built 1/1 valid task(s)
73+
agent 'bash .../examples/custom_agent/agent.sh' solved 1/1 task(s)
74+
PatchGym demo complete
75+
```
76+
77+
## Honest Boundaries
78+
79+
PatchGym does not provide strong sandboxing by default. Agent commands,
80+
validation commands, tests, and Git operations execute locally. For untrusted
81+
repositories or untrusted agents, run it inside a disposable container, VM, or
82+
separate machine.
83+
84+
PatchGym also does not claim package-registry publication, production adoption,
85+
hosted results, model rankings, or public leaderboard status.

‎docs/comparisons.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,18 @@
11
# Comparisons
22

3+
PatchGym is easiest to understand by what it is not. It is not trying to be the
4+
largest public benchmark, the smartest coding agent, or a repo summarizer. It is
5+
a local harness for turning a repository's own history into coding-agent repair
6+
tasks.
7+
8+
| Comparison | What They Optimize For | What PatchGym Optimizes For |
9+
|---|---|---|
10+
| SWE-bench-style public benchmarks | Shared public comparison across agents and models. | Local evaluation against a maintainer's own repository history. |
11+
| SWE-Gym / SWE-smith-style research environments | Large-scale benchmark generation, research workflows, or training/evaluation datasets. | A small readable reference harness that can be audited from source. |
12+
| Repo-to-prompt tools | Packaging repository context for a model. | Creating verifiable tasks, running agents, applying hidden tests, and writing reports. |
13+
| Coding agents | Producing code changes in a workspace. | Evaluating whether those changes survive hidden tests and validation commands. |
14+
| Plain test runners | Checking the current state of a codebase. | Replaying historical bug-fix tasks as agent-evaluation items. |
15+
316
## PatchGym vs SWE-bench
417

518
SWE-bench-style public benchmarks help compare agents on shared tasks. PatchGym

‎docs/walkthrough-90-seconds.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# 90-Second Walkthrough Script
2+
3+
Use this as the script for a short recorded demo.
4+
5+
## 0-15 Seconds: The Problem
6+
7+
Coding agents are easy to demo and hard to evaluate. Public benchmarks are
8+
useful, but they rarely answer the maintainer's practical question: can this
9+
agent fix my repository under my tests and constraints?
10+
11+
## 15-35 Seconds: The Idea
12+
13+
PatchGym turns Git history into local coding-agent benchmark tasks. A historical
14+
bug-fix commit often contains the base code, the tests that describe the desired
15+
behavior, and the patch that fixed the issue.
16+
17+
PatchGym splits that commit into hidden tests and an oracle solution.
18+
19+
## 35-55 Seconds: The Invariant
20+
21+
A task is valid only if:
22+
23+
```text
24+
base + hidden tests fails
25+
base + hidden tests + oracle patch passes
26+
```
27+
28+
That means the task actually grades behavior instead of just packaging a prompt.
29+
30+
## 55-75 Seconds: The Agent Run
31+
32+
PatchGym exports the base commit to a temporary workspace, runs an explicit
33+
agent command, captures the agent diff, applies hidden tests, runs validation,
34+
and writes JSON, Markdown, and HTML reports.
35+
36+
## 75-90 Seconds: The Boundary
37+
38+
PatchGym is local-first and inspectable. It is not a hosted leaderboard, not a
39+
model-ranking claim, and not a sandbox. It is a small evaluation harness for
40+
people who want evidence before trusting agents on their own code.

0 commit comments

Comments
 (0)