Skip to content

VALIDITY_PROOF.md

Latest commit

 

History

History
86 lines (62 loc) · 2.5 KB

VALIDITY_PROOF.md

File metadata and controls

86 lines (62 loc) · 2.5 KB
Raw

PatchGym Validity Proof

This proof uses the task generated by the kept local demo run.

Task id: 7b180f4c-fix-clamp-upper-bound-92d7b6b8

Base commit: 638b80e189a7ea04901f1934178858fe8a1a8c18

Solution commit: 7b180f4cc143ca13974d0d013e8a0e3daed7994a

Fixture repository: patchgym-demo-final/sample-repo

Task directory: patchgym-demo-final/tasks/7b180f4c-fix-clamp-upper-bound-92d7b6b8

Exact Command

TASK_ID="$(find patchgym-demo-final/tasks -mindepth 1 -maxdepth 1 -type d -exec basename {} \; | head -1)"
TASK_DIR="$(pwd)/patchgym-demo-final/tasks/$TASK_ID"
BASE_COMMIT="$(python3 -c 'import json,sys; from pathlib import Path; print(json.loads((Path(sys.argv[1]) / "task.json").read_text())["base_commit"])' "$TASK_DIR")"
TARGET_COMMIT="$(python3 -c 'import json,sys; from pathlib import Path; print(json.loads((Path(sys.argv[1]) / "task.json").read_text())["target_commit"])' "$TASK_DIR")"
PROOF_ROOT=/tmp/patchgym-hardening-validity
rm -rf "$PROOF_ROOT"
mkdir -p "$PROOF_ROOT/base" "$PROOF_ROOT/hidden" "$PROOF_ROOT/oracle"
for state in base hidden oracle; do
  git -C patchgym-demo-final/sample-repo archive "$BASE_COMMIT" | tar -x -C "$PROOF_ROOT/$state"
done
PYTHON_BIN="$(pwd)/.venv/bin/python"

cd "$PROOF_ROOT/base"
"$PYTHON_BIN" -m pytest -q

cd "$PROOF_ROOT/hidden"
git apply "$TASK_DIR/hidden_tests.patch"
"$PYTHON_BIN" -m pytest -q

cd "$PROOF_ROOT/oracle"
git apply "$TASK_DIR/hidden_tests.patch"
git apply "$TASK_DIR/oracle_solution.patch"
"$PYTHON_BIN" -m pytest -q

Observed Results

State A: base only

..                                                                       [100%]
2 passed in 0.00s
STATE_A_EXIT=0

State B: base + hidden tests

..F                                                                      [100%]
FAILED tests/test_calculator.py::test_clamp_upper_bound - assert 0 == 10
1 failed, 2 passed in 0.01s
STATE_B_EXIT=1

State C: base + hidden tests + oracle source patch

...                                                                      [100%]
3 passed in 0.00s
STATE_C_EXIT=0

Noop baseline:

Agent: noop
Score: 0/1

Oracle baseline:

Agent: oracle
Score: 1/1

Explanation

This is a valid coding-agent task because the base repository passes its original tests, the hidden regression test fails against the base implementation, and the historical oracle source patch makes the hidden test pass. The task prompt points to the likely source file and validation command, but it does not include the oracle patch.