fix(judge): a bad --judge model no longer discards a completed run - #314
Merged
Perry2004 merged 4 commits intoAug 30, 2026
Merged
Conversation
load_model_config() called sys.exit(1) on an unknown or malformed model name. run.py loads the *judge* model inside the judge stage, which only executes after the agent has finished. SystemExit derives from BaseException, so it escaped both the judge stage's `except Exception` and run()'s top-level `except Exception`, and the process died before write_run_meta() ever ran. For a first-time user whose models.yaml lacks deepseek-v4-pro — the default judge — that meant burning a full agent run of up to 30 minutes, quite possibly a successful interception, and ending with no run-meta.json at all: invisible to batch stats, to clawbench-rescore (which globs run-meta.json), and to HF upload. Both asks in the issue are implemented: - load_model_config() raises ModelConfigError, a plain Exception, instead of calling sys.exit(). load_models_yaml() does the same, since load_model_config() calls it and a missing models.yaml would otherwise still exit from inside the judge stage. Every failure branch on that path — missing file, unknown model, illegal characters, missing required field, absent api key — is now catchable. The top-level call site that loads the agent's own model catches it and exits as before, since no output directory exists yet and there is nothing to record. The judge stage's existing except-Exception handler now catches it too, records judge_setup_failed in judge.json, and the run proceeds to write run-meta.json as normal. - The judge model is resolved at startup, before the agent runs, so a --judge typo costs seconds rather than half an hour. The judge stage reuses that validated config; the lazy load remains for --human runs, which skip startup validation. This holds the invariant the issue asks for: a completed agent run always writes its metadata, whatever happens at scoring time. Fixes TIGER-AI-Lab#302.
Perry2004
approved these changes
Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #302, implementing both asks.
load_model_config()calledsys.exit(1)on an unknown or malformed model name.run.pyloads the judge model inside the judge stage, which only executes after the agent has finished.SystemExitderives fromBaseException, so it escaped both the judge stage'sexcept Exception(run.py:606) andrun()'s top-levelexcept Exception(run.py:674), and the process died beforewrite_run_meta()ever ran.For a first-time user whose
models.yamllacksdeepseek-v4-pro— the default judge, whichmodels.example.yamlwarns "must exist here or the scoring stage fails" — that meant burning a full agent run of up to 30 minutes, quite possibly a successful interception, and ending with norun-meta.jsonat all: invisible to batch stats, toclawbench-rescore(which globsrun-meta.json), and to HF upload.Ask 1 — validate the judge at startup
The judge model is now resolved before the agent runs (
run.py:220). A--judgetypo costs seconds instead of half an hour. The judge stage reuses that validated config; the lazy load stays for--humanruns, which skip startup validation.Ask 2 — make the failure catchable
load_model_config()raisesModelConfigError, a plainException, instead of exiting.load_models_yaml()had to change too:load_model_config()calls it, so a missingmodels.yamlwould still have exited from inside the judge stage and lost the run — the same bug through a different door. Every failure branch on that path is now catchable, which I verified by exercising each one:models.yamlmissingSystemExitModelConfigErrorSystemExitModelConfigErrorSystemExitModelConfigErrorbase_url/api_typeSystemExitModelConfigErrorapi_keyorapi_keysSystemExitModelConfigErrorThe top-level call site that loads the agent's own model catches it and exits as before — no output directory exists yet, so there is nothing for a
run-meta.jsonto record. The judge stage's existingexcept Exceptionnow catches it too, recordsjudge_setup_failedinjudge.json, and the run proceeds to writerun-meta.jsonas normal.That holds the invariant the issue asks for: a completed agent run always writes its metadata, whatever happens at scoring time.
Corpus
Host-side runner change; no task data involved.
Test plan
tests/test_run_judge_stage.py, 4 tests, drivingrun.main()through a fully mocked agent run: a bad--judgeexits 1 at startup withdocker_runnever called and no run-meta written; a judge failure arising mid-run still writesrun-meta.jsoncarryingjudge_setup_failed;ModelConfigErroris catchable byexcept Exceptionand is not aSystemExit; and a missingmodels.yamlraises rather than exits, both directly and throughload_model_config().models.yaml, confirming each raises rather than exiting, and that the happy path still normalizesapi_keys: [k1, k2]toapi_key: k1.load_model_config: three, all inrun.py, all now guarded.batch.py:65calls its own localload_models_yaml, so it is unaffected by the change here.196 passed, 3 skipped. The one failure,test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], reproduces identically on a cleanmainon this machine — those task files are git symlinks (mode120000) that Windows checks out as text. Unrelated.ruff checkandruff format --checkclean.Not verified: this is proven through the mocked harness, not a live containerized run — I don't have Docker or judge API credentials on hand. The judge-stage behaviour is exercised by injection rather than by a real 30-minute run.
Related issues
Fixes #302.
Heads-up on overlap: #313 moves
load_models_yaml/load_model_configout ofrun_support/config.pyintoutils/model_config.py, and this PR rewrites those same functions in place. Each merges cleanly intomainon its own, butgit merge-treereports a conflict inconfig.pybetween the two. Happy to rebase whichever you'd like to land second.