src/clawbench/eval/rescore.py:203-217 ships a maintainer's personal machine layout as the public defaults:
p.add_argument("--sweep-root", type=Path, default=Path.home() / "work/ClawBench/claw-output/sweep")
p.add_argument("--models-yaml", type=Path, default=Path.home() / "work/ClawBench/models/models.yaml")
clawbench-rescore is an installed console script (pyproject.toml:29) and both docs/scoring.md and eval/scoring.md advertise it as the way to reproduce every leaderboard number from public traces. On any machine that isn't that one, --sweep-root doesn't exist, the rglob matches nothing, and the tool prints "discovered 0 tasks" and exits 0 — a silent no-op for the exact audience we most want to succeed.
Every other code path resolves paths through WORKSPACE_ROOT / MODELS_YAML (run_support/config.py:91); rescore bypasses that.
Ask:
- Default
--models-yaml to the workspace-resolved config.MODELS_YAML.
- Default
--sweep-root to ./test-output (where runs actually land) or require it explicitly.
- Error loudly when the root does not exist, instead of reporting 0 tasks and exiting 0.
- While there: rescore only accepts the
api_key scalar form, rejecting the api_keys list form that load_model_config (config.py:192-196) normalizes — reuse load_model_config instead of re-parsing.
src/clawbench/eval/rescore.py:203-217ships a maintainer's personal machine layout as the public defaults:clawbench-rescoreis an installed console script (pyproject.toml:29) and bothdocs/scoring.mdandeval/scoring.mdadvertise it as the way to reproduce every leaderboard number from public traces. On any machine that isn't that one,--sweep-rootdoesn't exist, the rglob matches nothing, and the tool prints "discovered 0 tasks" and exits 0 — a silent no-op for the exact audience we most want to succeed.Every other code path resolves paths through
WORKSPACE_ROOT/MODELS_YAML(run_support/config.py:91); rescore bypasses that.Ask:
--models-yamlto the workspace-resolvedconfig.MODELS_YAML.--sweep-rootto./test-output(where runs actually land) or require it explicitly.api_keyscalar form, rejecting theapi_keyslist form thatload_model_config(config.py:192-196) normalizes — reuseload_model_configinstead of re-parsing.