Skip to content

rescore: public reproducibility CLI defaults to a maintainer's home paths and silently no-ops elsewhere #296

Description

@reacher-z

src/clawbench/eval/rescore.py:203-217 ships a maintainer's personal machine layout as the public defaults:

p.add_argument("--sweep-root",   type=Path, default=Path.home() / "work/ClawBench/claw-output/sweep")
p.add_argument("--models-yaml",  type=Path, default=Path.home() / "work/ClawBench/models/models.yaml")

clawbench-rescore is an installed console script (pyproject.toml:29) and both docs/scoring.md and eval/scoring.md advertise it as the way to reproduce every leaderboard number from public traces. On any machine that isn't that one, --sweep-root doesn't exist, the rglob matches nothing, and the tool prints "discovered 0 tasks" and exits 0 — a silent no-op for the exact audience we most want to succeed.

Every other code path resolves paths through WORKSPACE_ROOT / MODELS_YAML (run_support/config.py:91); rescore bypasses that.

Ask:

  1. Default --models-yaml to the workspace-resolved config.MODELS_YAML.
  2. Default --sweep-root to ./test-output (where runs actually land) or require it explicitly.
  3. Error loudly when the root does not exist, instead of reporting 0 tasks and exiting 0.
  4. While there: rescore only accepts the api_key scalar form, rejecting the api_keys list form that load_model_config (config.py:192-196) normalizes — reuse load_model_config instead of re-parsing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions