feat(session): tolerate throughput-only config diffs on --resume

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Relaxes the --resume strict-match guard so a resume no longer aborts when only pure scheduling / console-progress knobs differ from the persisted effective_config.yaml — concurrency (concurrency_limit / concurrency_limits), shard read/write concurrency, write-buffer sizing/flush, and console progress (show_progress, log intervals).
• Why: a run that partially fails because the inference service was over-subscribed (rate-limit / OOM) could previously only be recovered by starting completely fresh (discarding all completed samples) or reverting the concurrency change (re-triggering the same failure). Now you lower concurrency_limit / concurrency_limits and resume.
• The match scope is narrowed, not bypassed: no --force flag or env var. A field is resume-mutable iff changing it touches neither sample data nor any persisted artifact. Everything affecting on-disk content stays strict — sampling/seeds, max_iterations, shard_samples, record_*, max_retries (the failure signal in FAILED records), profile_*, detect_anomalies*, dump_progress / progress_dump_interval, and deterministic; infer_plans.yaml stays byte-for-byte strict. A classification-completeness test guards that every TaskRunnerConfig field is bucketed (throughput / strict / non-match), so a future field added without classification fails CI.
• Throughput knobs are recognized wherever they live — top-level concurrency_limit(s), the top-level runner_config defaults block, per-task runner_config, and models.*.args.concurrency_limit.
• On a tolerated diff, the effective_config.yaml body is rewritten with the new values and the header gains an appended # Resumed by sieval <ver> at <T>: record of exactly which fields changed — original provenance is preserved and the lineage accumulates across resumes. A tampered persisted file that no longer parses to a YAML mapping aborts as the same Resume aborted RuntimeError rather than an opaque error.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check)
• Unit + integration tests pass (pdm run pytest) — 353 passing across tests/unit/cli/leaderboard/ + tests/integration/resume/, including the TestStrictResumeMatch resume cases and 6 helper test classes (TestRunnerFieldClassification, TestStripNoncomparableFields, TestSplitHeader, TestDiffDicts, TestDiffLines, TestAppendResumeNote).

Manual

• Covered by automated tests — TestStrictResumeMatch exercises resuming with changed top-level / top-level-runner_config / per-task / per-model concurrency and console-progress (succeeds, body updated, resume note appended + accumulated across resumes), and aborts on a result-affecting (deterministic) or layout-affecting (shard_samples, max_retries, profile_*) change, and on a non-mapping persisted file. No manual run required.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ (changes confined to cli/)
• Deleted code verified — _strip_header, _brief_diff, and _diff_dicts are preserved as thin delegating wrappers (over _split_header / _diff_lines); no call sites broken