Promote staging to main: eval-ops toolkit

[charlesyhuang]

Caution

DO NOT USE THE MERGE BUTTON ON THIS PULL REQUEST.
THE BUTTON CREATES A NEW COMMIT AND BREAKS THE RELEASE MODEL.
COMMENT /promote TO COMPLETE THE PROMOTION.

Caution

DO NOT USE THE MERGE BUTTON ON THIS PULL REQUEST.
THE BUTTON CREATES A NEW COMMIT AND BREAKS THE RELEASE MODEL.
COMMENT /promote TO COMPLETE THE PROMOTION.

Summary

Ships eval-ops/, the operational toolkit used to drive the SWE-Marathon eval sweeps. It is a standalone directory of scripts — nothing in the CLI, backend, or frontend imports it, and no existing file is modified.

Commits in this promotion

2f7fafd Add the eval-ops toolkit used to drive the SWE-Marathon sweeps (#1269)

One commit. main is an ancestor of staging (git merge-base --is-ancestor origin/main origin/staging passes), so this fast-forwards cleanly.

Validation

Green on the staging tip 2f7fafd:

Workflow	Result
Staging Deploy	success
Branch Sync Guard	success

load_only Guard, E2E: CLI smoke and E2E: Dashboard did not run on this commit — their path filters exclude a change that touches only eval-ops/. That absence is expected here rather than a skipped check.

No automated tests: these are operational scripts that talk to the live Oddish API, and the repo's suite does not cover eval-ops/. They have instead been exercised continuously against the running experiments — the fill loop is driving experiment 826d7d88 right now, and passk.py produced the reported pass@1/3/8 numbers for the six-rung terra effort sweep.

Risk

Additive only. The scripts are not on any import path, so the blast radius on promotion is limited to the repository containing 19 new files. The behaviour they encode already runs today from a working copy; landing them means the next run inherits the accumulated gotchas instead of rediscovering them.

What the toolkit encodes

Three counting traps that each silently stalled a fill in practice, all now handled:

• The server counts every trial it still holds when resolving --n-trials, burnt ones included — and infra failures report status=success. Submitting held + 1 therefore creates nothing in any cell with a past failure, while still printing Task submitted!.
• Sweep payloads are deduplicated for 24h by a hash of the whole request, so once a cell plateaus every subsequent submit collapses into the first and creates nothing. Submits now carry a unique nonce.
• --ak parses agent kwargs into dict[str, str], so a nested value reaches Harbor as a string. Nested config goes through a sweep template instead.

It also classifies failures by Harbor's own result.harbor_exception.exception_type rather than by the error string — every failure shares the same Command failed (exit 1): opencode ... prefix regardless of cause, so the string cannot tell a rate limit from a context overflow.

---

Generated by Claude Code

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
oddish-app	Ready	Preview	Aug 15, 2026 11:09pm

[charlesyhuang]

/promote

[github-actions]

Promoted. main now points at 2f7fafdbaef513c5ecb6f6c189eb7b00cfa46733, the same commit as staging.