Render RewardKit reward design and outputs

[pfbyjy]

Some Harbor tasks now use RewardKit to score agents. A RewardKit verifier does not produce one number. It produces several named scores (for example exact, llmj, reward, soft_score) and a detailed report that lists every check, its weight, and what the LLM judge said.

Until now, oddish kept only the single reward number and threw the rest away. The detailed report reached S3, but you could only view it as raw JSON.

This PR keeps that data and shows it in the dashboard.

Backend (oddish/)

The worker stores two extra fields inside the existing trials.result JSON column. No database migration.

• _rewards — all the named scores from reward.json.
• _reward_details — a compact copy of reward-details.json: each dimension, its checks, their weights, and the judge's reasoning. Long reasoning is shortened so the row stays small, and the path of the full file is stored so the UI can fetch it later.

The single trial.reward number is computed exactly as before, so pass@k, baseline gates, and the matrix are unchanged. One rule for task authors: name the strict aggregate reward in reward.toml.

Both fields follow the same safety rules as the existing _verifier field. If the source file is missing, broken, or too large, oddish ignores it and the trial's reward is not affected. If a task tries to write its own _rewards key, it is dropped.

Frontend

Three additions. Tasks with a plain single-number reward look exactly the same as before.

1. Trial drawer: a "Reward Breakdown" card. It lists each dimension and each check inside it. Click a check to see what the judge answered ("no"), the score that became (0.00), the judge's full reasoning, and which files the judge read. If the stored copy was shortened, or the trial was imported and has no stored copy, the full report is fetched from the trial's files using the existing scoped files API.
2. Files and Artifacts tabs: reward.json and reward-details.json render as the same tree instead of raw JSON.
3. Task page: a "Reward Design" section. It shows how the task scores agents before any trial runs. It reads the task's tests/ files: the judge TOML configs and reward.toml are parsed (new dependency: smol-toml), and Python check files are scanned as text — never executed. It also has a what-if mode: toggle checks pass/fail and watch the final scores change, using the same aggregation math RewardKit uses.

Docs

• DOCS.md: a new section on writing RewardKit tasks for oddish, next to the metrics.json and ctrf.json sections.
• AGENTS.md: notes on how the UI loads this data.

Testing

• 18 new backend tests in test_rewardkit_capture.py; they pass along with the existing test_verifier_metrics.py (39 total).
• Three failures in test_harbor_ephemeral / test_harbor_runner are pre-existing in this environment — they fail the same way without this change.
• Frontend: typecheck, lint, prettier, and next build all pass. The TOML parsing was checked against the real judge files from the QA goldens gist.
• Not done yet: an end-to-end run against a live RewardKit trial. The QA goldens eval would be a good first exercise.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PKitApumfgZMroHfnNvGsH

---

Note

Medium Risk
Touches trial result persistence and major trial/task UI surfaces, but follows the existing CTRF embed pattern, leaves the headline reward scalar unchanged, and needs no schema migration.

Overview
Captures and displays RewardKit multi-dimensional rewards instead of discarding everything beyond the headline scalar.

Backend: Workers now embed _rewards (named scores from reward.json) and a bounded _reward_details summary into trials.result, with the full details file left in object storage for lazy load. Same forgiving rules as _verifier — missing/malformed/oversized input never affects the settled reward.

Frontend: Adds a Rewards tab in the trial drawer (embedded first, artifact fallback for imports/historical rows), renders reward.json / reward-details.json as reward trees, and a Reward Design section on the task page that reconstructs the program from tests/ TOMLs plus a static Python scan — including what-if pass/fail simulation. All surfaces are URL-addressable (?tab=rewards, ?taskView=reward, #reward-design).

Also forces preview backend redeploy when a live backend would otherwise keep a rotated DB password, and adds a small RewardKit demo task plus docs.

^{Reviewed by Cursor Bugbot for commit 94d712b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
oddish-app	Ready	Preview	Aug 11, 2026 11:45pm

[cursor]

Risk: medium. Left a non-blocking comment: Cursor Bugbot reported unresolved findings (including high severity) and its check finished as skipping, so this is not auto-approved. Two reviewers are already requested; no additional reviewers were assigned.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}

[github-actions]

Oddish preview

Commit: 94d712b07a9261bd1a0b03798eb769cbe1848337

Surface	Link	Target
Frontend	https://pr-1113.oddish.app	Vercel preview for 94d712b
Backend	oddish-pr-1113	oddish-pr-1113
Database	project nnmbrningmtnwwfnbaok	project nnmbrningmtnwwfnbaok

Vercel deployment URL: https://oddish-9lhmekgj0.oddish.app

Plan:

• Frontend deploy: true
• Backend deploy: true
• Migrations: false

This comment is updated by the PR Preview workflow.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ff1db78. Configure here.}