Skip to content
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
009f3b5
Capture RewardKit named rewards and criterion breakdown on trials
claude Aug 7, 2026
7afd6b4
Render RewardKit reward design and outputs in the dashboard
claude Aug 7, 2026
c391b1f
Match dimension weighting to rewardkit's actual aggregation
claude Aug 7, 2026
f1265d2
Add a RewardKit example task for exercising reward rendering
claude Aug 7, 2026
a3d65f1
Give rewards their own surfaces instead of nesting in summaries
claude Aug 7, 2026
9ecef2f
Raise the example task's agent timeout for real-agent runs
claude Aug 7, 2026
e8eb5eb
Open the example task's network for agent setup
claude Aug 7, 2026
43a4ac7
Merge origin/staging into claude/oddish-reward-rendering-evjp8w
claude Aug 7, 2026
0167d76
Merge branch 'staging' into claude/oddish-reward-rendering-evjp8w
pfbyjy Aug 7, 2026
6bd10a4
Make every reward surface URL-addressable
claude Aug 7, 2026
202b8c4
Merge remote-tracking branch 'origin/claude/oddish-reward-rendering-e…
claude Aug 7, 2026
8c2f286
Address review findings on the reward-details embed and what-if math
claude Aug 7, 2026
c5f69ac
Request metadata-only listings from the reward design card
claude Aug 7, 2026
54121ac
Keep reward surfaces off non-RewardKit tasks in the fallback paths
claude Aug 7, 2026
e19ad03
Match rewardkit's same-name collapse in what-if and enrichment
claude Aug 7, 2026
61e9639
Redeploy the preview backend whenever the DB branch is re-prepared
claude Aug 7, 2026
dcb6230
Model per-file judge rewards and flag every embed clip
claude Aug 7, 2026
33b5176
Merge staging into RewardKit rendering
kyle-compute Aug 11, 2026
20a1160
Defer RewardKit artifacts until tab intent
kyle-compute Aug 11, 2026
079aef3
Address RewardKit review findings
kyle-compute Aug 11, 2026
1e1de4f
Scope RewardKit enrichment and deep links
kyle-compute Aug 11, 2026
ff1db78
Merge branch 'staging' into claude/oddish-reward-rendering-evjp8w
kyle-compute Aug 11, 2026
94d712b
Keep valid RewardKit dimensions on parse errors
kyle-compute Aug 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -985,6 +985,16 @@ trials without a persisted `_verifier` summary lazily discover and parse their
`verifier/ctrf.json` artifact through the already-scoped trial files API; do
not add an unscoped artifact lookup for this fallback.

RewardKit output follows the same pattern: the drawer renders the reward
breakdown from the embedded `_rewards` / `_reward_details` summaries in
`trial.result` (written by `harbor_artifacts.py`), falling back to the scoped
trial files API for `verifier/reward.json` / `verifier/reward-details.json`
when the embedded copies are missing (imports) or truncated. The task page's
Reward design section (`reward-design-card.tsx`) reconstructs the reward
program client-side from the task's `tests/` TOMLs plus a best-effort static
scan of Python criteria, enriched with criteria observed in a completed
trial's breakdown — it executes nothing.

On an experiment page, removing a task always calls the scoped
`DELETE /experiments/{experiment_id}/tasks/{task_id}` proxy. It unlinks that
experiment membership and its scoped trials without deleting the task, even
Expand Down
39 changes: 39 additions & 0 deletions DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -782,3 +782,42 @@ dashboard shows those counts as a small passed/total line in the trial
drawer's summary. Missing, malformed, or oversized CTRF reports are ignored
and never change the settled `reward`; verifiers without a test report simply
show no test line.

## Reward Kit rewards (reward.json + reward-details.json)

Tasks whose verifier uses [Harbor's RewardKit](https://www.harborframework.com/docs/rewardkit)
(`harbor-rewardkit`) report **multiple named rewards** — one score per
dimension (`tests/<dimension>/`) plus the aggregates defined in
`tests/reward.toml` — in `/logs/verifier/reward.json`, and a per-criterion
breakdown (scores, weights, judge reasoning, errors) in
`/logs/verifier/reward-details.json`.

Oddish captures both:

- The trial's headline scalar (`trial.reward`) stays exactly as before: the
`"reward"` key of `reward.json`, or the sole value when only one is
reported. **Name your strict aggregate `reward`** in `reward.toml` so
pass@k, baseline gates, and the matrix keep working — other names are
treated as extra dimensions.
- The full named-rewards map is persisted under the reserved
`trial.result._rewards` key.
- A bounded summary of the breakdown (long judge reasoning truncated,
criterion lists capped) is persisted under `trial.result._reward_details`,
with the artifact path of the full document. The complete
`reward-details.json` stays with the trial artifacts.

The dashboard renders these in the trial drawer as a reward breakdown —
each dimension with its criteria, the judge's raw answer and reasoning, and
the files the judge read — and renders `reward.json` /
`reward-details.json` as reward trees in the Files and Artifacts tabs. On
the task page, RewardKit tasks additionally get a **Reward design** section
reconstructed from `tests/reward.toml`, the judge TOMLs, and a static scan
of Python criteria files, including a what-if mode that simulates how
criterion outcomes flow through each aggregation into the final rewards.

Both keys follow the metrics.json rules: `_rewards` and `_reward_details`
are Oddish-owned (task-authored keys of the same name are dropped), and a
missing, malformed, or oversized details file is ignored — it never changes
the settled `reward`. Imported trials rely on the artifact fallback: the
drawer lazy-loads `verifier/reward.json` and `verifier/reward-details.json`
through the scoped trial files API when the embedded copies are absent.
1 change: 1 addition & 0 deletions frontend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
"remark-breaks": "4.0.0",
"remark-gfm": "4.0.1",
"shiki": "4.0.2",
"smol-toml": "1.4.2",
"swr": "2.4.1",
"tailwind-merge": "3.5.0",
"tailwindcss-animate": "1.0.7",
Expand Down
9 changes: 9 additions & 0 deletions frontend/pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions frontend/src/app/(app)/tasks/[task_id]/task-detail-client.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ import {
Loader2,
Star,
} from "lucide-react";
import { RewardDesignCard } from "@/components/reward-design-card";

const TaskFilesPanel = dynamic(
() =>
Expand Down Expand Up @@ -1374,6 +1375,14 @@ export function TaskDetailClient({
error={judgeError}
/>

{/* RewardKit tasks get their reward program rendered as an
explorable design; other tasks render nothing here. */}
<RewardDesignCard
taskId={task.id}
taskVersion={selectedVersion?.version}
trials={trialsForVersion}
/>

<div className="space-y-3">
<div className="flex items-baseline justify-between">
<h2 className="font-mono text-[12px] font-semibold tracking-[0.06em] text-[color:var(--paper-ink-2)] uppercase">
Expand Down
18 changes: 18 additions & 0 deletions frontend/src/components/renderers/file-renderer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ const JsonRenderer = dynamic(
{ ssr: false, loading: () => <LoadingStub label="Rendering JSON..." /> }
);

const RewardJsonRenderer = dynamic(
() => import("./reward-json-renderer").then((m) => m.RewardJsonRenderer),
{ ssr: false, loading: () => <LoadingStub label="Rendering rewards..." /> }
);

function LoadingStub({ label }: { label: string }) {
return (
<div className="text-muted-foreground flex h-full items-center justify-center gap-2 p-8">
Expand All @@ -75,6 +80,7 @@ type FileRendererKind =
| "json"
| "config-json"
| "result-json"
| "reward-json"
| "diff"
| "csv"
| "log"
Expand Down Expand Up @@ -155,6 +161,16 @@ function getFileRendererKind(fileName: string): FileRendererKind {
if (lower.endsWith("/result.json") || lower === "result.json") {
return "result-json";
}
// RewardKit verifier output (also copied into task artifacts by some
// tasks): render as a reward tree rather than raw JSON.
if (
lower.endsWith("/reward.json") ||
lower === "reward.json" ||
lower.endsWith("/reward-details.json") ||
lower === "reward-details.json"
) {
return "reward-json";
}
if (ext === "json") return "json";
if (ext === "diff" || ext === "patch") return "diff";

Expand Down Expand Up @@ -269,6 +285,8 @@ export function FileRenderer({
return <ConfigJsonRenderer content={content ?? ""} />;
case "result-json":
return <ResultJsonRenderer content={content ?? ""} />;
case "reward-json":
return <RewardJsonRenderer content={content ?? ""} />;
case "diff":
return <DiffRenderer content={content ?? ""} fileName={fileName} />;
case "csv": {
Expand Down
40 changes: 40 additions & 0 deletions frontend/src/components/renderers/reward-json-renderer.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
"use client";

import { JsonRenderer } from "./json-renderer";
import { RewardBreakdownView } from "@/components/reward-breakdown-view";
import {
parseRewardDetailsDocument,
parseRewardsMap,
type RewardBreakdown,
type RewardsMap,
} from "@/lib/reward-kit";

/**
* Renders RewardKit verifier output files as a reward tree instead of raw
* JSON: `reward.json` (flat named-score map) and `reward-details.json`
* (per-criterion breakdown with judge reasoning). Anything that doesn't
* parse as either falls back to the plain JSON view.
*/
export function RewardJsonRenderer({ content }: { content: string }) {
let parsed: unknown;
try {
parsed = JSON.parse(content);
} catch {
return <JsonRenderer content={content} />;
}

const breakdown: RewardBreakdown | null = parseRewardDetailsDocument(parsed);
const rewards: RewardsMap | null = breakdown ? null : parseRewardsMap(parsed);

if (!breakdown && !rewards) {
return <JsonRenderer content={content} />;
}

// A bare reward.json map has no dimension entries to classify against, so
// every named score renders in the chip row.
return (
<div className="p-4">
<RewardBreakdownView breakdown={breakdown} rewards={rewards} />
</div>
);
}
Loading