agents: creative-pipeline eval — one commission across brief, sketch, storyboard, cut and review - #4620
Merged
Conversation
The eight existing tool-loop suites each score one surface. A model can pass sketch-tools and storyboard-tools outright and still lose the client's aspect ratio between them, or deliver a cut that runs half again as long as it was commissioned at. This suite scores the seams: one commission carried through brief, ideation, sketch, storyboard, cut and review. The three creative surfaces are composed, not reimplemented. The bridge builds the real sketch, storyboard and timeline bridges and merges their tool arrays, so the model drives the same contract those suites already cover and this file cannot drift from them. Only ui_storyboard_assemble_timeline is replaced: the storyboard bridge's version returns a sequence id and touches nothing, which is fine when the board alone is under test and useless here, because the handoff is the thing being measured. The replacement drives the timeline bridge's own tools, so a later trim acts on the storyboard's output. ui_brief_* and ui_review_* are eval instrumentation and nothing in web/ implements them. They exist because a brief passed only in the prompt cannot be told apart from one the model ignored. Rendered clips come back 1.35x the requested length, the way a video model that emits fixed-length takes does, so a cut planned to exactly fill the brief overruns. Trimming the last clip fixes it; trimming an earlier one only opens a gap and leaves the runtime unchanged, since the clips after it keep their start times. A model that re-measures sees that, and one that trims and declares victory does not. Two checks were rewritten after a live run against claude_agent_sdk/sonnet rather than kept and explained away. The review severity was a three-value enum that threw on "critical" — failing a run on this harness's vocabulary rather than on anything about the cut — so synonyms now map. And overrun detection grepped the note prose for runtime/duration/length, which scored a run that found the overrun, trimmed four clips and ripple-moved three to close the gaps as a miss, purely on wording; it now grades the severity the model assigned, which survives paraphrase. Tool-call ceilings came from the same run: 90 failed a case that did the job in 97 calls, so they are 130 and 150. Measured: review-catches-overrun scores 1.00 in ~25 calls, brief-constraints- hold 0.91 in 97. The full six-phase case spent 138 calls without converging at --max-iterations 140 and needs a higher cap; the harness test proves the case is satisfiable in 18 calls when driven optimally, so that is a finding about efficiency, not an unsatisfiable case. Harness tests use a scripted provider and no network. Alongside the passing run they assert the two failures the suite exists to catch: notes filed with nothing fixed afterwards, and a forbidden element reaching the board. Co-Authored-By: Claude <noreply@anthropic.com>
A live sonnet run assembled the cut at 16.20s, trimmed and ripple-moved it to exactly 12.00s, verified with ui_review_get_cut, then filed four notes as a compliance sign-off. The suite scored that as "review changed nothing". reviewActedOn counted timeline edits after the first review note, so it required report-then-fix and failed fix-then-verify-then-report. It also had no true positive that withinBriefRuntime did not already catch: a cut that overruns fails the runtime check whatever order the model worked in. So the check only ever fired on correct work done differently. It is now cutRevisedAfterAssembly, keyed on the handoff rather than on the report. Both orders pass; shipping the renderer's output unrevised still fails. That is the third check in this suite rewritten for the same reason — encoding one valid process shape and penalising the others. The first two were a severity enum that threw on "critical" and an overrun check that grepped the note prose. Replaying the recorded 93-call transcript through the corrected predicates scores 1.000 with no failed checks. Also adds scripts/dump-creative-run.ts, which writes what the model actually made — concepts, style-frame prompt, shot list, assembled cut with timings, review notes, phase snapshots, full transcript — to nodetool-debug/. The eval report carries pass/fail and call counts, which cannot show the work. full-pipeline is now verified live: 1.00 in 93 calls at --max-iterations 220. The earlier 138-call non-convergence was the SDK turn cap, not the case. Co-Authored-By: Claude <noreply@anthropic.com>
The suite fakes every generate and render, which is what makes it cost the agent loop and nothing else. That also means it never proves the pipeline can produce anything. --live wires a MediaBackend so the same tool calls additionally hit fal, and the run leaves real stills and clips on disk without changing a tool contract or a predicate. MediaBackend is an interface in the bridge and the fal wiring lives in the dump script: packages/agents has no fal dependency and should not grow one for an opt-in path. Headless stays the default, artifacts default to empty, and no predicate reads them — a case asserting on artifacts would fail every CI run. Media is the cheap half. flux/schnell bills $0.003 per megapixel and ltx-2-19b/distilled $0.0008, so nine artifacts cost about $0.17 against ~$2.60 for the loop that directs them. Verified end to end: a style frame, four keyframes and four clips, cut delivered at 11.80s inside a 12s brief. Two things the live run exposed, both documented rather than hidden. LTX returned 4.84s takes for 3s requests — a 1.61x overshoot against the 1.35x this eval simulates, so the planted defect is conservative rather than unfair. And the timeline still lays clips at the simulated overshoot, so the scored runtime is not the runtime of the files on disk; closing that needs the backend to report delivered duration. A generation failure is recorded on the artifact and never thrown. The model is scored on directing the pipeline, and failing its run because fal rate-limited would measure the weather. Also fixes two stale editsAfterReview references in the report writer left by the rename to editsAfterAssembly. Co-Authored-By: Claude <noreply@anthropic.com>
The suite is headless, so nobody can see what it produces without paying for a run. This is one --live run's output: a style frame, four keyframes and four clips that claude_agent_sdk/sonnet directed from a 9:16 / under-12s / hands and sunrise / no logo brief, delivered at 11.80s across 162 tool calls. Compressed for the repo — stills to WebP at 720px, clips to h264 at 540px. 15 MB of raw output became 1.3 MB, which is the difference between a fixture and a liability. Precedent for binaries this size already exists in marketing/public; the originals are not kept. The README records two things the eval itself cannot report. The planted 1.35x render overshoot is conservative: LTX returned 4.84s takes for 3s requests, 1.61x. And clip-shot_3 has neither hands nor sunrise, a per-shot brief violation the model caught in its own review while the suite's mustFeature check passed it, because that check only asks whether each element appears somewhere in the shot list. Co-Authored-By: Claude <noreply@anthropic.com>
georgi
force-pushed
the
eval/creative-pipeline
branch
from
July 31, 2026 13:13
1db5b0c to
0256488
Compare
flux/schnell was chosen on cost — $0.003 per megapixel — and it was the wrong trade. It mangles hands, and this brief requires them in three of four shots. A model that cannot draw the thing being commissioned is not cheap. Stills now come from openai/gpt-image-2; clips stay on LTX-distilled, which was never the problem. Both are overridable with CREATIVE_IMAGE_MODEL / CREATIVE_VIDEO_MODEL. Regenerated from the prompts recorded in the run dump rather than by re-running the agent, so this cost image and video calls and not another loop. The swap introduced a different brief violation, which is the more interesting half. flux could not render legible text, so it never breached the brief's no-logo rule by accident. gpt-image-2 can, and branded the bottle in shot 3 with raised COLD BREW COFFEE lettering. Better capability, new failure mode. The prompt is why: sonnet never carried no-logo into any image prompt. It read the brief and the constraint did not survive into the generation call. Shot 3 was regenerated with it appended. Neither violation is visible to the suite. forbiddenAvoided reads shot action text and layer names, so it can see the word logo in a prompt and never what the picture contains; mustFeature asks only whether each element appears somewhere in the shot list, so a shot with neither hands nor sunrise passes. The predicates grade the plan. Grading the artifact needs a human or a vision model, and that is now written down in both the fixture README and the package docs rather than left for someone to discover. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ninth tool-loop suite. The other eight each score one surface; this one scores the seams — one commission carried through brief → ideation → sketch → storyboard → cut → review. That's where a model that passes
sketch-toolsandstoryboard-toolsindividually still loses the client's aspect ratio between them.IS_SANDBOX=1 npm run dev:nodetool -- eval creative-pipeline \ -p claude_agent_sdk -m sonnet --max-iterations 220 --no-find-modelWhat one run produces
Brief: vertical social spot for a cold brew — 9:16, under 12s, must feature hands and sunrise, no logo.
sonnet read the brief, proposed three concepts, committed to one, built a style frame, wrote and rendered four shots, assembled at 16.20s, measured, trimmed and ripple-moved to 11.80s, then signed off with four review notes. 162 tool calls.
Clips: 1 · 2 · 3 · 4 — media checked in at
docs/evals/creative-pipeline/, 1.3 MB compressed from 15 MB.Design
Composed, not reimplemented. The bridge builds the real sketch, storyboard and timeline bridges and merges their tool arrays, so the model drives the same contract those suites already cover and this file cannot drift from them. Only
ui_storyboard_assemble_timelineis replaced — its own version returns a sequence id and touches nothing, and the handoff is the thing under test.ui_brief_*/ui_review_*are eval instrumentation, not a frontend contract. Nothing inweb/implements them. A brief passed only in the prompt cannot be told apart from one the model ignored.Planted defect. Rendered clips come back 1.35× longer than requested, like a video model that only emits fixed-length takes. Trimming the last clip fixes the runtime; trimming an earlier one only opens a gap, since later clips keep their start times.
--liveroutes the same tool calls to fal for real media. Headless stays default, artifacts default to empty, no predicate reads them — a case asserting on artifacts would fail every CI run. Media is the cheap half: ~$0.17 for nine artifacts against ~$2.60 for the agent loop.Three checks rewritten after live runs — all the same mistake
Each encoded one valid working order and failed models that used another.
"critical", failing a run on this harness's vocabulary. Synonyms now map.reviewActedOncounted edits after the first note, requiring report-before-fix. A run assembled at 16.20s, trimmed to 12.00s, verified, then filed notes as a sign-off — a complete loop scored as "review changed nothing". NowcutRevisedAfterAssembly, which accepts either order. It also had no true positivewithinBriefRuntimedidn't already catch.Tool-call ceilings were guesses too: 90 failed a case that did the job in 97 calls. Now 130/150, from measurement.
Two things the live run showed that the eval can't
The planted defect is conservative. LTX returned 4.84s takes for 3s requests — 1.61×, not the 1.35× simulated.
A per-shot brief violation the predicates miss.
clip-shot_3has neither hands nor sunrise. The model caught it in its own review;mustFeatureonly asks whether each element appears somewhere, so it passed.Verification
packages/agents+packages/clitypecheck clean; lint cleanfull-pipeline1.00 (93 calls),review-catches-overrun1.00 (25),brief-constraints-hold0.91 (97)Known gap
In
--livemode the timeline still lays clips at the simulated overshoot, so the scored runtime is not the runtime of the files on disk. Closing it needs the backend to report delivered duration. Documented in the source and inpackages/agents/CLAUDE.md.🤖 Generated with Claude Code