Skip to content

Commit 3264de1

Browse files
georgiclaude
andcommitted
docs: regenerate creative-pipeline media with gpt-image-2
flux/schnell was chosen on cost — $0.003 per megapixel — and it was the wrong trade. It mangles hands, and this brief requires them in three of four shots. A model that cannot draw the thing being commissioned is not cheap. Stills now come from openai/gpt-image-2; clips stay on LTX-distilled, which was never the problem. Both are overridable with CREATIVE_IMAGE_MODEL / CREATIVE_VIDEO_MODEL. Regenerated from the prompts recorded in the run dump rather than by re-running the agent, so this cost image and video calls and not another loop. The swap introduced a different brief violation, which is the more interesting half. flux could not render legible text, so it never breached the brief's no-logo rule by accident. gpt-image-2 can, and branded the bottle in shot 3 with raised COLD BREW COFFEE lettering. Better capability, new failure mode. The prompt is why: sonnet never carried no-logo into any image prompt. It read the brief and the constraint did not survive into the generation call. Shot 3 was regenerated with it appended. Neither violation is visible to the suite. forbiddenAvoided reads shot action text and layer names, so it can see the word logo in a prompt and never what the picture contains; mustFeature asks only whether each element appears somewhere in the shot list, so a shot with neither hands nor sunrise passes. The predicates grade the plan. Grading the artifact needs a human or a vision model, and that is now written down in both the fixture README and the package docs rather than left for someone to discover. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 0256488 commit 3264de1

12 files changed

Lines changed: 62 additions & 24 deletions

File tree

docs/evals/creative-pipeline/README.md

Lines changed: 35 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,12 @@ measured it, trimmed it, and signed it off — 162 tool calls, delivered at
2626
| `keyframe-shot_{1..4}.webp` | Storyboard — one still per shot |
2727
| `clip-shot_{1..4}.mp4` | Storyboard — each still animated |
2828

29+
Stills come from `openai/gpt-image-2`, clips from
30+
`fal-ai/ltx-2-19b/distilled/image-to-video`.
31+
2932
Compressed for the repo: stills to WebP at 720px, clips to h264 at 540px.
30-
15 MB of raw output became 1.3 MB, which is the difference between a fixture
31-
and a liability. The originals are not kept.
33+
Raw output became 1.8 MB, which is the difference between a fixture and a
34+
liability. The originals are not kept.
3235

3336
## Reproducing
3437

@@ -45,14 +48,39 @@ Roughly **$2.60** for the agent loop and **$0.17** for the media —
4548
$0.0008. The model driving the pipeline costs an order of magnitude more than
4649
the pipeline's output.
4750

51+
## Why not a cheaper image model
52+
53+
The first pass used `flux/schnell` at $0.003 per megapixel, chosen on cost. It
54+
was the wrong trade: flux mangles hands, and this brief requires them in three
55+
of four shots. A model that cannot draw the thing being commissioned is not
56+
cheap. gpt-image-2 costs materially more per image and renders anatomy that
57+
survives a look.
58+
59+
Swapping it in introduced a *different* brief violation, which is the more
60+
interesting half. flux could not render legible text, so it never breached the
61+
brief's `no logo` rule by accident. gpt-image-2 can, and branded the bottle in
62+
shot 3 with "COLD BREW COFFEE" in raised lettering. Better capability, new
63+
failure mode.
64+
65+
The prompt is why. Sonnet never carried `no logo` into any image prompt — it
66+
lives in the brief, and the model read the brief, but the constraint did not
67+
survive into the generation call. Shot 3 was regenerated with the constraint
68+
appended explicitly.
69+
4870
## Two things these files show that the eval cannot
4971

5072
**The planted defect is conservative.** The suite simulates rendered clips
5173
coming back 1.35× longer than requested. LTX returned 4.84s takes for 3s
5274
requests — **1.61×**. Real video models overshoot harder than the eval assumes.
5375

54-
**A per-shot brief violation the predicates miss.** `clip-shot_3` (condensation
55-
on a bottle) contains neither hands nor sunrise. The model caught this in its
56-
own review and filed it; the suite's `mustFeature` check only asks that each
57-
element appears *somewhere* in the shot list, so it passed. The model's
58-
judgement was finer than the check's.
76+
**Two brief violations the predicates cannot see.** `clip-shot_3` contains
77+
neither hands nor sunrise; the model caught that in its own review and filed
78+
it, while the suite's `mustFeature` check passed because it only asks whether
79+
each element appears *somewhere* in the shot list. And the same shot came back
80+
branded, which `forbiddenAvoided` also passed — that check reads shot action
81+
text and layer names, so it can see the word "logo" in a prompt and never what
82+
the picture actually contains.
83+
84+
Both are the same limit. The predicates grade the plan; only a human or a
85+
vision model grades the artifact. Worth knowing before trusting a green run to
86+
mean the deliverable is on brief.
96.9 KB
Binary file not shown.
-46.9 KB
Binary file not shown.
291 KB
Binary file not shown.
17.3 KB
Binary file not shown.
40.9 KB
Loading
5.95 KB
Loading
112 KB
Loading
27.3 KB
Loading
26.1 KB
Loading

0 commit comments

Comments
 (0)