Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
e69816d
feat(protocol): scenes, render record and setup stage on storyboards
claude Sep 5, 2026
b53691c
fix(protocol): drop the conditional empty spread in scene normalization
claude Sep 5, 2026
24a66cd
feat(storyboard): P1 storyboard contracts — prompts, scenes, ordering
claude Sep 5, 2026
e72c4aa
feat(protocol): derive version staleness from the render record
claude Sep 5, 2026
8bb0d92
feat(harness): select the creation-flow suites from the storyboard su…
claude Sep 5, 2026
8cfb13c
feat(storyboard): the ui_storyboard_* surface for the guided flow
claude Sep 5, 2026
3d8743f
fix(runtime): fake the clock in the run-budget precedence test
claude Sep 5, 2026
8fe0e26
docs(flows): correct where the flow's shipped assets live
claude Sep 5, 2026
9a341a4
feat(flows): P2 storyboard setup — hosts, steps, look, style presets
claude Sep 5, 2026
4063cfa
feat(studio): extract the linked script at Continue to storyboard
claude Sep 5, 2026
9c46b30
feat(storyboard): P3 board — scenes, card actions, staleness, reattac…
claude Sep 5, 2026
85642cc
feat(storyboard): P4 Edit Shot dialog and P5 imports
claude Sep 5, 2026
98e9ac8
Merge origin/main into the guided-creation-flow branch
claude Sep 6, 2026
8cb6a9d
fix(websocket): classify storyboards.stylePresets for the sandbox bou…
claude Sep 6, 2026
022631c
feat(flows): P6-P9 — the video, script, image and workflow flows
claude Sep 6, 2026
0e61f4d
fix(script): strip subtitle cue tags to a fixed point
claude Sep 6, 2026
9c4b24a
fix(script): make a written script's line ids unique per write
claude Sep 6, 2026
343f1f9
fix(review): settle direct-gen against the sending sequence, keep the…
claude Sep 6, 2026
5616938
fix(flows): fetch image-model samples on first use, restore PRD § 10.3
claude Sep 6, 2026
df0fd0f
Merge remote-tracking branch 'origin/main' into claude/tasks-creation…
claude Sep 6, 2026
b444334
fix(video): stamp the sequence's aspect, queue or omit the bed, bound…
claude Sep 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
28 changes: 20 additions & 8 deletions docs/creation-flows/prd.md
Original file line number Diff line number Diff line change
Expand Up @@ -896,18 +896,28 @@ setup?: {
pace?: "slow" | "normal" | "fast";
language?: string;
};

// scriptLine gains one, written only by a subtitle import:
targetDurationMs?: number; // the cue's own timing, the take's target
```

Nothing else changes. Reuse: cast and voice bindings, `useVoiceCostEstimate`,
`ui_script_*`, the § 7.6 extraction route, a new pure `parseSrt` beside
`parseFdx`.

The writer's prompt, schema and parse live in `@nodetool-ai/protocol`
(`script-authoring.ts`), as the Director's do, so the flow and the headless
`write_script` capability author the same artifact. Imported words never reach
the writer's schema: they are split by `splitImportedText` and attributed by a
call whose schema has no text field, so a model that ignores the schema can
misattribute a line but cannot reword it.

### 9.6 Headless parity

| Operation | Tool |
| --- | --- |
| Set brief, format, length, stage | new `ui_script_set_setup` |
| Write or rewrite | new `ui_script_write` (`rewrite: boolean`) |
| Set brief, format, length, stage | new `ui_script_set_setup`, mirrored by `edit_script`'s `set_setup` op |
| Write or rewrite | new `ui_script_write` (`rewrite: boolean`), mirrored headlessly by `write_script` |
| Edit lines and speakers | existing `ui_script_set_line_text`, `ui_script_set_speaker`, `ui_script_add_line`, `ui_script_add_speaker` |
| Bind voices, voice all | existing `ui_script_set_speaker_voice`, `ui_script_voice_all` |
| Import | `parseFdx`, `parseSrt`, then the line and speaker tools |
Expand Down Expand Up @@ -959,8 +969,10 @@ to look` sets stage `look`.
Heading "Choose the look". Size as preset tiles per aspect (square, portrait,
landscape, story, banner) with the pixel size shown. Style as the same twelve
`style` entities as E1, one library. Image model as `PresetTileGrid`, each
tile a sample of that model on a fixed prompt. Cost from the sketch generate
estimate times the variation count.
tile a sample of that model on a fixed prompt, fetched on first use rather
than shipped (R5), as E2's clip tiles are. A model with no published sample
falls back to its name. Cost from the sketch generate estimate times the
variation count.

`Generate your image` sets stage `done` and enqueues N generated layers
through the sketch editor's `text-to-image` binding, one per variation, each
Expand Down Expand Up @@ -997,10 +1009,10 @@ estimate.

| Operation | Tool |
| --- | --- |
| Set brief, use case, variations, stage | new `ui_sketch_set_setup` |
| Refine or re-refine | new `ui_sketch_refine_brief` |
| Generate variations | `ui_sketch_generate` called once per variation with the composed prompt |
| Pick | `ui_sketch_select_layer` plus visibility through `ui_sketch_adjust_layer` |
| Set brief, use case, variations, stage | new `ui_sketch_set_setup`, mirrored by `edit_sketch`'s `set_setup` op |
| Refine or re-refine | new `ui_sketch_refine_brief`, mirrored headlessly by `refine_image_brief` |
| Generate variations | `ui_sketch_generate` called once per variation with the composed prompt and its own `seed` |
| Pick | `ui_sketch_select_layer` plus visibility through `ui_sketch_set_layer_props` |

### 10.7 Acceptance criteria

Expand Down
37 changes: 23 additions & 14 deletions docs/creation-flows/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,13 @@ of each other. Inside a phase, tasks are listed in dependency order.
cards render disabled until P5. `Continue` writes `brief` and stage
`genre`. `/` completion off inside the flow. (PRD § 7.1)
- [ ] **Step 2: genre.** Fourteen cards through `OptionCardGrid`. Fourteen
`package://` stills under the example-board asset path, registered in
`PACKAGE_RUNTIME_ASSETS`; `npm run backend:smoke` passes. Picking
writes `genre`. (PRD § 7.2)
`package://` stills under the example-board asset path
(`packages/base-nodes/nodetool/assets/nodetool-base/storyboards/genres/`).
That tree needs no per-file registration: `scripts/bundle-backend.mjs`
copies `assets/nodetool-base/` wholesale, which is why no example-board
still appears in `PACKAGE_RUNTIME_ASSETS` either — that registry is for
files shipped beside a package's compiled `dist/`. Picking writes
`genre`. (PRD § 7.2)
- [ ] **Step 2: Direct.** `useDirectScreenplay` takes `genre`; `Review your
screenplay` runs it and sets stage `review` on success only. Test: the
Director prompt contains the genre (criterion 3); a rejected call leaves
Expand Down Expand Up @@ -302,33 +306,33 @@ of each other. Inside a phase, tasks are listed in dependency order.

## P7 — Script flow

- [ ] **Protocol.** Optional `setup` on `scriptDocument` in
- [x] **Protocol.** Optional `setup` on `scriptDocument` in
`packages/protocol/src/api-schemas/scripts.ts` per PRD § 9.5.
- [ ] **Format cards.** Five formats with cast shape and section layout, plus
- [x] **Format cards.** Five formats with cast shape and section layout, plus
the length row.
- [ ] **Writer.** `writeScript` in `web/src/hooks/script/useWriteScript.ts`:
- [x] **Writer.** `writeScript` in `web/src/hooks/script/useWriteScript.ts`:
`generate_text` with a structured script schema, imported text kept
verbatim and only split and attributed, applied through the existing
store, stage `review`. `Rewrite` keeps retained line ids. Tests:
criteria 3 and 4.
- [ ] **`parseSrt`.** Pure, SRT and VTT to lines with target durations and a
- [x] **`parseSrt`.** Pure, SRT and VTT to lines with target durations and a
`Narrator` speaker, in `web/src/lib/script/parseSrt.ts`. Fixture test.
- [ ] **Step 1 alternatives.** Paste or upload through the P5 route and
- [x] **Step 1 alternatives.** Paste or upload through the P5 route and
`parseFdx`, subtitles through `parseSrt`, blank sets `done`.
- [ ] **Review.** `PlanReview` over speaker, line, direction; edits through
- [x] **Review.** `PlanReview` over speaker, line, direction; edits through
`ui_script_set_line_text` and `ui_script_set_speaker`; word count and
spoken-length estimate at the flow's pace.
- [ ] **Voices step.** One `PresetTileGrid` row per speaker, a tile plays the
- [x] **Voices step.** One `PresetTileGrid` row per speaker, a tile plays the
speaker's first line in that voice (one TTS call per tile on demand,
cached per session), language and pace selects, cost from
`useVoiceCostEstimate`. `Voice your script` binds voices, runs
`ui_script_voice_all`, stage `done`, opens the editor. Test: one voice
per speaker, every line voiced once (criterion 5).
- [ ] **Tools.** `ui_script_set_setup`, `ui_script_write` in
- [x] **Tools.** `ui_script_set_setup`, `ui_script_write` in
`web/src/lib/tools/builtin/script.ts`, headless mirrors in
`packages/agents/src/capabilities/scripts.ts`, capability table rows,
eval case in `packages/agents/src/evals/surfaces/script.ts`.
- [ ] **Entry.** Enable the Script card on both hosts; resume by stage in the
- [x] **Entry.** Enable the Script card on both hosts; resume by stage in the
script surfaces; next-steps strip `Create storyboard`, `Send to
timeline`.

Expand Down Expand Up @@ -417,5 +421,10 @@ of each other. Inside a phase, tasks are listed in dependency order.
in the PR's Verification section.
- [ ] `docs/creation-flows/prd.md` is updated in the same PR when a phase
changes a contract it states.
- [ ] Shipped assets (genre stills, style thumbnails) are registered in
`PACKAGE_RUNTIME_ASSETS` and `npm run backend:smoke` passes.
- [ ] Shipped assets (genre stills, style thumbnails) live under
`packages/base-nodes/nodetool/assets/nodetool-base/`, which
`scripts/bundle-backend.mjs` copies wholesale — no `PACKAGE_RUNTIME_ASSETS`
entry, which is for files shipped beside a package's compiled `dist/`.
A registered path with no file behind it fails `npm run backend:smoke`,
so register nothing you have not shipped. Each asset stays under the
example-board size budget (PRD R5).
7 changes: 4 additions & 3 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 12 additions & 1 deletion packages/agents/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -869,13 +869,24 @@ result as an asset, and write it back onto the persisted board.
| `list_storyboards` | Boards newest first, with per-board still/clip counts |
| `create_storyboard` | Blank board (then `edit_storyboard` adds shots) |
| `get_storyboard` | Shots with ids, status, and whether each has a still/clip |
| `direct_storyboard` | Runs the Director over the board's brief/genre/style and writes the screenplay; `redirect` keeps retained shots' ids and media |
| `render_storyboard_stills` | `text_to_image` per shot → the shot's keyframe |
| `render_storyboard_clips` | the shot's clip: `image_to_video` seeded by the keyframe, or `text_to_video` from the prompt |
| `revise_storyboard_clip` | `video_to_video` revision of one shot's clip |
| `assemble_storyboard_timeline` | Rendered clips → a saved `timeline_sequences` row |

`edit_storyboard` is the document half: beside the shot ops it carries the
guided flow's `set_setup`, the scene ops (`move_shot`, `duplicate_shot`,
`update_scene`, `create_scene`, `merge_scene`), `set_style`, and the version ops
(`select_version`, `delete_version`, `add_keyframe_version`). Every one mirrors
a `ui_storyboard_*` browser tool and writes the same document shape, so a board
edited headlessly and one edited in the editor cannot diverge (PRD § 7.10).

Both render tools take `targets` (shot ids, indexes, or slugs) and default to
"whatever still needs this step", so a whole board is one call. Shots render
"whatever still needs this step", so a whole board is one call. `stale_only`
narrows that to the shots whose selected version was rendered from inputs the
shot no longer has (`isVersionStale`), which is how a style change is re-rendered
without re-rendering the board. Shots render
concurrently (default 3, max 8, 24 shots per call). Every write is a CAS on the
row's `updated_at` with a bounded retry, because concurrent renders all land on
the same board document; a conflicting write re-reads and re-applies rather than
Expand Down
6 changes: 6 additions & 0 deletions packages/agents/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,12 @@
"import": "./dist/tools/base-tool.js",
"default": "./dist/tools/base-tool.js"
},
"./host-modules/mammoth": {
"nodetool-dev": "./src/host-modules/mammoth.ts",
"types": "./dist/host-modules/mammoth.d.ts",
"import": "./dist/host-modules/mammoth.js",
"default": "./dist/host-modules/mammoth.js"
},
"./js-sandbox": {
"nodetool-dev": "./src/js-sandbox.ts",
"types": "./dist/js-sandbox.d.ts",
Expand Down
12 changes: 12 additions & 0 deletions packages/agents/src/capabilities/entities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,10 @@ const saveEntityAsset = async (
if (!asset) {
return { error: `Asset ${assetId} was not found.` };
}
const readOnly = Asset.systemEntityRefusal(asset);
if (readOnly) {
return { error: readOnly };
}
if (!asset.content_type.startsWith("image/")) {
return {
error: `${asset.name || asset.id} is a ${asset.content_type} asset; entities are image assets. Generate or upload an image first.`
Expand Down Expand Up @@ -406,6 +410,10 @@ const updateEntity: CapabilityExport = {
if (!sourceEntity || !sourceAsset) {
return { error: `Asset ${entityId} is not an entity — use create_entity to tag it.` };
}
const readOnly = Asset.systemEntityRefusal(sourceAsset);
if (readOnly) {
return { error: readOnly };
}
const targetAsset = await Asset.find(userId, targetId);
if (!targetAsset) {
return { error: `Asset ${targetId} was not found.` };
Expand Down Expand Up @@ -537,6 +545,10 @@ const deleteEntity: CapabilityExport = {
if (!entity) {
return { error: `Entity ${entityId} was not found.` };
}
const readOnly = Asset.systemEntityRefusal(asset);
if (readOnly) {
return { error: readOnly };
}

const nextMetadata = { ...(asset.metadata ?? {}) } as Record<string, unknown>;
delete nextMetadata[ENTITY_METADATA_KEY];
Expand Down
58 changes: 56 additions & 2 deletions packages/agents/src/capabilities/scripts.specs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -112,13 +112,18 @@ export const EDIT_SCRIPT_SCHEMA: JsonSchema = {
type: "array",
description:
'Operations in order. Each is {"op": <name>, ...arguments}: ' +
"set_setup {stage?, brief?, format?, length_seconds?, pace?, language?}, " +
"add_speaker {name, color?, provider?, model?, voice?, entityId?}, " +
"set_speaker {target, name?, color?, entityId?}, " +
"set_speaker_voice {target, provider, model, voice, settings?}, " +
"remove_speaker {target}, add_section {title?}, " +
"add_line {text, speaker?, section?, direction?, pause_after_ms?, index?}, " +
"set_line_text {target, text}, set_line_speaker {target, speaker}, " +
"remove_line {target}. `entityId` is an asset id from the ingredient " +
"remove_line {target}. `set_setup` writes the guided flow's own state — " +
"the stage the creator is standing at (idea, format, review, voices, " +
"done), the brief the script is written from, the format (voiceover, " +
"dialogue, interview, ad-read, tutorial), the length in seconds, the " +
"reading pace and the language; write_script reads them. `entityId` is an asset id from the ingredient " +
"library (list_entities) — when set the speaker is that character/entity " +
"and storyboard renders can season prompts with it. A line `target` is " +
"its id, its 0-based index across the script, or its exact text; a " +
Expand All @@ -132,6 +137,37 @@ export const EDIT_SCRIPT_SCHEMA: JsonSchema = {
required: ["script_id", "ops"]
};

export const WRITE_SCRIPT_SCHEMA: JsonSchema = {
type: "object",
properties: {
script_id: { type: "string", description: "Script id." },
provider: {
type: "string",
description:
"Language-model provider id (from find_model) for the writer."
},
model: {
type: "string",
description: "Language model id (from find_model) for the writer."
},
rewrite: {
type: "boolean",
description:
"Rewrite the script that is already there instead of writing a new " +
"one. Lines the rewrite keeps come back with their own ids, so their " +
"takes and storyboard links survive."
},
imported_text: {
type: "string",
description:
"Words the user supplied — pasted, or extracted from a file. They are " +
"split into lines and given speakers, never rewritten. Overrides the " +
"brief as the source of the words."
}
},
required: ["script_id"]
};

export const DERIVE_STORYBOARD_SCHEMA: JsonSchema = {
type: "object",
properties: {
Expand Down Expand Up @@ -253,7 +289,8 @@ export const assembleScriptTimelineSpec: CapabilitySpec = {
export const editScriptSpec: CapabilitySpec = {
name: "edit_script",
description:
"Edit a saved script headlessly: add cast members and assign their " +
"Edit a saved script headlessly: write its guided setup (brief, format, " +
"length, pace, language, stage), add cast members and assign their " +
"voices, add sections, and add, rewrite, reassign and remove lines. " +
"Operations run in order against the stored document and the result is " +
"saved; an open editor picks the change up live. Rewriting a line leaves " +
Expand All @@ -267,6 +304,22 @@ export const editScriptSpec: CapabilitySpec = {
}
};

export const writeScriptSpec: CapabilitySpec = {
name: "write_script",
description:
"Write a script's lines from its brief, format and length — set them " +
"first with edit_script's set_setup op — and save the cast and lines it " +
"produces, replacing what was there. Pass rewrite: true to rewrite an " +
"existing script; the lines it keeps keep their ids, so their takes and " +
"storyboard links survive. Words passed as imported_text are only split " +
"into lines and attributed to speakers, never reworded. Records no take: " +
"voice the result with voice_script_lines.",
inputSchema: WRITE_SCRIPT_SCHEMA,
category: "write",
userMessage: (params) =>
`${params["rewrite"] === true ? "Rewriting" : "Writing"} script ${String(params["script_id"])}`
};

export const deriveStoryboardFromScriptSpec: CapabilitySpec = {
name: "derive_storyboard_from_script",
description:
Expand Down Expand Up @@ -312,6 +365,7 @@ export const scriptsSpecs: readonly CapabilitySpec[] = [
voiceScriptLinesSpec,
assembleScriptTimelineSpec,
editScriptSpec,
writeScriptSpec,
deriveStoryboardFromScriptSpec,
deleteScriptSpec
];
Loading
Loading