Skip to content

Add reference-to-video support to providers and storyboards - #5696

Merged
georgi merged 3 commits into
mainfrom
codex/reference-to-video
Sep 9, 2026
Merged

Add reference-to-video support to providers and storyboards#5696
georgi merged 3 commits into
mainfrom
codex/reference-to-video

Conversation

@georgi

@georgi georgi commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

What changed

Adds an explicit reference-to-video operation to the existing providers and connects it to storyboard rendering. Previously, reference-conditioned models shared the image-to-video path even though a subject reference and an animation's first frame have different roles. A storyboard shot can now use Reference mode to generate a clip from its selected entities' reference images without requiring a keyframe.

Provider and workflow behavior

referenceToVideo accepts separate image and video reference arrays. FAL, KIE, Replicate, AtlasCloud, and MiniMax map these inputs to their supported models' declared fields, with NodeTool forwarding to the selected provider. Validation rejects unsupported media, counts, and reference-audio options before submission. Cancellation and timeout signals propagate through the new requests.

Model discovery distinguishes reference-to-video from first-frame animation and video editing. The new generic ReferenceToVideo node is available through workflow discovery and generated DSL/sandbox wrappers, and the operation participates in execution, cost tracking, and generation records.

Compatibility: the provider-level imageToVideo API now takes one image rather than an array. Its callers are migrated. The existing workflow node retains its serialized image-list input and uses the first non-empty image. Python/WanGP reference generation is deferred.

Storyboard behavior

The shot editor offers Reference alongside Keyframe and Direct modes. Model selection and render dispatch use the selected mode. Reference renders use each shot's ordered entity references, with ownership and image-type checks on stored assets. Render records capture the mode and reference asset IDs so replacing a reference marks the clip stale consistently in the UI and agent tools.

Verification

  • npm run test:affected — complete run passed with TMPDIR=/private/tmp and NODE_OPTIONS=--max-old-space-size=8192.
  • npm run typecheck — passed with NODE_OPTIONS=--max-old-space-size=8192.
  • npm run lint — passed.
  • npm run dev:nodetool -- harness gate --base origin/main — all 15 selfchecks passed.

npm run build:packages passed. The complete affected-test run passed all 127 backend build/test tasks, 1,445 web suites, 63 Electron suites, and 79 mobile suites. Workflow-runner Chromium E2E passed all 27 tests locally. Replicate retry coverage passed all 15 tests, including cancellation during rate-limit backoff with no second provider call. Provider requests were tested with mocks, not paid live generations.

CI follow-up: removed the new node:timers/promises import that prevented the workflow runner from loading in browsers, reusing the existing browser-safe delay helper. The Godot re-export integration test now has a 15-second budget for its two full exports after exceeding the default 5 seconds on CI; its assertions are unchanged.

The new agent-bridge regression was deliberately run without reference context and failed with Expected: false; Received: true for a fresh clip's staleClip flag. Restoring the context passed all 8 tests in storyboardSetupBridge.test.tsx.

Agent capabilities

  • npm run capabilities:check passes.
  • Updated find_model and render_storyboard_clips contracts have generated coverage mappings to the new reference-video capability suite.

@georgi
georgi enabled auto-merge (squash) September 9, 2026 21:46
@georgi
georgi merged commit 2157049 into main Sep 9, 2026
23 checks passed
@georgi
georgi deleted the codex/reference-to-video branch September 9, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant