Add reference-to-video support to providers and storyboards - #5696
Merged
Conversation
georgi
enabled auto-merge (squash)
September 9, 2026 21:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds an explicit reference-to-video operation to the existing providers and connects it to storyboard rendering. Previously, reference-conditioned models shared the image-to-video path even though a subject reference and an animation's first frame have different roles. A storyboard shot can now use Reference mode to generate a clip from its selected entities' reference images without requiring a keyframe.
Provider and workflow behavior
referenceToVideoaccepts separate image and video reference arrays. FAL, KIE, Replicate, AtlasCloud, and MiniMax map these inputs to their supported models' declared fields, with NodeTool forwarding to the selected provider. Validation rejects unsupported media, counts, and reference-audio options before submission. Cancellation and timeout signals propagate through the new requests.Model discovery distinguishes reference-to-video from first-frame animation and video editing. The new generic
ReferenceToVideonode is available through workflow discovery and generated DSL/sandbox wrappers, and the operation participates in execution, cost tracking, and generation records.Compatibility: the provider-level
imageToVideoAPI now takes one image rather than an array. Its callers are migrated. The existing workflow node retains its serialized image-list input and uses the first non-empty image. Python/WanGP reference generation is deferred.Storyboard behavior
The shot editor offers Reference alongside Keyframe and Direct modes. Model selection and render dispatch use the selected mode. Reference renders use each shot's ordered entity references, with ownership and image-type checks on stored assets. Render records capture the mode and reference asset IDs so replacing a reference marks the clip stale consistently in the UI and agent tools.
Verification
npm run test:affected— complete run passed withTMPDIR=/private/tmpandNODE_OPTIONS=--max-old-space-size=8192.npm run typecheck— passed withNODE_OPTIONS=--max-old-space-size=8192.npm run lint— passed.npm run dev:nodetool -- harness gate --base origin/main— all 15 selfchecks passed.npm run build:packagespassed. The complete affected-test run passed all 127 backend build/test tasks, 1,445 web suites, 63 Electron suites, and 79 mobile suites. Workflow-runner Chromium E2E passed all 27 tests locally. Replicate retry coverage passed all 15 tests, including cancellation during rate-limit backoff with no second provider call. Provider requests were tested with mocks, not paid live generations.CI follow-up: removed the new
node:timers/promisesimport that prevented the workflow runner from loading in browsers, reusing the existing browser-safe delay helper. The Godot re-export integration test now has a 15-second budget for its two full exports after exceeding the default 5 seconds on CI; its assertions are unchanged.The new agent-bridge regression was deliberately run without reference context and failed with
Expected: false; Received: truefor a fresh clip'sstaleClipflag. Restoring the context passed all 8 tests instoryboardSetupBridge.test.tsx.Agent capabilities
npm run capabilities:checkpasses.find_modelandrender_storyboard_clipscontracts have generated coverage mappings to the new reference-video capability suite.