Skip to content

CLAUDE.md

Latest commit

 

History

History
1417 lines (1111 loc) · 119 KB

CLAUDE.md

File metadata and controls

1417 lines (1111 loc) · 119 KB
Raw

Storyboard A3 — Project Instructions

What This Is

A Next.js 16 app that transforms the single-file storyboard.html prototype (from simple-infra) into a production agent-powered creative tool. Artists chat with AI agents (Gemini/Claude/OpenAI) to generate, edit, animate, and live-stream media using Livepeer's AI model network.

Current State

Phases 0-7 + Scope Advanced Integration complete. 21 tools, 17 skills, 5 agent plugins (Built-in, Claude, OpenAI, Gemini, Livepeer). Default agent: Livepeer (uses Daydream key for both LLM + inference — no separate Gemini/OpenAI/Claude key required). Default SDK: sdk.daydream.monster.

RESUME HERE — HyperFrames effort (last worked 2026-06-09)

Status: DONE + deployed + main-synced. The HyperFrames×AI hybrid (pixel-perfect HTML/UI over AI video) is fully shipped. Quick state for picking back up:

  • hyperframes-render cap is LIVE + durable on tool-staging-1 (simple-infra #70 + lead-in #72). Playwright + headless Chromium HTML→video compositor. Image us-docker.pkg.dev/livepeer-simple-infra/simple-infra/tool-hyperframes:phase-2-render-2026-06-09, pinned via TOOL_HYPERFRAMES_IMAGE in /opt/tool-host/.env. On sdk.daydream.monster/capabilities. SDK routes it to the tool orch via CAPABILITY_ORCH_MAP in /opt/sdk/.env (a VM-side config, NOT in any repo — re-add the "hyperframes-render":"https://tool-staging-1.daydream.monster:8935" entry if the SDK VM is ever rebuilt). See memory hyperframes-render-enabled.
  • MCP wired (storyboard #423/#424): create_media({model_override:"hyperframes-render", html, background_video_url, aspect_ratio}). Long reels (>15s) exceed create_media's duration cap → call the tool /run/hyperframes-render directly with duration:N (the showcases did this; see /tmp build scripts pattern).
  • i2v fix (#426): fal changed the ltx resolution schema — "1080p" now 422s; fixed to "auto". This had been silently breaking ALL ltx i2v. seedance's separate "fetch failed" is fal latency → use the Ken-Burns stills fallback or alt caps.
  • Phase 2 — all 3 showcases rendered E2E + shipped: Movie-grade Explainer (#427), Breaking-News Flash + E-commerce Lookbook (#428). Each = real reel + playbook (public/playbooks/) + showcase page (public/chapters/*-example.html) + registered on /playbooks + /case-studies. Proven recipe is in memory + the playbooks.
  • Security review fixes (in flight, finishing now): a code review flagged SSRF (caller HTML in --no-sandbox Chromium + unguarded _download). Fixes made in tool-hyperframes/.../tools.py: _assert_public_host() SSRF guard on _download + a Playwright page.route that blocks all non-file:/data: requests. If resuming: confirm these are committed to simple-infra main AND the tool image was rebuilt + redeployed (cd ~/hf-build && sudo docker build -f tool-hyperframes/Dockerfile -t <tag> . then docker compose -f /opt/tool-host/docker-compose.yaml --env-file /opt/tool-host/.env up -d --force-recreate tool-hyperframes). Re-run a render to verify.
  • Known minor follow-ups (not blockers): create_media duration capped at 15 for hyperframes-render via MCP (use direct call for longer); generate_project aspect_ratio still emits 4:3 keyframes (the Ken-Burns step cover-crops to 1080×1920 — a real generate_project bug worth a separate fix); orphaned-Chromium on render timeout (daemon thread, bounded).
  • Push reminder: storyboard + simple-infra both push via gh auth switch --user seanhanca (qianghan lacks access); switch back after.

GitHub

  • Repo: livepeer/storyboard — https://github.com/livepeer/storyboard
  • Push access: Use seanhanca account. qianghan does NOT have push access.
  • Before pushing: gh auth switch --user seanhanca
  • After pushing: gh auth switch --user qianghan

Key Commands

npm run dev          # local dev at localhost:3000
npm run build        # production build
npm run test         # vitest unit tests
npm run test:e2e     # playwright E2E tests
./scripts/dev.sh     # one-command dev setup

SKILL: Cowork / Code human-friendly MCP UX

When users work through Claude Cowork, Desktop, Code, or Cursor with Storyboard MCP, load cowork-ux (public/skills/cowork-ux.md · auto-discovery at .claude/skills/cowork-ux/SKILL.md). Core rules: never paste raw tool JSON; echo human_summary + permission_hint from tool responses before billable actions; dry-run with confirm: false before fan-outs > $0.20; render next_actions as numbered menus; loop subscribe_progress for async jobs. Auto mode = spend_cap + skill policy (no set_auto_mode tool). Extends storyboard-intro.md.

SKILL: When to call the storyboard MCP directly

The storyboard MCP server (mcp__storyboard__*) is loaded in this session. It is the right tool when the user wants you to create / render / generate visual artifacts, not to write a brief about them. Calling the MCP produces real images and videos hosted on fal.media plus a storyboard.daydream.monster/v/proj_* viewer URL the user can open. Writing a brief produces only text the user has to paste somewhere.

Trigger phrases that mean "call the MCP"

If the user's message contains any of these (alone or in combination), default to calling mcp__storyboard__generate_project (1-30 scenes) or mcp__storyboard__create_media (single asset). Do not write a brief and stop.

  • "give me [a/an] [N-]scene story / film / project / sequence …"
  • "create / render / generate / make / produce [N] images / scenes / video …"
  • "using storyboard / via storyboard / using storyboard mcp …" — explicit instruction
  • "produce / render this for me" following a prior brief
  • A standalone visual prompt phrased imperatively: "two bears catching salmon in fall, Studio Ghibli, 8 scenes" — this is a render request, not a discussion request
  • Follow-on after seeing rendered cards: "make it longer / add a scene / try again with a different style"
  • "/story", "/film", "/agent " patterns — these are storyboard chat commands; if asked outside the chat, the MCP equivalent is the right call

Trigger phrases that mean "write a brief"

Default to writing a paste-ready brief (no MCP call) only when the user is clearly asking for the brief itself, e.g.:

  • "give me a prompt I can paste"
  • "what should I type into the agent"
  • "write me a brief for the X agent"
  • "show me how I would ask the agent to do X"
  • An explicit "don't render — just describe"

When ambiguous, ask one short question — but only ONCE

If a single message could reasonably be either ("give me an 8-scene Ghibli bear story"), default to calling the MCP. Most users want the artifact. If the cost of a wrong guess is high (long video render, expensive model), say one line: "Rendering this now via MCP — say 'just the brief' if you only want the prompt." Then proceed.

Do NOT repeatedly ask "do you want me to render?" after the user has already told you what they want. The previous turn in conversation usually disambiguates: if the previous turn was a brief and the user says "try it" / "do it" / "use storyboard", they want the render.

Tool choice quick reference

Intent Tool Notes
Multi-scene visual story (≥2 scenes) generate_project Async by default for >4 scenes or any video model. Returns viewer_url once persisted.
Single image / video / audio create_media Action: generate (text-to-X) or animate (i2v with source_url).
Auto-plan from a brief, possibly long-running submit_creative_job + get_creative_job When auto_plan: true, Hermes plans scenes from the brief + agent's system_prompt.
Free-form long-running agent task submit_agent_task + get_agent_task Hermes executes a multi-step workflow that doesn't fit the per-scene model.
Stylization needs ("Ghibli", "anime", "watercolor", "kid-book") Use gpt-image (openai/gpt-image-2) — best stylization fidelity. recraft-v4 falls back to generic 2D watercolor when name brands are filtered. flux-dev is photoreal-leaning. gemini-image is solid for non-photoreal.
Cinematic photoreal video, single call veo-t2v (fal-ai/veo3.1/fast) Doesn't need an upstream image.
Animation from a keyframe seedance-i2v (bytedance/seedance-2.0/image-to-video) Requires source_url. Auto-plan can't chain — this is for explicit i2v calls.
Finishing — stitch / crop / watermark / export / mix create_media with model_override: "ffmpeg-concat" / "ffmpeg-overlay" / "ffmpeg-export" / "ffmpeg-trim" / "ffmpeg-grid" / "ffmpeg-loop" / "ffmpeg-audio-mix" / "ffmpeg-burn-subtitles" / "hyperframes-caption" / "hyperframes-lower-third" / "obscura-extract-text/markdown/links" / "yolo-detect" / "yolo-segment" etc. 17 live kind: "tool" capabilities. See docs/parity/finishing-primitives-audit-2026-05-28.md for the full live list + parity matrix. Note: opencv-smart-crop, opencv-face-detect, opencv-face-blur, opencv-best-frame appear in lib/sdk/capabilities.ts fallback lists but are NOT registered on production BYOC — code paths resolving to them will fail at runtime. For aspect cropping today, use ffmpeg-export with scale+crop. (hyperframes-render is now LIVE — the full HTML→video compositor; see the HyperFrames section.)
Publish — send to Slack / Drive / Linear / GitHub Not yet live. The kind: "mcp" bridge mechanism exists in lib/sdk/capabilities.ts but zero publisher bridges are registered on production BYOC (confirmed 2026-05-28 via curl https://sdk.daydream.monster/capabilities). Don't claim Publish-to-X is callable yet. Register slack-send-file, drive-upload, linear-create-issue in simple-infra BYOC adapter first.

Three capability kinds

list_capabilities partitions by kind:

  • ai — AI inference (50+ caps today). Default — what every existing entry was before.
  • tool — Deterministic Docker tool containers (ffmpeg, OpenCV, Pillow, hyperframes, obscura). Finish AI output: stitch, crop, watermark, export, audio mix, captions.
  • mcp — External MCP servers proxied through the Livepeer mcp-bridge. Publish to Slack / Drive / Linear / GitHub / Notion / custom internal tools.

All three speak the same /inference envelope and the same payment-ticket flow. Pick by capability name; kind is for grouping / SLA / reasoning, never for "AI is real and tools are different" branching.

High-finishing rule

After AI generation, plan a finishing pass before replying. Single MP4 from N clips → ffmpeg-concat. 16:9 cinematic → 9:16 for TikTok → ffmpeg-export with scale+crop (NOT opencv-smart-crop — not registered on production BYOC). Brand mark → ffmpeg-overlay. Platform spec → ffmpeg-export. The user wants a deliverable, not a punch list of fal.media URLs. See public/skills/finishing-quality.md for the full checklist + recipes.

Polling pattern after submit

Most calls return a job_id synchronously. Poll status via GET /api/creative/{job_id} (with the user's bearer token) at 8-15s intervals until status is done / partial / failed. Show the viewer_url when present. Per-scene urls are populated as scenes finish, even before the job-level done. Don't assume running means stuck — image jobs typically finish in 30-90s for 8 scenes; video jobs 2-5 min.

Known result quality expectations

  • Watercolor / hand-painted 2D: gpt-image > gemini-image > recraft-v4 > flux-dev. recraft drifts toward western kid-book illustration when stylization name-brands are filtered.
  • Cross-shot character consistency: any text-to-image model has none by default. For 6+ shots with a recognizable character, either pass a reference image (gpt-image-edit / kontext-edit) or train a per-character LoRA via /lora.
  • Photoreal product / actor: gpt-image > flux-dev. Both can do this; gpt-image-2 is more controllable.
  • Wordmark / brand text in image: never trust. Always handle text in post or via a static layer; AI text rendering is the hardest known failure mode.

Bearer token

The MCP routes inference through the user's Daydream API key. If you need to read job state outside MCP (e.g. via curl), the user's key lives at ~/.daydream/credentials on the dev machine. Never echo the key in user-visible output.

SKILL: Getting a user's file into a cap-usable URL (auto-ingest + upload verbs)

Caps take inputs as source_url / video_url / audio_url (a public https URL) — they can't read the user's disk or a base64 blob. As of the Upload/Ingest effort (2026-07-07, UPLOAD-INGEST-EXECUTION-REPORT.md) there are now three ingress paths, in order of preference:

  1. Auto-ingest (no extra call needed). create_media (and place_subject, director_re_render's animate path) now detect a data:<mime>;base64,… or gs://… value on source_url / audio_url / end_image_url and host it through resolveAssetUrl inline before dispatch. So "animate this pasted image" just works when the bytes actually arrive. Inline base64 is capped at ~3 MB decoded (the MCP JSON-RPC / Vercel ~4.5 MB body limit); oversize inputs fail cleanly with a pointer to create_upload_url / host-local-file.sh. gs:// has no size limit. Wiring: lib/mcp-server/ingest-asset-ref.ts → resolveAssetUrl.
  2. upload verb (explicit, any media type). upload({ data | source_url, mime_type?, kind?, filename? }) → a persistent public https URL. Accepts base64 for image / video / audio (MIME auto-detected: data: prefix → mime_type → filename ext → magic bytes), or re-hosts any public source_url (server-side fetch + SSRF guard, up to 50 MB). SVG is rejected (XSS). upload_image stays registered as the byte-identical image-only back-compat alias. Writes to Vercel Blob uploads/ (persistent). Same inline ~3 MB transport ceiling as auto-ingest.
  3. create_upload_url verb (large files, capable clients). create_upload_url({ content_type | kind, filename?, size? }) → a one-time signed Blob PUT recipe (upload_url, headers, public_url, ready-to-run curl, 1h expiry, 50 MB cap). A client with a shell (Codex / Cursor) PUTs bytes directly, bypassing the body limit, then uses public_url as source_url. Claude Desktop (no shell) can't PUT — use the webapp drag-drop or the livepeer upload CLI. Route: app/api/upload/signed-put/route.ts (reuses the @vercel/blob client-token machinery; PUT is /api/blob/?pathname=<encoded>, a query param).

Local file on an agent with gcloud (any size): scripts/host-local-file.sh <path> → prints a https://storage.googleapis.com/storyboard-uploads/ingest/… URL (2-day GCS lifecycle; --cleanup <url> to delete now). The MCP server on Vercel can't read local disk, so this is inherently an agent-side step.

Chat-pasted image in Claude Code with NO path/URL (the "animate this" case): a chat-pasted image is vision-only — the bytes are NOT on disk and CANNOT be transcribed to base64 by the model, so none of the three ingress paths above can fire on their own. In Claude Code specifically, the paste base64 IS persisted in the session transcript JSONL (~/.claude/projects/<cwd-slug>/*.jsonl, {type:"image",source:{type:"base64",media_type,data}}). Recover it hands-free with scripts/recover-pasted-image.mjs --host → prints a public URL to use as source_url (falls back to a data: URL for MCP auto-ingest if no gcloud). Skill: public/skills/recover-pasted-image.md; per-user Claude Code auto-discovery install goes in .claude/skills/recover-pasted-image/SKILL.md. Graceful fallback (any client, incl. Claude Desktop which has no readable transcript): ask the user to save the file + give a path, or drag it into the storyboard webapp for a URL. See UPLOAD-INGEST-CLAUDECODE-DIAGNOSIS.md for the full root cause.

Storage / lifecycle map (be precise — two different ingest/)

Prefix Backed by Written by Lifecycle
Blob uploads/ Vercel Blob upload / upload_image / create_upload_url / /api/upload Persistent (ephemeral:false)
Blob ingest/ Vercel Blob resolveAssetUrl base64 (auto-ingest) ~2-day TTL via /api/cron/blob-ingest-sweep (PR4). ephemeral:true.
GCS ingest/ gs://storyboard-uploads/ scripts/host-local-file.sh 2-day bucket lifecycle rule. ephemeral:true.

resolveAssetUrl() (lib/assets/resolve-asset.ts) is the single router: public https → passthrough, gs:// → https, base64 → Blob ingest/, local → throws (host agent-side first). It returns { url, host, ephemeral } and ephemeral is now truthful for both ingest/ prefixes.

GCS bucket: gs://storyboard-uploads/ (project livepeer-simple-infra), public-read. GCS lifecycle (set 2026-06-10, NOT in any repo): re-add if the bucket is rebuilt — gcloud storage buckets update gs://storyboard-uploads --lifecycle-file=<json> where json = {"rule":[{"action":{"type":"Delete"},"condition":{"age":2,"matchesPrefix":["ingest/"]}}]}. The Blob ingest/ TTL is enforced in-app by the daily blob-ingest-sweep cron (lib/assets/blob-ingest-sweep.ts), not a Blob-native lifecycle (Blob has none).

SKILL: Infrastructure Architecture

The Full Stack

Browser (Next.js on Vercel)
  → SDK Service (sdk.daydream.monster) — Python FastAPI on GCP VM
    → BYOC Orchestrator (byoc-staging-1.daydream.monster) — go-livepeer for inference
      → fal.ai / Gemini (actual AI model execution)
    → Scope Orchestrators (orch-staging-1/2.daydream.monster) — go-livepeer for LV2V
      → fal.ai Scope runner (live video-to-video pipeline)
  → Signer (signer.daydream.live) — payment ticket signing

VMs (GCP project: livepeer-simple-infra)

VM Purpose IP Domain
sdk-staging-1 SDK Service (inference + LV2V proxy) 34.168.200.215 sdk.daydream.monster
byoc-staging-1 BYOC orch (image/video/audio inference) 8.229.77.130 byoc-staging-1.daydream.monster
byoc-a3-staging-1 A3 BYOC orch (storyboard capabilities) 136.109.56.80 byoc-a3-staging-1.daydream.monster
orch-staging-1 Scope LV2V orch (shared, us-west) 34.169.235.70 orch-staging-1.daydream.monster
orch-staging-2 Scope LV2V orch (shared, us-east) — orch-staging-2.daydream.monster

SSH Access

gcloud compute ssh <vm-name> --zone=us-west1-b --project=livepeer-simple-infra
# Docker commands need sudo:
sudo docker ps
sudo docker logs sdk-service --tail 50
sudo docker exec sdk-service env | grep LV2V

Config Locations

  • SDK: /opt/sdk/docker-compose.yaml + /opt/sdk/.env
  • BYOC: /opt/byoc/docker-compose.yaml + /opt/byoc/.env
  • Orch: Docker run args (no compose file)

SKILL: SDK Service (sdk.daydream.monster)

What It Does

Python FastAPI app wrapping livepeer-python-gateway. Handles:

  • /inference — run AI model (image/video/audio) via BYOC orch
  • /capabilities — list available models (live from BYOC orch)
  • /stream/start|stop|status|publish|frame|control — LV2V proxy
  • /smart/inference — multi-step inference with model selection
  • /streams — list active LV2V streams (monitoring)
  • /streams/cleanup — emergency kill all streams

Key Environment Variables

ORCH_URL=https://byoc-staging-1.daydream.monster:8935    # BYOC for inference
SIGNER_URL=https://signer.daydream.live                   # Payment signer
LV2V_ORCH_URLS=https://orch-staging-1.daydream.monster:8935,https://orch-staging-2.daydream.monster:8935
LV2V_MODEL=scope
LV2V_PIPELINE=longlive                                    # Pipeline name on fal runner

Authentication Flow

  1. Browser sends Authorization: Bearer sk_... (Daydream API key) to SDK
  2. SDK's _extract_signer_headers() forwards it to signer for payment tickets
  3. SDK's _resolve_daydream_user_id() calls Daydream API to get Clerk user ID
  4. User ID passed to fal runner as daydream_user_id (required for auth)

CRITICAL: Without the Daydream API key, ALL inference fails (signer 401) and LV2V streams get ACCESS_DENIED from fal.

Updating SDK Code

gcloud compute scp app.py sdk-staging-1:/tmp/app.py --zone=us-west1-b --project=livepeer-simple-infra
gcloud compute ssh sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra --command="sudo docker cp /tmp/app.py sdk-service:/app/app.py && sudo docker restart sdk-service"

SKILL: BYOC Orchestrator

What It Does

go-livepeer in BYOC mode. Routes inference requests to fal.ai/Gemini providers via a serverless proxy adapter.

Capabilities (fal model IDs — MUST match fal.ai exactly)

Capability fal Model ID Verified
flux-dev fal-ai/flux/dev ✅
flux-schnell fal-ai/flux/schnell ✅
recraft-v4 fal-ai/recraft/v4/pro/text-to-image ✅
gemini-image gemini/gemini-2.5-flash-image ✅
ltx-i2v fal-ai/ltx-2.3/image-to-video ✅
ltx-t2v fal-ai/ltx-2.3/text-to-video ✅
kontext-edit fal-ai/flux-pro/kontext ✅
bg-remove fal-ai/birefnet ✅ (NOT fal-ai/bg-remove)
topaz-upscale fal-ai/aura-sr ✅ (NOT fal-ai/topaz-upscale)
chatterbox-tts fal-ai/chatterbox/text-to-speech ✅ (NOT fal-ai/chatterbox-tts)
gemini-text gemini/gemini-2.5-flash ✅
nano-banana fal-ai/nano-banana-2 ✅
seedream-5-lite fal-ai/bytedance/seedream/v5/lite/text-to-image ✅
seedance-i2v bytedance/seedance-2.0/image-to-video ✅
seedance-i2v-fast bytedance/seedance-2.0/fast/image-to-video ✅
grok-imagine-quality xai/grok-imagine-image/quality/text-to-image ✅
grok-imagine-edit xai/grok-imagine-image/quality/edit ✅
ideogram-bg-remove fal-ai/ideogram/remove-background ✅
nemotron-omni nvidia/nemotron-3-nano-omni ✅
grok-imagine-video xai/grok-imagine-video/v1.5/image-to-video ✅ 2026-06-02 — $0.147/s @ 720p, fluid camera + coherent multi-shot scenes
ltx-q-i2v fal-ai/ltx-2.3-quality/image-to-video ✅ 2026-06-02 — $0.056/s, LTX 2.3 Quality tier (higher fidelity than base ltx-i2v)
ltx-q-i2v-lora fal-ai/ltx-2.3-quality/image-to-video/lora ✅ 2026-06-02
ltx-q-t2v fal-ai/ltx-2.3-quality/text-to-video ✅ 2026-06-02
ltx-q-t2v-lora fal-ai/ltx-2.3-quality/text-to-video/lora ✅ 2026-06-02
ltx-q-a2v fal-ai/ltx-2.3-quality/audio-to-video ✅ 2026-06-02 — audio-driven (the only a2v cap on prod)
ltx-q-a2v-lora fal-ai/ltx-2.3-quality/audio-to-video/lora ✅ 2026-06-02
ltx-q-ref2v fal-ai/ltx-2.3-quality/reference-video-to-video ✅ 2026-06-02
ltx-q-ref2v-lora fal-ai/ltx-2.3-quality/reference-video-to-video/lora ✅ 2026-06-02
ltx-q-hdr fal-ai/ltx-2.3-quality/hdr ✅ 2026-06-02 — $0.063/s, EXR frames + MP4 preview for pro HDR pipelines
ltx-q-hdr-lora fal-ai/ltx-2.3-quality/hdr/lora ✅ 2026-06-02
cosmos-3-i2v nvidia/cosmos-3-super/image-to-video ✅ 2026-06-02 — $0.0525/s, NVIDIA omnimodal world model, strong motion + temporal coherence
cosmos-3-image nvidia/cosmos-3-super/text-to-image ✅ 2026-06-02 — $0.042/img
mirelo-sfx Mirelo-AI/sfx1.6/text-to-audio ✅ 2026-06-02 — $0.0105/s, premium SFX + ambiancer (loopable soundscapes)
mirelo-sfx-inpaint Mirelo-AI/sfx1.6/inpaint-audio ✅ 2026-06-02 — replace audio segments inside an existing track
mirelo-sfx-extend Mirelo-AI/sfx1.6/extend-audio ✅ 2026-06-02 — seamless tails on existing audio
mirelo-sfx-v2v Mirelo-AI/sfx1.6/video-to-video ✅ 2026-06-02 — auto-synced SFX track for any video, up to 60s
heygen-twin fal-ai/heygen/avatar5/digital-twin ✅ 2026-06-02 — $0.105/s, lifelike talking-head from a 15s training clip
flux-erase fal-ai/flux-pro/v1/erase ✅ 2026-06-02 — $0.0315/img, mask-based object/text removal + reconstruction
krea-2 krea/v2/medium/text-to-image ✅ 2026-06-02 — $0.0315/img, KREA's foundation image model (moodboard-driven)
krea-2-large krea/v2/large/text-to-image ✅ 2026-06-02 — $0.063/img, premium quality variant
flux-vto fal-ai/flux-pro/v1/vto ✅ 2026-06-02 — $0.05/img, BFL Virtual Try-On (multi-garment)

CRITICAL: fal renames models. Always verify with curl https://fal.ai/models/<model_id> before adding.

Adding / Updating BYOC Capabilities

Capabilities are registered by the inference adapter. The adapter reads CAPABILITIES_JSON from /opt/byoc/.env on startup and auto-registers each entry with the BYOC orch. No go-livepeer code changes needed.

Step 1: Find the fal model ID. Go to https://fal.ai/models/<model_path> and confirm the model exists. fal renames models — always verify.

Step 2: Add to CAPABILITIES_JSON on the VM. Use Python to avoid shell quoting issues with JSON:

gcloud compute ssh byoc-staging-1 --zone=us-west1-b --project=livepeer-simple-infra
sudo python3 -c "
import json
with open('/opt/byoc/.env') as f: content = f.read()
# Find the existing CAPABILITIES_JSON line and parse it
for line in content.splitlines():
    if line.startswith('CAPABILITIES_JSON='):
        caps = json.loads(line.split('=', 1)[1])
        break
# Append new capability
caps.append({'name': 'my-new-cap', 'model_id': 'fal-ai/some/model', 'capacity': 2, 'price_per_unit': 3})
# Write back
import re
content = re.sub(r'^CAPABILITIES_JSON=.*$', 'CAPABILITIES_JSON=' + json.dumps(caps), content, flags=re.MULTILINE)
with open('/opt/byoc/.env', 'w') as f: f.write(content)
print(f'Wrote {len(caps)} capabilities')
"

Step 3: Restart the adapter (must be down && up, not restart, so compose re-reads .env):

sudo bash -c 'cd /opt/byoc && docker compose down && docker compose up -d'

Step 4: Verify registration:

sudo docker logs byoc-adapter --tail 10  # Look for "Registered capability 'my-new-cap'"
curl -s https://sdk.daydream.monster/capabilities | python3 -c "import sys,json; [print(c['name']) for c in json.load(sys.stdin) if 'my-new' in c['name']]"

Step 5: Update storyboard code (in storyboard-a3 repo):

  • lib/sdk/capabilities.ts — add to FALLBACK_CAPABILITIES set + keyword resolution
  • lib/tools/compound-tools.ts — add to FALLBACK_CHAINS + selectCapability routing + user mention detection
  • CLAUDE.md — update capabilities table and count

Each capability entry:

{"name": "capability-name", "model_id": "fal-ai/vendor/model", "capacity": 2, "price_per_unit": 3}
  • name: short kebab-case name used in storyboard code (e.g., seedance-i2v)
  • model_id: exact fal model path (e.g., bytedance/seedance-2.0/image-to-video). Some models omit the fal-ai/ prefix.
  • capacity: max concurrent jobs (2 for video/3D, 4 for image/audio)
  • price_per_unit: relative cost tier (1=cheap, 5=expensive)

Common mistakes:

  • Using docker restart instead of down && up — won't re-read .env
  • Using sed or shell quoting for JSON — use Python instead, JSON with {} breaks shell interpolation
  • Wrong model_id — fal renames models; verify the URL works first
  • The BYOC VM is byoc-staging-1 (not byoc-a3-staging-1 — that VM no longer exists)

SKILL: LV2V (Live Video-to-Video) Streaming

Full debugging guide: docs/key-insights-scope.md

The 4 Required Fixes (all must be present)

  1. Orch timeout: -liveOutSegmentTimeout 300s on orch (prevents 30s watchdog kill)
  2. Fal keepalive: PR 864 ping/pong in fal app (prevents WebSocket disconnect)
  3. Graph params: start_stream MUST include graph with source/pipeline/sink nodes (creates input trickle channel)
  4. Edge format: Edges MUST use from/from_port/to_node/to_port/kind NOT source/target (enables pipeline wiring)

Graph Format (CRITICAL — must match Scope exactly)

start_stream_params = {
    "pipeline_ids": ["longlive"],
    "prompts": prompt,
    "graph": {
        "nodes": [
            {"id": "input", "type": "source", "source_mode": "video"},
            {"id": "longlive", "type": "pipeline", "pipeline_id": "longlive"},
            {"id": "output", "type": "sink"},
        ],
        "edges": [
            {"from": "input", "from_port": "video", "to_node": "longlive", "to_port": "video", "kind": "stream"},
            {"from": "longlive", "from_port": "video", "to_node": "output", "to_port": "video", "kind": "stream"},
        ],
    },
}

Trickle Channel URLs

Job-level (DON'T use for MediaPublish/MediaOutput):
  {id}, {id}-out, {id}-control, {id}-events

Per-stream (USE these — created by start_stream with graph):
  {id}-1-in   — input (MediaPublish writes here)
  {id}-2-out  — output (MediaOutput reads here)

Diagnostic Checklist (check in order)

Symptom Check
503 on stream start Daydream API key? daydream_user_id resolved? FAL_API_KEY on orch?
Stream dies <30s -liveOutSegmentTimeout 300s on orch? PR 864 deployed?
Publish 404 (fresh stream) Graph in start_stream? "start_stream returned 2 channels"?
Publish 410 Gone SDK restarted — stream session lost. Client should auto-stop on first 410. Restart the stream from the UI.
Publish OK but no output Edge format correct? Pipeline node ID = pipeline_id?
Output OK but black screen Card renders as <img> not <video>?
SDK unreachable from CLI but browser works GCP metadata server failure mode — gcloud compute instances reset sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra then cd /opt/sdk && sudo docker compose down && sudo docker compose up -d to recreate the stale docker network

Orch Requirements

-liveOutSegmentTimeout 300s
FAL_API_KEY=<key>
FAL_KEY=<key>
LIVE_AI_WS_PREFIX=wss://fal.run/daydream

fal Runner Auth

Requires daydream_user_id in start_lv2v params. SDK resolves from Daydream API key via _resolve_daydream_user_id(). Without it: ACCESS_DENIED.

Stream Health Monitoring

# Check active streams
curl -s https://sdk.daydream.monster/streams | python3 -m json.tool

# Kill zombie streams
curl -s -X POST https://sdk.daydream.monster/streams/cleanup

# Monitor a specific stream
./scripts/monitor-stream.sh <stream_id> [sdk_url] [api_key]

Zombie Stream Prevention

  • SDK has a stream reaper (checks every 30s, kills idle >2min or age >1hr)
  • Browser has beforeunload handler to stop streams on page close
  • Client-side dead stream detection (auto-stop after 30 consecutive publish failures)
  • SDK returns HTTP 410 Gone for unknown stream IDs — the publish handler checks both _stream_sessions and _lv2v_jobs first; if neither has the stream, it raises HTTPException(410, "Stream no longer exists"). Client treats first 410 as terminal and stops immediately. This is the primary defense against zombies after SDK restarts (the in-memory state is wiped but browsers don't know).

LV2V Failure Pattern: SDK Restart → Browser Zombies (RECURRING)

Symptom: User reports "videos work sometimes, fail other times" with 404 or 410 on /stream/{id}/publish?seq=NNNN where seq is in the thousands.

Root cause chain:

  1. SDK container crashes/restarts (commonly OOM Exited (137) under load, or the GCP metadata-server failure mode)
  2. _stream_sessions and _lv2v_jobs dicts are wiped (in-memory only)
  3. Browser tabs holding old streams keep publishing frames at 10fps to dead stream IDs
  4. Newly-started streams work fine (they get fresh IDs in the new SDK process)
  5. Old streams should auto-stop on the first 410 — verify the client has the 410-handling code in lib/stream/session.ts

Diagnosis steps:

# 1. Is the SDK reachable?
curl -s --max-time 10 -o /dev/null -w "%{http_code}\n" https://sdk.daydream.monster/capabilities

# 2. What containers are running?
gcloud compute ssh sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra --command="sudo docker ps -a --format '{{.Names}} {{.Status}}'"

# 3. Why did sdk-service exit? (137 = SIGKILL/OOM, 0 = clean restart)
gcloud compute ssh sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra --command="sudo docker inspect sdk-service --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}}'"

# 4. Recent healthcheck activity?
gcloud compute ssh sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra --command="sudo tail -30 /var/log/sdk-healthcheck.log"

# 5. List active streams (server-side state)
curl -s https://sdk.daydream.monster/streams | python3 -m json.tool

# 6. Test publish to a known-dead stream — should return 410, not 404
curl -X POST -H "Content-Type: image/jpeg" --data-binary "test" "https://sdk.daydream.monster/stream/DEADBEEF/publish?seq=1" -w "\nHTTP %{http_code}\n"

Recovery steps (in order):

# A. Server-side cleanup (kills any zombies still in SDK memory)
curl -X POST https://sdk.daydream.monster/streams/cleanup

# B. If SDK is unreachable but VM is RUNNING — likely the GCP metadata bug
gcloud compute instances reset sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra

# C. After reset, the docker network reference is stale — rebuild it
gcloud compute ssh sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra --command="cd /opt/sdk && sudo docker compose down && sudo docker compose up -d"

# D. Verify SDK is back
curl -s https://sdk.daydream.monster/capabilities | head -c 200

Why 410 instead of 404: The old SDK fell through to a non-existent BYOC trickle URL which returned 404. The client treated 404 as "transient" and only auto-stopped after 30 consecutive failures — which rarely happened because the orch returned mixed results. Now the SDK returns 410 immediately for unknown stream IDs, and the client stops on the very first one. See lib/stream/session.ts — look for if (r.status === 410) in startPublishing.

Why this keeps happening: LV2V stream sessions are intentionally ephemeral — persisting them across SDK restarts would create stale-state bugs and slow startup. The right pattern is "server says I'm gone clearly, client trusts it and stops." The 410 fix is the systematic defense; the underlying SDK crash is a separate operational concern (memory limits on the e2-medium VM, fal runner bugs, etc).

SKILL: Signer (signer.daydream.live)

What It Does

Signs payment tickets for on-chain Arbitrum payments. Used by BOTH inference and LV2V.

Endpoints

  • /sign-orchestrator-info — sign orch info for job creation
  • /generate-live-payment — create payment tickets for LV2V streams
  • /discovery/staging.json — orchestrator discovery

Auth

All signer requests require Authorization: Bearer sk_... (Daydream API key). The SDK forwards this from the browser's request headers via _extract_signer_headers().

SKILL: Capability Resolution (LLM Model Name Hallucination)

LLMs hallucinate model names (e.g., flux-pro, kling-i2v, lux-tts). The storyboard defends against this:

  1. Live capability list fetched from SDK /capabilities on app startup
  2. resolveCapability() fuzzy-matches invalid names: prefix match → keyword match → action default
  3. create_media tool has NO model_override in schema — agent can't pick models
  4. selectCapability() maps action → model deterministically
  5. Validation at execution time in both create_media and inference tools

Resolution chain

"flux-pro" → prefix "flux-" → "flux-dev"
"kling-i2v" → keyword "i2v" → "ltx-i2v"
"lux-tts" → keyword "tts" → "chatterbox-tts"

SKILL: Project-attached Voice + LoRA Auto-injection (MCP)

When create_media is called with a project_id, the MCP server auto-injects voices and LoRAs the user has bound to that project — so the agent doesn't need to remember to pass audio_url (for chatterbox-tts) or lora: { weights_url, trigger } (for flux-dev / flux-schnell) on every call.

Bind by surface:

  • voice_attach({ voice_id, project_id }) — binds a voice from the user's library (see voice_create / voice_list).
  • attach_lora_to_project({ lora_id, project_id }) — binds a trained LoRA.

Injection rules (see lib/mcp-server/auto-injection.ts):

  • Explicit always wins — passing audio_url or lora bypasses the lookup entirely.
  • Voice: only injects when the resolved capability is chatterbox-tts.
  • LoRA: only injects when the resolved capability is flux-dev or flux-schnell (the only LoRA-compatible base models on BYOC).
  • Best-effort: blob read failures, missing attachments, tombstoned records all return null → no injection, inference proceeds with the original payload. A transient Blob blip never breaks create_media.
  • Picks first match: if the project has multiple attached voices or LoRAs, the first one (insertion order in the attachments blob) wins. Pass explicit audio_url / lora to override.

Cross-surface scope:

  • MCP / CLI: auto-injection is live (CLI proxies MCP).
  • Browser create_media (compound-tools.ts) does NOT see MCP-side project attachments — the browser path has its own useLoraStore.getState().getActive() flow that injects whichever LoRA the user marked active via /lora apply. MCP-side attach_lora_to_project and browser-side active-LoRA are independent today. Bridging them needs a REST endpoint + Zustand sync (deferred).

Wiring: lib/mcp-server/tools/create-media.ts runs both helpers right after capability resolution, before the LoRA URL safety check (so injected weights_url still passes checkLoraWeightsUrl()).

SKILL: Director v2 + Brand-Locked Finishing

The chat-driven editorial layer on top of a project. Each intent is a separate MCP tool that reuses create_media (and so inherits entity routing + LoRA / voice / brand-kit auto-injection) where it makes sense, and goes straight to the SDK (sdkPost) for deterministic tool calls where create_media semantics don't apply.

The 6 MCP tools

Tool What it does Wiring
director_re_render({ project_id, scene_index, prompt?, action?, model_override?, prefer_fast? }) Re-prompt ONE scene; others stay locked. Reuses stored prompt if omitted. Delegates to createMedia with project_id; persists new url/capability/prompt back onto scene blob.
director_insert({ project_id, between: [a, a+1], prompt, title?, action? }) Insert a NEW scene at position b. Strict adjacency: b === a+1. Edge cases: [-1, 0] prepends, [N-1, N] appends. Delegates to createMedia; re-indexes subsequent scenes so .index stays in sync with array position.
director_retime({ project_id, scene_index, head_offset }) Trim-only v1. head_offset > 0 trims that many seconds off the start. tail_offset / target_duration / extend are accepted in the schema but rejected at runtime with a friendly "ships in v2" message. Calls ffmpeg-trim via sdkPost directly (no create_media detour — tool cap has no entity / LoRA / brand-kit semantics to inherit). Validates the scene is a video via CAPABILITY_KIND map + URL ext heuristic.
director_variant({ project_id, name, scenes?: number[], title_override? }) Create a child variant project. scenes omitted = copy all parent scenes verbatim. Re-indexes contiguously starting at 0. Sets parent_project_id link. Inherits brief/style/model/entities/consistency_priority from parent. Parent is invariant.
director_beat_lock({ project_id, bpm, beats_per_scene?, audio_ref?, start_offset_sec?, variant_name? }) Explicit-BPM v1. Creates a variant with uniform target_duration_sec = beats_per_scene × 60/bpm on every scene. Returns beat timestamps. Audio-driven BPM detection deferred to v2 (no audio-beat-detection cap is live; lib/agents/beat-extractor.ts is LLM scene-breakdown, not audio analysis). Creates a variant project; stamps target_duration_sec on every scene. Future director_retime v2 will read the field.
director_export({ project_id, brand_kit_id?, soundtrack? }) Stitch scenes via ffmpeg-concat, lay a soundtrack under the reel, then run the brand kit's finishing_chain sequentially. Each step's output URL flows into the next step's source_url param. Goes through sdkPost directly per step. Skips ffmpeg-concat when only one scene URL exists. soundtrack: "auto" (default) prefers an existing director_score bed; with none, auto-generates one via sonilo-t2m and mixes it via ffmpeg-mux + audio_fill:"loop" (Summit Gap B-2, "sound by default") — not ffmpeg-audio-mix, which is a multi-track audio-only mixer expecting tracks:[...] (see tests/unit/mux-audio-routing.test.ts for the exact failure mode this avoids). soundtrack: "off" skips any bed, including an existing scored one, for that export only. Best-effort throughout — a generation/mix failure warns and exports unmixed, never blocks. Skips kind:"mcp" publish bridges by name prefix (slack-/drive-/linear-/notion-/github-) with a structured warning — none are registered on production BYOC. Aborts chain on first upstream error with partial_url for diagnostics. Cross-surface: --soundtrack auto|off on the CLI's director export; a 🎵 toggle on the webapp Studio bar.

Brand Kit finishing_chain (the schema director_export consumes)

Added to MirroredContextKit as an optional field — backwards-compatible, max 12 steps. Each step is { capability, params?, label? }. Set via brand_kit_set_finishing_chain({ kit_id, steps }); pass steps: [] to clear. Surfaced in brand_kit_show / context_kit_show (text + structuredContent).

The non-publish chain (watermark + aspect + audio mix + LUFS + end-card) works today; publish steps are honestly skipped until the Slack/Drive/Linear/Notion/GitHub kind:"mcp" bridges register on production BYOC in simple-infra.

Schema fields added in P2

// In lib/mcp-server/storage.ts:
interface ProjectScene {
  // ...existing fields...
  target_duration_sec?: number; // beat_lock writes; retime v2 reads
}

interface StoredProject {
  // ...existing fields...
  parent_project_id?: string;   // variant / beat_lock link
  variant_name?: string;        // e.g. "teaser_30s", "beat_lock_120bpm"
}

interface FinishingStep {
  capability: string;
  params?: Record<string, unknown>;
  label?: string;
}

interface MirroredContextKit {
  // ...existing fields...
  finishing_chain?: FinishingStep[]; // max 12, consumed by director_export
}

Honest v2 deferrals (DO NOT re-attempt these without scope conversation)

  • director_retime: tail_offset, target_duration, extend (seedance-i2v continuation). Need a video-duration probe primitive that hasn't shipped.
  • director_variant: natural-language intent → scenes picker ("lead with the climax"). Needs an LLM planner subagent — explicit scenes: number[] is the deterministic v1.
  • director_beat_lock: auto-BPM detection from audio_ref. Needs a new BYOC cap or LLM-audio analysis pass.
  • director_export: aspect-ratio fanout (single output today). Per-aspect parallel runs land with v2 + cost aggregation across the chain.
  • brand_kit_set_finishing_chain publish steps: slack-send-file and friends are accepted in the schema but skipped at execution. Bridge registration in simple-infra BYOC adapter is the unlock.

Cross-surface parity (all 6 tools land on MCP + CLI + Webapp)

Tool MCP CLI (livepeer director … / livepeer brand-kit …) Webapp
re_render ✅ ✅ director re-render <p> <i> ["<prompt>"] ✅ right-click scene card
insert ✅ ✅ director insert <p> --between a,b "<prompt>" ✅ right-click scene card
retime ✅ ✅ director retime <p> <i> --trim-head <s> ✅ right-click scene card
variant ✅ ✅ director variant <p> <name> [--scenes 0,2,4] ✅ empty-space menu
beat_lock ✅ ✅ director beat-lock <p> --bpm <n> ✅ empty-space menu
export ✅ ✅ director export <p> [--brand-kit ckit_…] ✅ empty-space menu
brand_kit set_finishing_chain ✅ ✅ brand-kit set-finishing-chain <kit> --from <json> chat-only (no UI editor yet)

Webapp entries are all chat-routing (prefillChat / sendToAgent) — the agent plugins translate natural-language phrasing into the matching MCP tool call.

SKILL: Agent Plugins

4 Plugins Available

Plugin Model API Route Tool Format
Built-in None (SDK enrich) — Direct SDK calls
Claude claude-sonnet-4-6 /api/agent/chat Anthropic tool_use
OpenAI gpt-4o /api/agent/openai function calling
Gemini gemini-2.5-flash /api/agent/gemini functionDeclarations

Common Patterns

  • All plugins share the same 21-tool registry
  • System prompt loaded from skills/base.md + live capabilities + memory + canvas context
  • Tool-use loop: LLM calls tool → execute → send result → loop until done
  • Concurrent prompt execution (no queue unless depends_on)

Gemini Quirk

Gemini sometimes returns empty content (no parts) after tool execution. This is normal — the tool results are on the canvas. Don't show an error.

SKILL: Scope Domain Agent (Advanced LV2V)

Architecture

The Scope Domain Agent is an expert subsystem that translates natural language into precise Scope configurations. It runs as a tool layer accessible by all agent plugins (Gemini, Claude, OpenAI) — NOT a standalone plugin.

User: "stream my image with anime style"
  → Agent loads scope-agent skill (domain knowledge)
  → Agent calls scope_start tool (graph=simple-lv2v, preset=anime, source.ref_id=card)
  → scope_start builds ScopeStreamParams (graph, pipeline_ids, noise_scale, etc.)
  → session.ts passes scopeParams to SDK /stream/start
  → SDK passes full params through to fal runner (cloud/livepeer_app.py)
  → Scope validates graph, loads pipeline, starts stream

Key Principle: SDK as Passthrough

The SDK does NOT validate Scope params. It passes them through to the fal runner which validates against Scope's graph_schema.py. This means:

  • Adding new Scope features requires zero SDK changes — just send the right params
  • SDK only needs changes when the transport protocol changes (new channel types, etc.)
  • feat branch feat/scope-advanced-params in simple-infra contains the passthrough change

6 Scope Tools

Tool Purpose
scope_start Start LV2V with full config (graph, preset, LoRA, VACE, source)
scope_control Update params mid-stream (prompt, noise, denoising, LoRA scale)
scope_stop Stop a stream
scope_preset List/apply named presets (dreamy, cinematic, anime, etc.)
scope_graph List/build graph templates
scope_status Stream health, FPS, frames published/received

6 Graph Templates

Template Graph Use Case
simple-lv2v source→longlive→sink Default webcam/video transform
depth-guided source→depth_anything→longlive→sink Preserve depth/structure
scribble-guided source→scribble→longlive→sink Edge-guided generation
interpolated source→longlive→rife→sink 2x frame interpolation
text-only longlive→sink Pure text-to-video (no input)
multi-pipeline source→pipeline_a→pipeline_b→sink Chained transforms

7 Presets

dreamy (noise=0.7), cinematic (0.5), anime (0.6), abstract (0.95), faithful (0.2), painterly (0.65), psychedelic (0.9)

Key Scope Parameters (runtime, change mid-stream)

  • noise_scale (0.0-1.0): creativity level. 0=faithful, 1=ignore input
  • kv_cache_attention_bias (0.01-1.0): temporal consistency. Low=responsive, high=stable
  • denoising_step_list [1000,750,500,250]: quality vs speed
  • reset_cache: one-shot flush for dramatic style change
  • lora_scales: adjust LoRA strength without restart
  • prompts: string or [{text, weight}] for spatial blending

Scope Source Code Reference

  • Graph schema: Scope/scope/src/scope/server/graph_schema.py — node types, edge format, validation
  • Graph executor: Scope/scope/src/scope/server/graph_executor.py — builds queues, wires processors
  • Pipeline registry: Scope/scope/src/scope/core/pipelines/registry.py — auto-registration by VRAM
  • LongLive config: Scope/scope/src/scope/core/pipelines/longlive/schema.py — all params with ranges
  • Server schema: Scope/scope/src/scope/server/schema.py — Parameters model, VACE, transitions
  • LoRA manager: Scope/scope/src/scope/core/pipelines/wan2_1/lora/manager.py — 3 merge strategies
  • Cloud runner: Scope/scope/src/scope/cloud/livepeer_app.py — fal.ai WebSocket + trickle I/O
  • Modulation: Scope/scope/src/scope/server/modulation.py — beat-synced param oscillation

Multi-Source Input

FrameExtractor (lib/stream/frame-extractor.ts) supports: webcam, image, video, URL. All produce JPEG blobs at configurable FPS for trickle publish.

Stream Card Controls

Stream-type cards have: run/stop buttons, pub/recv stats, inline agent input (type natural language to control stream in real-time). Video cards have fullscreen button.

SKILL: Templates Gallery + Remix Funnel (Summit Gap B-3)

/templates is a curated, one-click-remix gallery — NOT the same "templates" as list_templates/apply_template (moodboard brief starters, lib/mcp-server/tools/templates.ts) or the Telegram bot's /template commands. Both concepts share the word by coincidence; see the disambiguating comment at the top of each file.

  • Source of truth: public/playbooks/*.md frontmatter (title/tier/format/persona/budget_usd/caps/skills/showcases/status/reliability), parsed at build time by scripts/build-template-index.mjs into public/template-index.json. Never throws on a single file's messy/missing frontmatter (many playbooks predate the frontmatter convention entirely, or have partial fields) — skip + log; exits 1 loudly only if the directory is unreadable or the index would end up empty.
  • vercel.json's buildCommand invokes the script explicitly (node scripts/build-template-index.mjs && ...) — Vercel calls next build directly, not npm run build, so npm's automatic prebuild lifecycle hook (also wired in package.json, for local npm run build convenience) never fires for the real deploy. Both wirings are needed; neither alone covers both paths.
  • Curation (lib/templates/curated.ts): exactly 12 slugs, objectively reproducible from decision D-5's filter (status starts with "live", reliability >= 4.0, non-empty showcases, has a title) — not hand-picked. A dedicated test re-derives the filter against the live index and asserts it matches the curated set exactly, so a future playbook edit that silently breaks reproducibility (e.g. a status-field casing typo — this happened once with realtime-music.md's "LIVE" vs every sibling's lowercase "live") fails CI instead of silently drifting.
  • Remix flow: each TemplateCard's "Remix ▸" link is /?template=<slug>. ChatPanel.tsx reads that query param in a mount-time useEffect (plain window.location.search, not useSearchParams() — avoids a new Suspense boundary), resolves it via lib/templates/resolve-remix-prompt.ts against CURATED_TEMPLATES, seeds the chat input, and strips the query param. User still presses Send — no auto-submit.
  • Webapp-only by design, not a cross-surface-parity gap: this is a discovery/UX funnel seeding the same chat input a CLI/MCP user would type by hand; every capability it routes to is already fully cross-surface. See the plan doc's "Scope note" for the full reasoning.
  • Featured on /start (top 3 by reliability: cad-to-3d-print, format-bridge, supervision-cv) via the same loadTemplateEntries() helper (lib/templates/load-index.ts), which fails safe (returns [] + logs) on a missing/malformed index rather than crashing the page.

SKILL: Canvas Layout & BatchId Grouping

BatchId System

Each create_media call tags all its cards with a shared batchId (batch_<timestamp>). This groups cards by the prompt that created them.

Layout Modes

  • autoLayout() — Grid layout. Cards with the same batchId stay contiguous in the grid.
  • narrativeLayout() — One row per batchId. Cards from the same prompt flow horizontally; different prompts stack as separate rows.
  • layoutTimeline() — Arrange specific refIds in grid order (used by project_generate).

Layout Commands

  • /organize or /organize grid — autoLayout (grid, grouped by batch)
  • /organize narrative or /organize flow — narrativeLayout (one row per prompt batch)

SKILL: Creative Pipeline (Primary Agent SDK API)

Architecture

The Creative Pipeline is the first-class API for all creative intent processing. Every user message flows through it before reaching the LLM agent.

User message → classify(text) → validate(plan) → execute(plan) → evaluate(results)
                    │                  │                │               │
              3-tier fallback    fix missing fields   parallel       user picks
              (deterministic     LLM sanity check     inference      favorites
               → LLM → regex)                        per task       (memory learns)

Pipeline API (createCreativePipeline)

import { createCreativePipeline } from "@livepeer/creative-kit";

const pipeline = createCreativePipeline({
  llmEndpoint: "/api/agent/gemini",     // optional: enables LLM classification
  skillContent: "...",                   // optional: intent skill markdown
  preferencesSummary: "prefers flux",    // optional: from creative memory
  executor: {
    infer(prompt, model) { ... },        // app-specific inference call
    addResult({ url, model, ... }) {},   // place result on canvas
    say(msg) { ... },                    // show message to user
  },
});

const result = await pipeline.run("compare gpt and flux for a sunset");
// result.handled === true → pipeline executed it
// result.handled === false → pass to LLM agent

Intent Types

Intent Detection Example
compare_models 2+ model names in text (deterministic) "using gpt, flux, recraft to make a cat"
batch_generate Multiple distinct subjects listed "make a cat, a dog, and a bird"
style_sweep "in X, Y, Z style" pattern "a sunset in watercolor, oil, and pencil"
variations "variations", "alternatives", "options" "show me 4 options for a logo"
story Scene markers or long narrative (>500 chars) "Scene 1 — intro..."
single Default — one generation "a cat on a roof"
unclear Low confidence — ask user "make something cool"

Classification Priority (prevents LLM from overriding facts)

  1. Deterministic — model names ARE in the text? → compare_models (0.99 confidence)
  2. LLM — reads skill file + preferences → structured JSON classification
  3. Regex — pattern-matching fallback (always works, no network)

Skill-Based Knowledge

The classifier's intent vocabulary comes from skill files:

  • Base: public/skills/intent-classifier.md
  • User-created: any skill with category: "intent"
  • Loaded via /skills/load intent-<name>

Skill content is injected into the LLM classifier's prompt. Users can teach new intents by creating skill files — no code changes needed.

Validation Step

After classification, validatePlan() ensures the plan is executable:

  • Required fields present (prompt, models, styles)
  • Auto-fixes missing fields from original text
  • Optional LLM review: "does this plan match what the user asked?"
  • Never returns an empty or unexecutable plan

Files

  • packages/creative-kit/src/agent/creative-pipeline.ts — pipeline API
  • packages/creative-kit/src/agent/intent-planner.ts — classify, validate, regex fallback
  • lib/agents/intent-planner.ts — storyboard wrapper (loads skills, preferences)
  • lib/agents/preprocessor.ts — wires pipeline into message flow
  • public/skills/intent-classifier.md — base intent skill
  • apps/creative-stage/app/page.tsx — creative-stage pipeline integration

SKILL: Model-native multi-shot (generation_mode, Summit Gap B-1)

submit_creative_job accepts generation_mode: "auto" | "scenes" | "native" (default "auto"). For a brief shaped like ONE short continuous piece, "auto" routes to Kling 3.0's multi_prompt param on kling-v3-turbo-pro-t2v — ONE inference call renders every shot with model-native continuity, instead of N independent per-scene renders. Verified live via docs/spikes/2026-07-05-kling-multishot.md.

Qualifying shape (lib/mcp-server/native-multishot.ts's decideGenerationMode):

  • 2-5 scenes (NATIVE_MAX_SHOTS = 5 — NOT 6. The fal doc says "1-6 shots", but each shot's duration is a whole-second enum with a 3s floor, so 6 shots × 3s = 18s exceeds the 15s total-duration hard cap. 5×3=15 is the true max under an even split.)
  • no per-scene model_override
  • delivery_promise isn't "source_led"
  • a positive video-intent signal: delivery_promise === "motion_led" (the existing brief/style keyword classifier in lib/production/delivery-promise.ts) OR an explicit video-shaped global_model_override. Absence of an override does NOT imply video — the codebase-wide default for an unspecified model is an image capability (ACTION_TO_DEFAULT_CAP.generate = 'flux-dev'), so an ordinary 2-5 scene photo brief with no signal correctly stays in scenes mode rather than wasting a billed Kling call before falling back.

tryNativeMultiShot bypasses create_media (its Zod schema has no multi_prompt passthrough) and calls sdkPost("/inference", ...) directly — but still runs the same 24h spend-cap preflight (estimateCost + spendLast24h) and recordSpend on success that create_media does, and reuses create-media.ts's exported classifyUpstreamError for humanized failure hints. Without this, native calls would be invisible to get_cost_report and the spend cap.

On success, submit_creative_job collapses workingScenes/state.scenes to a single terminal done scene with shot_prompts: string[] (the original per-shot prompts) stamped on it — the worker's resume-idempotency guard skips it, and the character_anchor block is explicitly skipped too (nothing left to hand an anchor image to). On failure, it falls back to the existing, unmodified scenes-mode fan-out with a native_fallback: <reason> warning. director_re_render on a scene with shot_prompts and no explicit new prompt regenerates the whole take via tryNativeMultiShot again (no silent fallback — an explicit user action gets an explicit result); passing an explicit prompt converts the scene back to an ordinary single-shot render. merge_projects copies scenes via a spread (not an explicit field list) specifically so shot_prompts survives a merge.

SKILL: Latency & quality tiers (quality: fast|balanced|hq) + ETA

create_media accepts an optional quality: "fast" | "balanced" | "hq" tier (video-focused). It maps to a default capability BEFORE resolveCapability — model_override still wins outright, then quality, then prefer_fast, then the action default. Tiers (ACTION_TO_TIER_CAP in lib/mcp-server/tools/create-media.ts):

Tier animate cap generate cap Prod-measured (5s clip)
fast pixverse-i2v flux-schnell p50 45.3s / p95 50.6s (n=20, ≤90s promise)
balanced seedance-mini-i2v flux-dev p50 ~185s / p95 ~213s (n=10)
hq kling-o3-i2v gpt-image ~127s (n=1; premium 4K, opt-in)

Default animate = pixverse-i2v (2026-07-06 promotion). When no quality is passed, animate routes to pixverse-i2v (ACTION_TO_DEFAULT_CAP.animate) — the fastest i2v (~50s p95, ~2–3× faster than the former default seedance-mini-i2v at ~185s p50). Because pixverse-i2v IS the fast-tier cap, a no-tier animate reports quality_tier: "fast" (defaultTierFor() in create-media.ts) so the label matches the routed cap. Other actions keep their neutral "balanced" default label (e.g. generate → flux-dev, the balanced cap). Balanced-tier decision: the balanced animate tier intentionally STAYS seedance-mini-i2v — it remains reachable via quality:"balanced", plus model_override:"seedance-mini-i2v" and the "seedance" keyword — so the higher-multi-shot-fidelity cap is never orphaned. hq/quality-led routing is unchanged (kling-o3-i2v). Explicit model_override ALWAYS wins.

Every create_media response now carries quality_tier, and the async submit path adds eta_seconds (p50) + eta_p95_seconds — computed by conductor.estimateEta() over the static CAPABILITY_SLA prior (no I/O on the hot path). The old hardcoded "60–180s" runtime string is gone. ETA accuracy was prod-validated at 30/31 runs (96.8%) within [0.5×, 1.5×] of the shown ETA.

The result's next_actions includes an "⬆︎ Re-render in HQ" affordance whenever the asset was NOT already hq — it re-issues the same render with quality:"hq" (video re-animates from the source frame; image regenerates from the same prompt). This is the user-facing "start fast, upgrade if you like it" loop.

Cross-surface parity note: the tier+ETA contract is live on the MCP path (the production-validated surface). The CLI run verb is capability-first — the user names the model directly, so the CLI equivalent of a tier is picking pixverse-i2v / seedance-mini-i2v / kling-o3-i2v explicitly (no separate --fast/--hq flag needed; director sub-verbs already carry --prefer-fast). The webapp agent reaches the same tiers by saying "make it fast" / "in HQ" (routed through its own selectCapability + the P4 planInference path) or via model_override; a dedicated Studio ⚡/💎 toggle is deferred UI polish, not a capability gap — every tier cap is already reachable from every surface.

P6 — blended list_capabilities SLA: the latency line in list_capabilities (and its structuredContent.sla) is now live-or-static blended — it shows measured P3 telemetry (tagged live) once a cap has ≥10 real samples, else the static CAPABILITY_SLA prior. So the displayed p50/p95 self-corrects after real traffic (the P5 pixverse fix generalizes to any cap, no hand-editing). The conductor already routes ALL actions (flag-guarded, not video-only) and telemetry already records ALL cap kinds — the perf layer is capability-agnostic. It stays DEFAULT OFF for dispatch; deleting the legacy loop to make the conductor the sole path is gated on the P4 A/B (deferred).

SKILL: Long Prompt Handling & Preprocessor

The Core Problem

Gemini (and other LLMs) choke on large prompts combined with many tool schemas. A 800-word storyboard brief + 500-char system prompt + 21 tool schemas = empty STOP or MALFORMED_FUNCTION_CALL. This is not fixable by prompt engineering — the model is token-overwhelmed.

The Solution: Client-Side Preprocessor (lib/agents/preprocessor.ts)

Multi-scene prompts are intercepted BEFORE reaching the LLM:

  1. Detect multi-scene: regex for Scene N —, numbered lists, or >1500 chars with "storyboard"/"scenes"
  2. Extract scenes client-side: title + first-sentence summary (≤25 words each)
  3. Extract style guide: visual_style, color_palette, mood → prompt_prefix
  4. Call project_create directly — no LLM needed for parsing
  5. Send 1-line instruction to agent: "Project created. Call project_generate."
  6. Agent just manages the generation loop (small context per call)

The LLM never sees the full brief. It gets a 50-word instruction instead of 800 words.

Stress-Tested

  • 20-scene, 8,651-char, 1,399-word prompt → all 20 scenes extracted, all prompts ≤25 words
  • Style guide correctly parsed (visual_style, color_palette, mood)
  • Preprocessing takes <50ms (pure regex, no LLM)
  • Short prompts ("cat eating cheez-it") bypass preprocessor correctly

Key Design Rules

  • System prompt must stay under ~500 chars. Don't list tool descriptions in base.md — Gemini sees them in function declarations already. Just routing rules.
  • Never send raw user text >500 words to Gemini. Preprocess or summarize first.
  • Empty STOP handler must detect prompt type. Multi-scene → route to project_create. Single image → enhance creatively. Don't try to "create a stunning image" from a storyboard brief.
  • MALFORMED_FUNCTION_CALL retry must be short. One sentence, not 4 lines of instructions.
  • Auto-continuation nudges must merge into the function response message (same user turn). Consecutive user messages cause Gemini 400 "function call turn" errors.

MALFORMED_FUNCTION_CALL Recovery

Gemini hits this when function calls are too large. Recovery: short 1-line instruction to use project_create or fewer steps.

Empty STOP Recovery

  • Multi-scene detected → replace with project_create instruction (from preprocessor)
  • Single image → enhance creatively into 30-word prompt
  • Round limit: 2 retries max, then show error

Gemini Turn Ordering

Gemini requires strict user/model alternation. The agent sanitizes messages before each API call — merges consecutive same-role messages. On 400 "function call turn" error, resets conversation and retries.

Completion Summary

After all tool calls finish, if Gemini didn't provide text, an auto-summary appears:

  • Done in 4.2s — media: 3 created (flux-dev)
  • 2/5 succeeded (12.1s) — media: 2 ok, 3 failed

Error Humanization

Raw errors are mapped to friendly messages in compound-tools.ts:

  • "Failed to fetch" → "Can't reach SDK — check connection & API key"
  • "503 / no orchestrator" → "No GPU available — try again in a moment"
  • Error messages show as red-bordered bubbles in chat, red icons on cards

Network Retry

sdkFetch() retries once after 2s on network errors (Failed to fetch, CORS, timeout). HTTP errors (4xx, 5xx) are NOT retried.

SKILL: VM Health & Auto-Recovery

SDK VM Health Check (current state — as of 2026-04-12)

/opt/sdk/healthcheck.sh runs every 2 minutes via cron on sdk-staging-1 but now does only a GCP metadata-server probe. If the metadata server fails twice in a row the script reboots the VM. There is no longer any probe of the SDK process itself — recovery for the SDK is entirely delegated to uvicorn (multi-worker, below) and Docker's restart: always policy.

Why the SDK probe was removed (important — don't put it back): the old script polled /capabilities with a 5s+15s timeout and docker restart sdk-service after 3 fails. Because /capabilities did a synchronous HTTP round-trip to the BYOC orch from inside an async handler on a single uvicorn worker, any ltx-i2v video render (20–60s on BYOC) would stall the event loop long enough to trip the probe. The cron then killed the container every 2 minutes during video gen, which also killed any concurrent /stream/start POSTs mid-flight — this looked like "LV2V failing before reaching fal.ai" because the CORS preflight 200'd but the POST was dropped when the container bounced. Script backup: /opt/sdk/healthcheck.sh.bak. Compose backup: /opt/sdk/docker-compose.yaml.bak.

SDK uvicorn config (current state — as of 2026-04-12, post-revert)

/opt/sdk/docker-compose.yaml overrides the image CMD with:

command: uvicorn app:app --host 0.0.0.0 --port 8000 --workers 1

Must stay at --workers 1. LV2V session state in app.py is module-level in-process dicts (_stream_sessions at line ~1330, _lv2v_jobs at line ~1191). With multi-worker uvicorn, each worker has its own copy, and /stream/start + subsequent /stream/{id}/publish / /stream/{id}/frame calls get load-balanced across workers by the OS accept queue. Non-owning workers return 410 Gone, lib/stream/session.ts:167 treats the first 410 as terminal and auto-stops the stream, and LV2V dies within one frame while the fal runner keeps rolling. An earlier attempt to set --workers 4 to fix an unrelated blocking problem broke LV2V this way — confirmed and reverted 2026-04-12.

How the original blocking problem is addressed:

  • Non-blocking I/O: Every blocking call inside an async def handler is wrapped in await asyncio.to_thread(fn, *args). That includes submit_byoc_job (/inference, /train, /enrich), submit_training_job, get_training_status, wait_for_training, list_capabilities (/capabilities cache miss), _llm_call, _resolve_daydream_user_id, _orch_request. A 64-thread default ThreadPoolExecutor is installed at app startup to handle this. asyncio.to_thread copies contextvars.Context into the worker thread so _current_signer_headers still propagates — this is why to_thread is used instead of run_in_executor.
  • /capabilities in-process cache with 60s TTL + stale-on-error. Hot path has no orch round-trip, so a stalled /inference (in the rare case one still slips through the non-blocking wrap) doesn't stall the probe the frontend hammers on every page load.
  • Healthcheck cron no longer probes the SDK at all, so a busy worker can't trigger a container restart.
  • Per-stream asyncio.Lock for /stream/{id}/publish, keyed by stream_id in _publish_locks: dict[str, asyncio.Lock]. Prevents the go-livepeer trickle server's 5-slot ring buffer (trickle_server.go:82) from slot-evicting an in-flight segment when a later seq catches up. Created in /stream/start, popped in /stream/stop and by the reaper.
  • LV2V stream reaper (new, scans _lv2v_jobs every 30s, kills idle >120s or age >3600s). Fixes a pre-existing leak where browser crashes without /stream/stop left orphan session state forever. Also cleans up _publish_locks entries.

All of the above landed in livepeer/simple-infra#11 (feat/sdk-nonblocking-io branch off feat/sdk-capabilities-cache). Verified by a 9-test e2e suite running against live sdk-staging-1. See docs/superpowers/specs/2026-04-12-sdk-nonblocking-io-design.md and docs/superpowers/plans/2026-04-12-sdk-nonblocking-io.md.

Do not "fix" this by adding more workers until LV2V session state is moved to a shared store (Redis, sqlite, or equivalent). With asyncio.to_thread on every blocking call, a single worker handles arbitrary concurrent load without session-state coherency issues. Multi-worker is neither needed nor safe.

Tier 1 resilience (as of 2026-04-16)

Five improvements deployed on feat/sdk-nonblocking-io (commit b7ed3ec), all in app.py only — zero gateway library changes:

  1. Structured error responses — /stream/start and /inference error paths now return {"error":"...", "rejections":[{"url":"...","reason":"..."}], "hint":"..."} instead of bare strings. Clients can parse rejection details and show actionable hints.
  2. Per-orch rejection logging — every NoOrchestratorAvailableError logs each orch's URL + rejection reason at ERROR level. Eliminates the "30-min debug cycle" pattern from 2026-04-15.
  3. Deposit-aware hint — when LV2V 503s because ticket faceValue > max faceValue, the error includes: "hint": "Broadcaster deposit too low for LV2V ticket. Fund the signer wallet on Arbitrum.". Would have diagnosed the 2026-04-15 LV2V outage in 2 seconds.
  4. Signer failover (SIGNER_URLS) — new env var, comma-separated. _signer_request_with_retry tries each URL with one retry on 5xx/network errors. Handles mid-restart LB gaps.
  5. Graceful shutdown — @app.on_event("shutdown") explicitly stops every active LV2V stream via job.stop(), cleans up _publish_locks, _stream_started_at, _stream_last_publish. Prevents the "SDK restart → browser publishes to dead streams → 404 → 410 cascade" pattern.

/capabilities caching (as of 2026-04-12)

/capabilities in app.py is wrapped with a 60s in-process TTL cache plus stale-on-error fallback. Capability lists only change on byoc-orch deploy, so 60s is fine. Single uvicorn worker = single cache, so worst case is 1 orch hit per minute. On refresh failure, the old list is returned rather than an empty response — a transient orch blip no longer blanks out the capability registry in every browser tab. The change is on feat/sdk-capabilities-cache in livepeer/simple-infra (commits 3019c51 baseline snapshot + cc2aa23 feat), and landed for real on feat/sdk-nonblocking-io (#11). Now baked into the SDK image at us-docker.pkg.dev/livepeer-simple-infra/simple-infra/sdk-service:nonblocking-2026-05-05 — /opt/sdk/.env pins SDK_IMAGE to that tag so docker compose down && up reproduces the patched code instead of regressing to the unpatched layer. The container also runs a startup self-check that fails fast if the patch is missing (grep -c asyncio.to_thread /app/app.py must be ≥15; assertion in app.py near the logger init).

GCP Monitoring

  • Uptime check every 5 minutes on https://sdk.daydream.monster/health (changed 2026-04-12 — was /capabilities; switched because /capabilities depends on the BYOC orch being reachable, so a BYOC outage was paging the SDK oncall).
  • /health is in-process (return {"status": "ok", "orchestrator": ORCH_URL}) — no I/O, instant response, only fires when the SDK is actually down.
  • Alert policy emails sean@livepeer.org if check fails for 5+ minutes.
  • Uptime check ID: sdk-service-health-xI5Sggu-Fq8
  • Alert policy ID: projects/livepeer-simple-infra/alertPolicies/10364248248890653369

Common VM Failure Mode

GCP metadata server (169.254.169.254) becomes unreachable → snapd crash-loops → SSH hangs → all services degrade. Fix: gcloud compute instances reset sdk-staging-1 --zone=us-west1-b --project=livepeer-simple-infra. After a reset the docker network reference can be stale — follow with cd /opt/sdk && sudo docker compose down && sudo docker compose up -d to rebuild it.

Lessons from the 2026-04-11 outage (don't relearn these)

  • "SDK keeps crashing during video gen" is almost never OOM or a code crash. Check OOMKilled, ExitCode, and kernel dmesg first — if they're clean, it's healthcheck-driven or worker-exhaustion, not a real death.
  • Symptom correlation check: grep SDK logs for Started server process (restart marker) and correlate with ltx-i2v dispatch lines. A 1:1 cadence = healthcheck killing a busy server, not a memory problem.
  • "LV2V stream failed without hitting fal" + no POST to /stream/start in SDK logs (only the OPTIONS preflight) = the POST was dropped because the container bounced between preflight and POST. Root cause is upstream of LV2V — look at why the container bounced.
  • Do not add a new external process killer as a reaction to "SDK feels slow." The right fix is always either more workers or cheaper endpoints. External killers misdiagnose busy as dead.
  • Hot-patching app.py: gcloud compute scp app.py sdk-staging-1:/tmp/ ; docker cp /tmp/app.py sdk-service:/app/app.py ; docker restart sdk-service. This is ephemeral — a docker compose up -d (e.g. from editing compose) recreates the container from the image and wipes the hot patch. Don't hot-patch as the durable fix — bake into the image. To rebuild the SDK image: extract app.py from simple-infra feat/sdk-nonblocking-io (commit b7ed3ec), build a thin overlay FROM us-docker.pkg.dev/livepeer-simple-infra/simple-infra/sdk-service:latest + COPY app.py /app/app.py, push to AR with a dated tag, update /opt/sdk/.env to pin SDK_IMAGE to that tag. Don't pin to :latest — a future re-tag would silently take effect on the next pull.
  • The deployed SDK image is built from source that isn't in any branch of simple-infra (1788 lines vs the 2172 on feat/scope-advanced-params). The baseline is now captured on feat/sdk-capabilities-cache — future SDK source changes should branch from there.
  • Multi-worker uvicorn breaks LV2V on this service. _stream_sessions and _lv2v_jobs are per-process dicts; 4 workers means 3 out of every 4 publish/poll requests hit a worker that doesn't know about the stream and return 410 Gone, which the client treats as terminal. Symptom: stream "starts" (card shows streaming), then card turns black within one frame while the fal runner keeps running (it was already dispatched and doesn't know the SDK lost track). If you're ever tempted to add workers again to fix a latency problem, move LV2V state to a shared store first — or wrap the blocking BYOC call in asyncio.to_thread instead so a single worker can handle concurrent requests.
  • Test thresholds calibrated to single happy-path runs are flaky. During the non-blocking I/O PR (#11), T3 was initially asserting on /stream/start total duration under 15s — which was true in one run (34.1s), false in another (17.9s), and catastrophically false in a third (111.3s) because fal Scope runner cold-start variance swings wildly. The fix was to decouple the test from total duration and instead measure the actual invariant ( /health p95 during the crossfire window) which is immune to fal warmth. Similarly T1/T2/T7 thresholds had to be relaxed from "sub-100ms" (calibrated for in-VM probes) to "sub-1500ms" (calibrated for over-the-internet through Caddy+TLS). Rule: tests should measure invariants, not proxies, and thresholds should be loose enough to tolerate ambient jitter while tight enough to catch multi-second regressions.
  • Verifying a lock from inside the lock is tautological. T4 initially tried to verify the per-stream publish lock by reading Lp-Trickle-Seq response headers, but the SDK's MPEG-TS publisher batches frames and doesn't expose per-frame seqs. Then tried timing-based verification, but the un-locked portions of each request (HTTP parse, body read, lock acquire wait) dwarf the in-lock work, so wall-clock measurements show high "parallelism" even when the lock is serializing the protected region. Final design: direct in-lock instrumentation via a thread-safe _publish_in_lock_max: dict[str, int] counter updated inside async with lock: and exposed via /debug/stream/{id}/publish-stats. Assert in_lock_max == 1. This IS tautological if you trust your code, but that's fine — the test's purpose is to fail-loudly if a future refactor moves the lock out of scope.

SKILL: Context Menu + Card Transformations

Right-click Context Menu

  • Uses card-context-menu CustomEvent (dispatched from Card.tsx onContextMenu)
  • ContextMenu.tsx listens for the event, shows at cursor position
  • Dismiss race condition fix: 10ms delay before attaching dismiss listeners (prevents the opening right-click from immediately closing the menu)

Direct vs Agent Actions

  • Direct: Upscale, Remove BG (one-click), Restyle/Animate/Transform (prompt then execute)
  • Agent: Restyle with AI, Animate with AI, Ask Claude (routes to chat)
  • LV2V from card: Captures card image as input for live stream

create_media source_url

When restyling/animating an existing canvas card via chat, the agent must:

  1. Call canvas_get to find the card's URL
  2. Pass source_url in the create_media step Without source_url, restyle has no input image and fails.

SKILL: Hydration & SSR

The page renders null on server and defers all UI to client-side useEffect. This avoids hydration mismatches from localStorage, Zustand stores, and Date.now().

NEVER use Date.now(), Math.random(), or localStorage in initial state or render. Use 0 or "" as defaults, set real values in useEffect.

SKILL: Slash Commands — /story, /film, /stream, /briefing

/story — Multi-scene story generator

/story <concept>        — generate 6-scene story with style + characters
/story list             — recent stories
/story apply [id]       — create project + generate images
/story show <id>        — re-display a saved story

Architecture: Gemini generates JSON (title, audience, arc, context, 6 scenes) → user reviews in StoryCard → apply calls project_create + project_generate directly (image fast path, 0 LLM tokens). Natural-language apply: "yes", "apply them", "I like it". Files: lib/story/, components/chat/StoryCard.tsx, skills/storyteller.md

/film — 4-shot mini-film with camera directions

/film <concept>         — generate 4-shot film script
/film/load <genre>      — load genre skill (animation, action, documentary, noir, scifi)
/film apply [id]        — generate key frames → animate each to video via kling-i2v
/film skills            — list available genre skills

Architecture: Gemini generates JSON (title, style, character_lock, 4 shots with camera directions) → user reviews in FilmCard → apply calls project_create + project_generate + per-shot create_media(animate) + canvas_organize(narrative). Auto-detects genre from keywords. Files: lib/film/, components/chat/FilmCard.tsx, skills/film-*.md

/stream — Live stream with prompt traveling

/stream <concept>       — plan multi-scene live stream
/stream apply [id]      — start stream, scenes transition automatically
/stream stop            — stop active stream
/stream list            — recent stream plans

Architecture: Gemini generates JSON (title, style, graph_template, 3-6 scenes with prompts/presets/durations) → user reviews in StreamPlanCard with visual timeline → apply calls scope_start with Scene 1 → setTimeout schedules scope_control for each subsequent scene → scope_stop auto-fires after total duration. Each transition updates the prompt + preset via the Scope control API. Like "prompt traveling" through a visual story.

Key concept: prompt traveling — the stream's visual content evolves over time as the prompt changes scene-by-scene. The viewer sees the stream morph from one scene to the next, creating a narrative arc in real-time. Files: lib/stream-cmd/, components/chat/StreamPlanCard.tsx

Daily Briefing — Email-powered visual deck

daily briefing [style]  — fetch Gmail → generate visual slides

Styles: modern, dark, light, colorful, corporate, scenic, vivid, isometric, iso, lego Requires Gmail MCP connection (local server at scripts/gmail-mcp-server.ts). Architecture: briefing fast path in gemini/index.ts → gmail_list via MCP → LLM summarization → analyzeEmail (urgency/action/date) → project_create + project_generate → canvas_organize → caption banners with CaptionBanner component (date=cyan, action=amber, expandable). Cover slide uses coverText field.

/talk — Talking Video with Voice Cloning

/talk <text> --face <card>                Generate talking video with default voice
/talk <text> --face img-1 --voice aud-2   Clone voice from audio card

Pipeline: chatterbox-tts (text + optional audio_url for voice clone) → talking-head (image + audio → lip-synced video). Right-click image card → "Talking Video" for the UI version (multi-step dialog: text → voice picker). Import voice samples: right-click canvas → Import → select .wav/.mp3 file. Files: lib/skills/commands.ts (handleTalk), components/canvas/ContextMenu.tsx (talking-video action)

/project — Project Management

/project list              — show all projects (blue clickable names)
/project show [name]       — details + scene list of active or named project
/project switch <name>     — set as active (partial match works)
/project add <brief>       — create empty project
/project replay [name]     — regenerate all scenes from stored prompts
/project clear             — remove all projects

Projects auto-created by agent. Friendly names from brief ("ev-bikes", "sunset-story"). Capped at 30 in localStorage. Cards get project-prefixed refIds: ev-bikes.img-1 instead of bare img-1. ProjectListCard renders with blue names, switch buttons, active badge. Files: lib/projects/commands.ts, components/chat/ProjectListCard.tsx, lib/projects/store.ts

/analyze — Image/Video Analysis via Gemini Vision

/analyze <card-name>       — extract style, characters, setting, mood from image/video

Right-click card → "Analyze Media" for the UI version. Sends media to Gemini 2.5 Flash vision → extracts CreativeContext (style, palette, characters, setting, mood, description). Auto-applies as creative context if none exists. Supports images and short videos (<15MB). Files: lib/tools/image-analysis.ts, lib/skills/commands.ts (handleAnalyze)

/stream graphs — Scope Graph Management

/stream graphs              — list built-in + saved graph templates
/stream graphs save <name>  — save last stream's graph for reuse
/stream graphs remove <name> — delete a saved graph

6 built-in templates: simple-lv2v, depth-guided, scribble-guided, interpolated, text-only, multi-pipeline. User-saved graphs persisted in localStorage (max 20). Referenced by name in /stream plans. Skill reference: skills/scope-graph-builder.md — full pipeline/node/param reference. Files: lib/stream-cmd/graph-store.ts, lib/stream-cmd/commands.ts, lib/stream/scope-graphs.ts

Canvas Time Machine — undo/redo + snapshots

Cmd+Z                        Undo last canvas action
Cmd+Shift+Z                  Redo
/snapshot save <name>         Named checkpoint (persists in localStorage)
/snapshot restore <name>      Restore saved state (pushes current to undo first)
/snapshot list                Show all named snapshots
/snapshot delete <name>       Remove a snapshot

History manager: packages/creative-kit/src/stores/history-manager.ts. In-memory undo/redo (max 50), localStorage snapshots (max 20). Canvas store wraps every mutating action with pushUndo().

Variation Grid — generate 4, pick 1

/vary <card-refId>            Generate 4 alternatives (mixed strategy)
Right-click card → "Variations (x4)"

Strategies: seed (same model, different seeds), model (alternate models), prompt (prompt tweaks), mixed (default: 1 seed + 1 model + 2 prompt variations). All 4 run in parallel via create_media. Creative memory learns from which you keep. Files: packages/creative-kit/src/agent/variation-engine.ts

Final Cut Composer — render canvas to video

/render                       Render all canvas cards into a single video
/render <project>             Render a specific project's scenes
/render <project> --music aud-1  Add background music from an audio card

Browser-side via MediaRecorder + canvas.captureStream(30fps). Supports crossfade/cut/fade-black transitions. Music mixed via Web Audio API. Output: WebM video (auto-download + canvas card). Files: packages/creative-kit/src/agent/render-engine.ts

Face Lock — character consistency across scenes

/facelock <card-refId>        Lock character reference for this project
/facelock                     Show current lock status
/facelock clear               Remove the lock

How it works: When face lock is active, all generate/restyle actions route through kontext-edit with the locked image as image_url. For animate, the locked image becomes the first frame. This preserves face identity because kontext-edit is an image-edit model that retains the source face. Limitations: This is reference-image consistency, not face-ID embedding (no IP-Adapter on BYOC). Works well for same-character-different-scene, less well for radically different poses. Injection point: lib/tools/compound-tools.ts — 15-line conditional block before inference closure capture.

Social Export — platform-ready crops

/export social instagram      1080x1080 center-crop
/export social tiktok          1080x1920 portrait
/export social youtube         1920x1080 landscape
/export social twitter         1200x675
/export social all             All platforms at once

Smart crop with face-bias heuristic (biases toward top 1/3 when cropping vertically). Currently images only; video cropping planned. Files: packages/creative-kit/src/agent/social-export.ts

Creative Tools — context menu + slash commands

Tool Context Menu Slash Command Capability
Variations 🔀 right-click card /vary <card> mixed strategy
LEGO Style 🧱 right-click card /lego <desc> kontext-edit
Make Logo 🎨 right-click card /logo <desc> kontext-edit / flux-dev
Replace Object 🔄 right-click card — kontext-edit
Isometric ◆ right-click card /iso <desc> kontext-edit / flux-dev
Virtual Try-On 👕 right-click card /tryon <person> <garment> fashn-tryon
Video Try-On 🎥 right-click card — fashn-tryon → seedance/veo/ltx
Talking Video 🗣 right-click card /talk <text> --face <card> chatterbox-tts → talking-head
Weather Effect ⛅ right-click card — kontext-edit → kling-i2v
Cinematic Video 🎬 right-click card — seedance-mini-i2v (preferred; ½ cost/2× speed) → seedance-i2v (up to 15s + audio)
Analyze Media 🔍 right-click card /analyze <card> Gemini Vision
Edit with GPT Image ✏️ right-click card — gpt-image-edit
Product Briefing 📋 right-click card — gpt-image-edit
GPT Image 2 🎨 right-click canvas — gpt-image (text, logos, products)
Convert to 3D 🖥 right-click card — tripo-i3d
Import Media 📁/🔗 right-click canvas — image/video/audio (GCS upload)

Capabilities (150 on BYOC orch — last expanded 2026-06-25 (+1): Seed Audio 1.0 — seed-audio (bytedance/seed-audio-1.0, fal-exclusive, ~$0.05/audio-sec) — an all-in-one audio model that generates voice + music + SFX in one pass: multi-speaker scenes from a single prompt, voice defined by a text description / character image (image_url) / up to 3 reference clips (audio_url). Routed via the "seed audio" mention + multi-speaker/dialogue/scene-audio intent detection (compound-tools case tts|music) + resolveCapability keywords (dialogue/multi-speaker/scene-audio); plain single-voice TTS stays on chatterbox-tts. Skill public/skills/seed-audio.md. Output: one mixed audio_url. Prior 2026-06-24 (+3): Seedance 2.0 Mini — seedance-mini-t2v (bytedance/seedance-2.0/mini/text-to-video) + seedance-mini-i2v (.../mini/image-to-video) + seedance-mini-ref2v (.../mini/reference-to-video), all ~$0.0394/s (≈½ the cost of standard Seedance 2.0, ~2× speed, strong multi-shot character/wardrobe/style consistency). Preferred over seedance-i2v for default animate routing — ACTION_TO_DEFAULT_CAP.animate, the browser animate default, and the bare "seedance" mention all resolve to Mini; standard seedance-i2v stays reachable via model_override. duration is a STRING enum '4'-'15'/'auto' (create_media's seedance path already String()-casts it). Prior 2026-06-24 (+3): Krea 2 Open-Source — krea-2-os (fal-ai/krea-2/turbo, the most aesthetic OPEN-WEIGHTS image base, LoRA-trainable, ~$0.032/MP) + krea-2-lora-training (fal-ai/krea-2-trainer — the 2nd trainable base on BYOC after flux; train a brand/character aesthetic LoRA) + krea-2-lora (fal-ai/krea-2/turbo/lora — run a trained Krea-2 LoRA). The LoRA workflow (submit_lora_train base_model:"krea-2" → apply_lora model:"krea-2-lora") is wired across MCP+CLI+webapp; krea-2-lora-training accepts+meters jobs but full SDK weight-retrieval has a known 202-polling follow-up. Skill public/skills/krea-2-open-source.md; playbook+showcase train-your-brand-aesthetic. Prior 2026-06-22 (+1): Sync Labs sync-3 avatar (sync-lipsync-v3 — fal-ai/sync-lipsync/v3/image-to-video, $8/min: a single still — illustration / animated frame / photo — + an audio track → a talking character with accurate lipsync; best for stylized/illustrated characters; sibling to lipsync/talking-head; showcase public/chapters/talking-character-example.html). Prior 2026-06-19 (+3): Hyper3D Rodin v2.5 (rodin-i3d / rodin-t3d — production 3D, PBR + HD textures, 50K→2M tris; skill public/skills/rodin-3d.md) + LTX 2.3 Ingredient (ltx-q-ingredient — reference sheet → video scene; skill public/skills/ltx-ingredient.md). Prior 2026-06-18 batch 2 (+14): Pixelcut + Bria 3.0 video bg removal, Bernini-R (unified gen+edit video), Luma Ray 3.2, Luma UNI-1. Earlier 2026-06-18: sensenova-u1-infographic + Kling 3.0 Turbo (retired pre-turbo kling-v3))

Image t2i: flux-dev, flux-schnell, recraft-v4, gemini-image, nano-banana, flux-flex, seedream-5-lite, krea-2 ($0.0315/img — moodboard-driven foundation), krea-2-large ($0.063/img — premium variant), krea-2-turbo ($0.032/img — fast iteration on the krea-2 family, same aesthetic understanding as krea-2-medium with turbo speed), krea-2-os ($0.032/MP — Krea 2 Open-Source Turbo fal-ai/krea-2/turbo; the most aesthetic OPEN-WEIGHTS base, the malleable RAW checkpoint, and the only krea cap that is LoRA-trainable — default for fashion/brand/editorial/interiors/travel), krea-2-lora ($0.032/MP — fal-ai/krea-2/turbo/lora, runs a trained Krea-2 LoRA), cosmos-3-image (NVIDIA omnimodal world model), mai-image-2.5 ($0.063/img — Microsoft photoreal foundation, natural lighting + skin tones + refined text rendering for branding/packaging/commercial design; t2i + edit), ideogram-v4 ($0.04/img — best open-weight image model, strong text rendering for posters / signage), sensenova-u1-infographic ($0.04/MP — SenseNova U1: one native multimodal model doing image-gen + visual understanding; turns a single prompt into a data-rich infographic, optional thinking mode for complex prompts. Route "infographic"/"data-viz" intents here, not a generic image model. Skill: public/skills/infographics.md) Edit: kontext-edit, flux-fill, flux-erase ($0.0315/img — mask-based object/text removal + reconstruction; cleanest for sign / watermark / unwanted-object cleanup), gpt-image-edit, grok-imagine-edit, uni-1-edit / uni-1-max-edit ($0.04 / $0.063/MP — Luma UNI-1 instruction edit, strong reference following), bernini-r-edit-image ($0.04/MP — Bernini-R single-image edit, sibling to its video edit) Image (multimodal reasoning): uni-1-t2i / uni-1-max ($0.04 / $0.063/MP — Luma UNI-1, multimodal reasoning image model with strong taste + reference following; -max = premium tier) Video T2V: veo-t2v, ltx-t2v, pixverse-t2v, seedance-mini-t2v ($0.0394/s — Seedance 2.0 Mini text-to-video, cheap + fast + multi-shot consistent), kling-v3-turbo-t2v ($0.0525/s — Kling 3.0 Turbo 720p: faster gen, lower cost, superior lip-sync, stable motion), kling-v3-turbo-pro-t2v ($0.0735/s — Turbo 1080p; also the model-native multi-shot engine — see "Model-native multi-shot" skill below), kling-o3-t2v (Omni — strongest prompt/ref consistency, 4K, up to 15s), ltx-q-t2v ($0.056/s — LTX 2.3 Quality tier, higher fidelity than base ltx-t2v), ltx-q-t2v-lora (Quality + LoRA style transfer), ray-32-t2v ($0.42/s — Luma Ray 3.2, premium: multi-keyframe control, expressive faces, HDR/EXR, motion transfer), bernini-r-t2v ($0.07875/s — Bernini-R, one model that both generates AND edits video) Video I2V: veo-i2v, ltx-i2v, pixverse-i2v, kling-v3-turbo-i2v ($0.0525/s — Kling 3.0 Turbo 720p), kling-v3-turbo-pro-i2v ($0.0735/s — Turbo 1080p), kling-o3-i2v (Omni 4K/15s), pixverse-i2v ($0.063/s — the DEFAULT animate cap since 2026-07-06: fastest i2v at ~50s p95 / ~2–3× faster than seedance-mini; also the fast tier cap), seedance-mini-i2v ($0.0394/s — Seedance 2.0 Mini, the balanced-tier / higher-multi-shot-fidelity option, reachable via quality:"balanced" or model_override: ½ cost + 2× speed of seedance-i2v, strong multi-shot consistency), seedance-i2v, seedance-i2v-fast, grok-imagine-video ($0.147/s — xAI v1.5, fluid camera + coherent multi-shot scenes, best for "polished in seconds"), cosmos-3-i2v ($0.0525/s — NVIDIA omnimodal, strong motion + temporal coherence), ltx-q-i2v ($0.056/s — LTX 2.3 Quality), ltx-q-i2v-lora (Quality + LoRA), ray-32-i2v ($0.42/s — Luma Ray 3.2 premium) Video edit / ref (Bernini-R, unified gen+edit, guide with up to 5 ref images, scene stays intact): bernini-r-edit (instruction edit — swap objects/weather/backgrounds/camera/style), bernini-r-ref-edit (reference-guided edit), bernini-r-ref2v (reference-to-video). All $0.07875/s. Video background removal (commercial-use matting): bria-video-bg-remove ($0.063/s — Bria VRMBG 3.0, high accuracy; custom bg color / image / adjustable blur), bria-video-bg-remove-rt ($0.063/s — real-time tier for live streams / webcam), pixelcut-video-bg-remove ($0.063/s — ultra-crisp fast production alternative). Skill: public/skills/video-background-removal.md. Route via action='animate' with source_url. Video audio-driven: ltx-q-a2v ($0.056/s — first audio-to-video cap on BYOC; drives video from an audio track), ltx-q-a2v-lora Video ref-driven: pixverse-ref2v, kling-o3-ref2v, ltx-q-ref2v (Quality reference-video-to-video; closest to true style-transfer-from-video), ltx-q-ref2v-lora, ltx-q-ingredient ($0.056/s — LTX 2.3 Ingredient: one reference sheet → a video scene that composites your characters / products / props / environments; for creative ads + character workflows. Skill: public/skills/ltx-ingredient.md. Route via action='animate', source_url = the reference sheet + a scene prompt) Video HDR: ltx-q-hdr ($0.063/s — outputs EXR frames + MP4 preview for pro HDR pipelines), ltx-q-hdr-lora Video misc: veo-transition, pixverse-transition, void-inpaint TTS: chatterbox-tts, gemini-tts, inworld-tts, grok-tts, seed-audio ($0.05/audio-sec — Seed Audio 1.0 all-in-one: voice + music + SFX in one pass, multi-speaker scenes from one prompt, voice via text/character-image/reference clips; route dialogue / multi-speaker / scene-audio intents here, not plain single-voice TTS. Skill: public/skills/seed-audio.md) SFX / audio: music (minimax 60s tracks), sfx (mmaudio short clips), mirelo-sfx ($0.0105/s — text-to-SFX with loop support, premium fidelity), mirelo-sfx-extend (seamless tails on existing audio), mirelo-sfx-inpaint (replace audio segments inside a track), mirelo-sfx-v2v (video → auto-synced SFX track, up to 60s — alternative to MMAudio for higher-quality foley), sonilo-t2m ($0.105/s — first professionally licensed text-to-music; sibling to minimax-music with cleaner licensing for commercial use), sonilo-v2m ($0.105/s — video-to-music: upload a video, get an original soundtrack matching pacing + emotion automatically; sibling pattern to mirelo-sfx-v2v but for music instead of SFX) ASR / transcription: wizper (fal-ai/wizper, Whisper-v3, $1/per), nemotron-asr (nvidia/nemotron-asr-multilingual; open streaming ASR, 40 language-locales, native punctuation + capitalization, ultra-low latency — built for real-time voice agents and live captions) Talking-head: lipsync, talking-head, heygen-twin ($0.105/s — HeyGen Avatar 5 Digital Twin; trained personalized avatar from a 15s recording, lifelike identity stable across camera angles), sync-lipsync-v3 ($8/min — Sync Labs sync-3 avatar; a single still — illustration / animated frame / photo — + an audio track → a talking character with accurate lipsync; the best pick for STYLIZED / illustrated / cartoon characters where talking-head/heygen expect a real face. Inputs: image_url + audio_url → video.url. Route via action='lipsync' or mention "sync-3"/"sync-lipsync") 3D: tripo-t3d, tripo-i3d, tripo-mv3d, tripo-p1-t3d, tripo-p1-i3d, triposplat ($0.10/asset — single image to high-quality 3D Gaussian splats in <5s, learned density control, controllable Gaussian budgets for hero assets vs backgrounds vs LOD; sibling to tripo-i3d but Gaussian-splat output instead of mesh), rodin-i3d / rodin-t3d ($0.15/mesh — Hyper3D Rodin v2.5: production-ready 3D with PBR materials + HD textures, from text OR up to 5 reference images, quality tiers 50K→2M triangles. Premium tier above tripo for finished/hero meshes; image-to-3D via action='animate' + source_url, text-to-3D via action='generate'. Skill: public/skills/rodin-3d.md) GPT: gpt-image, gpt-image-edit Grok Imagine (xAI): grok-imagine-quality (t2i), grok-imagine-edit (instruction-based image edit), grok-imagine-video (premium fluid-camera i2v) Multimodal reasoning (NVIDIA): nemotron-omni — text+image+video+audio in one loop BG removal: bg-remove (birefnet — fast), ideogram-bg-remove (high-fidelity subject isolation) Try-on: fashn-tryon, flux-vto ($0.05/img — BFL Virtual Try-On, fast low-latency packshots + multi-garment outfit composition) Face-swap (added 2026-06-03): face-swap-image ($0.009/gen, half-moon-ai/ai-face-swap/faceswapimage — replaces the deprecated easel backend the original face-swap cap pointed at) · face-swap-video ($0.024/s, half-moon-ai/ai-face-swap/faceswapvideo, ≤25 fps, ≤25 min input — first true video face-swap cap on BYOC). The bare face-swap name is kept as an alias to face-swap-image for backward compatibility. See public/skills/face-swap.md + public/playbooks/one-face-six-lives.md for usage + flagship showcase pattern. Live LV2V face-swap is deferred to a Scope graph-node PR (fal has no streaming face-swap endpoint). Video understanding (added 2026-06-03): marlin-video ($0.015/1K tokens, fal-ai/marlin, 2B video-VLM — accepts mp4/mov/webm/m4v/gif up to ~2 min, returns structured JSON with scene boundaries + timestamps + camera motion + subjects + dominant colors). First-ever video understanding cap on BYOC. Used in 5 skill recipes: scene_tag_reel (reverse-engineer video into a remixable project — the flagship), auto_critique_video (post-gen drift gate), lv2v_quality_monitor (real-time stream scoring — designed, not yet wired), storyboard_from_video (paste-link onboarding), moderation_check (pre-publish safety). See public/skills/video-understanding.md + public/playbooks/paste-link-get-project.md + flagship showcase at /chapters/paste-link-get-project-example.html. Other: topaz-upscale, sam3 Fallback chains: all video/image/TTS models have 2-4 siblings for automatic retry on failure. Default animate cap = pixverse-i2v (2026-07-06): the fastest i2v (~50s p95, ~2–3× faster than seedance-mini) and also the fast tier cap. Seedance 2.0 Mini (seedance-mini-i2v/-t2v/-ref2v): the balanced-tier / higher-multi-shot-fidelity option — ½ cost + 2× speed of standard Seedance 2.0 with strong multi-shot character/wardrobe/style consistency; reachable via quality:"balanced", model_override, or the "seedance" keyword. Standard seedance-i2v (highest fidelity, slow/premium) + seedance-i2v-fast remain available via model_override. Up to 15s cinematic video with audio.

Picking the right new cap (added 2026-06-02)

  • Polished i2v in seconds, fluid camera, coherent multi-shot: grok-imagine-video (premium; ~$0.147/s).
  • High-fidelity i2v with motion strength + temporal coherence from a multimodal brief: cosmos-3-i2v (~$0.0525/s; cheaper than kling-v3-i2v).
  • Higher-quality LTX with optional LoRA / audio-driven / ref-video / HDR: ltx-q-* family. Always prefer ltx-q-* over base ltx-* for finished work; use base ltx-* only when speed > quality.
  • Audio drives the video: ltx-q-a2v (only a2v cap on production).
  • HDR EXR delivery for pro VFX: ltx-q-hdr.
  • Sound-effects with seamless loops / inpaint / extend / video-sync: mirelo-sfx*. Use over base sfx when the work needs >5s of audio, looping, or video-sync.
  • Personalized talking-head from a 15s recording: heygen-twin (>~$0.10/s but best identity stability across angles). For one-off lipsync of an existing avatar, stick with talking-head.
  • Erase object / sign / watermark cleanly: flux-erase.
  • Brand-consistent, moodboard-driven foundation image: krea-2 (mid-tier) or krea-2-large (premium). Pair with brand_kit / context_kit for art-directed pipelines.
  • Garment try-on: flux-vto (BFL, fast packshot) or fashn-tryon (existing, multi-step outfit). Both stay available — the fallback chain bridges them.

Picking among the 7 caps added 2026-06-04

  • Photoreal with accurate skin tones + readable text in image (logos, packaging, signage): mai-image-2.5 (Microsoft, $0.063/img). Best when the deliverable IS a branded surface.
  • Open-weight image with strong text rendering, lower cost than mai-image-2.5: ideogram-v4 ($0.04/img). Use for posters / signage / non-photoreal compositions where Ideogram's typography excels.
  • Fast iteration on the krea-2 aesthetic (moodboard pipelines, brand-locked): krea-2-turbo ($0.032/img). Same look as krea-2-medium, faster — pick for the iteration phase, then bump to krea-2-large for final.
  • Image → 3D for hero assets (game / AR / interactive): triposplat ($0.10/asset, Gaussian-splat output). Prefer over tripo-i3d (mesh) when downstream is a splat renderer (Three.js, Unity SplatVFX). Use tripo-i3d when the consumer needs a mesh.
  • Real-time / streaming transcription, 40 languages, voice-agent context: nemotron-asr. Use over wizper when you need streaming ingestion or punctuation/casing in non-English. wizper remains the right call for batch transcription.
  • Auto-generate a soundtrack for a video (no music spec, just video): sonilo-v2m — uploads a video, returns a soundtrack matched to its pacing + emotion. The sibling pattern to mirelo-sfx-v2v (which does SFX, not music).
  • Music gen with cleaner commercial licensing than minimax-music: sonilo-t2m. Same role as music cap (text-to-music) but professionally licensed for commercial reuse. Pick when client work / brand deliverables need clear rights.

SKILL: magenta music DJ demo — named start/stop

Demo name: magenta music DJ demo. Named triggers (act immediately, don't re-confirm):

  • "start music demo" / "start the DJ" / "start magenta music demo" → run the START steps, then hand the user the demo page URL.
  • "stop music demo" → run the STOP steps (stop the A100 to save cost), confirm stopped.

The demo: a virtual DJ — type or tap a prompt and the real Magenta RealTime 2 (MRT2, 2.4B) model on a GPU re-steers the music live over WebSocket, gapless (not pre-baked clips). Use case: show how a realtime inference cap becomes an agent-driven, steerable instrument.

The one VM (GCP project livepeer-simple-infra)

VM Role Zone Machine
magenta-rt-a100 Magenta RT 2 WS server + Caddy TLS us-central1-f a2-highgpu-1g (1× A100-40GB)
  • Reserved regional static IP 34.42.244.33 (magenta-rt-ip, us-central1) — persists across stop/start, so the URLs below never change.
  • WS endpoint: wss://34-42-244-33.sslip.io/stream · Health: https://34-42-244-33.sslip.io/health → {"status":"ok","model":"mrt2_base","rtf":1.26}.
  • MUST stay in us-central1 (the static IP + the 34-42-244-33.sslip.io Caddy cert + the demo's hardcoded wss URL are IP-pinned). If us-central1 A100 is stocked out, the migration recipe in memory magenta_realtime_music covers re-pointing.
  • A100 is required for gapless (RTF0.8 = faster than realtime). The old L4 (magenta-rt-gpu/magenta-rt-l4) was RTF2.0 = audible 2s gaps — do NOT run the DJ on an L4.

The viewer URL to hand the user: https://storyboard.daydream.monster/demo/storyboard-demo.html → Music tab → ▶ Start the DJ → type/tap a prompt (e.g. "classic chinese guzheng", "driving synthwave").

START steps

P=livepeer-simple-infra; Z=us-central1-f; VM=magenta-rt-a100
gcloud compute instances start $VM --zone=$Z --project=$P
# startup-script auto-restarts magenta.service + caddy; model reload ~2-3 min. Poll health:
for i in $(seq 1 18); do
  curl -s --max-time 12 https://34-42-244-33.sslip.io/health | grep -q '"status":"ok"' && { echo LIVE; break; }; sleep 15; done

Then open the demo page → Music tab → Start the DJ. First audio takes a few seconds (warmup + ~6s client prebuffer), then it plays gapless and morphs on each prompt.

If the WS doesn't respond after boot

gcloud compute ssh magenta-rt-a100 --zone=us-central1-f --project=livepeer-simple-infra \
  --command="sudo systemctl status magenta caddy --no-pager; sudo journalctl -u magenta -n 30 --no-pager"

(magenta.service runs the WS server; caddy terminates TLS for 34-42-244-33.sslip.io. Both are Restart=always + restored by the startup-script on boot.)

STOP (save cost — A100 is ~$4/hr running)

gcloud compute instances stop magenta-rt-a100 --zone=us-central1-f --project=livepeer-simple-infra

Stopped cost ≈ ~$42/mo (200 GB pd-ssd boot disk + idle static IP + snapshots) — GPU/compute not billed. Backup boot snapshot: magenta-rt-l4-boot-20260609.

Implementation: demo page public/demo/storyboard-demo.html (Music tab, vanilla WS DJ mirroring lib/stream/realtime/magenta-audio-transport.ts); server simple-infra/magenta-rt/server.py (branch feat/magenta-rt-server, on the VM). Full migration/infra detail in memory magenta_realtime_music.

SKILL: supervision-live demo — named start/stop

Demo name: supervision-live demo. Named triggers (act immediately, don't re-confirm):

  • "start supervision-live demo" (also "show the demo") → run the START steps below, then hand the user the viewer URL.
  • "stop supervision-live demo" → run the STOP steps below (stop both VMs to save cost), confirm stopped.

The demo: a live YouTube street-cam → per-frame CV (YOLO+ByteTrack+zone) on a GPU → MediaMTX media server → viewable live stream.

The viewer URL to hand the user (constant — anchored to MediaMTX's static IP):

http://34.168.194.124:8888/superlive-rt (browser player · NOT /index.m3u8)

The two always-on components (GCP, project livepeer-simple-infra)

VM Role Zone Notes
mediamtx-staging-1 Media server (egress). Static IP 34.168.194.124 us-west1-b MediaMTX via docker run --restart always → auto-starts on boot.
magenta-rt-gpu GPU supervision worker (L4) us-west1-a sv-worker.service (systemd) auto-starts; pushes to MediaMTX's static IP, so the viewer URL is constant even though this VM's own IP changes on restart.

The viewer URL is ALWAYS http://34.168.194.124:8888/superlive-rt (MediaMTX is the static anchor).

Steps to show it

P=livepeer-simple-infra
# 1. Start both VMs if stopped (idempotent — already-running is fine).
gcloud compute instances start mediamtx-staging-1 --zone=us-west1-b --project=$P
gcloud compute instances start magenta-rt-gpu    --zone=us-west1-a --project=$P
# 2. Wait ~2-3 min: MediaMTX (docker restart=always) + sv-worker (systemd) auto-resume;
#    the worker re-resolves the live HLS and starts pushing. Then verify it's flowing:
for i in $(seq 1 18); do
  [ "$(curl -sL --max-time 10 -o /dev/null -w '%{http_code}' http://34.168.194.124:8888/superlive-rt/index.m3u8)" = "200" ] && { echo LIVE; break; }; sleep 15; done
# 3. Prove the overlay is real (optional): grab a live frame and view it.
ffmpeg -y -v error -i "http://34.168.194.124:8888/superlive-rt/index.m3u8" -frames:v 1 -q:v 3 /tmp/rt.jpg   # then Read /tmp/rt.jpg

How to VIEW it (give the user one of these)

  • Browser player (easiest): http://34.168.194.124:8888/superlive-rt — NOT /index.m3u8 (that downloads the manifest). The bare path is MediaMTX's built-in HLS player.
  • WebRTC (sub-second): http://34.168.194.124:8889/superlive-rt
  • VLC / ffplay: open http://34.168.194.124:8888/superlive-rt/index.m3u8
  • To screenshot it for the user: use the Playwright MCP — browser_navigate to http://34.168.194.124:8888/superlive-rt (http page → http stream, no mixed-content issue; file:// is blocked), wait ~7s, browser_take_screenshot, then Read the saved jpeg.

Self-serve (web UI — paste ANY live URL)

The demo's 🎥 Live-Stream CV tab has a "Process your own live stream" card: presets + a paste-your-own URL box (YouTube live / HLS .m3u8 / RTMP) + ▶ Start & process. It calls /api/supervision-live (Next.js route) which mints a token from the least-privilege GCP SA (Vercel env GCP_SA_KEY_B64), writes the URL to gs://storyboard-uploads/superlive/source.txt, and starts the GPU + media VMs. The worker reads that object via curl (not gsutil — not on the systemd PATH) and hot-swaps within seconds. The page polls GET /api/supervision-live until live, then shows ▶ Open the processed live stream (opens http://34.168.194.124:8888/superlive-rt in a new tab — http stream can't embed in the https page). Guardrails: SINGLE stream (one GPU/worker) + auto-stop after 15min idle via /api/cron/supervision-idle (reads lastactive.txt). Verified E2E (posted the Amsterdam cam → live processed Amsterdam stream). The SA + key + token are already provisioned; the worker (infra/supervision-worker/) re-pulls on GPU boot via the startup-script.

So when the user says "let them paste a URL" / "self-serve" → it already works on the deployed demo page; you just need the VMs reachable (the route starts the GPU; keep MediaMTX startable). To test the route by hand: curl -X POST https://storyboard.daydream.monster/api/supervision-live -d '{"url":"<live url>"}' then poll GET until {"live":true}.

The combined demo page (music + CV)

https://storyboard.daydream.monster/demo/storyboard-demo.html — presenter-ready, two tabs. The Music tab plays real Magenta samples (always works). The CV tab's "🔴 Live (realtime GPU)" button points at the realtime stream, BUT the page is https and the stream is http → mixed-content blocked in-browser; for the live CV in front of an audience, open the MediaMTX player URL directly (above) or front MediaMTX with Caddy/TLS + a DNS name (orch-VM pattern) to fix permanently.

If the stream looks stalled

The live-cam HLS source URL expires; the worker's launcher re-resolves it and sv-worker.service restarts on failure, but a hung ffmpeg can wedge. Kick it: gcloud compute ssh magenta-rt-gpu --zone=us-west1-a --project=$P --ssh-flag="-o StrictHostKeyChecking=no" --command="sudo systemctl restart sv-worker" (SSH to this VM is flaky — retry, or gcloud compute instances reset magenta-rt-gpu … which reruns the startup-script). Check: sudo journalctl -u sv-worker -n 20 (look for advancing {"frame": N…}).

STOP after the demo (to save cost — the L4 is ~$0.70/hr)

gcloud compute instances stop magenta-rt-gpu --zone=us-west1-a --project=livepeer-simple-infra
gcloud compute instances stop mediamtx-staging-1 --zone=us-west1-b --project=livepeer-simple-infra

Implementation: infra/supervision-worker/ (the GPU worker), infra/mediamtx/ + scripts/deploy-mediamtx.sh (the media server), scripts/live-stream-supervision*.sh (the no-GPU segment/rolling tiers), docs/supervision-live-phase4-design.md (the full design). NOTE: magenta-rt-gpu (us-west1-a, L4) runs the supervision worker. The realtime music DJ has its OWN dedicated A100 VM now — magenta-rt-a100 (us-central1-f) — see the magenta music DJ demo skill above. Do NOT repurpose magenta-rt-gpu for music; the two demos are independent VMs.

Key Files

App

  • app/page.tsx — Main page, plugin registration, mounted gate
  • app/api/agent/chat/route.ts — Claude API proxy + MCP tool routing
  • app/api/agent/gemini/route.ts — Gemini API proxy
  • app/api/agent/openai/route.ts — OpenAI API proxy

Canvas

  • components/canvas/InfiniteCanvas.tsx — Pan/zoom, dot grid, card/arrow rendering
  • components/canvas/Card.tsx — Drag, resize, media display, info bar, stream controls + inline agent, video fullscreen
  • components/canvas/ArrowEdge.tsx — SVG arrows + HTML click targets + inline popup
  • components/canvas/ContextMenu.tsx — Right-click actions (direct + agent)
  • components/canvas/CameraWidget.tsx — Webcam + LV2V with prompt control, info bar (FPS/pub/recv), 5 preset buttons

Agent & Tools

  • lib/agents/gemini/index.ts — Gemini plugin (default), MALFORMED_FUNCTION_CALL recovery, auto-continuation
  • lib/agents/claude/index.ts — Claude plugin
  • lib/tools/compound-tools.ts — create_media (main tool for all media creation), batchId tagging
  • lib/tools/canvas-tools.ts — canvas_get/create/update/remove/organize
  • lib/tools/scope-tools.ts — scope_start/control/stop/preset/graph/status (Scope Domain Agent)
  • lib/tools/project-tools.ts — project_create/generate/iterate/status (Director)
  • lib/sdk/capabilities.ts — Live capability registry + resolveCapability

Stream & Scope

  • lib/stream/session.ts — LV2V session lifecycle (start, publish, poll, control, stop), scopeParams passthrough
  • lib/stream/scope-params.ts — Scope TypeScript types, presets, validation
  • lib/stream/scope-graphs.ts — 6 graph templates (simple-lv2v, depth-guided, etc.)
  • lib/stream/frame-extractor.ts — Multi-source frame extraction (webcam, image, video, URL)

Slash Commands

  • lib/story/ — /story generator, store, commands, storyteller prompt
  • lib/film/ — /film generator, store, commands, film prompt + 5 genre skills
  • lib/stream-cmd/ — /stream generator, store, commands, stream prompt
  • components/chat/StoryCard.tsx — Story card (purple theme, per-scene copy/edit)
  • components/chat/FilmCard.tsx — Film card (orange theme, camera icons)
  • components/chat/StreamPlanCard.tsx — Stream card (cyan theme, visual timeline)

MCP

  • lib/mcp/client.ts — MCP tool discovery + execution (discoverToolsViaProxy, executeToolCallViaProxy)
  • lib/mcp/store.ts — MCP server persistence (localStorage)
  • lib/mcp/types.ts — McpServerConfig, McpToolDef, MCP_PRESETS (5 presets including Gmail Local)
  • app/api/mcp/discover/route.ts — Server-side MCP discovery proxy (CORS bypass)
  • app/api/mcp/call/route.ts — Server-side MCP tool execution proxy
  • app/api/mcp/auth/route.ts — OAuth 2.0 + PKCE flow for Anthropic remote MCP
  • scripts/gmail-mcp-server.ts — Local Gmail MCP server (Google OAuth + 3 tools)

Skills (23 total)

  • skills/scope-agent.md — Scope Domain Agent: full parameter reference + natural language mapping
  • skills/director.md — Director workflow + Scope integration for multi-stream orchestration
  • skills/base.md — Base system prompt (rules for create_media, project_create routing)
  • skills/scope-lv2v.md, skills/live-director.md, skills/scope-graphs.md — LV2V reference
  • skills/seedance-cinematic.md — Seedance 2.0 cinematic video: prompt craft, film integration, duration guide

Infrastructure

  • simple-infra/sdk-service-build/app.py — SDK service code
  • simple-infra/environments/staging/byoc-a3.yaml — BYOC config
  • simple-infra/environments/staging/fleet.yaml — Orch fleet config

Plans & Docs

  • docs/scope-adv-plan.md — Scope Advanced Integration plan (6 phases, architecture, parameter reference)
  • docs/key-insights-scope.md — LV2V debugging guide

E2E Tests

  • tests/e2e/scope-phase{1-6}.spec.ts — Scope integration tests (58 tests total)
  • tests/e2e/storyboard.spec.ts — Core app tests
  • Run: npx playwright test tests/e2e/scope-phase*.spec.ts tests/e2e/storyboard.spec.ts

Source Material

  • Original storyboard: /Users/qiang.han/Documents/mycodespace/simple-infra/storyboard.html
  • SDK service: /Users/qiang.han/Documents/mycodespace/simple-infra/sdk-service-build/app.py
  • SDK feat branch: feat/scope-advanced-params in simple-infra (Scope param passthrough)
  • Scope source: /Users/qiang.han/Documents/mycodespace/Scope/scope/src/scope/
  • Scope client: /Users/qiang.han/Documents/mycodespace/Scope/scope/src/scope/server/livepeer_client.py
  • Scope cloud app: /Users/qiang.han/Documents/mycodespace/Scope/scope/src/scope/cloud/livepeer_app.py
  • go-livepeer LV2V: /Users/qiang.han/Documents/mycodespace/livepeer/go-livepeer/server/ai_live_video.go