Timeline Video Render#664
Open
cale-bradbury wants to merge 14 commits into
Open
Conversation
Clock and LFOInput self-scheduled their own requestAnimationFrame loops,
decoupled from HedronEngine's actual frame loop. That drifted invisibly
during live playback but broke outright during fixed-framerate video
rendering, since nothing called their rAF callbacks in lockstep with the
render loop's synthetic per-frame deltaTime.
Adds IPlugin.update(engine, { deltaFrame, deltaTime }), mirroring the args
shape sketches already receive, called from HedronEngine.advanceFrame for
both the live loop and render sequences. Clock gains a deterministic
step(deltaMs) driven the same way. LFOInput now implements update() instead
of self-ticking, and caches its store reference in the constructor again.
TimelineManager no longer self-schedules requestAnimationFrame; it exposes step(deltaMs), driven by TimelineInput's new IPlugin.update hook, both live and during a fixed-framerate render. play() gains a `silent` option to advance without starting audio playback, for headless rendering. Timeline length is now a `durationSeconds` option node (editable, feeding useTimelineData) instead of the hardcoded TIMELINE_DURATION constant. Exports DEFAULT_TIMELINE_ID and TimelineOptionNodes from the package so other packages (e.g. video-render) can read the timeline's duration/audio without the timeline package knowing they exist. Since TimelineManager no longer self-drives, the Storybook story now runs its own small rAF loop to keep the preview animating.
The hook only accepted ParamNode option nodes, warning and skipping anything else - but engine.addOptionNodes already supports ConfigShot on the write side. Widens the read side to match, so a shot-type option node (e.g. a plugin's own trigger-pad "shot") can be looked up and rendered the same way as any param.
An input-less plugin (no InputNode type of its own), modeled on scene-control's structure. It depends on @hedron-gl/timeline (reads the timeline's duration/audio option nodes, drives it during a render), but the timeline package has no knowledge of it - keeping the dependency one-way. VideoRenderPlugin owns a captureFrame shot (a normal ShotNode, so any MIDI/LFO/Timeline input can target it) and all the engine orchestration for rendering: resizing, driving the timeline deterministically and muted for the render's duration, and stepping engine.renderFramesSequence. Only genuinely Electron-specific I/O - writing files, invoking ffmpeg, showing a native directory picker - is injected via the constructor as a VideoRenderCallbacks object, so the package itself never touches fs/IPC.
Replaces main/handlers/frameHandlers.ts with main/handlers/renderVideo/, split into saveFrame (single-frame capture), renderSequence (frame-sequence + video orchestration), and ffmpeg (command construction/execution, pulled out since it's grown non-trivial with audio muxing). Adds a DialogEvents.OpenOutputDirDialog handler (same dialog.showOpenDialog pattern as the existing sketches-dir picker) so rendering can target a user-chosen directory instead of always writing under Documents. handleResourceFiles.ts now tracks the current resources directory so the render handler can resolve the timeline's audio resource (a filename) to an absolute path for ffmpeg, without a renderer round-trip.
engine.ts registers VideoRenderPlugin with videoRenderCallbacks - a thin adapter (renamed from frameCapture.ts) that's now just three IPC calls (save a frame, build a video, pick an output directory). All the engine orchestration that used to live here (resize, timeline sync, the render loop itself) moved into the plugin in the previous commit, so this file no longer touches the engine at all. Also registers TimelineGlobalPanel and VideoRenderGlobalPanel as sibling entries in pluginViews.globalPanel, replacing the old standalone "Video Controls" popup (App.tsx) entirely: its Render tab is now the video-render panel, Capture is the captureFrame shot, and Test Loop is dropped. Adds @hedron-gl/video-render as a workspace dependency.
- Drop RenderSequenceOptions.video and the "if (!video)" no-op branch in saveFrameSequenceHandler: buildVideo is only ever called when a video is actually wanted, so the flag could never be false. - Drop the unused `path` field from SaveFrameSequenceResponse/BuildVideoResult (never read by any caller). - Drop the `typeof engine.renderFramesSequence === 'function'` guard in VideoRenderPlugin.renderFrames: it's a real, always-present method, not optional, so the check and its error branch could never fire. - Delete the temporary trimmed-audio WAV file after muxing it into the video, instead of leaving it in the output directory indefinitely.
There was a problem hiding this comment.
Pull request overview
This PR introduces a dedicated @hedron-gl/video-render plugin/package to handle frame capture and timeline-based video rendering, and standardizes time-based behavior by moving per-plugin requestAnimationFrame loops into the engine’s per-frame update path.
Changes:
- Added new
@hedron-gl/video-renderpackage (plugin + UI panel) and wired it into the desktop app via injected Electron callbacks. - Refactored time-based systems (clock, timeline, LFO/gamepad/audio inputs) to advance via
HedronEngine.advanceFrame()instead of self-scheduling withrequestAnimationFrame. - Extended timeline to support configurable
durationSecondsvia an option node and propagate duration changes throughTimelineManager.
Reviewed changes
Copilot reviewed 45 out of 47 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-lock.yaml | Adds workspace link for the new @hedron-gl/video-render package and updates lockfile entries. |
| packages/video-render/vite.config.ts | Vite library build config for the new plugin package. |
| packages/video-render/tsconfig.json | TypeScript config for the new plugin package. |
| packages/video-render/src/VideoRenderPlugin.ts | Implements the render/capture orchestration plugin against engine + timeline. |
| packages/video-render/src/types.ts | Defines callback interfaces and render/capture option/result types. |
| packages/video-render/src/index.ts | Public exports for the video-render package. |
| packages/video-render/src/env.d.ts | Adds Vite client types reference for the package. |
| packages/video-render/src/constants.ts | Defines the option-node parent ID constant used by the plugin/UI. |
| packages/video-render/src/components/VideoRenderGlobalPanel/VideoRenderGlobalPanel.tsx | Adds UI to configure and trigger timeline-based rendering from the desktop app. |
| packages/video-render/src/components/VideoRenderGlobalPanel/VideoRenderGlobalPanel.module.css | Styles for the new global panel UI. |
| packages/video-render/src/components/VideoRenderGlobalPanel/useCaptureFrameShot.ts | Registers the shot handler behind the capture-frame option shot node. |
| packages/video-render/package.json | Declares the new publishable workspace package and its deps/peers. |
| packages/video-render/.eslintrc.cjs | ESLint config for the new package. |
| packages/video-render/.eslintignore | Ignores build output and node_modules for the new package. |
| packages/ui-core/src/hooks/useNodeOptionNodes.ts | Extends useNodeOptionNodes typing/runtime checks to support shot nodes. |
| packages/timeline/src/TimelineManager.ts | Replaces internal rAF ticking with deterministic step(deltaMs) and adds setDuration. |
| packages/timeline/src/TimelineInput.ts | Adds durationSeconds option node and drives managers via plugin update. |
| packages/timeline/src/stories/Timeline.stories.tsx | Manually ticks TimelineManager.step() for storybook usage without an engine. |
| packages/timeline/src/index.ts | Exports TimelineOptionNodes and DEFAULT_TIMELINE_ID. |
| packages/timeline/src/components/TimelineGlobalPanel/useTimelineData.tsx | Uses the timeline’s durationSeconds param to derive durationMs. |
| packages/timeline/src/components/TimelineGlobalPanel/TimelineGlobalPanel.tsx | Adds a UI control for durationSeconds. |
| packages/lfo-input/src/LFOInput.ts | Converts LFO driving loop to engine-driven update. |
| packages/gamepad-input/src/GamepadInput.ts | Removes self-scheduled rAF loop and relies on engine frame updates. |
| packages/engine/src/plugins/Plugin.ts | Adds PluginUpdateArgs and the optional plugin.update() hook. |
| packages/engine/src/HedronEngine/HedronEngine.ts | Calls clock.step + plugin.update each frame and shares it with render sequencing. |
| packages/clock/src/index.ts | Replaces internal rAF ticking with explicit step(deltaMs) called by the engine. |
| packages/audio-input/src/AudioInput.ts | Removes self-scheduled rAF loop and relies on engine frame updates. |
| apps/desktop/src/shared/FrameEvents.ts | Updates frame/video IPC payload types for the new render flow. |
| apps/desktop/src/shared/Events.ts | Adds OpenOutputDirDialog IPC event + response type. |
| apps/desktop/src/renderer/utils/renderVideo/videoRenderCallbacks.ts | Implements desktop-side callbacks for the new plugin (IPC to main). |
| apps/desktop/src/renderer/utils/frameCapture.ts | Removes the old window-based frame/render helpers. |
| apps/desktop/src/renderer/main.tsx | Stops importing the removed frameCapture utility. |
| apps/desktop/src/renderer/engine.ts | Registers the new video-render plugin and adds its global panel to pluginViews. |
| apps/desktop/src/renderer/components/VideoControls/VideoControls.tsx | Removes legacy VideoControls UI. |
| apps/desktop/src/renderer/components/VideoControls/VideoControls.module.css | Removes legacy VideoControls styling. |
| apps/desktop/src/renderer/components/VideoControls/RenderTab.tsx | Removes legacy render UI tab (window-based). |
| apps/desktop/src/renderer/components/VideoControls/LoopTab.tsx | Removes legacy loop testing UI tab. |
| apps/desktop/src/renderer/components/VideoControls/CaptureTab.tsx | Removes legacy capture UI tab (window-based). |
| apps/desktop/src/renderer/components/App/App.tsx | Removes the legacy VideoControls widget from the app shell. |
| apps/desktop/src/main/index.ts | Adds output-dir dialog handler and routes frame/video IPC to new handlers. |
| apps/desktop/src/main/handlers/renderVideo/saveFrame.ts | New main-process handler for writing frames to disk. |
| apps/desktop/src/main/handlers/renderVideo/renderSequence.ts | New main-process handler to build a video from already-saved frames. |
| apps/desktop/src/main/handlers/renderVideo/paths.ts | Shared path helpers for render output directories/filenames. |
| apps/desktop/src/main/handlers/renderVideo/ffmpeg.ts | New ffmpeg integration to build mp4 + optional audio muxing. |
| apps/desktop/src/main/handlers/frameHandlers.ts | Removes legacy combined frame/video handler implementation. |
| apps/desktop/src/main/handleResourceFiles.ts | Tracks current resources directory for audio resolution during muxing. |
| apps/desktop/package.json | Adds dependency on @hedron-gl/video-render. |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+159
to
+165
| const isPlayingNode = engine.getNodeOptionNode(timelineId, 'isPlaying') | ||
| if (engine.getParamValue(isPlayingNode.id)) { | ||
| manager.goTo(0) | ||
| manager.play() | ||
| } else { | ||
| manager.pause() | ||
| } |
Comment on lines
+6
to
+10
| function runFfmpeg(cmd: string): Promise<void> { | ||
| return new Promise((resolve, reject) => { | ||
| exec(cmd, (error, _stdout, stderr) => { | ||
| if (error) { | ||
| console.error('ffmpeg error:', error, stderr) |
Comment on lines
+1
to
+19
| import fs from 'fs' | ||
| import { getDocumentsPath, getFrameFilePath, getTimestampedFilePath } from './paths' | ||
| import { SaveFrameOptions, SaveFrameResponse } from '@shared/FrameEvents' | ||
|
|
||
| export async function saveFrameHandler( | ||
| _: unknown, | ||
| base64Data: string, | ||
| options?: SaveFrameOptions, | ||
| ): Promise<SaveFrameResponse> { | ||
| try { | ||
| const { name, frameIndex, outputDirAbsolute } = options ?? {} | ||
|
|
||
| let filePath: string | ||
| if (name !== undefined && frameIndex !== undefined) { | ||
| filePath = getFrameFilePath(outputDirAbsolute ?? getDocumentsPath(), name, frameIndex) | ||
| } else { | ||
| filePath = getTimestampedFilePath() | ||
| } | ||
| fs.writeFileSync(filePath, base64Data, 'base64') |
Comment on lines
634
to
641
| private advanceFrame(deltaTime: number) { | ||
| this.clock?.step(deltaTime * 1000) | ||
| Object.values(this.plugins).forEach((plugin) => | ||
| plugin.update?.(this, { deltaFrame: 1, deltaTime }), | ||
| ) | ||
|
|
||
| // Flush buffered node value updates before processing the frame | ||
| flushParamValueBuffer(this.store.setState) |
Comment on lines
+10
to
+12
| let currentResourcesServer: ResourcesServer | null = null | ||
| let currentResourcesDir: string | null = null | ||
| export const getCurrentResourcesDir = (): string | null => currentResourcesDir |
# Conflicts: # packages/timeline/src/TimelineInput.ts
- ffmpeg.ts: build ffmpeg commands as an args array via execFile instead of a shell string via exec, closing a shell injection vector on filenames/paths. - saveFrame.ts: use async fs.writeFile instead of the sync variant so frame writes don't block Electron's main process during a render. - VideoRenderPlugin.ts: pause() before re-calling play() when restoring playback state after a render, since TimelineManager.play() is a no-op when already "playing" (which the silent render pass leaves it as) - audio was never resuming otherwise. - HedronEngine.ts: flush the param value buffer both before and after plugin.update() in advanceFrame(), so plugins read current-frame values instead of one-frame-stale ones, and their own buffered updates are visible to sketches the same frame. - handleResourceFiles.ts: clear currentResourcesDir when shutting down the previous server, before validating the new directory, so it can't stay pointed at a directory no server is actually serving. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add Output Directory/Name/Create Video/Width/Height/FPS as option nodes on the video-render node (via defineOptionNodeConfigs), so render settings use the standard param UI everywhere else in the app and are saved/loaded with the project via the engine's normal state serialization, instead of living in component-local state backed by sessionStorage. - Browse and Render are now shot nodes (like the existing Capture Frame), rendered as TriggerPads, so they can be triggered from MIDI/keyboard/etc like any other shot, not just a plain button click. - Panel layout: Output Directory is a standard string NodeContainer at 2/3 width with Browse beside it at 1/3; Width/Height/FPS on one row; Name/Create Video on one row; vertical padding to match sketch param panels. - Fix apps/desktop/electron.vite.config.ts: @hedron-gl/video-render was missing from the renderer's HMR alias list (every other workspace package has one), so dev-mode edits to this package were silently resolving against the stale built dist instead of live source. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will likely have some conflicts after the other PRs, I'll clean this up once those are merged.
Moves video rendring into a plugin that uses values from the timeline.
Moves timing control to global hedron update loop instead of per-plugin
requestAnimationFrameCleaned up hacky parts of the existing video code, watching for signals in console.logs, calling functions from
window