Add agentic assistant panel to the image editor - #4372
Merged
Conversation
Bring the timeline/storyboard editor's agent-chat pattern to the image (sketch) editor: a toggleable right-side assistant panel whose LLM can drive the canvas through a set of ui_sketch_* frontend tools. - sketchAgentBridge: serializable handler interface + snapshot types, with get/set/has, mirroring timelineAgentBridge. - useSketchAgentBridge: builds the handler from the editor's per-instance stores (editor, session bindings, canvas refs) plus the direct-gen job runner, and registers it while the editor is the focused surface. - ui_sketch_* tools (lib/tools/builtin/sketch.ts): get_state, add/remove/ duplicate/select layer, set_layer_props, reorder, merge_down, flatten_visible, generate (text-to-image / image-to-image), set_color, set_tool, resize_canvas, selection, and get_layer_image (vision). Wired into the frontend-tools IPC manifest. - SketchAgentPanel: reuses ChatView + GlobalChatStore, passing the open document id as workflow_id so the server forwards the ui_sketch_* tools. - Mounted as a toggleable column in SketchEditor with an Assistant toggle in the prompt bar; assistantPanelOpen flag added to the ui slice. - Extend SketchCanvasRefStore with getLayerData / fillLayerWithColor so the bridge can read and fill layers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WyF6AfWZPdihy1TLKLi9ca
Let the image editor render a layer — or the flattened composite — to a PNG
and upload it as a throwaway asset, returning the asset id + URL so the agent
can hand the artwork to another tool or workflow that needs an asset id
rather than raw pixels.
- lib/sketch/renderLayerToAsset.ts: reusable helper. Composite prefers the
live canvas (flattenToDataUrl) and falls back to re-flattening the
serialized document; a single layer renders from its committed raster via
exportLayer. Uploads via AssetStore.createAsset("file"), with a
deleteRenderedAsset best-effort cleanup companion.
- sketchAgentBridge: renderLayerToAsset handler method + result type.
- useSketchAgentBridge: wires it to the helper using the instance's document
and live-canvas flatten getter.
- ui_sketch_render_to_asset tool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WyF6AfWZPdihy1TLKLi9ca
Extend the render-to-asset capability so the agent can materialize several
layers at once, not just one layer or the composite.
- renderLayersMerged: composite an arbitrary subset of layers (original
bottom-to-top order, honoring per-layer opacity/blend, group nesting
flattened away) into a single temporary asset.
- sketchAgentBridge / useSketchAgentBridge: renderLayersToAssets(targets,
{merge, name}) — one asset per layer, or a single merged asset when
merge is true. Targets are resolved up front so a bad id fails before any
upload.
- ui_sketch_render_to_asset now accepts `targets` (list) + `merge`, and
always returns an `assets` array (the single/composite path returns a
one-element array).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WyF6AfWZPdihy1TLKLi9ca
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Brings the timeline/storyboard editors' agent-chat pattern to the image (sketch) editor: a toggleable right-side Assistant panel whose LLM can drive the canvas through a new set of
ui_sketch_*frontend tools. No backend changes — the unified/wschat loop already forwards any registeredui_*tool whose manifest it received, as long as the turn carries aworkflow_id.Until now the editor's only AI surfaces were single-shot: the full-frame prompt bar and the selection inpaint/edit bar. This adds a conversational agent that can plan and execute multi-step edits (add layers, generate, recolor, reorder, resize, inspect) like a real editor.
How
components/sketch/sketchAgentBridge.ts— serializableSketchAgentHandlerinterface + snapshot/node types, withget/set/hasSketchAgentHandler(mirrorstimelineAgentBridge). Nothing but plain JSON crosses the bridge.hooks/sketch/useSketchAgentBridge.ts— builds the handler from the editor's per-instance stores (editor, session bindings, canvas refs) plususeDirectGenJob, and registers it while the editor is the focused surface (cleared on blur/unmount). Layers are addressed by id, case-insensitive name, or the literal"active".lib/tools/builtin/sketch.ts— registers theui_sketch_*tools:get_state,add_layer,remove_layer,duplicate_layer,select_layer,set_layer_props,reorder_layer,merge_down,flatten_visible,generate(text-to-image / image-to-image),set_color,set_tool,resize_canvas,selection, andget_layer_image(returns a PNG data URL so the agent can see the artwork). Registered infrontendToolsIpc.ts.components/sketch/SketchAgentPanel.tsx— reusesChatView+GlobalChatStore, passing the open document id asworkflow_id(context only — noworkflow_target, so the document is never run as a workflow).SketchEditor.tsx— mounts the panel as a toggleable right-most column, callsuseSketchAgentBridge(active), and threads anactiveprop fromStandaloneSketchEditor. An Assistant toggle button was added to the prompt bar; theassistantPanelOpenflag lives in the UI slice.SketchCanvasRefStore.ts— extended withgetLayerData/fillLayerWithColorgetters (kept optional for existing callers) so the bridge can read and fill layers, reusing the canvas's real fill logic.Testing
npm run typecheck(web): 0 errors.npm run lint(web): passes (exit 0); the panel'suseMemodep warning matches the existingTimelineAgentPanel/StoryboardAgentPanelsiblings.src/lib/tools/__tests__/sketchTools.test.ts(13 tests) covers manifest registration, schema shape, the no-editor error path, handler delegation, and Zod validation rejections — all pass. ExistingSketchCanvasRefStore/SelectionActionBarsuites still pass. (The unrelated canvas-rendering suites fail only because the nativecanvasdep is skipped in this sandbox; verified identical failures without this change.)🤖 Generated with Claude Code
https://claude.ai/code/session_01WyF6AfWZPdihy1TLKLi9ca
Generated by Claude Code