AI Assistant: in-editor scene-editing agent (experiment)

[nearnshaw]

What this is

An experimental AI Assistant panel inside the Creator Hub editor: a chat sidebar that edits the open scene's files locally using a coding agent, with real-time streaming, per-scene conversation persistence, and subscription-based sign-in. The agent runs entirely on the user's machine against the scene folder — no server component.

How it works

Agent harness

• The Electron main process spawns pi (@mariozechner/pi-coding-agent, the agent underlying OpenDCL) directly in RPC/JSONL mode, one process per open scene, bridged to the renderer chat UI over IPC (main/src/modules/ai.ts).
• The @dcl-regenesislabs/opendcl package is kept purely as an asset carrier: we build the system prompt from its prompts/system.md + SDK7 context docs, and cherry-pick two of its extensions — dcl-context (injects scene.json metadata into the prompt) and dcl-validate (auto-runs tsc --noEmit after every TypeScript write).
• dcl-asset-path is deliberately excluded: its case-sensitive path guard rejects the Creator Hub assets/Models/ convention and steers downloads to a root models/ dir the Creator Hub can't see.
• The model is pinned per auth provider (Claude Sonnet 4.5 on Anthropic/Copilot credentials, GPT-5.1 Codex Max on ChatGPT).

Official SDK skills, auto-installed

• On agent start, the latest decentraland/sdk-skills are downloaded (cached app-wide by commit sha, 24h refresh) and synced into <project>/.agents/skills/ (main/src/modules/sdk-skills.ts). A marker file tracks ownership so user-added skills are never touched and upstream removals are cleaned up. opendcl's bundled (outdated) skills are only an offline-first-run fallback.

Auth: sign in with an existing AI subscription

• Setup panel offers Sign in with Claude (Anthropic) / ChatGPT / GitHub Copilot, using pi's own OAuth flows (main/src/modules/ai-auth.ts) — browser-based PKCE for Anthropic/Codex, device-code for Copilot — with progress, device-code instructions, and manual-code fallback streamed to the UI. An Anthropic API key remains available as the manual alternative.
• Credentials are stored in pi's ~/.opendcl/agent/auth.json; the spawned agent reads and auto-refreshes them itself (PI_CODING_AGENT_DIR is passed explicitly so main-process writes and agent reads can never diverge).
• Note: these flows authenticate the way pi/opencode-style tools do (first-party client IDs), which is a ToS gray zone accepted for this experiment; the longer-term shipping answer is a Decentraland-hosted proxy.

Conversation persistence

• Sessions ride pi's built-in session files: the agent is spawned with --continue, and pi keys sessions by cwd — so history is per scene and survives panel close and app restarts.
• Reopening the panel rebuilds the transcript via the get_messages RPC command (request/response correlation with timeouts in ai.ts); a header button starts a fresh conversation via new_session.

Editor integration

• After any turn that modified files, the inspector reloads the scene from disk so the agent's changes become visible immediately.

Testing

• Unit tests: JSONL splitter, sdk-skills cache/sync, chat reducer (all suites green: main 38, preload 23, renderer 147, shared 26).
• Verified live: pi boots over RPC, extensions provably load, OAuth-only auth streams claude-sonnet-4-5, session restore + fresh-session both work against a real scene.

🤖 Generated with Claude Code