TDPilot DPSK4 v2.4 — Implementation Plan (Cold-Start Executable)

For the receiving agent: Drop this file into a fresh Claude Code session inside the TDPilot DPSK4 repo and say "execute this plan." Everything below is self-contained — no prior session memory required. The plan is organized into five phases (A–E) sequenced by dependency, with subagent dispatch points marked explicitly. Default operating mode is maximize intelligence; cost is not a constraint.

0. Bootstrap Context

Repo location

Main repo (deepseekv4 fork): <REPO_ROOT>/
Worktree where prior session worked: <REPO_ROOT>/.claude/worktrees/quizzical-heisenberg-3cd974
Branch: claude/quizzical-heisenberg-3cd974 (working branch — confirm or create a new branch from main if user prefers)
Upstream reference (original TDPilot): <USER_DESKTOP>/TDPilot/TDPilot-main/ (read-only reference; do not commit there)

Current release state

Shipped on main: v2.3.0 (commit ef0aec2) + 4-bug audit (commit 8f087cd)
Pending in working tree (from prior session — DECIDE FIRST: commit or discard):
- M skills/tdpilot-dpsk4-core/SKILL.md — added §0 "Mandatory STOP Rules" + extended Reference Files index
- ?? skills/tdpilot-dpsk4-core/references/anti-patterns.md (NEW)
- ?? skills/tdpilot-dpsk4-core/references/audio-reactive-glsl.md (NEW)
- ?? skills/tdpilot-dpsk4-core/references/glsl-idioms.md (NEW)
- ?? skills/tdpilot-dpsk4-core/references/recording-and-export.md (NEW)
First action of this plan: decide whether to commit those 5 files as the opening commit of v2.4 work, or roll them into Phase A's commit. Recommended: commit them first under a docs(skills): expand core reference library message so v2.4 work starts from a clean tree.

Critical derived-artifacts checklist (from CLAUDE.md)

After ANY change in this repo, confirm whether each artifact is affected:

td_component/tdpilot.tox — rebuild required if any of _TOX_SOURCE_FILES (mcp_webserver_callbacks.py, event_emitter.py, ws_callbacks.py, tdpilot_startup.py) is touched. Rebuild only possible inside a live TouchDesigner session.
tdpilot.plugin — ZIP of skills, .tox, README, commands, .mcp.json, plus pyproject.toml/src/td_mcp/uv.lock. Rebuild via uv run python scripts/build_plugin_zip.py.
Tool count — currently 104. Source of truth: EXPECTED_MIN_TOOL_COUNT in src/td_mcp/release_gates.py. v2.4 will add ~3 new tools (DMX inspect/status, MIDI devices) — bump that constant and all user-facing copies (README, npm/README, plugin_README, docs/API_REFERENCE, etc.).
CHANGELOG.md — entry per phase.
Seven version files must stay in sync (enforced by scripts/check_versions.py):
- pyproject.toml
- src/td_mcp/__init__.py
- .claude-plugin/plugin.json
- .claude-plugin/marketplace.json (the Update-button-driving one)
- npm/package.json
- mcp/manifest.json
- td_component/mcp_webserver_callbacks.py (API_VERSION)
GitHub Release — after git push origin vX.Y.Z, also run gh release create vX.Y.Z so the .mcpb / .plugin assets land on the Release page.

Operating philosophy for v2.4

Cost is not a constraint. Maximize intelligence and capability per turn. Latency and reliability still matter.

This flips several defaults vs. earlier versions:

Be verbose in tool descriptions and system prompts — embed examples, anti-patterns, rationale. The model uses this context to make better decisions.
Dispatch parallel subagents aggressively — any 2+ independent inspection or research tasks should run in parallel.
Use td_tool_batch liberally for multi-tool inspections. Don't serialize what can be parallelized.
Default thinking.budget_tokens HIGH (e.g., 8000) — deep reasoning compounds.
Eager memory + knowledge recall every turn, even when not explicitly asked.
Generous pre-turn retrieval — return 5–10 candidate docs and let the model rank, not 1–2.
LLM-assisted compaction with the full-tier model, not a heuristic.
Aggressive screenshot capture for visual verification — token cost is not a reason to skip a td_screenshot.

What DOESN'T flip:

Reliability matters — retries, circuit breakers, error handling still critical.
Latency matters — SSE streaming, tool batching still critical.
Security matters — Authmode migration still Phase B.
Correctness matters — BM25 content_type fix still needed.

1. Phase Overview

Phase	Theme	Items	Effort	TouchDesigner rebuild needed?
A	Zero-Risk Quick Wins	5	~Half day	Yes (1 .tox rebuild for items A.3, A.4, A.5)
B	Vision + Security + Onboarding	5	~1 week	Yes (1 .tox rebuild)
C	Hardware Integration + Discovery + Observability	9	~1–1.5 weeks	Yes (1–2 .tox rebuilds)
D	Streaming + LLM-Assisted Compaction	2	~1 week	Yes (1 .tox rebuild)
E	Polish, Content, Commercial Readiness	7	Ongoing	No

Total scope: 28 items. Excluded by user request: projection mapping reference + td_window_inspect (deferred to v2.6 or later).

Phases A and B should release as v2.4.0. Phase C should release as v2.4.1 (additive tools). Phase D should release as v2.5.0 (architectural change). Phase E rolls in opportunistically.

2. Phase A — Zero-Risk Quick Wins

Goal: Ship 5 high-leverage edits in a single session. One .tox rebuild required.

Subagent dispatch suggestion: none — all items are small focused edits the main agent can do directly. Optionally dispatch ONE explorer subagent to confirm current line numbers in target files before editing.

A.1 — Update `docs/NEW_SESSION_PROMPT.md`

Why: Currently references v2.1.5 as "latest." Any fresh agent picking it up plans work already shipped.
File: docs/NEW_SESSION_PROMPT.md
Steps:
1. Update the "Current state" block to reflect main @ 8f087cd, "v2.3.0 shipped, v2.3.1 fixes landed (uncommitted in some sessions), v2.4 in planning."
2. Update "Most recent merged PR" to #43.
3. Add a maintenance convention to AGENTS.md: "After every release, update docs/NEW_SESSION_PROMPT.md's Current state block before closing the release PR."
Acceptance: grep -n "v2.1.5" docs/NEW_SESSION_PROMPT.md returns no matches in the "Current state" block.
Gotchas: none.

A.2 — Reorder README install-decision table

Why: New users hit a wall of release notes before they see how to install. The decision table belongs above the fold.
File: README.md
Steps:
1. Locate the install-decision table (standalone .tox vs Claude Code CLI vs npm).
2. Move it to immediately after the project tagline / hero section, before any changelog or feature lists.
3. Add a one-sentence "start here" rule: "If you just want to chat with TouchDesigner → standalone .tox. If you already use Claude Code → plugin. If you want MCP in another client → npx tdpilot."
4. Wrap the changelog/feature block beneath in a <details> collapsible.
Acceptance: The first 60 lines of README.md show the decision table and the start-here rule. The full release history is one <details> link away.
Gotchas: preserve all anchor links — some external sites may deep-link to README sections.

A.3 — Teach `tool_batch` in `SYSTEM_PROMPT_BASE`

Why: tool_batch exists and works, but the system prompt never mentions it, so the model defaults to serial tool_use blocks. Saves 3–9s of latency per multi-tool turn. Highest leverage-per-byte change available.
File: td_component/tdpilot_api_runtime.py — find the SYSTEM_PROMPT_BASE constant.

Steps:

Locate the "Operating Protocol" section of SYSTEM_PROMPT_BASE.

Add a new paragraph teaching tool_batch:

**Parallel inspection via tool_batch.** When you need 2+ independent
read-only lookups in the same turn (e.g., td_get_nodes + td_get_errors
+ td_audit_project, or memory_recall + knowledge_search), issue them as
a single `tool_batch` call rather than sequential tool_use blocks. This
halves to quarters your turn latency on inspection-heavy work. Reserve
serial tool_use for genuinely sequential operations (create then connect
then set parameters).

Add 2 worked examples inline:
- Example 1: pre-build inspection → tool_batch([td_get_nodes, td_get_errors, td_audit_project])
- Example 2: memory recall before creation → tool_batch([memory_recall, knowledge_search, td_get_hints])

Acceptance: grep -n "tool_batch" td_component/tdpilot_api_runtime.py shows the new prompt text. Manual test in a live TD session: ask the agent to "audit the project" and confirm it issues a single tool_batch call.
Gotchas:
- Touches _TOX_SOURCE_FILES → .tox rebuild required after this batch.
- System prompt changes break DeepSeek's cache prefix for every new session — accept the one-time cold-turn cost.

A.4 — Add `gpuCookTime` + `cudaMemory` to `td_cooking_info`

Why: Current tool reports CPU cook time only. For GPU-heavy projects (every visual project) the bottleneck is invisible. TD 2025 also exposes TOP.cudaMemory(pixelFormat=...) for per-TOP VRAM footprint.
Files:
- td_component/callbacks/handlers/inspect.py — find handle_cooking_info (around lines 332–344 per prior research)
- src/td_mcp/registry/tools_data.py — find td_cooking_info tool definition

Steps:

In handle_cooking_info's collect_cook function, add:

entry["gpuCookTime"] = float(getattr(node, "gpuCookTime", 0.0))
if node.family == "TOP":
    try:
        entry["cudaMemoryBytes"] = node.cudaMemory()
    except Exception:
        entry["cudaMemoryBytes"] = None

In tools_data.py, extend the sort_by enum / parameter docs to include "gpuCookTime" and "cudaMemoryBytes".
Update the tool docstring to mention GPU diagnostics.

Acceptance: Manual test in a live TD project — call td_cooking_info(sort_by="gpuCookTime", limit=10) and confirm GPU-heavy TOPs (feedback loops, large GLSL TOPs) appear at the top.
Gotchas:
- gpuCookTime is 0 for non-TOP operators — sort_by="gpuCookTime" will sort lots of zeros; document this in the docstring.
- cudaMemory is per-pixel-format; for the default pass we can omit pixelFormat and TD picks the current format.
- Touches _TOX_SOURCE_FILES → .tox rebuild required (already in A.3's bundle).

A.5 — Retry-with-backoff for DeepSeek 429/503

Why: Currently any 429 or 503 raises AgentError and kills the turn. DeepSeek frequently rate-limits at peak. Users must manually resend.
File: td_component/tdpilot_api_agent.py — find the API-call function.
Steps:
1. Wrap the urllib.request.urlopen block in a helper _call_api_with_retry(max_retries=3, initial_backoff=2.0).
2. Inside the wrapper:
  - On HTTP 429: parse Retry-After header if present; else exponential backoff with jitter (backoff = initial * 2**attempt + random.uniform(0, 0.5)).
  - On HTTP 503: same backoff strategy.
  - On HTTP 5xx (other): retry up to max_retries.
  - On HTTP 400/401/403/404: raise immediately (non-retryable).
  - Check the agent stop_flag between retries; abort if set.
3. Per-attempt timeout stays at the existing value (request_timeout=120 or whatever is current). Cumulative wait is bounded by max_retries * (timeout + max_backoff).
4. On final failure (after exhausting retries), surface EV_HINT with diagnosis ("API rate-limited 3× — wait a minute and retry") before raising.
Acceptance: Unit test in tests/ that mocks a 429 response and asserts the retry path executes; manual test triggering rate limit and confirming auto-recovery.
Gotchas:
- Touches _TOX_SOURCE_FILES → .tox rebuild required.
- Test path: add a fixture under tests/_mock_deepseek.py (per existing PR-20 mock-deepseek infrastructure).

Phase A bundle: `.tox` rebuild + verification

After items A.3, A.4, A.5 land, rebuild the .tox:

Open td_component/build_export_mcp_tox.py in TouchDesigner Textport (use the canonical 4-line recipe from memory file feedback_td_tox_rebuild_recipe.md).
Confirm td_component/.tox-source-hash.json updates.
Confirm auto-mirror to main repo's td_component/tdpilot.tox (per Phase 1.2.2+ behavior).
Run python scripts/check_tox_freshness.py — should pass.
Run python scripts/check_versions.py — should pass (no version bumps yet).
Run ruff: uv run ruff check src/ td_component/ tests/.

Phase A definition of done: All 5 items merged, .tox rebuilt, ruff clean, manual smoke test in a live TD session passes.

3. Phase B — Vision, Security, Onboarding

Goal: Fix the highest-impact correctness/security/UX issues. Releasable as v2.4.0.

Subagent dispatch suggestion:

For B.1 (screenshot/cache): dispatch ONE sonnet subagent to deep-read the agent loop, runtime, and screenshot handler to produce a concrete patch plan with exact line numbers and a tested sequencing for stripping base64 from history vs. surfacing it to the model as a proper image content block.
For B.2 (Authmode wizard): dispatch ONE sonnet subagent to map the existing /firstrun endpoint surface in tdpilot_api_introspect.py::firstrun_status and tdpilot_api_web_callbacks.py, plus the chat HTML, and produce a wizard step spec.
For B.4 (BM25 content_type): dispatch ONE sonnet subagent to map all existing memory/recipe/knowledge store schemas (in tdpilot_api_memory.py, tdpilot_api_recipes.py, and the knowledge store) and propose the migration sequence (backfill default, filter location, rollout safety).
B.3 (worked examples) and B.5 (slash commands) can run in parallel with the above as separate subagents writing the content.

Dispatch B.1, B.2, B.4, B.3, B.5 subagents in a single message → 5 parallel research/scoping reports back. Synthesize into 5 focused implementation PRs (or one combined PR).

B.1 — Screenshot vision pipeline + history pruning

Why (under cost-no-issue): the REAL value isn't token savings — it's that the model literally cannot see screenshots through tool_result JSON. DeepSeek's Anthropic-compat layer does not surface base64 inside tool_result. We need to:
- (a) keep the screenshot accessible to the chat UI broadcast,
- (b) preserve it in some form for the model,
- (c) wire it as a proper image content block in the user turn (not buried in tool_result).
Files:
- td_component/tdpilot_api_agent.py — agent loop where tool_result is appended to self.messages
- td_component/callbacks/handlers/inspect.py::handle_screenshot — where the payload is generated
Approach:
1. Split the screenshot payload into a display payload (full base64, broadcast via WS to the chat UI) and an agent payload (path + format + size_bytes + a content_type hint).
2. In the agent loop, when assembling self.messages:
  - Append a placeholder tool_result with the agent payload (no base64).
  - Immediately follow with a synthetic user-turn message containing an image content block referencing the same screenshot.
3. Test that DeepSeek's compat layer accepts image in user content. If not, fall back to base64-in-tool_result with a logged warning (currently the model is blind anyway).
Acceptance:
- Model can describe what's in a fresh screenshot it just took (manual test: ask "describe what you see" after td_screenshot).
- Cache hit rate (visible in EV_USAGE) does not regress on multi-screenshot turns.
Gotchas:
- Sequencing matters: the image block must appear in user content, not assistant content, per Anthropic API spec.
- DeepSeek's image handling is undocumented — test with a real call, not just mocks.

B.2 — Authmode=open → token migration wizard

Why: v2.2.1's drag-and-go default disables token auth. Drive-by RCE risk via td_exec_python from any local-origin browser tab. Need a guided migration without breaking the drag-and-go UX.
Files:
- td_component/tdpilot_api_web_callbacks.py — add new HTTP route POST /set-authmode
- td_component/tdpilot_api_introspect.py::firstrun_status — extend to report Authmode state
- td_component/tdpilot_api_chat.html — add wizard step
Steps:
1. Extend /firstrun response with authmode_open: bool and recommended_action: "switch_to_token".
2. Add new route POST /set-authmode that:
  - Accepts {"mode": "token"}.
  - Writes the COMP param Authmode = "token".
  - Generates a new token via the existing token-generation code.
  - Returns {"ok": true, "token": "<new_token>"} in the response body.
3. Add a wizard step in the chat HTML that:
  - Detects authmode_open: true on first load.
  - Shows a banner: "Drag-and-go mode is convenient but insecure. Switch to token auth?"
  - On click → POST /set-authmode → shows the new token + a "Reload chat" button (the existing chat tab will need to reload to pick up the token).
Acceptance: Fresh install on a new machine shows the wizard. Clicking switches Authmode and surfaces the token. Insecure-mode banner disappears on reload.
Gotchas:
- The wizard must NOT auto-switch — user must opt in (some users genuinely want drag-and-go).
- After switch, the open chat tab will have a stale (empty) token — wizard must instruct reload.

B.3 — Worked examples in `docs/examples/`

Why: README ends at "you're chatting" — no concrete path to first value. 5 examples cover the most-asked workflows.
Files: new directory docs/examples/
Examples to write:
1. 01-audio-reactive-noise-field.md — uses the empirically verified chain from references/audio-reactive-glsl.md. Exact prompts, expected tool calls, screenshots.
2. 02-feedback-loop-from-scratch.md — references the canonical feedback recipe from tdpilot-dpsk4-core SKILL.md §11.
3. 03-project-health-audit.md — opens an existing .toe, runs td_audit_project, surfaces errors and instability.
4. 04-glsl-shader-driven-by-mouse.md — value0/value1 Values-page pattern from glsl-idioms.md.
5. 05-record-1080p-clean-export.md — uses the pre-recording checklist from recording-and-export.md.
Format per example:
- Headline: what you'll build (1 sentence)
- Prerequisites: TD build, plugin version
- Setup: empty .toe project, what to open
- The prompts: literal user messages to type, in order
- The expected tool sequence: what tools the agent should call (so user knows what to expect)
- The result: screenshot or short GIF
- Variations: 2–3 prompts to try after the base example works
Acceptance: Each example is smoke-tested in a live TD session before merging.
Gotchas: keep the examples concise — 10 prompts max per example.

B.4 — BM25 retrieval `content_type` tagging

Why: The over-eager-tool-use bug (memory entry project_tdpilot_api_agent_overeager_tool_use.md): BM25 retrieval matches instruction-shaped docs on short prompts and the model executes those instructions. Root cause is content conflation — search results and user intent land in the same context window with no structural barrier.
Files:
- td_component/tdpilot_api_runtime.py::_run_pre_turn_retrieval
- td_component/tdpilot_api_memory.py (memory store schema)
- td_component/tdpilot_api_recipes.py (recipe store schema)
- Any knowledge-store schema files (search for knowledge_save definitions)
Steps:
1. Add a content_type field to each store's schema: "instruction" | "reference" | "fact".
2. Backfill existing entries: default to "reference" for safety (instruction-typed entries should be opt-in via re-save).
3. In _run_pre_turn_retrieval, filter out content_type == "instruction" docs unless the user's message contains an explicit reference to the doc's name/title.
4. Update memory_save / knowledge_save / recipe-save tools to accept and surface content_type as a parameter (with the docstring explaining the difference).
5. Document the field in the relevant skill/reference files.
Acceptance: Reproduce the over-eager-tool-use scenario from the memory entry; after fix, the model no longer drive-by-creates nodes from short prompts.
Gotchas:
- Migration must not lose existing data; default to "reference" is safe but means instruction-typed docs (recipe step lists) won't be retrieved until re-saved.
- This touches _TOX_SOURCE_FILES → .tox rebuild required.

B.5 — New slash commands: `/td-explain`, `/td-tour`, `/td-build`

Why: Only /td-check and /td-snapshot ship today. Comparable tools have 5–10. Discoverability is zero.
Files: new files in commands/
Commands to add:
- commands/td-explain.md — invokes td_get_nodes + td_get_errors + td_describe_surface and renders a structured project summary. Pure markdown spec; the agent does the work.
- commands/td-tour.md — auto-runs a 5-step capability walkthrough: health check → create a test node → screenshot it → save a memory entry → list recipes. Designed for first-time users.
- commands/td-build.md — accepts a template name or freeform description; invokes the recipe / macro system to scaffold a starting structure.
Acceptance: Each command, invoked in Claude Code with TDPilot installed as a plugin, produces the expected behavior.
Gotchas: commands only run in the Claude Code plugin variant, not the standalone .tox chat. Document this in each command's preamble.

Phase B bundle: `.tox` rebuild + verification

After B.1, B.2, B.4 land, rebuild the .tox (B.3 and B.5 are pure docs/commands, no rebuild needed).

Phase B definition of done: all 5 items merged + .tox rebuilt + smoke test for vision (B.1), Authmode wizard (B.2), and BM25 fix (B.4) all pass. Worked examples (B.3) live and linked from README. New slash commands (B.5) work in a fresh Claude Code session.

Release here as v2.4.0 if Phase C is going to take more than a week. Otherwise bundle A+B+C into a single v2.4.0 release.

4. Phase C — Hardware Integration, Discovery, Observability

Goal: Cover the heavy TD-domain gaps (DMX, MIDI), surface capability information, add cost/state observability.

Subagent dispatch suggestion: This phase has 9 items, mostly independent. Recommended dispatch pattern:

Wave 1 (parallel research): 3 sonnet subagents in parallel:
- Agent C-research-1: deep-research TD's DMX Fixture POP, DMX Out POP, Pan Tilt CHOP, DMX Map DAT API surface (web search TD docs if needed) → output a tool spec for td_dmx_inspect and td_dmx_status.
- Agent C-research-2: deep-research TD's app.midiDeviceManager API and midiinmapCHOP parameter surface → output a spec for td_midi_devices.
- Agent C-research-3: deep-research the chat HTML, EV_USAGE event payload, and tracing infrastructure → output a spec for cost tracking surfacing in the UI.
Wave 2 (parallel implementation): dispatch 4 sonnet subagents to write the references (NDI, MIDI, depth-sensors) and capability index in parallel.
Wave 3 (main agent): integrate the tools, run rebuilds, verify.

C.1 — `td_dmx_inspect` + `td_dmx_status` + DMX hint pack

Why: Flagship TD 2025.32820 feature, zero tool coverage. Lighting designers ask "what channels is fixture 3 using?" and the agent has to drop to td_exec_python.
Files:
- src/td_mcp/registry/tools_hardware.py (NEW) or extend tools_data.py
- src/td_mcp/hints/packs/topics/dmx_rig.yaml (NEW)
- src/td_mcp/macros/templates.py — add "dmx_basic_rig" template
Tools to add:
- td_dmx_inspect(path: str) — reads a dmxFixturePOP path, returns per-point fixture addresses, universe assignments, attribute overrides, DMX Map DAT universe layout.
- td_dmx_status(path: str) — reads a dmxOutPOP, returns protocol (Art-Net/sACN/KiNET/FTDI), IP/port, packet rate, last error, NIC binding.
Acceptance: Manual test with a real DMX rig setup in TD. Agent answers "what universe is fixture 5 on?" without td_exec_python.
Gotchas:
- DMX operators are build-gated (2025.30000+) → guard with td_get_build_compatibility.
- Mac FTDI path is absent → status tool must handle gracefully.
- Bumps tool count by 2 — update EXPECTED_MIN_TOOL_COUNT in release_gates.py and all user-facing copies (CLAUDE.md derived-artifacts item 3).

C.2 — `td_midi_devices` + MIDI hint pack

Why: Second most common hardware input. Currently agents guess device names and can't verify bindings without screenshots.
Files:
- src/td_mcp/registry/tools_system.py or tools_hardware.py
- src/td_mcp/hints/packs/op_types/midiinmapCHOP.yaml (NEW)
- src/td_mcp/macros/templates.py — add "midi_controller_bind" template
Tool to add: td_midi_devices() — executes app.midiDeviceManager.inputDevices and app.midiDeviceManager.outputDevices in full exec mode, returns device name + ID + active state.
Acceptance: With a USB MIDI controller plugged in, the tool lists the device. Agent maps modwheel → bloom intensity without screenshots.
Gotchas:
- app.midiDeviceManager requires full exec mode, not restricted. Guard with the exec-mode check used by td_python_env_status.
- Bumps tool count by 1.

C.3 — Reference: `references/ndi-and-streaming.md`

Why: NDI / Spout / Syphon are the dominant inter-app video routing standards. Zero coverage today.
File: skills/tdpilot-dpsk4-core/references/ndi-and-streaming.md (NEW)
Content outline:
- NDI In TOP (ndisourceTOP) source discovery via td_exec_python
- NDI Out TOP parameters
- Syphon In/Out (macOS-only; build-check pattern)
- Spout In/Out (Windows-only; build-check pattern)
- Cross-platform conditional wiring
- Multi-machine sync via timeline CHOP
- Verified-build header: Verified: TD 2025.32820
Acceptance: New reference appears in the SKILL.md Reference Files index.

C.4 — Reference: `references/midi-osc-control.md`

Why: Covers MIDI/OSC live input and mapping — the second most-asked TD workflow.
File: skills/tdpilot-dpsk4-core/references/midi-osc-control.md (NEW)
Content outline:
- MIDI In CHOP, MIDI Out CHOP, MIDI Map CHOP
- OSC In DAT, OSC Out DAT, OSC In CHOP, OSC Out CHOP
- Auto-learn patterns (cross-link to preset-systems-and-ui.md §11)
- Bidirectional feedback (motorized faders, LED rings)
- Common controllers: Ableton Push, Novation Launchpad, generic HID
- Verified-build header
Acceptance: Linked from the SKILL.md Reference Files index.

C.5 — Reference: `references/depth-and-body-tracking.md`

Why: Depth-camera workflows (Kinect, Azure Kinect, ZED, RealSense, MediaPipe) are common installation-art primitives.
File: skills/tdpilot-dpsk4-core/references/depth-and-body-tracking.md (NEW)
Content outline:
- Kinect TOP / Kinect CHOP (legacy Kinect v1/v2)
- Azure Kinect TOP
- ZED pipeline via central ZED TOP (TD 2025+ restructure)
- RealSense TOP
- MediaPipe via Script TOP + Python
- Body tracking → particle systems (POP-side)
- Verified-build header
Acceptance: Linked from SKILL.md.

C.6 — Capability index + featured prompt chips

Why: Discoverability — users don't know what TDPilot can do.
Files:
- src/td_mcp/registry/tools_introspect.py — add td_get_capabilities_summary
- td_component/tdpilot_api_chat.html — add featured prompt chips below the input field
Steps:
1. New tool td_get_capabilities_summary() returns a grouped human-readable index: "Things I can build / Things I can diagnose / Things I can remember / Hardware I can talk to / etc." Each group has 2–3 example prompts.
2. In the chat HTML, fetch the summary on first load and render 5–6 featured prompt chips below the input field. Chips vanish after the first user message.
Acceptance: Fresh chat session shows chips. Clicking a chip pre-fills the input field. After first send, chips disappear.

C.7 — Cost tracking: per-turn and per-session

Why: EV_USAGE tracks per-call but doesn't accumulate. Under cost-no-issue this is less critical, but it's still useful telemetry — and shipping cost visibility makes the open-core / paid-tier story credible for the commercial pivot.
Files:
- td_component/tdpilot_api_runtime.py::AgentRuntime — add session accumulators
- td_component/tdpilot_api_tracing.py — extend Tracer per-turn record with token totals
Steps:
1. In AgentRuntime, accumulate _session_input_tokens, _session_output_tokens, _session_cache_hits, _session_cache_misses from every EV_USAGE.
2. Emit EV_USAGE_SESSION with session_total keys after each turn.
3. Add a USD estimate using DeepSeek's published rates (label as "approximate," versioned with publish date).
4. Expose via a /stats HTTP endpoint (new route).
5. Surface in the chat UI footer.
Acceptance: After a 5-turn session, /stats returns totals consistent with the sum of individual EV_USAGE events.

C.8 — `CookThreadDispatcher` circuit breaker on TD pause

Why: When TD is paused mid-turn, every queued tool call times out at 60s. A 10-tool turn = 600s of hangs.
Files: td_component/tdpilot_api_runtime.py::CookThreadDispatcher
Steps:
1. On dispatcher timeout, call cancel_pending() (currently only called from stop()).
2. In pump(), after a 5-second grace period of waiting, check _is_td_paused(); if paused, drain pending queue with {"error": "TD paused — tool dispatch suspended"} errors immediately.
3. Push EV_HINT with a paused-TD diagnosis when the breaker trips.
Acceptance: Pause TD mid-turn. The current tool call times out at 60s (existing behavior); subsequent queued tools error immediately with the helpful message rather than each hanging 60s.
Gotchas: Grace period prevents false positives on legitimate long-running ops (e.g., td_screenshot on a heavy scene).

C.9 — Configurable `thinking.budget_tokens`

Why under cost-no-issue: uncap thinking depth. Default to 8000+ (was effectively unbounded but uncontrolled; now we set a high explicit ceiling).
Files:
- td_component/tdpilot_api_agent.py — include "thinking": {"type": "enabled", "budget_tokens": N} in request body
- td_component/tdpilot_api_runtime.py::AgentRuntime.__init__ — wire from COMP param Thinkingbudget
- Build script: add Thinkingbudget COMP param (default 8000)
Acceptance: COMP param visible in TD. Setting it to 16000 produces deeper reasoning on complex multi-tool turns (verifiable via EV_USAGE thinking-token counts).
Gotchas: DeepSeek's compat layer support for the thinking block is undocumented — test with a real call before committing the default.

Phase C bundle: `.tox` rebuild + tool-count bump + verification

Items C.1, C.2 bump tool count by 3 → update EXPECTED_MIN_TOOL_COUNT and all user-facing copies (the CLAUDE.md derived-artifacts checklist lists every file).
Rebuild .tox after C.1, C.2, C.6, C.7, C.8, C.9 land.
Run python scripts/check_versions.py and python scripts/check_tox_freshness.py.

Phase C definition of done: all 9 implementation items merged + 3 references shipped + .tox rebuilt + tool count bumped to ~107 in all 11+ user-facing copies + manual smoke test passes.

Release as v2.4.1 (additive — no breaking changes).

5. Phase D — Streaming + LLM-Assisted Compaction

Goal: The two biggest user-visible quality improvements. SSE streaming changes perceived latency by an order of magnitude. LLM-assisted compaction preserves long-session coherence.

Subagent dispatch suggestion:

Pre-work agent: dispatch ONE sonnet subagent to do a deep architectural read of the API-call function, the entire message-history-assembly flow, and the WS event broadcast pipeline → produce a sequencing diagram for SSE streaming that respects thinking-block echo requirements (memory: feedback_deepseek_thinking_blocks_must_echo.md).
Compaction-research agent: dispatch ONE sonnet subagent to deep-read tdpilot_api_compaction.py::_summarise_old_turns and propose the flash-tier LLM-call architecture, including auth sharing, async dispatch, fallback behavior.
Main agent integrates both specs.

D.1 — SSE streaming for LLM responses

Why: Currently the API call uses urllib.request.urlopen (blocking) — user sees nothing for 10–120s on complex turns. Module docstring explicitly admits "Streaming is NOT implemented in this revision." Chat UI already has token-event rendering (WS EV_TEXT fan-out) — UI side is ready.
File: td_component/tdpilot_api_agent.py
Approach:
1. Replace urllib.request.urlopen with http.client.HTTPSConnection for streaming.
2. Use chunked transfer / SSE parsing — each data: line is one event.
3. For each text-block chunk, call on_text(token) to fan out via WS.
4. Critical: thinking blocks MUST be assembled in full before the response closes — DeepSeek requires the COMPLETE thinking block to be echoed back on the next turn (per memory: feedback_deepseek_thinking_blocks_must_echo.md). Don't flush partial thinking blocks to self.messages.
5. On stop_reason event, finalize the assembled message and append to history.
6. Preserve the _strip_reasoning contract — strip reasoning_content sub-keys but KEEP the full type: thinking blocks.
Acceptance:
- First token visible in UI within ~1s of API call.
- End-of-turn message history matches the non-streaming flow byte-for-byte (run a comparison test).
- Thinking-block echo on subsequent turn does not produce HTTP 400.
Gotchas:
- DeepSeek's Anthropic-compat SSE format may diverge from non-streaming — test with real endpoint, not just mocks.
- request_timeout=120 no longer means "blocking timeout" — interpret as "total wallclock budget"; reads inside the stream should respect a per-chunk timeout (~30s) to detect stalls.
- Test extensively with mock_deepseek fixtures (PR-20 infrastructure at tests/_mock_deepseek.py) AND with real DeepSeek API.

D.2 — LLM-assisted compaction (replace heuristic)

Why under cost-no-issue: current compaction summary is a pure local heuristic (_summarise_old_turns) — loses node paths, parameter values, error messages. After compaction, model often asks "what have we done?" wasting a full round trip. With cost-no-issue, replacing the heuristic with a flash-tier LLM call is clearly correct.
Files: td_component/tdpilot_api_compaction.py
Approach:
1. Add a _summarise_via_llm(messages_to_compact) method using a flash-tier model (DeepSeek's cheaper variant if available, else the same tier).
2. Structured prompt asks for: nodes created/deleted, params changed, errors encountered, current project state, why decisions were made.
3. The compaction call shares the Agent's SSL context + API key.
4. Async dispatch — don't block the cook thread.
5. On failure (rate limit, timeout), fall back to the existing local heuristic.
6. Raise the compaction threshold from 20 to 40 messages (we can preserve more context under cost-no-issue).
Acceptance: After compaction in a 50-message session, model retains awareness of node paths created earlier and doesn't ask "what have we done?"
Gotchas:
- Failed flash call must not halt the turn — fall back gracefully.
- Flash call uses its own circuit breaker (don't retry inside compaction; one attempt is enough).

Phase D bundle: `.tox` rebuild + verification

Both items touch _TOX_SOURCE_FILES → rebuild required.

Phase D definition of done: streaming visible in UI + compaction preserves coherence in a 50-message session smoke test.

Release as v2.5.0 (architectural — major version bump).

6. Phase E — Polish, Content, Commercial

Goal: Long-tail improvements that compound. Roll in opportunistically; no hard deadline.

These are mostly content/docs, can run in parallel by dispatching subagents.

E.1 — Demo video / GIF / community link

Why: Comparable tools have demo videos. TDPilot's README has a static screenshot. For the open-core commercial pivot, social proof matters.
What:
- Record a 2-minute screen capture of one complete turn (create a node, screenshot, result).
- Convert to a GIF for the README hero section.
- Add a community link (Discord or GitHub Discussions).
- Add a ## Demo section near the top of README.
Files: README.md, docs/images/ (new GIF asset).
Acceptance: GIF visible in README. Community link clicks through.

E.2 — Free-vs-Pro section in README

Why: Commercial direction (open-core + api.tdpilot.io) is in memory only. Set expectations BEFORE the SaaS pivot ships.
File: README.md
Content (aspirational, not contractual):

TDPilot Free & Pro TDPilot core is and will remain open-source (MIT). A hosted Pro tier with [server-resident skill hosting, cloud corpus sync, technique marketplace] is in development at api.tdpilot.io. Join the waitlist [link].
Acceptance: Section visible. Waitlist link works (or marked TBD).

E.3 — `references/3d-asset-import.md` + FBX import macro

File: skills/tdpilot-dpsk4-core/references/3d-asset-import.md (NEW)
Plus macro: "fbx_import" in src/td_mcp/macros/templates.py
Content: FBX/OBJ/Alembic/Point cloud import workflows, the null-SOP terminator pattern, FBX sub-hierarchy issue, display/render flag requirements.

E.4 — Extension COMP scaffolding macro

File: src/td_mcp/macros/templates.py — add "extension_comp" template
What: creates baseCOMP + extensionDAT (with skeleton class body using TDStoreTools) + Common>Extensions>Object binding + optional Version/Help/Creator custom param page.

E.5 — Python hot-reload reference

File: add a section "External Python Modules and Hot-Reload" to skills/tdpilot-dpsk4-core/references/preset-systems-and-ui.md (natural home for Python patterns)
Content: mod.<DATname> access, importlib.reload(mod.mymodule), app.reloadCustomOperators(), tdu.Dependency, file-watch + chopexec pattern.

E.6 — Document standalone-only trigger-skills in plugin README

File: plugin_README.md (and a note in README.md CLI section)
Content: explicit list of standalone-only auto-loading skills (performance-mode.md, popx-mode.md). Explain the gap. Optionally port as /td-performance and /td-popx slash commands (M effort, separate item).

E.7 — Verified-build header convention

What: add a > Verified: TD 2025.32820 header to every reference file so the agent knows which build the recipe was tested on.
Files: all skills/tdpilot-dpsk4-core/references/*.md + the new C.3, C.4, C.5 references.
Acceptance: every reference has the header. Update protocol documented in AGENTS.md.

7. Execution Strategy

When to dispatch subagents

Any phase with 3+ independent research/scoping needs → dispatch in parallel (single message, multiple Agent tool calls).
Any item with significant unknowns (DMX API surface, MIDI API surface, SSE format) → dispatch a research subagent FIRST, then implement based on the spec.
Content authoring (references, examples) → dispatch one subagent per file; main agent reviews and merges.

When to use `td_tool_batch`

Any inspection sequence (e.g., pre-build scope lock: td_get_nodes + td_get_errors + td_audit_project) — always batch.
Any verification sequence (e.g., post-edit: td_get_errors + td_screenshot + td_cooking_info) — always batch.

When NOT to parallelize

Task B depends on task A's output → sequence.
Shared mutable state (both agents would edit the same file) → sequence.
Single trivial operation → just do it.

Verification per item

Every item has explicit acceptance criteria. Do not mark complete without verifying. Per memory feedback_ci_check.md: after git push, immediately check gh pr checks or gh run list.

`.tox` rebuild protocol

Per memory feedback_td_tox_rebuild_recipe.md, use the canonical one-line Textport recipe (NOT the build-script docstring example — that's broken on TD macOS). The rebuild auto-mirrors to the main repo's td_component/ since v2.2.1.

Version bump protocol

For each phase that ships as a release:

Bump 7 version files (per CLAUDE.md derived-artifacts item 5).
python scripts/check_versions.py — must pass.
python scripts/check_tox_freshness.py — must pass.
CHANGELOG.md — add an entry under the new version.
Open PR via gh pr create.
After merge, push tag: git tag vX.Y.Z && git push origin vX.Y.Z.
Critical: run gh release create vX.Y.Z to create the GitHub Release (per memory feedback_create_github_release_after_tag.md). The release-assets.yml workflow only fires on Release-creation events, not on tag-push.
Confirm .mcpb and .plugin assets land on the Release page.
npm publish via Trusted Publisher OIDC (do NOT add NPM_TOKEN, per memory feedback_npm_trusted_publisher_oidc.md).

8. Risks & Gotchas

Risk	Mitigation
`_TOX_SOURCE_FILES` edits without rebuild → CI `check_tox_freshness` fails	Bundle all `_TOX_SOURCE_FILES` edits per phase into a single rebuild. Do NOT push without rebuilding.
Adding `# noqa:` comments to tox-baked files → CI fails on staleness	Never add comment-only edits to `_TOX_SOURCE_FILES` files unless you're ready to rebuild (memory: `feedback_noqa_on_tox_source_breaks_ci.md`).
DeepSeek thinking blocks stripped from history → next API call 400s	KEEP `type: thinking` blocks in messages; strip only `reasoning_content` sub-keys (memory: `feedback_deepseek_thinking_blocks_must_echo.md`). Especially critical for D.1 (SSE streaming) and B.1 (vision pipeline).
Tag push without GitHub Release creation → Update button greys out	After every `git push origin vX.Y.Z`, run `gh release create vX.Y.Z` (memory: `feedback_create_github_release_after_tag.md`).
npm publish 404 → reflexively adding NPM_TOKEN breaks OIDC	Retry once first; OIDC has transient hiccups. Only audit binding if retry also 404s. Don't add NPM_TOKEN (memory: `feedback_npm_trusted_publisher_oidc.md`).
TD Textport runs only single-line statements	Use the canonical one-line recipe from memory file `feedback_td_textport_single_line_exec.md` (compile + builtins-resolution wrapped in a single expression) for any multi-line operation.
TD MCP restricted mode blocks `isinstance`/`hasattr`/`Exception`/`type`/`dir`	If a tool fails in restricted mode but works in full mode, that's the cause — switch the relevant tool to full exec mode with the proper guard (memory: `feedback_td_mcp_restricted_mode_overzealous.md`).
tdpilot_API `.tox` rebuild only refreshes `/project1/tdpilot_API` or `/local/tdpilot_API`	COMPs at arbitrary user paths stay on OLD baked code until manually drag-in (memory: `feedback_tdpilot_api_install_parent_quirk.md`). Document in release notes.
Tool count drift across 11+ files	After C.1 (DMX) and C.2 (MIDI), update `EXPECTED_MIN_TOOL_COUNT` and run a sweep: `grep -rn "104 tools\|103 tools" .`

9. Deferred to Later (Not in v2.4 / v2.5)

Projection mapping reference + td_window_inspect — explicitly deferred by user. Save for v2.6 or when there's a concrete user request.
Web/voice/mobile chat surfaces — out of scope (per memory: cut from v2.2.0 scope).
Cloud-deps + local-model support — out of scope.

10. Quick Reference — File Map

Topic	Primary files
MCP server entry point	`src/td_mcp/__init__.py`, `src/td_mcp/server.py`
Tool registry	`src/td_mcp/registry/tools_*.py`
Tool count gate	`src/td_mcp/release_gates.py`
Hint packs	`src/td_mcp/hints/packs/`
Macros	`src/td_mcp/macros/templates.py`
TD-side chat pipe (.tox-baked)	`td_component/{mcp_webserver_callbacks, event_emitter, ws_callbacks, tdpilot_startup}.py`
TD-side agent runtime	`td_component/tdpilot_api_agent.py`, `tdpilot_api_runtime.py`
TD-side compaction	`td_component/tdpilot_api_compaction.py`
TD-side memory/recipes	`td_component/tdpilot_api_memory.py`, `tdpilot_api_recipes.py`
TD-side web routes	`td_component/tdpilot_api_web_callbacks.py`, `tdpilot_api_introspect.py`
Chat UI	`td_component/tdpilot_api_chat.html`
Skills	`skills/tdpilot-dpsk4-{core,production}/SKILL.md`, `skills/popx-touchdesigner/SKILL.md`
References	`skills/tdpilot-dpsk4-core/references/*.md`
Slash commands	`commands/*.md`
Build scripts	`scripts/build_.py`, `scripts/check_.py`
Tests	`tests/`, with mock-deepseek at `tests/_mock_deepseek.py`
Docs	`docs/` (this file lives here)
Release config	`.claude-plugin/{plugin,marketplace}.json`, `npm/package.json`, `mcp/manifest.json`, `pyproject.toml`, `td_component/mcp_webserver_callbacks.py` (API_VERSION)

11. Day 1 Cold-Start Checklist

When you open the new session and load this plan, execute in order:

cd <REPO_ROOT>/.claude/worktrees/quizzical-heisenberg-3cd974 (or switch to a fresh worktree from main).
git status → confirm working tree state. If the 5 prior-session files are still uncommitted, commit them first as docs(skills): expand core reference library with STOP rules and 4 reference files.
git log --oneline -5 → confirm you're starting from 8f087cd or later on main.
Read this file end-to-end. Confirm understanding before executing.
Open Phase A. Dispatch any subagents needed (optional for Phase A; mandatory for Phase B).
Mark each item as complete only after acceptance criteria pass.
After Phase A bundle, rebuild .tox inside TouchDesigner using the canonical Textport recipe.
Run python scripts/check_versions.py + python scripts/check_tox_freshness.py + uv run ruff check.
Open PR via gh pr create. After merge, decide whether to ship as v2.3.2 (just A) or continue to Phase B before releasing.
Repeat for Phases B → C → D → E.

12. Definition of Done for v2.4

v2.4 is shippable when:

Phases A + B + C complete.
Tool count = 107 (104 + DMX inspect + DMX status + MIDI devices).
All 7 version files = 2.4.0.
.tox rebuilt + auto-mirrored to main repo.
CHANGELOG entry under v2.4.0 documenting:
- 5 quick wins (Phase A)
- Vision pipeline + Authmode wizard + worked examples + BM25 fix + slash commands (Phase B)
- DMX + MIDI tools + 3 references + capability index + cost tracking + circuit breaker + thinking budget (Phase C)
GitHub Release created with .mcpb and .plugin assets.
npm publish via OIDC succeeded.
README updated to reflect new tool count and feature surface.
Smoke test in a live TD session: create a node, take a screenshot, agent describes what it sees (B.1 vision), agent uses tool_batch for the inspection (A.3), agent recovers from a synthetic 429 (A.5).

v2.5 ships separately after Phase D (SSE streaming + LLM compaction) — major version bump warranted by the architectural change.

Phase E rolls in opportunistically as patch releases (v2.4.x, v2.5.x).

FilesExpand file tree

v2.4_IMPLEMENTATION_PLAN.md

Latest commit

History