TDPilot DPSK4 v2.6 — Implementation Plan (Cold-Start Executable)

Theme: Retrieval + knowledge. Structural fix for Bug 8 + skill packs + web ingestion.

Authored: 2026-05-18 (alongside v2.5 + v2.7 plans).

Status: not_started. Update this front-matter as phases complete.

Prerequisite: v2.5 SHIPPED. Activity log + journal hints land first so we can measure retrieval-quality impact properly.

For the receiving agent: This is the cold-start executable plan for v2.6 of TDPilot DPSK4. Same conventions as v2.5_IMPLEMENTATION_PLAN.md. Drop into a fresh session, say "execute v2.6", and follow phases sequentially.

0. Bootstrap context

Pre-conditions

v2.5.0 must be shipped before starting v2.6 work. Verify:

git tag --list "v2.5.*"                  # at least v2.5.0 present
grep "EXPECTED_MIN_TOOL_COUNT" src/td_mcp/release_gates.py  # ≥ 110

Activity log + journal hints are live (phase v2.5.1 completed). v2.6 hybrid retrieval will use the same args_hash deep-canonical from B-010.

Why this theme

v2.4 Bug 8 (over-eager tool use from short prompts) was patched with a 16-char prompt floor + length-relative BM25 threshold. That's a workaround, not a fix. The root cause is BM25's vulnerability to instruction-shaped memory entries — semantic mismatch the BM25 ranker can't detect. Hybrid retrieval with vector-side semantic similarity is the structural fix.

Skill packs solve a separate problem: our SYSTEM_PROMPT_BASE in tdpilot_api_runtime.py is monolithic and ~2000 tokens. Splitting into per-task skill packs (POPx, shader, audio-reactive, debugging) lets the agent specialize per-turn without bloating context.

Web ingestion completes the trio: with hybrid retrieval working, we can ingest external docs (official TD pages, POPX examples, general tutorial sites) into the knowledge corpus on demand.

Critical derived artifacts

Same checklist as v2.5 — see v2.5_IMPLEMENTATION_PLAN.md.

1. Phase overview

Phase ID	Item	Effort	Tox rebuild	Status
`v2.6.1-hybrid-retrieval`	Embeddings sidecar + sqlite-vec store + BM25/vector/RRF fusion	1-1.5 weeks	API only	not_started
`v2.6.2-skill-packs`	Skill pack loader + system-prompt split + 6 built-in packs	1 week	API only	not_started
`v2.6.3-web-ingestion`	`td_ingest_url` + markitdown/crawl4ai sidecar	3 days	MCP only	not_started
`v2.6.4-bug8-relax`	Relax 16-char prompt floor now that vector handles semantic gap	1 day	API only	not_started

Total scope: 4 phases, ~3 weeks of work.

Sequencing constraint: 2.6.1 → 2.6.4 (relax depends on hybrid retrieval being live). 2.6.2 and 2.6.3 are independent of each other and can run in parallel agents.

New optional dependency groups (via pyproject extras):

[retrieval] — sentence-transformers, sqlite-vec
[web] — markitdown, optionally crawl4ai for JS-heavy pages

2. Phase v2.6.1 — Hybrid retrieval (BM25 + vector + RRF)

Status: not_started Branch suggestion: claude/v2.6.1-hybrid-retrieval Tox rebuild: API .tox only

Why

BM25 misses semantic matches and over-matches on instruction-shaped entries. Vector retrieval (sentence-transformers) captures semantic similarity but misses exact keyword matches. Reciprocal Rank Fusion (RRF) combines both robustly without score normalization. This is the structural fix for Bug 8.

Pre-flight checks

ls td_component/tdpilot_api_bm25.py                            # current BM25 path exists
grep -E "bm25_score" td_component/tdpilot_api_runtime.py       # find current call sites
mkdir -p ~/.tdpilot-dpsk4/api/models/                          # model cache dir

Files to create

td_component/tdpilot_api_embeddings.py — subprocess manager (same sidecar pattern as OCR in v2.5.2)
td_component/tdpilot_api_embed_worker.py — standalone worker: sentence_transformers.SentenceTransformer('all-MiniLM-L6-v2'), accept text on stdin (JSON), emit 384-dim float32 vectors on stdout
td_component/tdpilot_api_vector_store.py — sqlite-based store with sqlite-vec extension (cosine similarity)
td_component/tdpilot_api_retrieval.py — unified retrieve(query, k=10) -> list[Entry] API; runs BM25 + cosine in parallel, fuses with RRF
tests/test_v26_embeddings.py — worker lifecycle (~8 tests)
tests/test_v26_vector_store.py — CRUD + cosine + migration (~10 tests)
tests/test_v26_rrf.py — RRF correctness pinned against known examples (~7 tests)
tests/test_v26_hybrid_retrieval.py — end-to-end + Bug 8 regression cases (~10 tests)

Files to modify

td_component/tdpilot_api_runtime.py — replace direct bm25_score(...) calls with retrieval.retrieve(...); keep all existing thresholds initially (relax in v2.6.4)
td_component/tdpilot_api_bm25.py — keep as-is; now called from retrieval.py
td_component/build_tdpilot_api_tox.py::_API_TOX_SOURCE_FILES — add 4 new files
pyproject.toml — [project.optional-dependencies] retrieval = ["sentence-transformers>=2.5", "sqlite-vec>=0.1"]
CHANGELOG.md — v2.6.1 entry

Architecture

chat-pipe pre-turn retrieval:
  user prompt
    → embed via embed_worker subprocess (BLOCKING for short query; ~10ms after warmup)
    → BM25 score (existing path, parallel)
    → cosine similarity vs all entries (sqlite-vec, fast — <5ms for 1000 entries)
    → RRF fusion (rank-based, k=60)
    → top-K context entries (default K=10, configurable)
    → injected into system prompt as INFORMATIONAL CONTEXT (per Bug 8 prefix)

Vector store schema

CREATE TABLE embeddings (
  entry_id TEXT PRIMARY KEY,
  entry_kind TEXT NOT NULL,                  -- 'memory' | 'knowledge' | 'recipe'
  model_version TEXT NOT NULL,               -- 'all-MiniLM-L6-v2-v1'
  vector BLOB NOT NULL,                      -- 384-dim float32, packed
  created_at REAL NOT NULL,
  updated_at REAL NOT NULL
);
CREATE INDEX idx_kind ON embeddings(entry_kind);
-- sqlite-vec virtual table for cosine
CREATE VIRTUAL TABLE vec_embeddings USING vec0(
  entry_id TEXT PRIMARY KEY,
  vector FLOAT[384]
);

RRF formula

Standard reciprocal-rank-fusion:

RRF_score(d) = Σ_i 1 / (k + rank_i(d))   where k = 60

Sum over rankers (BM25 + cosine). No score normalization needed; the formula is rank-based. Top-K by RRF_score. Robust to scale differences.

Migration on first start

On first chat-pipe start with v2.6.1 installed:

Detect: vector store empty AND memory entries exist.
Stream progress events to chat UI: {"type": "status", "status": "indexing", "progress": "47/123"}.
Encode all entries in batches of 32. Persist as you go.
On success: chat ready. On failure: log + fall back to BM25-only mode for the session (degrades but doesn't break).

One-shot, persisted. Subsequent saves to memory/knowledge encode-on-write.

Model versioning

model_version field allows hot-swap. On mismatch detection at retrieval time, lazy re-encode that entry (don't block startup). Future model upgrade is non-breaking.

Model location + caching

~/.tdpilot-dpsk4/api/models/all-MiniLM-L6-v2/ — bundled-cache to avoid first-run HuggingFace download in air-gapped environments. Document HF_HOME override in README.

Tests (~35 total)

8 in test_v26_embeddings.py — worker spawn/warmup/idle-kill/restart-on-crash
10 in test_v26_vector_store.py — CRUD, cosine correctness, model-version migration, index integrity
7 in test_v26_rrf.py — known cases pinned (single-ranker passthrough, equal-rank averaging, k=60 default)
10 in test_v26_hybrid_retrieval.py — end-to-end on curated corpus + Bug 8 regression: "Reply: KICK1" against KICK1-procedure memory returns 0 results structurally (no length workaround needed)

Validation gates

pip install -e .[retrieval]
uv run pytest tests/test_v26_embeddings.py tests/test_v26_vector_store.py tests/test_v26_rrf.py tests/test_v26_hybrid_retrieval.py -v
uv run pytest tests/ --ignore=tests/agent_evals -x -q          # ~2070 pass
# Manual: trigger migration on a fresh ~/.tdpilot-dpsk4/api/, watch indexing progress
# Manual: send "Reply: KICK1" to chat-pipe, verify NO retrieval-driven td_create_node fires

Risks + mitigations

Risk: First-run indexing slow on machines with 1000+ memory entries (~30s). Mitigation: background indexing with progress events; BM25 fallback during indexing.
Risk: Sentence-transformers + torch is a ~500MB dependency. Mitigation: optional via extras; document size in README.
Risk: sqlite-vec doesn't install on some platforms. Mitigation: pure-Python cosine fallback (slower but works); detect at startup.
Risk: Model upgrade breaks vectors. Mitigation: model_version field + lazy re-encode (designed in).

Resume from here

After this PR merges, mark v2.6.1-hybrid-retrieval as completed. Next: v2.6.2 or v2.6.3 (independent — can run in parallel). v2.6.4 depends on this AND on observed real-world performance, so it's last.

3. Phase v2.6.2 — Skill pack loader

Status: not_started Branch suggestion: claude/v2.6.2-skill-packs Tox rebuild: API only

Why

SYSTEM_PROMPT_BASE in tdpilot_api_runtime.py is monolithic (~2000 tokens). It contains protocol points, intent gate, cycle-detect rules, output format, tool reference hints, AND domain knowledge (POPx specifics, shader debugging, audio-reactive patterns). Splitting domain knowledge into loadable skill packs:

Keeps the always-on prompt lean (better cache hits per DeepSeek prefix-cache discipline).
Lets the agent specialize per-task without bloating context.
Mirrors Claude Code's Skill paradigm — same mental model for users coming from that ecosystem.

Pre-flight checks

wc -l td_component/tdpilot_api_runtime.py                      # baseline
grep -c "SYSTEM_PROMPT_BASE" td_component/tdpilot_api_runtime.py
ls skills/ 2>/dev/null                                          # may have tdpilot-dpsk4-core

Files to create

src/td_mcp/skills/__init__.py
src/td_mcp/skills/loader.py — discovers skills from two locations:
1. Built-in: src/td_mcp/skills/builtin/<name>/SKILL.md
2. User-defined: ~/.tdpilot-dpsk4/api/skills/<name>/SKILL.md (overrides built-in if same name)
src/td_mcp/skills/builtin/popx/SKILL.md — POPx generators/falloffs/modifiers/simulations
src/td_mcp/skills/builtin/shader/SKILL.md — GLSL TOP/POP debugging idioms
src/td_mcp/skills/builtin/audio_reactive/SKILL.md — CHOP → audio analysis patterns
src/td_mcp/skills/builtin/debugging/SKILL.md — systematic-debugging workflow specific to TD
src/td_mcp/skills/builtin/feedback_loops/SKILL.md — the kaleidoscope class (v2.4 canonical test)
src/td_mcp/skills/builtin/python_extensions/SKILL.md — TD COMP extensions patterns
tests/test_v26_skills.py — ~20 tests

Files to modify

td_component/tdpilot_api_runtime.py — skill loader hook in build_system_prompt(...); skill content prepended to base prompt
td_component/tdpilot_api_runtime.py — split SYSTEM_PROMPT_BASE content: keep CORE (protocol points, intent gate, cycle-detect rules, output format) in base; migrate DOMAIN content into skill packs
td_component/build_tdpilot_api_tox.py::_API_TOX_SOURCE_FILES — verify tdpilot_api_runtime.py triggers rebuild
CHANGELOG.md

Skill frontmatter format

---
name: popx
description: TouchDesigner POP operators — generators, falloffs, modifiers, simulations
when_to_use: prompt mentions POPx, POP, particle, generator, falloff, simulation, GPU geometry
auto_load_keywords: [popx, pop generator, particle, falloff, simulation]
size_tokens: 850                              # author-declared; loader verifies
---

# POPx — TouchDesigner GPU Operators

## Mental model
Generator → Falloff → Modifier → Tool → Simulation
...

Tools

@mcp.tool()
async def td_load_skill(name: str) -> dict:
    """Load a skill pack for subsequent turns. Auto-unloads on task-done detection."""

@mcp.tool()
async def td_unload_skill(name: str) -> dict:
    """Explicit unload."""

@mcp.tool()
async def td_list_skills(active_only: bool = False) -> list:
    """List available skill packs; if active_only, only currently-loaded ones."""

Runtime

Loaded skills' SKILL.md content gets prepended to system prompt for subsequent turns
Skill is auto-loaded when prompt matches any auto_load_keywords (case-insensitive substring match)
Auto-unload on _TASK_DONE_RE match (reuse v2.4 sticky-pro pattern from B-008-T)
Hard limit: 2 active skills at once (avoid context bloat). Loading a 3rd evicts least-recently-used.

v2.6 vs v2.7 split (per audit decision)

v2.6: OPT-IN — SYSTEM_PROMPT_BASE unchanged. Skills ADD to it. Users can test skill mechanism without behavior shift.
v2.7: Full split — DOMAIN content removed from SYSTEM_PROMPT_BASE, moved to skills. Real behavior change.

This two-step rollout gives users one release of overlap to validate.

Tests (~20)

Load/unload roundtrip
Auto-load on keyword match
Auto-unload on task-done regex
2-skill limit enforcement (LRU eviction)
Frontmatter validation (missing fields, malformed YAML)
Missing-file fallback (skill name doesn't exist → log + continue)
User-skill overrides built-in
Skill content actually prepended to system prompt (integration)
Skill size budget respected (sum of loaded < 4000 tokens warning)
Tool registration discoverable via td_list_skills

Validation gates

uv run pytest tests/test_v26_skills.py -v
uv run pytest tests/ --ignore=tests/agent_evals -x -q
# Manual: send a "build POP particle system" prompt, verify popx skill auto-loads in chat UI

Risks + mitigations

Risk: Skill content bloats context past LLM limits. Mitigation: hard 2-skill limit + size budget warning + LRU eviction.
Risk: Skill collisions (two skills with conflicting guidance). Mitigation: loader logs warning when concurrent skills mention same operator family; future enhancement: skill-compatibility metadata.
Risk: Auto-load false-positives. Mitigation: opt-in keyword match; user can disable auto-load via Skillautoload=off COMP param.

Resume from here

Mark completed. Move to v2.6.3 (web ingestion) or v2.6.4 (Bug 8 relax) — both independent of skill packs.

4. Phase v2.6.3 — Web ingestion (`td_ingest_url`)

Status: not_started Branch suggestion: claude/v2.6.3-web-ingestion Tox rebuild: MCP only

Why

With hybrid retrieval live (v2.6.1), the knowledge corpus is queryable. The agent should be able to grow it on demand — ingest official TD docs, POPx examples, general reference material. Closes the corpus-completeness gap that motivates many "I don't know that operator" turn failures.

Pre-flight checks

grep "td_search_official_docs" src/td_mcp/registry/                # confirm corpus search exists
ls src/td_mcp/knowledge/                                            # corpus dir

Files to create

src/td_mcp/knowledge/web_ingestion.py — crawler manager + markdown extractor
src/td_mcp/knowledge/ingestion_worker.py — subprocess running markitdown (lightweight) or crawl4ai (JS-capable fallback)
src/td_mcp/registry/ingestion_tools.py — register td_ingest_url
tests/test_v26_web_ingest.py — 12 tests

Tool surface

@mcp.tool()
async def td_ingest_url(
    url: str,
    tags: list[str] = [],
    knowledge_kind: Literal["fact", "reference", "instruction"] = "reference",
    max_size_bytes: int = 5 * 1024 * 1024,
) -> dict:
    """Crawl URL, extract markdown, store in knowledge corpus with embeddings.
    Refuses file:// and local IPs. Strips script tags + external resource refs."""
    return {
        "entry_id": "kb_2026_05_18_abc",
        "title": "POP Trace - Derivative",
        "url": url,
        "size_chars": 12450,
        "tags": tags,
        "embedding_indexed": True
    }

Safety constraints

Refuse file:// schemes (path traversal vector)
Refuse local IPs — 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, IPv6 ULA fc00::/7 (SSRF prevention)
Refuse private LAN suffixes — .local, .lan, .internal
Max body 5MB (configurable via param)
Max crawl-time 30s
Strip <script>, <style>, <iframe> tags; rewrite <a href> to absolute URLs
User-Agent identifies as TDPilot so site operators can block if needed

Engines

Primary: markitdown (Microsoft, simple, no JS, fast, small dep)
Fallback: crawl4ai (JS-capable, larger dep, only loaded when markitdown returns empty content)

Persistence

Ingested URLs become knowledge corpus entries:

~/.tdpilot-dpsk4/api/knowledge/projects/<project>/entries/<id>.md
content_type: "reference" by default (added in v2.4 Phase B)
Auto-embedded by v2.6.1 hybrid retrieval pipeline on save

Tests (12)

URL refusal: file://, 127.x.x.x, .local (3 tests)
Size cap honored (1 test)
Time cap honored (1 test)
markitdown extraction roundtrip on fixture HTML (2 tests)
crawl4ai fallback when markitdown returns empty (1 test)
Knowledge-store integration (1 test)
Embedding indexed automatically on save (1 test)
Tag preservation (1 test)
Tool registered (1 test)

Validation gates

pip install -e .[web]
uv run pytest tests/test_v26_web_ingest.py -v
# Manual: td_ingest_url("https://docs.derivative.ca/Trace_POP"), verify entry exists + retrievable

Risks + mitigations

Risk: SSRF if safety checks have a hole. Mitigation: test refusal paths exhaustively; pin in CI.
Risk: Ingested malicious content poisons corpus. Mitigation: content_type marker + INFORMATIONAL CONTEXT ONLY prefix (per Bug 8 fix) survives.
Risk: crawl4ai dependency heavy. Mitigation: lazy-load only when markitdown empty.

Resume from here

Mark completed. Move to v2.6.4.

5. Phase v2.6.4 — Bug 8 workaround relax

Status: not_started Branch suggestion: claude/v2.6.4-bug8-relax Tox rebuild: API only Depends on: v2.6.1 SHIPPED AND verified stable in production for ≥ 1 week.

Why

v2.4 Bug 8 fix has a 16-char prompt floor + length-relative BM25 threshold (<40 chars requires score ≥0.5). With hybrid retrieval live (v2.6.1), the vector side handles semantic mismatch structurally. The length workarounds become unnecessary AND cost real recall on legitimate short questions ("Define CHOP family.").

Pre-flight checks

git tag --list "v2.6.1*"                                        # v2.6.1 shipped
# Manually verify in production for ≥ 1 week before starting this phase.
grep "16-char" td_component/tdpilot_api_runtime.py             # find current workaround
grep "0.5" td_component/tdpilot_api_runtime.py | grep -i score # find score threshold

Files to modify

td_component/tdpilot_api_runtime.py — remove 16-char prompt floor; relax length-relative score threshold to a constant low threshold (e.g., RRF score ≥ 0.015)
tests/test_tdpilot_api_runtime.py::test_pre_turn_retrieval_handler_failure_is_isolated — update to assert the new behavior (per v2.3.0 CHANGELOG note about this test)
tests/test_v26_bug8_regression.py (new) — pin the "Reply: KICK1" case still returns 0 results structurally via vector-side semantic mismatch

Migration

Keep workaround behind a feature flag for 1 release. Default Retrievalmode=hybrid (new behavior); Retrievalmode=legacy falls back to v2.4 workaround. Document in README. Remove flag in v2.7.

Tests (8)

5 in test_v26_bug8_regression.py covering the original Bug 8 failure modes (each one should now return 0 results via hybrid retrieval without length workarounds)
3 covering legitimate short questions return correct retrievals ("CHOP family", "POP generator", "GLSL TOP")

Validation gates

uv run pytest tests/test_v26_bug8_regression.py -v
uv run pytest tests/ --ignore=tests/agent_evals -x -q
# Manual: send "Reply: KICK1" — verify no td_create_node fires
# Manual: send "Define CHOP family." — verify retrieval returns matching knowledge entry

Risks + mitigations

Risk: Regression — the Bug 8 case fires again. Mitigation: dedicated regression test suite; rollback to v2.4 workaround behind Retrievalmode=legacy flag for 1 release.

Resume from here

Mark completed. v2.6 done — proceed to release engineering or v2.7.

6. Release engineering — v2.6.0 ship checklist

Version bumps

Same 7 manifests as v2.5 (see v2.5 §10). All to 2.6.0.

Tool count

v2.6 adds: td_load_skill, td_unload_skill, td_list_skills, td_ingest_url = +4 MCP tools. Bump EXPECTED_MIN_TOOL_COUNT: int = 114 (assuming v2.5 hit 110).

`.tox` rebuild discipline

Both .tox files need rebuild (chat-pipe gets hybrid retrieval + skill loader; MCP gets web ingestion + check-updates).

Release narrative

Lead with "Structural fix for over-eager tool use (Bug 8) via hybrid retrieval" — concrete user-visible improvement. Then skill packs (cleaner prompts, faster cache hits). Then web ingestion (growable corpus).

Migration messaging

First chat-pipe start with v2.6 will index ~50-200 existing entries (~30s). Document in release notes so it's not surprising. Banner in chat UI during indexing.

7. Risk register — v2.6 cross-cutting

Risk	Phase	Probability	Impact	Mitigation
First-run indexing too slow	2.6.1	Med	Med	Background indexing + progress events + BM25 fallback
Sentence-transformers dep size scares users	2.6.1	Med	Med	Optional via `[retrieval]` extras
Bug 8 regression after relax	2.6.4	Low	High	Dedicated regression suite + `Retrievalmode=legacy` flag for 1 release
Skill packs bloat context	2.6.2	Med	Med	Hard 2-skill limit + LRU + size budget warning
Web ingestion SSRF	2.6.3	Low	High	Refuse file:// + local IPs + .local; test exhaustively
crawl4ai install fails	2.6.3	Med	Low	Lazy fallback; markitdown handles 80% of cases alone
Model upgrade breaks vectors	2.6.1	Low	High	`model_version` field + lazy re-encode (designed in)

8. Resume instructions per phase

Same convention as v2.5 §12. Edit the §1 overview table status field after each phase merges.

Decision points

Before starting v2.6.2: decide opt-in vs full-split. Plan defaults to opt-in for v2.6; full split deferred to v2.7. Reconfirm.
Before starting v2.6.4: verify v2.6.1 has been in production ≥1 week with no retrieval-regression reports. If not, defer.

End of v2.6 plan. Next: v2.7_IMPLEMENTATION_PLAN.md (orchestration + distribution maturity).

FilesExpand file tree

v2.6_IMPLEMENTATION_PLAN.md

Latest commit

History

v2.6_IMPLEMENTATION_PLAN.md

File metadata and controls

TDPilot DPSK4 v2.6 — Implementation Plan (Cold-Start Executable)

0. Bootstrap context

Pre-conditions

Why this theme

Critical derived artifacts

1. Phase overview

2. Phase v2.6.1 — Hybrid retrieval (BM25 + vector + RRF)

Why

Pre-flight checks

Files to create

Files to modify

Architecture

Vector store schema

RRF formula

Migration on first start

Model versioning

Model location + caching

Tests (~35 total)

Validation gates

Risks + mitigations

Resume from here

3. Phase v2.6.2 — Skill pack loader

Why

Pre-flight checks

Files to create

Files to modify

Skill frontmatter format

Tools

Runtime

v2.6 vs v2.7 split (per audit decision)

Tests (~20)

Validation gates

Risks + mitigations

Resume from here

4. Phase v2.6.3 — Web ingestion (td_ingest_url)

Why

Pre-flight checks

Files to create

Tool surface

Safety constraints

Engines

Persistence

Tests (12)

Validation gates

Risks + mitigations

Resume from here

5. Phase v2.6.4 — Bug 8 workaround relax

Why

Pre-flight checks

Files to modify

Migration

Tests (8)

Validation gates

Risks + mitigations

Resume from here

6. Release engineering — v2.6.0 ship checklist

Version bumps

Tool count

.tox rebuild discipline

Release narrative

Migration messaging

7. Risk register — v2.6 cross-cutting

8. Resume instructions per phase

Decision points

4. Phase v2.6.3 — Web ingestion (`td_ingest_url`)

`.tox` rebuild discipline