feat(skill-optimizer): SkillOpt-flavored offline training loop (#64)

* feat(skill-optimizer): SkillOpt-flavored offline training loop

Adds a new skill, slash command, and Node trainer that treats accumulated learn-rule rows as training trajectories and runs a budget-capped offline optimization over an existing SKILL.md.

Mirrors Microsoft SkillOpt's ReflACT pipeline adapted to pro-workflow's SQLite plane: rollout pulls recent learnings, reflect calls an optimizer LLM for bounded add/delete/replace patches, aggregate vote-merges across minibatches, clip enforces an LR budget, apply patches deterministically, evaluator LLM scores the candidate against a frozen held-out validation set built from the same trajectories, and the gate accepts only when delta >= accept threshold.

Rejected candidates feed a buffer that biases the next reflect step; epoch boundaries trigger a slow-update consolidation pass that must itself pass the gate. Kill switch at ~/.pro-workflow/STOP halts between steps. Provider dispatch supports anthropic / openai / openrouter / fireworks, matching the existing llm-council pattern.

Schema: 5 new tables (optimization_runs, optimization_candidates, optimization_patches, optimization_validation, optimization_rejections). CLI: scripts/optimize-skill.js with --json mode. Skill: skills/skill-optimizer/SKILL.md. Command: /skill-optimize.

Pure layer (apply / aggregate / clip / hash) covered by 9 node:test cases. LLM-driven layer (reflect / validate / slow / trainer) wires the dispatch but is tested live against a real provider key on invocation.

Version bump 3.3.0 -> 3.4.0 for the new top-level skill + command + schema surface.

* fix(skill-optimizer): address PR #64 review findings

13 review findings, all valid, all fixed:

1. CLI infers provider from model id (claude-* -> anthropic, gpt-*/o* -> openai). Mixed-provider example in commands/skill-optimize.md no longer requires --evaluator-provider for gpt-4o-mini.

2. NaN guard on numeric flag parsing: --epochs notanumber now errors fast instead of producing NaN in config.

3. SKILL.md drops the 'ReflACT' label since it is not a SkillOpt term; loop is referred to by stage names only.

4. optimization_validation gains prompt_hash + unique(skill_slug, prompt_hash); upsertValidation now uses ON CONFLICT DO UPDATE and persists learning_id from trajectoriesToValidation. Eliminates duplicate validation rows on repeated train() runs.

5. aggregate.patchKey now includes payload, so two distinct add proposals at the same anchor are preserved. Conflict counter moved to a separate anchor-level set so 'same anchor, different payloads' is still surfaced as a conflict.

6. apply.ts replace branch uses skill.replace(anchor, () => payload) function form. Payloads containing $1 / $& / $` are no longer mangled as backreferences.

7. llm.ts fails closed on unknown models: PRICE_PER_M_TOKENS lookup throws with the known-model list instead of silently returning {0,0} and bypassing budget cap.

8. llm.ts postJson gains an abortable timeout (default 120s, override via SKILL_OPTIMIZER_TIMEOUT_MS). Timer cleared on response end and on error to avoid leaks.

9. trainer.ts adds a preflight guardBudget() check before every paid LLM call (baseline eval, reflect, evaluate, slow update, slow update eval). Loop short-circuits when remaining budget cannot cover the call.

10. trainer.ts flushes spent / accepted / rejected / epochs / bestSkillHash / bestScore via flushAndEnd() before every endRun() call (completed / failed / stopped). Counters now persist even on early-exit branches.

11. trainer.ts slow update now passes recentlyAcceptedDiffs (last 8) and uses delta >= acceptThreshold against the gate, identical to the per-step gate. Previously consolidation saw an empty diff list and used raw >= bestScore.

12. trainer.ts writeBestSkill strips any prior <!-- skill-optimizer: ... --> stamp before writing the new one, so re-runs do not accumulate stamps. stripExistingStamp exported for test coverage.

13. Shared parse.ts (stripFencesAndParse) extracted from reflect.ts / validate.ts / slow.ts. All three modules now share identical fence-strip + try/catch JSON parse behavior.

Tests: 13/13 pass (4 new cases for the new behavior). Build + typecheck clean.

* fix(skill-optimizer): self-review pass on PR #64

Five issues caught reading the prior commit cold.

1. trainer.ts writeBestSkill: skip the write entirely when bestHash === initialHash. Previously every run rewrote SKILL.md with a fresh stamp even when no candidate beat the baseline, polluting the file with optimizer metadata on no-op runs.

2. trainer.ts makeMinibatches: replaced biased Array.sort(() => Math.random() - 0.5) with Fisher-Yates shuffle. The sort-based shuffle is non-uniform because the comparator violates transitivity; Fisher-Yates is the standard fix.

3. trainer.ts step loop: moved the budget guard above store.saveCandidate / store.recordPatches. Previously a budget exhaustion right before the evaluator left orphaned 'pending' candidates and their patches persisted in the DB with no eval to follow.

4. trainer.ts validation guard: throws early if the validation set has fewer than 2 items (e.g. --holdout 0 or pathological trajectories). Empty validation made every candidate score 0, delta 0 >= 0 threshold, trivially accepting garbage edits.

5. trainer.ts STAMP_RE: dropped the trailing $ anchor so a global replace strips ALL accumulated <!-- skill-optimizer: ... --> stamps, not just the last one. Pre-existing risk only on files that already had multiple stamps from older runs.

Tests: 14/14 pass (1 new case for multi-stamp strip).

Author: rohitg00

commands/skill-optimize.md

@@ -0,0 +1,39 @@
+---
+description: Train a skill's SKILL.md by running a SkillOpt-flavored offline loop over accumulated learn-rule corrections
+---
+
+# /skill-optimize - SkillOpt-flavored offline training
+
+Run an offline, budget-capped optimization loop over a skill's accumulated `learn-rule` trajectories. Proposes bounded patches via an optimizer LLM, validates each candidate against a held-out portion of the same trajectories, and overwrites SKILL.md only when the candidate strictly improves the weighted score.
+
+## Quick Start
+
+```text
+/skill-optimize <slug> [--epochs 3] [--budget-usd 0.50]
+```
+
+## What it does
+
+1. Pulls recent `learnings` rows scoped to the skill slug (or global)
+2. Splits them into train + validation (~25% holdout, freezes validation set)
+3. Runs `epochs` x `minibatches` rounds of: reflect → aggregate → clip → apply → evaluate → gate
+4. Stops on: budget exhausted, kill switch (`~/.pro-workflow/STOP`), no improvement, or epochs done
+5. If any candidate beat the baseline, overwrites SKILL.md and stamps the new hash
+
+## Requirements
+
+- 8 or more existing `learnings` rows for the slug
+- `ANTHROPIC_API_KEY` (or `OPENAI_API_KEY` / `OPENROUTER_API_KEY` / `FIREWORKS_API_KEY` with matching `--optimizer-provider`)
+- `npm run build` has been run in the pro-workflow plugin directory at least once
+
+## Examples
+
+```text
+/skill-optimize pro-workflow
+/skill-optimize wiki-research-loop --budget-usd 1.0 --epochs 5
+/skill-optimize wrap-up --optimizer-model claude-opus-4-7 --evaluator-model gpt-4o-mini
+```
+
+The third example mixes providers. The CLI infers the provider from the model id (`claude-*` → anthropic, `gpt-*` / `o*` → openai), so you do not need `--evaluator-provider openai` for `gpt-4o-mini`. Pass an explicit `--optimizer-provider` / `--evaluator-provider` to override inference.
+
+See [skills/skill-optimizer/SKILL.md](../skills/skill-optimizer/SKILL.md) for full mechanics, defaults, and the SkillOpt provenance.