Skip to content

Commit d4bc5b5

Browse files
cowwoccowwoc
committed
feature: improve status-agent instructions with SPRT benchmark, adversarial hardening, and GetOutput guardrails
1 parent 132deb6 commit d4bc5b5

17 files changed

Lines changed: 3226 additions & 2406 deletions

File tree

‎.cat/issues/v2/v2.1/STATE.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Finalize naming conventions and UI polish before recording demo videos.
2424
- rename-satisfies-to-parent-requirements
2525
- fix-disable-model-invocation-docs
2626
- refactor-adversarial-tdd-protocol
27+
- refactor-benchmark-artifact-storage
2728
- refactor-eliminate-content-relay
2829
- split-execution-waves-into-agent-sections
2930
- extract-shared-planning-approach
@@ -119,6 +120,8 @@ Finalize naming conventions and UI polish before recording demo videos.
119120
- defer-plan-generation-to-work-phase
120121
- fix-session-end-jvmscope-methods
121122
- parallelize-review-concern-fixing
123+
- add-failure-investigation-to-skill-builder
124+
- fix-skill-builder-fresh-subagent-per-run
122125

123126

124127
## Issues Closed

‎.cat/issues/v2/v2.1/add-failure-investigation-to-skill-builder/PLAN.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Plan: add-failure-investigation-to-skill-builder
2+
3+
## Goal
4+
5+
When `cat:instruction-builder-agent`'s SPRT benchmark rejects one or more test cases, automatically
6+
run a structured failure investigation before presenting results to the user. The investigation mirrors
7+
the methodology in `plugin/skills/learn/phase-investigate.md`: use `cat:get-history-agent` and
8+
`session-analyzer` to examine raw conversation transcripts, thinking blocks, agent context at the time
9+
of failure, and sources of priming (e.g., prior runs sharing subagent context, model defaults, escape
10+
clauses in instructions).
11+
12+
## Background
13+
14+
Currently, when SPRT rejects, the instruction-builder shows aggregated pass/fail counts and asks the
15+
user what to do next. The root cause is not investigated — the assumption is that the skill instructions
16+
are at fault. This assumption can be wrong (see: batch contamination producing spurious TC5 failures
17+
where runs 1-13 were 100% pass but runs 14-27 contaminated by shared context were ~7% pass).
18+
19+
## Approach
20+
21+
Add a new investigation sub-step after SPRT completes and before presenting results to the user:
22+
23+
1. Identify which test cases were Rejected
24+
2. For each rejected test case, retrieve the subagent IDs for the failing runs using session-analyzer
25+
3. Examine the subagent conversation logs: what did the agent receive as context? What was in its
26+
`<output>` tag injection? Were there thinking blocks showing the agent rationalizing adding follow-ups?
27+
4. Look for priming sources:
28+
- Batch contamination (multiple runs in one subagent context)
29+
- Model-default behaviors overriding "Do not..." instructions
30+
- Escape clauses ("unless user requests") being exploited
31+
- Prior successful patterns in context being replicated
32+
5. Present findings to the user alongside the SPRT results
33+
34+
## Sub-Agent Waves
35+
36+
### Wave 1
37+
- Read `plugin/skills/instruction-builder-agent/SKILL.md` (worktree copy)
38+
- Read `plugin/skills/learn/phase-investigate.md` for investigation methodology
39+
- Design the investigation sub-step: where it fits in Step 3 of the SPRT loop, what it reads, what it outputs
40+
- Update `plugin/skills/instruction-builder-agent/SKILL.md` to add the investigation sub-step after SPRT and before the user-facing results presentation
41+
- The investigation should:
42+
- Use `session-analyzer analyze <SESSION_ID>` to discover subagent IDs for failing benchmark runs
43+
- Use `session-analyzer search <SESSION_ID>/subagents/agent-<ID> "Would you like|What would you"` to find failure instances
44+
- Report: which runs failed, what the agent output was, whether batch contamination is present (multiple runs in one subagent), and what priming sources were detected
45+
46+
## Post-conditions
47+
48+
- [ ] `plugin/skills/instruction-builder-agent/SKILL.md` contains an investigation sub-step that runs automatically on SPRT Reject
49+
- [ ] The investigation step uses `cat:get-history-agent` / `session-analyzer` to examine raw subagent conversations
50+
- [ ] The investigation identifies batch contamination, thinking block patterns, and instruction priming sources
51+
- [ ] Investigation findings are presented to the user before asking whether to improve the skill
52+
- [ ] No regressions to the existing SPRT benchmark loop

‎.cat/issues/v2/v2.1/add-failure-investigation-to-skill-builder/STATE.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# State
2+
3+
- **Status:** open
4+
- **Progress:** 0%
5+
- **Dependencies:** []
6+
- **Blocks:** []

‎.cat/issues/v2/v2.1/fix-skill-builder-fresh-subagent-per-run/PLAN.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Plan: fix-skill-builder-fresh-subagent-per-run
2+
3+
## Goal
4+
5+
Fix `cat:instruction-builder-agent`'s SPRT benchmark runner to ensure each benchmark run spawns a
6+
completely fresh subagent. This enforces the independence assumption that SPRT requires for statistically
7+
valid results.
8+
9+
## Background
10+
11+
SPRT requires each trial to be an independent Bernoulli draw from the same underlying distribution.
12+
When multiple benchmark runs execute inside a single subagent context, trial N is conditioned on
13+
trials 1…N-1 (the agent has already seen the prior runs in its conversation history). This is
14+
batch contamination.
15+
16+
Observed effect: In `2.1-improve-status-agent-instructions`, runs 1-13 (individual fresh subagents)
17+
passed TC5 100% of the time. Runs 14-27 (batched into one subagent context) passed only ~7% of the
18+
time — the agent learned from the conversation history to add follow-up questions. This caused a
19+
spurious SPRT Reject.
20+
21+
## Approach
22+
23+
Audit the instruction-builder's Step 3 benchmark runner:
24+
25+
1. Read `plugin/skills/instruction-builder-agent/SKILL.md` to understand current benchmarking design
26+
2. Identify any code paths where multiple runs for the same test case are batched into one subagent
27+
3. Enforce: each individual run (TC + run number) spawns its own fresh Task subagent with no prior
28+
conversation context from other runs
29+
4. The SPRT orchestrator that accumulates results should be in the parent agent, not inside a
30+
benchmarking subagent
31+
32+
## Sub-Agent Waves
33+
34+
### Wave 1
35+
- Read `plugin/skills/instruction-builder-agent/SKILL.md` (worktree copy) — specifically Step 3 and any
36+
benchmark-runner sub-steps
37+
- Identify the current batching design: where does the skill instruct the agent to spawn subagents for runs?
38+
Is there any instruction that allows batching multiple runs into one context?
39+
- Fix the skill instructions to enforce one-subagent-per-run
40+
- Add an explicit guard: "Each run MUST spawn a fresh subagent. Do NOT execute multiple runs in the same
41+
subagent context — context from prior runs contaminates later runs and invalidates SPRT independence."
42+
- Commit the fix
43+
44+
## Post-conditions
45+
46+
- [ ] `plugin/skills/instruction-builder-agent/SKILL.md` explicitly requires one fresh subagent per benchmark run
47+
- [ ] No instruction in the SPRT benchmark loop allows batching multiple runs into one subagent context
48+
- [ ] A comment or guard is present explaining WHY (SPRT independence requirement)
49+
- [ ] No regressions to the existing benchmark design

‎.cat/issues/v2/v2.1/fix-skill-builder-fresh-subagent-per-run/STATE.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# State
2+
3+
- **Status:** open
4+
- **Progress:** 0%
5+
- **Dependencies:** []
6+
- **Blocks:** []

‎.cat/issues/v2/v2.1/improve-status-agent-instructions/STATE.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# State
22

3-
- **Status:** open
4-
- **Progress:** 0%
3+
- **Status:** closed
4+
- **Resolution:** implemented
5+
- **Progress:** 100%
56
- **Dependencies:** []
67
- **Blocks:** []
8+
- **Target Branch:** v2.1

‎.cat/issues/v2/v2.1/refactor-benchmark-artifact-storage/PLAN.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Plan: refactor-benchmark-artifact-storage
2+
3+
## Current State
4+
The instruction-builder-agent writes benchmark artifacts (test-cases.json, benchmark.json,
5+
compressed-SKILL.md) to a session-scoped directory: `benchmark-artifacts/<session-id>/`.
6+
These files are effectively ephemeral — they change with every session and cannot be used to
7+
compare results across sessions. The stable per-skill benchmark directory (e.g.,
8+
`plugin/skills/status-agent/benchmark/`) exists but is not the canonical write target.
9+
10+
## Target State
11+
The instruction-builder-agent writes benchmark artifacts exclusively to a stable directory
12+
adjacent to the skill file being improved: `<skill-dir>/benchmark/`. For example:
13+
- Plugin skill: `plugin/skills/status-agent/SKILL.md` → `plugin/skills/status-agent/benchmark/`
14+
- End-user skill: `my-skills/my-skill/SKILL.md` → `my-skills/my-skill/benchmark/`
15+
16+
The session-scoped `benchmark-artifacts/<session-id>/` directory is no longer created.
17+
This allows benchmark results to be committed alongside the skill and compared across sessions
18+
to detect regressions or improvements.
19+
20+
## Parent Requirements
21+
None
22+
23+
## Risk Assessment
24+
- **Risk Level:** LOW
25+
- **Breaking Changes:** None — behavior-preserving change to artifact write location
26+
- **Mitigation:** The stable per-skill benchmark/ directory already exists for skills that
27+
have been through the instruction-builder; the change only redirects writes
28+
29+
## Files to Modify
30+
- plugin/skills/instruction-builder-agent/SKILL.md - update artifact write paths to use
31+
skill-adjacent benchmark/ directory instead of benchmark-artifacts/<session-id>/
32+
33+
## Pre-conditions
34+
- [ ] All dependent issues are closed
35+
36+
## Sub-Agent Waves
37+
38+
### Wave 1
39+
- Read the current instruction-builder-agent SKILL.md to identify all locations where
40+
benchmark-artifacts/<session-id>/ paths are constructed or used
41+
- Files: plugin/skills/instruction-builder-agent/SKILL.md
42+
- Update all benchmark artifact write paths to use the skill-adjacent benchmark/ directory:
43+
derive the benchmark dir from the skill file path (e.g., dirname of SKILL.md + "/benchmark/")
44+
- Files: plugin/skills/instruction-builder-agent/SKILL.md
45+
- Ensure the approach works for both plugin skills (under plugin/skills/) and end-user skills
46+
(any path the instruction-builder receives as its argument)
47+
- Files: plugin/skills/instruction-builder-agent/SKILL.md
48+
- Remove any instruction to create or populate benchmark-artifacts/<session-id>/ directory
49+
- Files: plugin/skills/instruction-builder-agent/SKILL.md
50+
51+
## Post-conditions
52+
- [ ] instruction-builder-agent SKILL.md writes test-cases.json and benchmark.json to
53+
`<skill-dir>/benchmark/` (sibling of the SKILL.md being improved)
54+
- [ ] No references to `benchmark-artifacts/<session-id>/` remain in instruction-builder-agent SKILL.md
55+
- [ ] The path derivation works for end-user skills at any path (not just plugin/skills/)
56+
- [ ] E2E: Run /cat:instruction-builder-agent on a test skill and confirm benchmark artifacts
57+
appear in the skill-adjacent benchmark/ directory, not in benchmark-artifacts/

‎.cat/issues/v2/v2.1/refactor-benchmark-artifact-storage/STATE.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# State
2+
3+
- **Status:** open
4+
- **Progress:** 0%
5+
- **Dependencies:** []
6+
- **Blocks:** []

‎.cat/retrospectives/index.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
"trigger_interval_days" : 7
66
},
77
"last_retrospective" : "2026-03-16T18:30:00Z",
8-
"mistake_count_since_last" : 3,
8+
"mistake_count_since_last" : 4,
99
"files" : {
1010
"mistakes" : [ "mistakes-2026-01.json", "mistakes-2026-02.json", "mistakes-2026-03.json" ],
1111
"retrospectives" : [ "retrospectives-2026-01.json", "retrospectives-2026-02.json", "retrospectives-2026-03.json" ]

0 commit comments

Comments
 (0)