Skip to content

Commit 041164b

Browse files
forkrulinstallerclaude
authored
hone: a green suite is not evidence a fix worked (#12)
Found in forkrul-woodpecker-CI spec 002. Round 3 of a hone run pointed a critic at the fixes rounds 1-2 had produced, rather than at the implementation. Four of six blocking findings were regressions HONE ITSELF had introduced -- three of them credential leaks -- and every one shipped under a fully green suite. The Cool phase required only "the full test suite must be green again", which is exactly the condition that held while all four landed. A gate that cannot distinguish a working fix from a broken one is the cannot-fail shape, sitting in the skill that exists to find cannot-fail shapes. The pipeline's own Golden Rule says fix the prompt before the code; for the review stage this SKILL.md is the prompt. - every fix must be shown to FAIL without itself (revert, watch red, restore) - round N+1's panel reviews round N's fixes as part of the diff; a round whose fixes went unreviewed cannot count as clean -- the rating measures what the panel examined, same principle as the overlap cap - a fix that moves a threshold must re-measure BOTH directions, including the one that was fine before - a spec edit written to justify a code change is the highest-risk edit in the pipeline, because it removes the thing that would have contradicted the code; its premise must be measured and the measurement shown Co-authored-by: installer <installer@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent ebfc01e commit 041164b

1 file changed

Lines changed: 33 additions & 0 deletions

File tree

skills/hone/SKILL.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,10 +98,33 @@ Dispatch **1 judge subagent** with all critic findings. The judge:
9898
log with reasoning)
9999
- **After each fix, the full test suite must be green again.** The test freeze holds
100100
through hone: a fix never edits a test unless `spec.md`/`tasks.md` changed first
101+
- **Every fix must be shown to fail without itself.** Revert the fix, watch the guarding
102+
test go red, restore it. A fix whose test passes with the fix removed has not been
103+
verified — only asserted — and does not count as applied. Record the check in the log.
101104
- A finding that reveals a *spec* gap is not fixed here — Golden Rule: halt and route
102105
back through smithy to `anvil`/`temper`
103106
- Append the round to `code-review.md`; if not converged, run the next round
104107

108+
**The repair is the least-reviewed code in the pipeline, and it is written under the
109+
worst conditions.** It is authored at the end of a session, against a defect just
110+
understood, by the author of that defect, under pressure to close it — and then merged
111+
because it "fixes the finding". The critics scrutinised the implementation; nobody
112+
scrutinises the repair. In the run that produced this rule, **four of six blocking
113+
findings in round 3 were regressions rounds 1 and 2 had introduced, three of them
114+
credential leaks, every one under a fully green suite.** Two guards follow:
115+
116+
- **Round N+1's panel reviews round N's fixes as part of the diff**, not just the
117+
original implementation. A round whose fixes went unreviewed is not a clean round.
118+
- **A fix that moves a threshold must re-measure BOTH directions**, including the one
119+
that was fine before. Optimising for the failure you were just shown, without
120+
re-checking its opposite, is how a fix aimed at over-masking opened a credential leak.
121+
122+
**A spec edit written to justify a code change is the highest-risk edit in the pipeline**,
123+
because it removes the very thing that would otherwise have contradicted the code. If a
124+
fix requires amending an A++-tempered artifact, the amendment's premise must be
125+
*measured and the measurement shown* — not reasoned, however obvious it sounds. Halt and
126+
re-temper instead where the change is a real requirement change.
127+
105128
## Critic Prompt Template
106129

107130
```
@@ -144,6 +167,11 @@ Return:
144167
round earns at most A+; the second confirms the first wasn't luck. A round capped by the
145168
overlap signal cannot count as clean — disjoint findings mean undiscovered defects remain.
146169

170+
**A round is clean only if the previous round's fixes were in the diff its critics
171+
reviewed.** Zero blocking findings against code nobody looked at is not a signal. This is
172+
the same principle as the overlap cap: the rating measures what the panel *examined*, not
173+
what it happened to return.
174+
147175
## Iteration Cap
148176

149177
**Maximum 3 rounds**, then escalate to the user with the stuck findings. Quench's
@@ -159,6 +187,11 @@ spec (back to `temper`) or an oversized task (back to `anvil`).
159187
a test is wrong, that's a spec conversation (Golden Rule), not a hone edit.
160188
- **Don't fix beyond the findings.** Refactor-while-you're-in-there at this stage is
161189
scope creep on green code; put it in the log as a NIT for a future feature.
190+
- **Don't treat a green suite as evidence a fix worked.** It was green before the fix and
191+
it is green after; that is a statement about the tests, not the repair. Revert the fix
192+
and watch the test fail, or you have not checked anything.
193+
- **Don't let the author of a fix be its only reviewer.** Same rule as the diff itself,
194+
and for the same reason — a repair inherits every blind spot that produced the defect.
162195
- **Don't review a unit larger than ~400 changed lines.** Split by task/commit first.
163196
- **Don't accept conformance findings without the FR → file:line trace** (nor any
164197
critic's finding without its procedure artifact). Evidence, not vibes.

0 commit comments

Comments
 (0)