Add overfitting audit to workflow review

Author: Redwarx008

.agents/skills/review/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: review
-description: Independently verifies an implementation against its authorized final Design, shared working tree, real production call paths, and validation entrypoints. Use automatically through an independent subagent after Act completes or requests follow-up review; run read-only in the existing workspace.
+description: Independently verifies an implementation against its authorized final Design, shared working tree, real production call paths, validation entrypoints, and intended input class rather than visible examples. Use automatically through an independent subagent after Act completes or requests follow-up review; run read-only in the existing workspace.
 ---
 
 # Workflow Review
@@ -20,10 +20,11 @@ Work read-only in the shared working tree. Never create a worktree, edit files,
 1. Build a checklist from every final Design success criterion, constraint, interface, failure behavior, production integration obligation, and durable project-documentation obligation.
 2. Trace each requirement through the real runtime/editor/tooling call path. Confirm wiring, ownership, error/cleanup behavior, and compatibility boundaries.
 3. Inspect the actual diff and surrounding consumers, not just named declarations or tests.
-4. Cross-check the actual diff against the supplied validation entrypoints and any raw evidence. Verify that each evidence type proves the claimed behavior and follows the project's deterministic-versus-visual validation rules.
-5. Reverse-audit every added dependency, abstraction, interface or factory, wrapper, configuration or flag, state copy or cache, compatibility path, fallback, file, and duplicated helper. Identify the Design criterion or inspected project constraint that requires it, and check whether existing project code, a shared seam, the standard library, the engine/platform, or an installed dependency already provides a complete solution. Report unsupported complexity, but do not optimize for raw line or file count and do not remove required validation, failure handling, security, accessibility, or project ownership boundaries.
-6. Inspect version-control ownership and documentation boundaries.
-7. Report only evidence-backed P0/P1/P2 findings with tight file/line references and causal mechanisms.
+4. Audit generalization from the Design, public contracts, and domain invariants. Inspect implementation and validation for behavior coupled to visible examples, fixture/test names, exact names or IDs, incidental ordering or counts, current dimensions or data values, or the current dataset. Vary those irrelevant properties mentally or through already-authorized evidence and trace whether the same rule still holds. Do not demand behavior outside the authorized input class, and do not treat protocol-, schema-, or domain-required constants as overfitting.
+5. Cross-check the actual diff against the supplied validation entrypoints and any raw evidence. Verify that each evidence type proves the claimed behavior and follows the project's deterministic-versus-visual validation rules. Passing examples are insufficient when the implementation or evidence does not exercise the governing invariant.
+6. Reverse-audit every added dependency, abstraction, interface or factory, wrapper, configuration or flag, state copy or cache, compatibility path, fallback, file, and duplicated helper. Identify the Design criterion or inspected project constraint that requires it, and check whether existing project code, a shared seam, the standard library, the engine/platform, or an installed dependency already provides a complete solution. Report unsupported complexity, but do not optimize for raw line or file count and do not remove required validation, failure handling, security, accessibility, or project ownership boundaries.
+7. Inspect version-control ownership and documentation boundaries.
+8. Report only evidence-backed P0/P1/P2 findings with tight file/line references and causal mechanisms.
 
 ## Output
 

.agents/skills/review/agents/openai.yaml

@@ -1,7 +1,7 @@
 interface:
   display_name: "Workflow Review"
-  short_description: "Independently verify implementation completeness in place"
-  default_prompt: "Use $review to inspect the shared working tree and validation evidence against its authorized Design."
+  short_description: "Verify completeness and reject example-specific implementations"
+  default_prompt: "Use $review to inspect the shared working tree, production paths, generalization, and validation evidence against its authorized Design."
 
 policy:
   allow_implicit_invocation: true

.agents/skills/review/references/review-contract.md

@@ -8,6 +8,9 @@ Determine whether the shared working tree fully implements the authorized final
 - Reconstruct requirement-to-production-call-path relationships independently.
 - Treat declarations, selectors, tests, or registration as insufficient unless the real runtime/editor/tooling path consumes them.
 - Verify every final Design success criterion, constraint, contract, failure behavior, and required production integration path has evidence.
+- Derive the intended input class and governing invariants only from the final Design, public contracts, inspected project rules, and domain model. Do not silently broaden the feature while reviewing it.
+- Audit implementation and validation for example-specific coupling: fixture or test names, exact names/IDs, incidental order/count/dimensions, visible sample values, current dataset membership, and branches that recognize known cases instead of implementing the governing rule. Consider equivalent inputs with those irrelevant properties changed and trace the production behavior.
+- Treat a fixed value as legitimate when an explicit protocol, schema, resource format, approved requirement, or domain invariant requires it. Report overfitting only when evidence connects an incidental example property to incorrect or unsupported behavior.
 - For every Design correction that affects implementation, verify the final selected Design is the sole scope authority; if the diff exceeds it, report the excess rather than inferring user authorization.
 - Confirm the supplied workspace is the workspace whose diff is reviewed. When a linked worktree was created, verify the local Design was not copied or staged and that the implementation diff belongs to the supplied workspace.
 - Verify every selected Design decision or amendment that creates a durable project rule has the required project-documentation target; a workflow-only record is insufficient.
@@ -19,7 +22,7 @@ Determine whether the shared working tree fully implements the authorized final
 ## Severity
 
 - **P0:** Data loss, security boundary break, destructive behavior, or unusable core workflow.
-- **P1:** Unmet success criterion, broken production integration, false validation claim, or material regression.
-- **P2:** Non-blocking maintainability or resilience issue with a concrete risk.
+- **P1:** Unmet success criterion, broken production integration, false validation claim, material regression, or implementation that only satisfies visible examples while failing the authorized input class.
+- **P2:** Non-blocking maintainability or resilience issue with a concrete risk, including suspicious example coupling whose failure is not yet demonstrated.
 
 `PASS` requires no P0/P1, no unaccounted success criterion, and no implementation beyond final Design scope. If a finding admits multiple reasonable fixes or changes intent, state the decision needed; the main agent must ask the user.

.agents/skills/review/references/reviewer-output.md

@@ -5,9 +5,13 @@
 | Success criterion | Evidence inspected | Result |
 | --- | --- | --- |
 
+## Generalization audit
+
+For each materially changed behavior, state the intended input class or invariant, the irrelevant example properties varied or counterexamples inspected, the production evidence, and the result. Do not claim broader support than the final Design requires.
+
 ## Findings
 
-For each finding provide severity, concise title, exact file/line, observed evidence, causal mechanism, and violated criterion/constraint. Omit the section when there are no findings.
+For each finding provide severity, concise title, exact file/line, observed evidence, causal mechanism, and violated criterion/constraint. For overfitting, identify the incidental example property the implementation depends on and an equivalent authorized input that would take the wrong path. Omit the section when there are no findings.
 
 ## Validation gaps
 

.claude-plugin/marketplace.json

@@ -4,13 +4,13 @@
     "name": "Redwarx008"
   },
   "description": "Agent Workflow Kit marketplace for Claude Code.",
-  "version": "0.3.0+codex.20260723043714",
+  "version": "0.3.0+codex.20260725135218",
   "plugins": [
     {
       "name": "agent-workflow-kit",
       "source": "./",
       "description": "A decision-complete Design, adaptive Act, and independent Review workflow for coding agents.",
-      "version": "0.3.0+codex.20260723043714",
+      "version": "0.3.0+codex.20260725135218",
       "author": {
         "name": "Redwarx008"
       }

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-workflow-kit",
-  "version": "0.3.0+codex.20260723043714",
+  "version": "0.3.0+codex.20260725135218",
   "description": "A decision-complete Design, adaptive Act, and independent Review workflow for coding agents.",
   "author": {
     "name": "Redwarx008",

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-workflow-kit",
-  "version": "0.3.0+codex.20260723043714",
+  "version": "0.3.0+codex.20260725135218",
   "description": "A decision-complete Design, adaptive Act, and independent Review workflow for coding agents.",
   "author": {
     "name": "Redwarx008",

README.md

@@ -87,7 +87,7 @@ npm run test:powershell
 - Design 是讨论过程中持续维护的记录，不是最后凭记忆补写。
 - 除非用户显式要求，否则不新增或修改测试；现有测试仍可作为验证证据运行。
 - 不把 TDD 机械套到视觉、GPU、shader 或探索性原型；证据形式服从真实因果链。
-- Reviewer 不接受主 agent 的完成结论，从成功标准反向检查生产路径。
+- Reviewer 不接受主 agent 的完成结论，从成功标准反向检查生产路径，并检查实现是否依赖固定示例、名称/ID、顺序、数量、尺寸或当前数据集，而非 Design 与领域不变量。
 
 进一步说明见 [工作流哲学](docs/workflow-philosophy.md)、[验证策略](docs/validation-strategy.md)、[独立 Review](docs/independent-review.md)、[会话记忆](docs/session-memory.md) 和 [参考实现取舍](docs/reference-lessons.md)。
 

docs/independent-review.md

@@ -2,7 +2,7 @@
 
 ## 为什么需要独立检查
 
-实现 agent 容易沿着自己写代码时的假设检查，因而漏掉“局部存在但没有接进生产路径”的问题。独立 reviewer 应从成功标准反向重建调用链，而不是相信完成摘要。
+实现 agent 容易沿着自己写代码时的假设检查，因而漏掉“局部存在但没有接进生产路径”的问题，也可能把当前示例、固定 ID、顺序或数据集误当成领域规则。独立 reviewer 应从成功标准反向重建调用链和输入不变量，而不是相信完成摘要或通过的样例。
 
 真实项目中，独立检查曾发现 selector 和 pass 已存在，但 render object 没有进入 shadow caster 集合；也发现失败前置校验发生在文件复制之后，留下孤儿文件。这些问题仅检查符号或 happy-path 测试很难发现。
 
@@ -15,7 +15,8 @@ Reviewer 必须读取实现所在的共享工作区。为 Review 创建隔离 wo
 1. 从 Design 的每个成功标准建立覆盖表。
 2. 把 Design 指定的验证入口、实际 diff 和可复现的原始输出交叉核对，不把主 agent 的完成标记当作事实。
 3. 从外部入口沿真实运行时/编辑器路径追踪到变更点。
-4. 检查失败、清理、所有权、兼容和版本控制边界。
-5. 验证证据确实证明行为，而非只证明文件存在或构建通过。
+4. 从 Design、公开契约和领域模型确定预期输入类别，改变与语义无关的名称、ID、顺序、数量、尺寸和样例值，检查实现是否仍由同一不变量驱动；协议、格式或领域明确规定的常量不算过拟合。
+5. 检查失败、清理、所有权、兼容和版本控制边界。
+6. 验证证据确实证明一般规则，而非只证明文件存在、构建通过或已知样例通过。
 
 P0/P1 或未满足成功标准必须阻断完成。若修复方式不唯一，主 agent 回到用户决策，而不是 reviewer 或主 agent自行挑选方案。