v0.5.0: Goodhart tests — hidden acceptance criteria for contract compliance

Add adversarial hidden tests that implementation agents never see, countering
Goodhart's Law (optimizing for visible test inputs rather than true contract
compliance). Graduated-disclosure remediation re-implements failing components
with behavioral hints that get progressively more specific.

New features:
- Goodhart test author (single LLM call, adversarial prompt)
- Hidden test storage isolated from visible tests
- Polish-phase Goodhart evaluation + remediation loop
- Graduated hints: Level 1 (behavioral), Level 2 (contract invariant)
- Wizard, lifecycle polish phase, integrator improvements
- Fix Haiku pricing in budget tests (0.80 -> 1.00)

1573 tests, all passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: jmcentire

CLAUDE.md

@@ -35,7 +35,7 @@ Every agent follows 3 phases: Research -> Plan+Evaluate -> Execute. Research and
 2. **Shape** -- (Optional) Produce a Shape Up pitch: appetite, breadboard, rabbit holes, no-gos
 3. **Decompose** -- Task -> DecompositionNode tree (2-7 components), guided by shaping context
 3. **Contract** -- For each component (leaves first), generate ComponentContract
-4. **Test** -- For each contract, generate ContractTestSuite with executable tests
+4. **Test** -- For each contract, generate ContractTestSuite with executable tests + hidden Goodhart tests
 5. **Validate** -- Mechanical gate: all refs resolve, no cycles, test code parses
 6. **Implement** -- Each component independently by code_author agent, verified by contract tests
 7. **Integrate** -- Parent components: glue code wiring children, parent-level tests
@@ -85,6 +85,19 @@ Opt-in (`monitoring_enabled: true`). Pact-generated code embeds `PACT:<project_h
 
 **Budget hypervisor**: `estimate_tokens()` provides content-aware token estimation (symbol ratio → chars/token: 3.5 for code, 4.5 for prose). `record_tokens_validated()` cross-validates reported vs estimated counts using `max()` for conservative accounting. The `claude_code` backend no longer falls back to `len(text) // 4`. The `claude_code_team` backend now tracks spend via estimation.
 
+### Goodhart Tests (Hidden Acceptance Criteria)
+
+During the Test phase, Pact generates two test suites per component: **visible tests** (shown to the implementation agent) and **Goodhart tests** (hidden, never shared with agents). This counters Goodhart's Law: when agents can see all tests, they optimize for passing those specific inputs rather than truly satisfying the contract.
+
+Goodhart tests are adversarial — they probe for hardcoded returns, boundary-adjacent inputs, invariant generalization, and postcondition universality. They live in `.pact/contracts/<cid>/goodhart/` and are never loaded by `load_all_test_suites()` or `render_handoff_brief()`.
+
+During the **polish phase**, Goodhart tests run against all implementations. Failures trigger **graduated-disclosure remediation**: the component is re-implemented with behavioral hints (not exact errors) that get more specific on each attempt (max 2 by default). The agent never sees the actual test code.
+
+- **Level 1**: Vague behavioral hint from test description (e.g., "your add() function may not correctly handle the commutative property")
+- **Level 2**: Specific contract invariant (e.g., "The contract requires: add(a,b) == add(b,a). Your implementation appears to violate this.")
+
+Config: `max_goodhart_attempts` in pact.yaml (default: 2). Cost: ~$0.07/component for generation (1 LLM call, no research/plan).
+
 ### Casual-Pace Scheduling
 
 Poll-based, not event-loop. Agents invoked for focused bursts, state fully persisted between bursts.
@@ -162,7 +175,7 @@ src/pact/
     state.json         # Run lifecycle state
     audit.jsonl        # All actions + decisions
     decomposition/     # Tree + decisions
-    contracts/         # Per-component contracts + tests
+    contracts/         # Per-component contracts + tests + goodhart/
     implementations/   # Per-component code + attempts/
     standards.json     # Global standards (auto-generated after decomposition)
     tasks.json         # Phased task list (auto-generated after decomposition)
@@ -235,7 +248,7 @@ pact incident <project-dir> <id>      # Show incident details + diagnostic repor
 ## Testing
 
 ```bash
-make test          # 1482 tests, ~7s
+make test          # 1573 tests, ~7s
 make test-quick    # Stop on first failure
 ```
 

README.md

@@ -82,20 +82,24 @@ Interview -----> Shape (opt) -----> Decompose -----> Contract -----> Test
                                          Integrate (glue + parent tests)
                                                                        |
                                                                        v
+                                         Polish (Goodhart tests + regression check)
+                                                                       |
+                                                                       v
                                                                  Diagnose (on failure)
 ```
 
-**Nine phases, all mechanical gates:**
+**Ten phases, all mechanical gates:**
 
 1. **Interview** -- Identify risks, ambiguities, ask clarifying questions
 2. **Shape** -- (Optional) Produce a Shape Up pitch: appetite, breadboard, rabbit holes, no-gos
 3. **Decompose** -- Task into 2-7 component tree, guided by shaping context if present
 4. **Contract** -- Each component gets a typed interface contract
-5. **Test** -- Each contract gets executable tests (the enforcement)
+5. **Test** -- Each contract gets executable tests plus hidden Goodhart tests (adversarial acceptance criteria the implementation agent never sees)
 6. **Validate** -- Mechanical gate: refs resolve, no cycles, tests parse
 7. **Implement** -- Each component built independently by a code agent
 8. **Integrate** -- Parent components composed via glue code
-9. **Diagnose** -- On failure: I/O tracing, root cause, recovery
+9. **Polish** -- Cross-component regression check + Goodhart test evaluation with graduated-disclosure remediation
+10. **Diagnose** -- On failure: I/O tracing, root cause, recovery
 
 ## Health Monitoring
 
@@ -277,7 +281,7 @@ my-project/
     state.json         # Run lifecycle
     audit.jsonl        # Full audit trail
     decomposition/     # Tree + decisions
-    contracts/         # Per-component interfaces + tests
+    contracts/         # Per-component interfaces + tests + goodhart/
     implementations/   # Per-component code
     compositions/      # Integration glue
     learnings/         # Accumulated learnings
@@ -288,7 +292,7 @@ my-project/
 
 ```bash
 make dev          # Install with LLM backend support
-make test         # Run full test suite (1501 tests)
+make test         # Run full test suite (1573 tests)
 make test-quick   # Stop on first failure
 make clean        # Remove venv and caches
 ```

docs/index.html

@@ -554,6 +554,11 @@ <h3>Wavefront Scheduling</h3>
         <h3>Prompt Caching</h3>
         <p>Static prompt prefixes cached across API calls. 50-70% input token savings. Cache hit rates tracked in budget metrics. Research results persisted and reused across phases.</p>
       </div>
+      <div class="feature-card">
+        <span class="feature-icon">&#x1f3af;</span>
+        <h3>Hidden Acceptance Criteria</h3>
+        <p>Goodhart tests: adversarial hidden tests the agent never sees. Catches hardcoded returns, missing validation, and invariants that hold only for visible inputs. Graduated-disclosure remediation on failure.</p>
+      </div>
       <div class="feature-card">
         <span class="feature-icon">&#x1f6e1;&#xfe0f;</span>
         <h3>Drift Detection</h3>
@@ -655,7 +660,7 @@ <h2>If AI writes the code, who debugs it at 3am?</h2>
         <div class="step"><span class="arrow">&#9656;</span> <span>Fixer agent spawned with contract context</span></div>
         <div class="step"><span class="arrow">&#9656;</span> <span>Reproducer test added to contract</span></div>
         <div class="step"><span class="arrow">&#9656;</span> <span>Implementation rebuilt from scratch</span></div>
-        <div class="step"><span class="arrow">&#9656;</span> <span><span class="label">1,501 tests pass</span> <span class="dim">// verified</span></span></div>
+        <div class="step"><span class="arrow">&#9656;</span> <span><span class="label">1,573 tests pass</span> <span class="dim">// verified</span></span></div>
         <div class="step"><span class="arrow">&#9656;</span> <span>Contract got stricter. Bug can't recur.</span></div>
       </div>
     </div>

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pact-agents"
-version = "0.4.0"
+version = "0.5.0"
 description = "Contract-first multi-agent software engineering. Contracts before code. Tests as law. Agents that can't cheat."
 readme = "README.md"
 license = {text = "MIT"}

src/pact/agents/test_author.py

@@ -351,3 +351,147 @@ async def author_tests(
     )
 
     return suite, research, plan
+
+
+# ── Goodhart (Hidden) Test Author ─────────────────────────────────
+
+GOODHART_SYSTEM = """You are an adversarial test author for contract-driven development.
+Your job is to generate hidden acceptance tests that catch implementations that
+"teach to the test" — passing visible tests through shortcuts rather than truly
+satisfying the contract.
+
+Think like a code reviewer who suspects the implementation was written by an agent
+that could see the exact test inputs and assertions. Ask yourself:
+- What shortcuts could an agent take if it only saw the visible tests?
+- Hardcoded returns that happen to match visible test inputs
+- Missing validation that visible tests don't exercise
+- Invariants that hold only for the specific values in visible tests
+- Boundary conditions adjacent to (but distinct from) visible edge cases
+- Postconditions that should hold generally but visible tests only check with specific values
+
+Key principles:
+- Tests verify behavior at boundaries (inputs/outputs), not internals
+- All test functions MUST be prefixed with test_goodhart_
+- Each test's description field MUST explain the behavioral property being tested,
+  NOT the specific assertion. These descriptions become graduated hints during remediation.
+- Dependencies must be mocked — tests verify one component in isolation
+- Generated code must be syntactically valid
+- Do NOT duplicate coverage already in the visible tests — find gaps"""
+
+GOODHART_SYSTEM_TS = """You are an adversarial test author for contract-driven development.
+Your job is to generate hidden acceptance tests that catch implementations that
+"teach to the test" — passing visible tests through shortcuts rather than truly
+satisfying the contract.
+
+Think like a code reviewer who suspects the implementation was written by an agent
+that could see the exact test inputs and assertions. Ask yourself:
+- What shortcuts could an agent take if it only saw the visible tests?
+- Hardcoded returns that happen to match visible test inputs
+- Missing validation that visible tests don't exercise
+- Invariants that hold only for the specific values in visible tests
+- Boundary conditions adjacent to (but distinct from) visible edge cases
+- Postconditions that should hold generally but visible tests only check with specific values
+
+Key principles:
+- Tests verify behavior at boundaries (inputs/outputs), not internals
+- All test names MUST contain "goodhart" (e.g., "goodhart: should handle...")
+- Each test's description field MUST explain the behavioral property being tested,
+  NOT the specific assertion. These descriptions become graduated hints during remediation.
+- Dependencies must be mocked — tests verify one component in isolation
+- Generated code must be syntactically valid TypeScript for Vitest
+- Do NOT duplicate coverage already in the visible tests — find gaps"""
+
+
+async def author_goodhart_tests(
+    agent: AgentBase,
+    contract: ComponentContract,
+    visible_suite: ContractTestSuite,
+    dependency_contracts: dict[str, ComponentContract] | None = None,
+    language: str = "python",
+) -> ContractTestSuite:
+    """Generate adversarial hidden tests (single LLM call, no research/plan).
+
+    These tests are never shown to the implementation agent. They catch
+    Goodhart's Law violations: implementations that optimize for visible
+    test inputs rather than truly satisfying the contract.
+
+    Args:
+        agent: The LLM agent backend.
+        contract: The component contract.
+        visible_suite: The visible test suite (for gap analysis).
+        dependency_contracts: Contracts for dependencies (mock info).
+        language: Test language — "python", "typescript", or "javascript".
+
+    Returns:
+        ContractTestSuite with adversarial test cases.
+    """
+    contract_summary = _render_focused_contract(contract)
+
+    # Summarize visible tests so the LLM knows what's already covered
+    visible_summary = "\n".join(
+        f"  - {tc.id}: {tc.description} [{tc.category}] (function: {tc.function})"
+        for tc in visible_suite.test_cases
+    )
+
+    dep_mock_info = ""
+    if dependency_contracts:
+        for dep_id, dc in dependency_contracts.items():
+            dep_mock_info += f"\nDependency '{dep_id}' functions to mock:\n"
+            for func in dc.functions:
+                inputs_str = ", ".join(f"{i.name}: {i.type_ref}" for i in func.inputs)
+                dep_mock_info += f"  - {func.name}({inputs_str}) -> {func.output_type}\n"
+
+    # Select language-specific system prompt and instructions
+    if language in ("typescript", "javascript"):
+        system_prompt = GOODHART_SYSTEM_TS
+        import_hint = f"import {{ ... }} from '../src/{contract.component_id}'"
+        framework = "Vitest"
+        test_lang = language
+    else:
+        system_prompt = GOODHART_SYSTEM
+        import_hint = f"from src.{contract.component_id} import *"
+        framework = "pytest"
+        test_lang = "python"
+
+    prompt = f"""Generate adversarial hidden acceptance tests for component '{contract.name}'.
+
+Contract:
+{contract_summary}
+
+Visible tests already covering this contract:
+{visible_summary}
+
+{f"Dependencies to mock:{dep_mock_info}" if dep_mock_info else ""}
+
+Requirements:
+- component_id must be "{contract.component_id}"
+- contract_version must be {contract.version}
+- All test functions prefixed with test_goodhart_ (Python) or described as "goodhart: ..." (TS/JS)
+- Each test_case description must explain the BEHAVIORAL PROPERTY, not the assertion
+  Good: "The add function should be commutative for all numeric inputs"
+  Bad: "Test that add(2,3) equals add(3,2)"
+- Find gaps in visible test coverage — do NOT duplicate existing tests
+- Focus on: hardcoded-return detection, boundary adjacency, invariant generalization,
+  postcondition universality, input-space exploration beyond visible values
+- Import from: {import_hint}
+- Use {framework} conventions
+- test_language must be "{test_lang}"
+- generated_code must contain the COMPLETE test file"""
+
+    cache_prefix = f"Contract:\n{contract_summary}\n\nVisible tests:\n{visible_summary}"
+
+    suite, in_tok, out_tok = await agent.assess_cached(
+        ContractTestSuite, prompt, system_prompt, cache_prefix=cache_prefix,
+    )
+
+    # Ensure required fields
+    suite.component_id = contract.component_id
+    suite.contract_version = contract.version
+    suite.test_language = language
+
+    logger.info(
+        "Goodhart tests authored for %s: %d cases (%d tokens)",
+        contract.component_id, len(suite.test_cases), in_tok + out_tok,
+    )
+
+    return suite

src/pact/budget.py

@@ -19,9 +19,16 @@
 # Built-in defaults — overridable via config.yaml model_pricing
 # Format: model_id -> (input_cost_per_million, output_cost_per_million)
 DEFAULT_MODEL_PRICING: dict[str, tuple[float, float]] = {
-    # Anthropic
-    "claude-haiku-4-5-20251001": (0.80, 4.00),
+    # Anthropic — https://platform.claude.com/docs/en/about-claude/pricing
+    "claude-haiku-3": (0.25, 1.25),
+    "claude-haiku-3-5": (0.80, 4.00),
+    "claude-haiku-4-5-20251001": (1.00, 5.00),
+    "claude-sonnet-4": (3.00, 15.00),
     "claude-sonnet-4-5-20250929": (3.00, 15.00),
+    "claude-sonnet-4-6": (3.00, 15.00),
+    "claude-opus-4": (15.00, 75.00),
+    "claude-opus-4-1": (15.00, 75.00),
+    "claude-opus-4-5": (5.00, 25.00),
     "claude-opus-4-6": (5.00, 25.00),
     # OpenAI
     "gpt-4o": (2.50, 10.00),

src/pact/cli.py

@@ -251,6 +251,12 @@ def main() -> None:
     p_pricing.add_argument("--export", action="store_true", help="Export pricing to ~/.config/pact/model_pricing.json")
     p_pricing.add_argument("--json", action="store_true", dest="json_output", help="Output as JSON")
 
+    # wizard
+    p_wizard = subparsers.add_parser("wizard", help="Guided project setup wizard")
+    p_wizard.add_argument("project_dir", help="Project directory path")
+    p_wizard.add_argument("--config", default=None, metavar="FILE", help="JSON/YAML config file for non-interactive mode")
+    p_wizard.add_argument("--budget", type=float, default=None, help="Override budget (skips budget question)")
+
     args = parser.parse_args()
 
     if args.verbose:
@@ -332,6 +338,67 @@ def main() -> None:
         cmd_health(args)
     elif args.command == "pricing":
         cmd_pricing(args)
+    elif args.command == "wizard":
+        cmd_wizard(args)
+
+
+def cmd_wizard(args: argparse.Namespace) -> None:
+    """Guided project setup wizard."""
+    from pathlib import Path
+
+    import yaml
+
+    from pact.project import ProjectManager
+    from pact.wizard import (
+        build_wizard_questions,
+        generate_pact_yaml,
+        generate_sops_md,
+        generate_task_md,
+        load_wizard_config_from_file,
+        run_wizard_interactive,
+    )
+
+    project_dir = args.project_dir
+
+    if args.config:
+        config = load_wizard_config_from_file(Path(args.config))
+    else:
+        questions = build_wizard_questions()
+        print("Pact Project Wizard")
+        print("=" * 40)
+        print("Answer each question to configure your project.")
+        print("Press Enter to accept defaults shown in [brackets].\n")
+        config = run_wizard_interactive(questions)
+
+    if args.budget is not None:
+        config.budget = args.budget
+
+    project = ProjectManager(project_dir)
+    project.init(budget=config.budget)
+
+    project.task_path.write_text(generate_task_md(config))
+    project.sops_path.write_text(generate_sops_md(config))
+
+    pact_yaml = generate_pact_yaml(config)
+    with open(project.config_path, "w") as f:
+        yaml.dump(pact_yaml, f, default_flow_style=False, sort_keys=False)
+
+    print(f"\nProject initialized: {project.project_dir}")
+    print(f"  task.md   - Pre-filled from your description")
+    print(f"  sops.md   - Tailored for {config.language}")
+    print(f"  pact.yaml - Configured with your preferences")
+
+    if config.run_interview:
+        print("\nStarting interview phase...")
+        asyncio.run(cmd_interview(argparse.Namespace(
+            project_dir=project_dir, verbose=False,
+        )))
+    else:
+        print(f"\nNext steps:")
+        print(f"  1. Review and refine task.md")
+        print(f"  2. pact interview {project_dir}")
+        print(f"  3. pact approve {project_dir}")
+        print(f"  4. pact run {project_dir}")
 
 
 def cmd_pricing(args: argparse.Namespace) -> None:

src/pact/decomposer.py

@@ -15,7 +15,7 @@
 
 from pact.agents.base import AgentBase
 from pact.agents.contract_author import author_contract
-from pact.agents.test_author import author_tests
+from pact.agents.test_author import author_goodhart_tests, author_tests
 from pact.contracts import validate_all_contracts
 from pact.project import ProjectManager
 from pact.schemas import (
@@ -418,6 +418,19 @@ async def decompose_and_contract(
                 "tests",
                 f"{component_id}: {len(suite.test_cases)} cases",
             )
+
+            # Generate Goodhart (hidden) tests — skip if already exist
+            if not project.load_goodhart_suite(component_id):
+                goodhart_suite = await author_goodhart_tests(
+                    agent, contract, suite,
+                    dependency_contracts=dep_contracts,
+                    language=project.language,
+                )
+                project.save_goodhart_suite(goodhart_suite)
+                project.append_audit(
+                    "goodhart_tests",
+                    f"{component_id}: {len(goodhart_suite.test_cases)} hidden cases",
+                )
         else:
             logger.info("Skipping tests for %s — already exist", component_id)