Skip to content

v0.5.0: Goodhart Tests — Hidden Acceptance Criteria

Choose a tag to compare

View all tags
@jmcentire jmcentire released this 04 Mar 22:55
v0.5.0
33fc2d5

What's New

Goodhart Tests — hidden adversarial acceptance criteria that implementation agents never see. Counters Goodhart's Law: when agents can see all tests, they optimize for those specific inputs rather than truly satisfying the contract.

Features

  • Goodhart test author: Single-LLM-call adversarial test generation during the Test phase. Probes for hardcoded returns, boundary-adjacent inputs, invariant generalization, and postcondition universality.
  • Isolated storage: Hidden tests live in .pact/contracts/<cid>/goodhart/, completely isolated from visible test loading and handoff briefs.
  • Polish-phase evaluation: After all visible tests pass, Goodhart tests run as a final acceptance gate.
  • Graduated-disclosure remediation: Failing components get behavioral hints (never actual test code):
    • Level 1: Vague behavioral hint from test description
    • Level 2: Specific contract invariant/postcondition
    • Max 2 attempts (configurable via max_goodhart_attempts)
  • Wizard: New pact wizard command for guided project setup
  • Lifecycle improvements: Polish phase integration, integrator enhancements

Cost Impact

~$0.07/component for Goodhart test generation (1 LLM call, no research/plan). For a 5-component project: ~$0.35 total overhead.

Stats

  • 1,573 tests, all passing
  • 23 files changed, +2,131 lines

Full changelog: v0.4.0...v0.5.0

Assets 2
Loading