Make the audit trio comprehensive, on any repo (PP-PLAN-040)

[jeremylongshore]

Make the audit trio comprehensive, on any repo

Umbrella tracking issue for PP-PLAN-040 — make /audit-tests + /implement-tests +
@intentsolutions/audit-harness comprehensive (every audit dimension) and make the harness
work on any repo without Claude in the loop.

Canonical plan: intent-eval-lab/000-docs/040-PP-PLAN-audit-trio-comprehensive-2026-06-04.md
Rollback baseline (safety net): intent-eval-lab/000-docs/041-RR-LAND-iep-rollback-baseline-2026-06-04.md
Task source-of-truth: bd workspace ~/000-projects/.beads/ (this issue is the public face; bd wins for state).

Resolved model — inspector + provisioner + toolbox (not competitors)

• /audit-tests = inspector — decides what to look for, runs checks, reports.
• /implement-tests = provisioner — installs the testing setup, staged for review, never committed.
• @intentsolutions/audit-harness = toolbox — the no-Claude CLI engine that runs unattended (CI, pre-commit, terminal) on any repo.

Key decisions

• Harness is a read-only deterministic classifier + gate-runner + orchestrator of external tools (no filesystem mutation, no live-fetch).
• Reuse, don't reinvent: conform shells out to the existing 7 /validate-* validators + ajv/spectral; security/hygiene shell out to gitleaks/osv-scanner/syft/markdownlint/Vale/lychee; skill-quality consumes j-rig verdict rows.
• Data-first: audit-profile/v1 (closed/versioned/hashed) + one canonical dimension→gate registry, before any verb.
• New gates ship advisory; blocking promotion is engineer-pinned + FP-rate-gated. Safety levers (INDETERMINATE, per-gate supervision, .audit-harness.yml kill-switch, canary, signed install.sh, rollback baseline) ship first.
• Auto-update = Renovate + a scheduled ecosystem-sync orchestrator (weekly cron + on-release dispatch) homed in the org umbrella, driven by an ecosystem.json manifest.

Epics (bd is canonical; plain-English titles)

1. Establish the IEP rollback baseline as the safety net before any audit-harness surgery. — bd_000-projects-853u (in progress)
2. Build the audit-harness data and safety spine (audit-profile/v1, canonical registry, INDETERMINATE, supervision, kill-switch, promotion rule). — bd_000-projects-5ilh (in progress)
3. Add the read-only classify verb with a golden fixture corpus authored first. — bd_000-projects-22sk
4. Add the conform verb by reusing the /validate-* validators + bundled content-addressed schemas. — bd_000-projects-b6qm
5. Add testing-depth gates (L2/L4/L5 + property/fuzz/flakiness), advisory-first, fast/deep split. — bd_000-projects-a7sb
6. Add security/hygiene/skill-quality gates that consume j-rig behavioral verdicts. — bd_000-projects-qew3
7. Add the currency advisory report with a per-upstream-identity pin relation. — bd_000-projects-e0l4
8. Stand up the ecosystem-sync home (ecosystem.json + scheduled orchestrator). — bd_000-projects-sujb
9. Refactor /audit-tests + /implement-tests to call the harness brain once trusted. — bd_000-projects-7elp

Guard — claude-code-plugins has bespoke, in-flight CI/CD (do not disrupt)

Epic 8 reaches into claude-code-plugins (validator-authoring home + manifest member). That repo runs a
large purpose-built plugin-testing pipeline (validate-plugins, pr-prescreen, e2e-tests, cli-test,
publish-changed-packages, secret-scan, security-audit, deploy-marketplace, sync-external,
plane-sync) with active feature work in flight. Any ecosystem-sync work touching it must study + coordinate
with the existing pipelines first and never bolt generic automation on blindly; it is sequenced last in
any fan-out.

Progress so far (2026-06-04)

• ✅ Master plan + rollback baseline filed.
• ✅ Phase 0 data-first: audit-profile/v1 schema + golden fixtures + spec landed in audit-harness (ajv-verified; escape-scan clean).
• ✅ bd epic tree filed (9 epics + 28 children + verb-sequence edges).

Beads: bd_000-projects-853u bd_000-projects-5ilh bd_000-projects-22sk bd_000-projects-b6qm bd_000-projects-a7sb bd_000-projects-qew3 bd_000-projects-e0l4 bd_000-projects-sujb bd_000-projects-7elp

• Jeremy Longshore
  intentsolutions.io

[jeremylongshore]

Phase 2 conform complete — audit-harness PR #54, all required CI green (conform suite 31/31, classify, ruff, shellcheck, self-check 18/20/22, regression, manifest-alignment; only advisory Vale fails — known CVEs false-positive).

audit-harness conform [repo] is read-only + advisory-first, emitting gate-result/v1 rows. Bundled content-addressed schemas at schemas/conform/v1/ (skillmd/mcp/plugin/agent); each row records the bundled schema's sha256 in policy_hash. E4 epic + 3 children (wire validators, bundle schemas + policy_hash, pass/fail fixtures) all closed. AAR: audit-harness/000-docs/011-AA-AACR-conform-verb-phase2-2026-06-05.md.

Next in the plan: Phase 3 (E5) testing-depth gates.

• Jeremy Longshore
  intentsolutions.io