Skip to content

Commit a56687c

Browse files
committed
feat: add sensing evidence claim gate
1 parent a3b6e1d commit a56687c

14 files changed

Lines changed: 2712 additions & 15 deletions

.github/CODEOWNERS

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Evidence policy and its enforcement path require explicit repository-owner
2+
# review. Branch protection must require Code Owner approval for this to bind.
3+
/.github/CODEOWNERS @ruvnet
4+
/evidence/policies/ @ruvnet
5+
/.github/workflows/model-release-gate.yml @ruvnet
6+
/v2/crates/wifi-densepose-train/ @ruvnet
7+
/README.md @ruvnet
8+
/benchmarks/ @ruvnet
9+
/docs/benchmarks/ @ruvnet
10+
/docs/releases/ @ruvnet
11+
/docs/huggingface/ @ruvnet

.github/workflows/model-release-gate.yml

Lines changed: 191 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,17 @@ name: Model release gate (ADR-298)
55
# boundary, near-constant output, degenerate class balance, a metric
66
# surfaced under a task name it wasn't computed as) before it ships.
77
#
8-
# Checker: v2/crates/wifi-densepose-train/src/model_gates.rs
8+
# Checkers:
9+
# * v2/crates/wifi-densepose-train/src/model_gates.rs
10+
# * v2/crates/wifi-densepose-train/src/sensing_claim_gate.rs (ADR-328)
911
#
10-
# IMPORTANT — the honest scope of this job: it protects the *checker itself*
11-
# from regressing (the gate logic + its issue-1521 regression fixture are
12-
# exercised on every push/PR that touches this crate), and running it is
13-
# required before ADR-298 can be called "wired in" at all. It does NOT gate
14-
# an actual model publish — this repository does not automate uploading to
15-
# the HuggingFace model repo (`ruvnet/wifi-densepose-pretrained`); that
16-
# remains a manual, human-run step. Before publishing or replacing a model
17-
# artifact there, run this gate against the real head weights locally:
12+
# IMPORTANT — the honest scope of this job: it protects the structural model
13+
# checker from regressing and hard-fails every committed sensing claim manifest
14+
# that does not satisfy the repository-owned ADR-328 policy. It still does NOT
15+
# gate an actual HuggingFace model publish because this repository does not
16+
# automate uploads to `ruvnet/wifi-densepose-pretrained`; that remains a manual,
17+
# human-run step. Before publishing or replacing a model artifact there, run
18+
# the structural model gate against the real head weights locally:
1819
#
1920
# cargo test -p wifi-densepose-train model_gates
2021
#
@@ -29,14 +30,45 @@ on:
2930
- master
3031
paths:
3132
- "v2/crates/wifi-densepose-train/**"
33+
- "evidence/claims/**"
34+
- "evidence/fixtures/**"
35+
- "evidence/policies/**"
36+
- "README.md"
37+
- "benchmarks/**"
38+
- "docs/benchmarks/**"
39+
- "docs/releases/**"
40+
- "docs/huggingface/**"
41+
- "docs/adr/ADR-298-model-release-sanity-gates.md"
42+
- "docs/adr/ADR-304-evidence-engine.md"
43+
- "docs/adr/ADR-328-sensing-evidence-claim-gate.md"
44+
- ".github/workflows/model-release-gate.yml"
45+
- ".github/CODEOWNERS"
3246
pull_request:
3347
paths:
3448
- "v2/crates/wifi-densepose-train/**"
49+
- "evidence/claims/**"
50+
- "evidence/fixtures/**"
51+
- "evidence/policies/**"
52+
- "README.md"
53+
- "benchmarks/**"
54+
- "docs/benchmarks/**"
55+
- "docs/releases/**"
56+
- "docs/huggingface/**"
57+
- "docs/adr/ADR-298-model-release-sanity-gates.md"
58+
- "docs/adr/ADR-304-evidence-engine.md"
59+
- "docs/adr/ADR-328-sensing-evidence-claim-gate.md"
60+
- ".github/workflows/model-release-gate.yml"
61+
- ".github/CODEOWNERS"
3562
workflow_dispatch:
3663

3764
permissions:
3865
contents: read
3966

67+
env:
68+
# Update only with the CODEOWNERS-reviewed policy. This prevents an unnoticed
69+
# threshold edit from changing the policy consumed by the same workflow.
70+
CLAIM_POLICY_SHA256: 1ba2b73ede726a789aa50f4eb60a443429bb2f38ff3a2acf27755708265e303f
71+
4072
jobs:
4173
model-release-gate:
4274
name: Model release gate check
@@ -46,22 +78,167 @@ jobs:
4678
with:
4779
persist-credentials: false
4880
submodules: recursive
81+
fetch-depth: 0
82+
83+
- name: Verify protected claim-policy digest
84+
run: |
85+
actual="$(sha256sum evidence/policies/sensing-claim-policy-v1.json | cut -d ' ' -f 1)"
86+
test "$actual" = "$CLAIM_POLICY_SHA256"
87+
88+
- name: Require evidence manifest for changed public claim surfaces
89+
if: github.event_name == 'pull_request'
90+
env:
91+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
92+
run: |
93+
mapfile -d '' -t changed < <(git diff --name-only -z --diff-filter=ACMRT "$BASE_SHA" "$GITHUB_SHA")
94+
changed_manifests=()
95+
for path in "${changed[@]}"; do
96+
case "$path" in
97+
evidence/claims/research/*.json|evidence/claims/production/*.json|evidence/claims/safety_critical/*.json)
98+
changed_manifests+=("$path")
99+
;;
100+
esac
101+
done
102+
103+
for surface in "${changed[@]}"; do
104+
required_class=""
105+
case "$surface" in
106+
README.md)
107+
if git diff --unified=0 "$BASE_SHA" "$GITHUB_SHA" -- README.md \
108+
| grep -Eiq '^\+[^+].*(accuracy|auc|precision|recall|sensitivity|specificity|false[ -]?positive|confidence|detect|through walls|heart[ -]?rate|breathing|occupancy|presence|pose|pck|mpjpe|latency|held[ -]?out|benchmark|score|production[ -]?ready|[0-9]+([.][0-9]+)?(%|[[:space:]]*(ms|hz|bpm)))'; then
109+
required_class="production"
110+
else
111+
continue
112+
fi
113+
;;
114+
benchmarks/*|docs/benchmarks/*)
115+
if git diff --unified=0 "$BASE_SHA" "$GITHUB_SHA" -- "$surface" \
116+
| grep -Eiq '^\+[^+].*(production|ready|deploy|ship|safety|medical|commercial)'; then
117+
required_class="production"
118+
else
119+
required_class="any"
120+
fi
121+
;;
122+
docs/releases/*|docs/huggingface/*)
123+
required_class="production"
124+
;;
125+
*)
126+
continue
127+
;;
128+
esac
129+
130+
matched=false
131+
for manifest in "${changed_manifests[@]}"; do
132+
[[ -f "$manifest" ]] || continue
133+
relative="${manifest#evidence/claims/}"
134+
class="${relative%%/*}"
135+
filename="${relative#*/}"
136+
[[ "$filename" != */* ]] || continue
137+
if [[ "$required_class" == production && "$class" != production ]]; then
138+
continue
139+
fi
140+
if jq -e --arg surface "$surface" \
141+
'(.claim_surface_paths | type == "array") and (.claim_surface_paths | index($surface) != null)' \
142+
"$manifest" >/dev/null; then
143+
matched=true
144+
break
145+
fi
146+
done
147+
if [[ "$matched" != true ]]; then
148+
echo "Public claim surface $surface changed without a matching $required_class class-bound evidence manifest." >&2
149+
exit 1
150+
fi
151+
done
49152
50153
- name: Install Rust toolchain
51-
run: rustup toolchain install stable --profile minimal
154+
run: rustup toolchain install 1.89 --profile minimal
52155

53156
- name: Run the model-release gate's own test suite
54157
working-directory: v2
55158
run: cargo test -p wifi-densepose-train --no-default-features model_gates -- --nocapture
56159

160+
- name: Run sensing evidence and claim gate tests
161+
working-directory: v2
162+
run: cargo test -p wifi-densepose-train --no-default-features sensing_claim_gate -- --nocapture
163+
164+
- name: Validate repository policy with research-only fixture
165+
working-directory: v2
166+
run: |
167+
mkdir -p ../evidence-receipts
168+
cargo run -p wifi-densepose-train --no-default-features \
169+
--bin sensing-claim-gate -- \
170+
--manifest ../evidence/fixtures/research-synthetic.json \
171+
--policy ../evidence/policies/sensing-claim-policy-v1.json \
172+
--required-class research \
173+
--receipt ../evidence-receipts/research-synthetic.receipt.json
174+
175+
- name: Gate every committed sensing claim manifest
176+
working-directory: v2
177+
run: |
178+
mkdir -p ../evidence-receipts
179+
manifest_count=0
180+
gate_failed=false
181+
while IFS= read -r -d '' manifest; do
182+
relative="${manifest#../evidence/claims/}"
183+
class="${relative%%/*}"
184+
filename="${relative#*/}"
185+
if [[ ! -f "$manifest" || -L "$manifest" ]]; then
186+
echo "Claim manifest must be a regular non-symlink file: $manifest" >&2
187+
gate_failed=true
188+
continue
189+
fi
190+
if [[ "$filename" == */* || ! "$filename" =~ ^[a-z0-9][a-z0-9._-]*\.json$ ]]; then
191+
echo "Claim JSON must be exactly one level below a class directory and use a canonical filename: $manifest" >&2
192+
gate_failed=true
193+
continue
194+
fi
195+
case "$class" in
196+
research|production|safety_critical) ;;
197+
*)
198+
echo "Unsupported claim class directory for $manifest" >&2
199+
gate_failed=true
200+
continue
201+
;;
202+
esac
203+
cli_class="${class//_/-}"
204+
stem="${filename%.json}"
205+
manifest_count=$((manifest_count + 1))
206+
if ! cargo run -p wifi-densepose-train --no-default-features \
207+
--bin sensing-claim-gate -- \
208+
--manifest "$manifest" \
209+
--policy ../evidence/policies/sensing-claim-policy-v1.json \
210+
--required-class "$cli_class" \
211+
--receipt "../evidence-receipts/${class}-${stem}.receipt.json"; then
212+
gate_failed=true
213+
fi
214+
done < <(find ../evidence/claims -name '*.json' -print0 | sort -z)
215+
if (( manifest_count == 0 )); then
216+
echo "Claim inventory is empty; at least one class-bound manifest is required." >&2
217+
exit 1
218+
fi
219+
if [[ "$gate_failed" == true ]]; then
220+
echo "One or more sensing claim manifests failed closed." >&2
221+
exit 1
222+
fi
223+
224+
- name: Upload machine-readable evidence receipts
225+
if: always()
226+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
227+
with:
228+
name: sensing-claim-receipts
229+
path: evidence-receipts/
230+
if-no-files-found: warn
231+
57232
- name: Summarize result
58233
if: always()
59234
run: |
60235
{
61236
echo '### Model release gate (ADR-298)'
62237
echo ''
63-
echo 'This job protects `model_gates.rs` from regressing. It does not itself'
64-
echo 'gate a real HuggingFace model publish — that upload is a manual step'
65-
echo 'outside this repository; run `cargo test -p wifi-densepose-train model_gates`'
66-
echo 'against real head weights before publishing one.'
238+
echo 'This job protects `model_gates.rs` and the ADR-328 claim gate from'
239+
echo 'regressing, and gates every JSON manifest in `evidence/claims/`.'
240+
echo 'This is repository evidence lint. The committed policy disables'
241+
echo 'production and safety claims until artifact retrieval, a real presence'
242+
echo 'reproducer, authenticated evaluator attestation, and maintainer review.'
243+
echo 'It does not gate the external HuggingFace upload.'
67244
} >> "$GITHUB_STEP_SUMMARY"

docs/adr/ADR-298-model-release-sanity-gates.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,13 @@ for any classifier artifact proposed for release, fails on:
3939
Each gate emits a structured, human-readable failure explaining the defect and
4040
the offending numbers.
4141

42+
ADR-328 adds the complementary evidence-release boundary. ADR-298 answers
43+
"is this model artifact structurally degenerate?" ADR-328 answers "does the
44+
submitted physical-sensing evidence support this class of public claim under a
45+
pre-registered policy?" A model must pass both checks. Structural model health
46+
cannot substitute for real-hardware held-out evidence, and strong benchmark
47+
evidence cannot excuse a degenerate classifier head.
48+
4249
## Consequences
4350

4451
- The specific degenerate presence head cannot ship again, and the
@@ -55,6 +62,9 @@ the offending numbers.
5562
metric cannot be constructed with a presence label.
5663
- `cargo test -p wifi-densepose-train`; the CI gate runs in the model-check
5764
workflow.
65+
- `cargo test -p wifi-densepose-train --no-default-features sensing_claim_gate`;
66+
`.github/workflows/model-release-gate.yml` also evaluates committed claim
67+
manifests and retains machine-readable receipts.
5868
- This ADR does **not** withdraw the already-published artifact (an
5969
outward-facing action requiring maintainer sign-off) — it prevents
6070
recurrence and documents the model-card correction.

docs/adr/ADR-304-evidence-engine.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# ADR-304: Evidence engine — MLflow for physical sensing
22

3-
- **Status**: Accepted — initial implementation planned (ADR-300 phase 1)
3+
- **Status**: Accepted — ADR-328 claim receipt edge implemented; signed ledger planned
44
- **Date**: 2026-08-11
55
- **Deciders**: ruv
66
- **Tags**: evidence, provenance, ledger, accuracy, drift, benchmark, honesty, substrate
@@ -52,6 +52,14 @@ that unifies them per deployment context:
5252
Build an **evidence engine**: a per-`(room, device, subject)` append-only
5353
accuracy ledger that every model automatically writes to.
5454

55+
ADR-328 now supplies the first enforceable edge of this decision: a strict
56+
claim manifest, a separately versioned policy, fail-closed production/safety
57+
rules, and a content-addressed JSON receipt. It deliberately does **not** claim
58+
to be the append-only signed ledger described below. Until the ledger and
59+
ADR-319 witness anchoring land, the receipt carries artifact digests and CI
60+
decisions, production cannot exceed `metadata_attested`, and the committed
61+
policy denies production entirely. It cannot authorize a production claim.
62+
5563
### 1. The evidence record
5664

5765
- An `EvidenceRecord` keyed by context — space id (ADR-306), signed device id
@@ -104,6 +112,10 @@ accuracy ledger that every model automatically writes to.
104112

105113
## Validation
106114

115+
- ADR-328 gate: simulator evidence cannot release a production claim; held-out
116+
environment overlap, insufficient samples, missing thresholds, and a failed
117+
confidence bound each produce a deterministic denial receipt. CI evaluates
118+
every manifest in `evidence/claims/` against the repository-owned policy.
107119
- `cargo test` on the evidence-engine crate — append-only invariant (no
108120
in-place mutation; corrections are new records); per-context aggregation math
109121
against fixtures; evidence-level is set by provenance and cannot be upgraded;

0 commit comments

Comments
 (0)