Skip to content

Commit 0c38976

Browse files
authored
fix(governance): restore causal gatekeeper bool contract and isolate test mocks
* refactor(nemo): consolidate GFA LLMRails to single harness singleton - Add reload_nemo_rails() with asyncio.Lock to nemo_node_factory.py as the canonical hot-reload entry point for the shared LLMRails singleton - Remove independent rails global from server.py and _rails/get_rails() from tools/api.py; both now delegate to the harness singleton - Hot-reload via /v1/nemo/approve-refinement now propagates atomically to all GFA pod consumers (graph nodes, endpoints, tools) - Quarantine infra/modules/nemo_guardrails/main.tf as HISTORICAL-ONLY - Add nemo-freshness-check CI job to gate configmap snapshot drift * feat(compliance): add S3-compat ledger provider and fafef04 paper fixes - ObjectStoreLedgerProvider (boto3): supports AWS S3, GCS S3 Interop, MinIO, Ceph via S3_RECONCILIATION_BUCKET / S3_ENDPOINT_URL env vars - Register 's3' and 'object-store' aliases in _PROVIDERS factory - Update reconciliation-worker.yaml: default provider=s3, add S3 Secret keys, extend CiliumNetworkPolicy FQDN to *.amazonaws.com - P2.1: context_accumulator.py — separators in _content_hash json.dumps - P2.2: fiscal_limit_guard.py — per-reservation TTL sentinel key - P2.3: routing_seal.py — dot-sanitization in generate_seal() - P2.4: reconciliation_worker.py — GcsLedgerProvider (already present) - P2.6: safety_node.py — live Redis risk-metric reads with sentinel fallback (cbf:portfolio_drawdown / portfolio:daily_vol) - P2.7: causal_gatekeeper.py — MIN_SAMPLES guard before regression - P2.8: measure_paper_metrics.py — re-enable ungoverned baseline - P2.9: measure_reconciliation_metrics.py — honour REDIS_PASSWORD - POAM: add findings 038-042 - CAGE_ARXIV.MD: update Table 2 latency (fafef04, n=200), §6.1 infra note with Cloud Build IDs, §6.6 FPR note for live-Redis path - Archive fafef04 measurement evidence * docs: update markdown docs for S3-compat provider and fafef04 fixes - README.md: RECONCILIATION_PROVIDER table row + tree description - CHANGELOG.md: add Unreleased entries for ObjectStoreLedgerProvider, CronJob manifest, POAM-038-042, fiscal TTL sentinel, routing-seal dot-sanitization, context-accumulator canonical JSON, causal MIN_SAMPLES guard, safety-node live Redis metrics, baseline re-enable, reconciliation REDIS_PASSWORD fix; flag superseded 2026-08-05 entry - docs/architecture/GATEWAY_ARCHITECTURE.md: add GcsLedgerProvider + ObjectStoreLedgerProvider to backend list - docs/governance/CAUSAL_AND_CBF_GOVERNANCE.md: MIN_SAMPLES guard bullet + updated provider list in balance-provenance paragraph - deployment/README.md: add reconciliation-worker.yaml row to table - docs/governance/GOVERNANCE_OVERVIEW.md: POAM-023 open -> closed * docs(governance): address CAGE arXiv peer review 5 feedback * docs(governance): sync markdown docs with review-5 fixes * fix(governance): restore causal gatekeeper bool contract and isolate test mocks - causal_gatekeeper.py: insufficient-telemetry guard was returning a (False, message) tuple; callers using `if not result` would silently treat non-empty tuples as truthy, inverting fail-closed to fail-open. Fixed to return bare False per the documented bool contract. - server.py: add load_rails import so the symbol is patchable at src.governed_financial_advisor.server.load_rails in unit tests. - test_deployment_verification: update container name to match manifest. - test_causal_gatekeeper: mock Redis cache calls in unit tests. - test_hitl_toctou_revalidation: correct patch target to revalidate_post_hitl (actual method called by post_hitl_revalidate_node).
1 parent 94f920b commit 0c38976

48 files changed

Lines changed: 2068 additions & 219 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,83 @@ jobs:
177177
run: |
178178
python scripts/check_stpa_freshness.py --verbose
179179
180+
nemo-freshness-check:
181+
name: "NeMo Rail Actions Freshness Check"
182+
runs-on: ubuntu-latest
183+
steps:
184+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
185+
with:
186+
fetch-depth: 0
187+
persist-credentials: false
188+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
189+
with:
190+
python-version: "3.11"
191+
- name: Check nemo-rails-configmap actions.py snapshot is current
192+
run: |
193+
python3 - <<'EOF'
194+
import sys
195+
196+
CONFIGMAP = "deployment/k8s/nemo-rails-configmap.yaml"
197+
CANONICAL = "config/rails/actions.py"
198+
199+
# Extract the actions.py block from the ConfigMap YAML.
200+
# The block starts at the line containing "actions.py: |" (or "actions.py: |-")
201+
# and ends when the indentation drops back to the data: level.
202+
try:
203+
with open(CONFIGMAP) as f:
204+
lines = f.readlines()
205+
except FileNotFoundError:
206+
print(f"ERROR: {CONFIGMAP} not found — add the configmap snapshot or update this check.")
207+
sys.exit(1)
208+
209+
snapshot_lines = []
210+
in_block = False
211+
block_indent = None
212+
for line in lines:
213+
stripped = line.lstrip()
214+
indent = len(line) - len(stripped)
215+
if not in_block:
216+
if stripped.startswith("actions.py:"):
217+
in_block = True
218+
# block_indent is the indentation of the content lines (one level deeper)
219+
block_indent = indent + 2
220+
continue
221+
# Inside the block: collect lines that are indented at block_indent or deeper
222+
if stripped == "" or indent >= block_indent:
223+
snapshot_lines.append(line[block_indent:] if stripped else "\n")
224+
else:
225+
break # dedented back — end of block
226+
227+
if not snapshot_lines:
228+
print(f"ERROR: Could not extract actions.py content from {CONFIGMAP}.")
229+
print("Ensure the ConfigMap has a 'actions.py:' key under 'data:'.")
230+
sys.exit(1)
231+
232+
snapshot = "".join(snapshot_lines).rstrip("\n") + "\n"
233+
234+
try:
235+
with open(CANONICAL) as f:
236+
canonical = f.read()
237+
except FileNotFoundError:
238+
print(f"ERROR: {CANONICAL} not found.")
239+
sys.exit(1)
240+
241+
if snapshot == canonical:
242+
print(f"OK: {CONFIGMAP} actions.py snapshot matches {CANONICAL}.")
243+
else:
244+
import difflib
245+
diff = list(difflib.unified_diff(
246+
snapshot.splitlines(keepends=True),
247+
canonical.splitlines(keepends=True),
248+
fromfile=f"{CONFIGMAP}:actions.py",
249+
tofile=CANONICAL,
250+
))
251+
print("ERROR: ConfigMap actions.py snapshot is stale relative to config/rails/actions.py.")
252+
print("Regenerate the snapshot: copy config/rails/actions.py into the ConfigMap data block.")
253+
print("".join(diff[:80]))
254+
sys.exit(1)
255+
EOF
256+
180257
no-direct-bind-proof:
181258
name: "NoDirectBind State-Space Proof"
182259
runs-on: ubuntu-latest

CAGE_ARXIV.MD

Lines changed: 123 additions & 87 deletions
Large diffs are not rendered by default.

CHANGELOG.md

Lines changed: 59 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,66 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
99

1010
## [Unreleased]
1111

12+
### Added
13+
- `KmsSigner.sign()` now embeds `signed_at` Unix timestamp in every signed payload; `verify()` rejects payloads older than `MAX_KMS_PAYLOAD_AGE_SECONDS` (300 s), closing replay-attack vector.
14+
- `CbfGovernor._local_debits` intra-window debit ledger: `verify_action()` computes `effective_balance = snapshot - local_debits` to prevent double-spend within KMS TTL window; `reset_local_debits()` added for reconciliation daemon.
15+
- `ConsensusGate`: degraded-quorum routing (`ERROR + APPROVE → ESCALATE`) now explicitly handled before catch-all case.
16+
- `FiscalLimitGuard.rollback_state(amount, audit_id)`: Saga compensation stub — logs `[SAGA-ROLLBACK]`, reverses Redis debit, re-raises on failure.
17+
- `tests/test_provenance_chain.py`: `test_link_hash_is_deterministic` asserts hash stability across calls.
18+
19+
### Fixed
20+
- `CausalGatekeeper`: Redis connection errors are now fail-closed (raise `RuntimeError`) rather than returning a zero-deflection sentinel (fail-open). Absent keys remain first-boot safe.
21+
- Terminology: "TOCTOU gap" for the rollback atomicity issue renamed to "saga-atomicity gap" throughout docs and paper.
22+
- Redis access model for gateway corrected in documentation: gateway has read-write access (Tier 4 FiscalLimitGuard uses `WATCH/MULTI/EXEC`), not read-only as previously documented.
23+
24+
### Documentation
25+
- `CAGE_ARXIV.MD`: 58 peer-review items addressed — bibliography fixes, formal-proof caveats (under-approximation, saga-atomicity, CBF conditional implication), security notes (FTRA trust boundary, replay vulnerability, intra-window double-spend), new Appendix D (adversarial payload examples), expanded roadmap (NoDirectBind, POAM-TIER2-001, FTRA formal verification).
26+
1227
---
1328

29+
## [Unreleased — prior]
30+
31+
### Added
32+
33+
- `src/compliance_bridge/reconciliation_worker.py``ObjectStoreLedgerProvider` (S3-compatible via boto3: AWS S3, GCS S3 Interop, MinIO, Ceph). Registered `"s3"` and `"object-store"` aliases in the `_PROVIDERS` factory.
34+
- `deployment/k8s/reconciliation-worker.yaml` — new CronJob manifest running `ExternalLedgerReconciler` every 5 minutes; default `RECONCILIATION_PROVIDER` changed to `"s3"`; added `S3_RECONCILIATION_BUCKET`, `S3_ENDPOINT_URL`, `S3_REGION_NAME`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` env vars; CiliumNetworkPolicy egress extended to `*.amazonaws.com`.
35+
- `docs/POAM.md` — added POAM-2026-038 through -042.
36+
37+
### Fixed
38+
39+
- `src/gateway/governance/fiscal_limit_guard.py` — per-reservation TTL sentinel key `fiscal:reservation:{uuid}` (`ex=reservation_ttl`, default 300 s) bounds the crash-leakage window between `reserve()` and `confirm()`/`release()`.
40+
- `src/gateway/governance/routing_seal.py``generate_seal()` / `_canonical_payload()` sanitize dots (`.replace(".", "-")`) in the action slug to guarantee an unambiguous 3-part `.` split during `verify_seal()`.
41+
- `src/compliance_bridge/context_accumulator.py``_content_hash()` now passes `separators=(",", ":")` to `json.dumps()` for canonical, whitespace-free serialization.
42+
- `src/gateway/governance/causal_gatekeeper.py` — added `_MIN_CAUSAL_SAMPLES` guard (default 30, overridable via `CAUSAL_MIN_SAMPLES`) before `backdoor.linear_regression` to fail closed on sparse telemetry.
43+
- `src/governed_financial_advisor/graph/nodes/safety_node.py` — replaced hardcoded zero sentinels for `drawdown`, `order_size`, `daily_vol` with `_fetch_live_risk_metrics()`, reading live values from Redis (`cbf:portfolio_drawdown:{account_id}`, `portfolio:daily_vol:{account_id}`) with 200 ms socket timeout and safe-sentinel fallback.
44+
- `scripts/measure_paper_metrics.py` — re-enabled `measure_ungoverned_baseline()`.
45+
- `scripts/measure_reconciliation_metrics.py``_make_sync_redis()` now honours `REDIS_PASSWORD`.
46+
47+
### Changed
48+
49+
- `refactor(nemo): consolidate GFA LLMRails to single harness singleton`
50+
- Added `reload_nemo_rails(config_path)` (async, `asyncio.Lock`-guarded) and
51+
`_get_reload_lock()` to
52+
`src/gateway/governance/langgraph_harness/nemo_node_factory.py`; the
53+
module-level `_nemo_rails` singleton is now the sole `LLMRails` instance
54+
for the entire GFA pod.
55+
- Removed the module-level `rails = load_rails()` global and all
56+
`global rails` declarations from `src/governed_financial_advisor/server.py`;
57+
both hot-reload endpoints (`/v1/nemo/propose-refinement` and
58+
`/v1/nemo/approve-refinement/{id}`) now call `await reload_nemo_rails()`
59+
from the harness instead of maintaining their own `LLMRails` instance.
60+
- Removed the `_rails` singleton and `get_rails()` helper from
61+
`src/governed_financial_advisor/tools/api.py`; it now calls
62+
`get_nemo_rails()` from the harness directly.
63+
- Net result: one `LLMRails` instance per GFA pod (down from three); a
64+
single approved refinement now propagates to every consumer
65+
simultaneously instead of only the instance it was applied against.
66+
- Quarantined `infra/modules/nemo_guardrails/main.tf` with a
67+
`HISTORICAL-ONLY — DO NOT APPLY` banner (predates and diverges from the
68+
canonical `config/rails/` source) and added a `nemo-freshness-check` CI
69+
job (`.github/workflows/ci.yml`) that diffs `config/rails/actions.py`
70+
against `deployment/k8s/nemo-rails-configmap.yaml`.
71+
1472
## [v2.1.1-post — 2026-08-05]
1573

1674
> Post-release fixes and paper measurement improvements. No version bump — these
@@ -37,7 +95,7 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
3795
of conditionally passing them from the LLM execution plan. Closes UCA-5/UCA-2
3896
100% benign `trade_execution` FPR (75.0% → 0.0%). **Architectural invariant:**
3997
safety enforcement is purely deterministic LangGraph node execution — never
40-
dependent on LLM plan output (`fix(governance)`).
98+
dependent on LLM plan output (`fix(governance)`). *(Superseded 2026-08-06: `drawdown`/`daily_vol` are now read live from Redis with sentinel fallback — see `[Unreleased]`.)*
4199
- `config/rails/actions.py` — added Stage 1C structural-attack blocklist inside
42100
`custom_self_check_input()` between Stage 1B (illegal-finance) and Stage 2
43101
(allowlist). Stage 1C blocks SQL injection markers (`;`, `--`, `'; DROP`,
@@ -58,8 +116,6 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
58116

59117
---
60118

61-
## [Unreleased — prior]
62-
63119
### Added
64120
- `.github/CODEOWNERS` — single-maintainer review enforcement for architectural paths
65121
- `.github/pull_request_template.md` — reference implementation verification checklist

COMPLIANCE.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ h(S(t+1)) ≥ (1−γ) · h(S(t)), γ ∈ (0,1)
108108

109109
This guarantees the cash balance never drops below the minimum threshold in a single step. The decay factor `γ` bounds the maximum permissible drawdown per evaluation cycle. External CBF state reconciliation is implemented via [`src/compliance_bridge/reconciliation_worker.py`](src/compliance_bridge/reconciliation_worker.py)**POAM-2026-031 closed 2026-07-27**. Reconciled balances are KMS-signed before Redis write; the CBF fails closed on TTL expiry.
110110

111+
> **CBF intra-window hardening:** `CbfGovernor.verify_action()` now computes `effective_balance = snapshot_balance - self._local_debits`, where `_local_debits` accumulates approved-trade costs within the current 300 s KMS snapshot TTL window. `reset_local_debits()` is called by the reconciliation daemon on each snapshot refresh. This closes the intra-window double-spend gap previously documented as a limitation.
112+
111113
### 2.2 Confabulation Risk Formula
112114

113115
**Source:** [`src/gateway/governance/confabulation_scorer.py`](src/gateway/governance/confabulation_scorer.py) · **Control:** `CTRL_AGT_001`
@@ -156,6 +158,8 @@ Scores below `FRIA_ZONE_DEFER` trigger a local hard deny without invoking the ex
156158
| Contention strategy | Exponential backoff |
157159
| Redis failure mode | Fail-closed (blocks request) |
158160

161+
> **Consensus degraded-quorum routing:** The `ERROR + APPROVE` verdict combination is now explicitly routed to `ESCALATE` (HITL) before the catch-all case.
162+
159163
### 2.6 Provenance Hash Chain Integrity
160164

161165
**Source:** [`src/gateway/governance/provenance_chain.py`](src/gateway/governance/provenance_chain.py) · **Control:** `CTRL_CTX_007`
@@ -180,6 +184,10 @@ HMAC-SHA256 token format:
180184

181185
TTL: **30 seconds**. Unsigned or expired requests return HTTP 403.
182186

187+
### 2.8 FTRA Reachability Verification Gap (Documented)
188+
189+
> **FTRA verification gap (documented):** Tier 0.5 (FTRA) is not included in the 21-state BFS automaton. This is a documented under-approximation. Gap closure requires integrating FTRA into the state tuple or enforcing it at the controller boundary.
190+
183191
---
184192

185193
## 3. STPA Unsafe Control Actions (UCAs)
@@ -209,7 +217,9 @@ Full STPA hazard analysis (UCAs 1–9, Saga pattern, FiscalLimitGuard): [`docs/s
209217
### B. ISO/IEC 42001 & DORA (Digital Operational Resilience Act)
210218
* **Status:** Technical Controls Implemented & Observable.
211219
* **Mechanism:**
212-
* **Transaction Atomicity (DORA Art. 12):** The LangGraph SAGA Write-Ahead Log (WAL) pattern isolates tool calls and model actions, guaranteeing LIFO (Last-In, First-Out) rollbacks during system or execution faults to prevent partial "ghost states" in ledger positions.
220+
* **Transaction Atomicity (DORA Art. 12):** The LangGraph SAGA Write-Ahead Log (WAL) pattern isolates tool calls and model actions, guaranteeing LIFO (Last-In, First-Out) rollbacks during system or execution faults to prevent partial "ghost states" in ledger positions. A residual **saga-atomicity gap** exists where Tier 3a commitment precedes downstream tier execution; this is addressed by the compensation stub described below.
221+
222+
* **Saga compensation stub:** `FiscalLimitGuard.rollback_state(amount, audit_id)` reverses the Redis debit if a downstream tier fails after Tier 3a commitment. This implements the Saga pattern compensation step and closes the residual saga-atomicity gap.
213223
* **Continuous Telemetry Validation (DORA Art. 10):** Real-time Langfuse OpenTelemetry spans are piped through the placebo refuter at runtime to verify that the agent's world-model matches execution reality, rather than drifting on synthetic variables.
214224
* **Tamper-Proof Audit Logging:** All decisions and system exceptions generate a cryptographically hash-chained SHA-256 ledger (`cage-intent/1.0`) to satisfy strict non-repudiation and lifecycle logging policies.
215225
* **Companion Documentation:**
@@ -235,6 +245,8 @@ Full STPA hazard analysis (UCAs 1–9, Saga pattern, FiscalLimitGuard): [`docs/s
235245
* **Zero-Trust Network Hardening:** Deploys Linkerd SPIFFE/SVID mTLS for cryptographic workload validation (**POAM-007 / IA-3**, closed 2026-05-17) and Cilium Layer 7 network policies for default-deny egress lockdown (**POAM-011 / SC-8**, Open). Both controls are technically active in the `governance-stack` Kubernetes namespace; POAM-011 (SC-8) and POAM-012 (SC-12) remain Open pending formal assessment closure.
236246
* **Programmatic Evidence:** The automated script `oscal_ssp_exporter.py` automatically compiles these exact control configurations and implementation narratives into the authoritative 1,330-line Open Security Controls Assessment Language (OSCAL) document on every build pipeline run. OSCAL artifacts are persisted to GCS using the native GCS SDK (boto3 S3-compat fallback) at schema version **OSCAL v1.0.4**.
237247
* **KMS Batch Signing for Audit Evidence:** All OSCAL findings and AARM conformance reports are asymmetrically signed via Google Cloud KMS HSM (`src/gateway/governance/kms_signer.py`) before GCS persistence. The private key never leaves the HSM; Cloud Audit Logs provide external, immutable attestation of every signing operation. This constitutes the audit evidence chain for FedRAMP HIGH AU-9 and AU-10.
248+
249+
* **KMS replay-attack closure:** `KmsSigner.sign()` now embeds `"signed_at": int(time.time())` in every signed payload. `KmsSigner.verify()` raises `ValueError` if `now - signed_at > 300 s`. This closes the replay-attack vector where a compromised agent with Redis write access could reset the 300 s TTL indefinitely.
238250
* **⚠️ Gaps to Authorization:** The CAGE software runtime does not inherently possess an official **Authority to Operate (ATO)**. To close this loop, the parent organization must deploy independent assessors to complete RMF Step 5 (Assess) and Step 6 (Authorize), as well as remediate the remaining 11 open infrastructure POA&M infrastructure tickets.
239251
* **Companion Documentation:** For infrastructure configurations, Linkerd policy files, and security posture tracking, see [docs/SECURITY_STATUS.md](docs/security/SECURITY_STATUS.md) and [docs/POAM.md](docs/compliance/cross-region/POAM.md).
240252

0 commit comments

Comments
 (0)