Commit 51a3b01
authored
feat(security): pilot hardening — stateless guarantee, model pinning, CI security gate (headroomlabs-ai#1515)
## Description
Engineering hardening derived from the Box vendor security assessment.
Each change turns a "No/Partial" questionnaire answer into a genuine
"Yes" by making the product safer — not by editing the form. The
throughline is Headroom's core promise to enterprise pilots: **it runs
inside the customer's environment and never persists or leaks their
data.** These changes make that provable.
Three themes: (1) a complete **stateless write guarantee** (a stateless
proxy writes nothing to the workspace during serving), (2)
**data-at-rest** protection (no cleartext prompts written on errors),
and (3) **supply-chain integrity** (all model downloads pinned;
SCA/SAST/secret-scanning in CI).
Closes # (no tracking issue)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
> Note: two deliberate, reversible default changes (not breaking): the
upstream-error debug dump is now off by default
(`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are
pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a
pure no-op when not stateless.
## Changes Made
- **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`),
and the output-savings recorder now honor stateless (in-memory only);
persistent memory is disabled under stateless with a warning. Added a
process-wide flag `headroom.paths.process_is_stateless()` (also honors
`HEADROOM_STATELESS`).
- **Debug dump** — the Anthropic *and* OpenAI handlers wrote full
requests (cleartext prompts/tools/system) to
`~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by
default, stateless-aware, with a redacted middle tier; helpers extracted
to `handlers/_debug_dump.py`.
- **Model pinning** — all model downloads pin an immutable commit SHA:
our repos, kompress, image router/SigLIP, the third-party Qdrant memory
embedder (centralized in `onnx_runtime`), and the fastembed relevance
model (via the `revision` kwarg fastembed forwards to
`snapshot_download`). `HEADROOM_HF_PIN=off` bypasses.
- **CI security gate** — new `security.yml`: dependency audit
(pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python +
JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff
scoped). `.gitleaks.toml` allowlists SBOM/lockfiles.
- **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins,
docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check <10 changed source files>
All checks passed!
$ mypy <changed source files>
Success: no issues found in 7 source files # + handlers/server: no issues (annotation-unchecked notes only)
$ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \
tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \
tests/test_proxy_savings_history.py tests/test_observability_metrics.py \
tests/test_toin.py tests/test_paths.py -q
================= 176 passed, 6 skipped, 2 warnings in 13.92s ==================
```
New tests (18): `tests/test_stateless_writers.py`,
`tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`,
`tests/test_hf_revision_pinning.py`, plus stateless control assertions
in `tests/test_proxy_savings_history.py`. They include the non-stateless
control cases (savings/TOIN still persist) and a regression guard that
fails if any handler writes a debug dump without gating it.
## Real Behavior Proof
- **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`.
- **Exact command / steps:**
- Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` +
`set_process_stateless(True)` → `flush()`; TOIN
`ToolIntelligenceNetwork(TOINConfig(storage_path=""))`;
`create_app(ProxyConfig(memory_enabled=True, stateless=True))`.
- Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))`
across env values.
- Model pinning: model SHAs fetched/verified against the live
HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers
tested.
- **Observed result:** under stateless, no `proxy_savings.json` /
`savings_events.jsonl` / `toin.json` / `output_savings.json` /
`memory.db` is created; `proxy.memory_handler is None`. With
`stateless=False` the control tests confirm each still persists. Debug
dump resolves to `off` by default and is forced off in stateless. CI:
dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the
gitleaks binary.
- **Not tested:** `pip-audit` was not run on the local machine (broken
`ensurepip`); CI is the first real run (the committed all-extras grype
scan is clean). The fastembed download path is exercised by CI/runtime,
not in unit tests (the revision resolver is unit-tested).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- **Concurrency:** `stateless` is a per-process config flag, never
per-request/per-session. Many sessions share one proxy's setting; a
stateless and a stateful proxy are separate OS processes with isolated
state. The one in-process edge (two proxies, different settings —
essentially tests) fails closed to in-memory, so a stateless proxy can
never leak.
- **Memory under stateless** is *disabled* (not in-RAM): the memory
subsystem is multi-component (SQLite + vector + markdown bridge) and a
partial in-RAM mode would be risky; ephemeral containers and
cross-session learning are contradictory. An ephemeral in-RAM memory
mode is a possible follow-up.
- **Docs/CHANGELOG** left unchecked: the two new env vars
(`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior
changes are documented in code comments; happy to add user docs + a
CHANGELOG entry if preferred.
- CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side
and out of scope here.1 parent 5771a80 commit 51a3b01
18 files changed
Lines changed: 884 additions & 101 deletions
File tree
- .github
- workflows
- headroom
- proxy
- handlers
- relevance
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
11 | 42 | | |
12 | 43 | | |
13 | 44 | | |
| |||
21 | 52 | | |
22 | 53 | | |
23 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
24 | 59 | | |
25 | 60 | | |
26 | 61 | | |
| |||
33 | 68 | | |
34 | 69 | | |
35 | 70 | | |
| 71 | + | |
| 72 | + | |
36 | 73 | | |
37 | | - | |
| 74 | + | |
38 | 75 | | |
39 | 76 | | |
40 | 77 | | |
41 | | - | |
| 78 | + | |
42 | 79 | | |
43 | 80 | | |
44 | 81 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
98 | 127 | | |
99 | 128 | | |
100 | 129 | | |
| |||
364 | 393 | | |
365 | 394 | | |
366 | 395 | | |
| 396 | + | |
| 397 | + | |
367 | 398 | | |
368 | 399 | | |
369 | 400 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
0 commit comments