Skip to content

Commit 51a3b01

Browse files
authored
feat(security): pilot hardening — stateless guarantee, model pinning, CI security gate (headroomlabs-ai#1515)
## Description Engineering hardening derived from the Box vendor security assessment. Each change turns a "No/Partial" questionnaire answer into a genuine "Yes" by making the product safer — not by editing the form. The throughline is Headroom's core promise to enterprise pilots: **it runs inside the customer's environment and never persists or leaks their data.** These changes make that provable. Three themes: (1) a complete **stateless write guarantee** (a stateless proxy writes nothing to the workspace during serving), (2) **data-at-rest** protection (no cleartext prompts written on errors), and (3) **supply-chain integrity** (all model downloads pinned; SCA/SAST/secret-scanning in CI). Closes # (no tracking issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) > Note: two deliberate, reversible default changes (not breaking): the upstream-error debug dump is now off by default (`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a pure no-op when not stateless. ## Changes Made - **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`), and the output-savings recorder now honor stateless (in-memory only); persistent memory is disabled under stateless with a warning. Added a process-wide flag `headroom.paths.process_is_stateless()` (also honors `HEADROOM_STATELESS`). - **Debug dump** — the Anthropic *and* OpenAI handlers wrote full requests (cleartext prompts/tools/system) to `~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by default, stateless-aware, with a redacted middle tier; helpers extracted to `handlers/_debug_dump.py`. - **Model pinning** — all model downloads pin an immutable commit SHA: our repos, kompress, image router/SigLIP, the third-party Qdrant memory embedder (centralized in `onnx_runtime`), and the fastembed relevance model (via the `revision` kwarg fastembed forwards to `snapshot_download`). `HEADROOM_HF_PIN=off` bypasses. - **CI security gate** — new `security.yml`: dependency audit (pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python + JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff scoped). `.gitleaks.toml` allowlists SBOM/lockfiles. - **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins, docs). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check <10 changed source files> All checks passed! $ mypy <changed source files> Success: no issues found in 7 source files # + handlers/server: no issues (annotation-unchecked notes only) $ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \ tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \ tests/test_proxy_savings_history.py tests/test_observability_metrics.py \ tests/test_toin.py tests/test_paths.py -q ================= 176 passed, 6 skipped, 2 warnings in 13.92s ================== ``` New tests (18): `tests/test_stateless_writers.py`, `tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`, `tests/test_hf_revision_pinning.py`, plus stateless control assertions in `tests/test_proxy_savings_history.py`. They include the non-stateless control cases (savings/TOIN still persist) and a regression guard that fails if any handler writes a debug dump without gating it. ## Real Behavior Proof - **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`. - **Exact command / steps:** - Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` + `set_process_stateless(True)` → `flush()`; TOIN `ToolIntelligenceNetwork(TOINConfig(storage_path=""))`; `create_app(ProxyConfig(memory_enabled=True, stateless=True))`. - Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))` across env values. - Model pinning: model SHAs fetched/verified against the live HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers tested. - **Observed result:** under stateless, no `proxy_savings.json` / `savings_events.jsonl` / `toin.json` / `output_savings.json` / `memory.db` is created; `proxy.memory_handler is None`. With `stateless=False` the control tests confirm each still persists. Debug dump resolves to `off` by default and is forced off in stateless. CI: dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the gitleaks binary. - **Not tested:** `pip-audit` was not run on the local machine (broken `ensurepip`); CI is the first real run (the committed all-extras grype scan is clean). The fastembed download path is exercised by CI/runtime, not in unit tests (the revision resolver is unit-tested). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - **Concurrency:** `stateless` is a per-process config flag, never per-request/per-session. Many sessions share one proxy's setting; a stateless and a stateful proxy are separate OS processes with isolated state. The one in-process edge (two proxies, different settings — essentially tests) fails closed to in-memory, so a stateless proxy can never leak. - **Memory under stateless** is *disabled* (not in-RAM): the memory subsystem is multi-component (SQLite + vector + markdown bridge) and a partial in-RAM mode would be risky; ephemeral containers and cross-session learning are contradictory. An ephemeral in-RAM memory mode is a possible follow-up. - **Docs/CHANGELOG** left unchecked: the two new env vars (`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior changes are documented in code comments; happy to add user docs + a CHANGELOG entry if preferred. - CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side and out of scope here.
1 parent 5771a80 commit 51a3b01

18 files changed

Lines changed: 884 additions & 101 deletions

.github/dependabot.yml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,35 @@ updates:
4040
update-types:
4141
- "minor"
4242
- "patch"
43+
44+
# Rust dependency updates (cargo workspace: crates/*)
45+
- package-ecosystem: cargo
46+
directory: /
47+
schedule:
48+
interval: weekly
49+
commit-message:
50+
prefix: "deps"
51+
open-pull-requests-limit: 5
52+
groups:
53+
cargo-minor-patch:
54+
update-types:
55+
- "minor"
56+
- "patch"
57+
58+
# npm dependency updates (TS SDK, plugins, docs site)
59+
- package-ecosystem: npm
60+
directories:
61+
- "/sdk/typescript"
62+
- "/plugins/openclaw"
63+
- "/plugins/opencode"
64+
- "/docs"
65+
schedule:
66+
interval: weekly
67+
commit-message:
68+
prefix: "deps"
69+
open-pull-requests-limit: 5
70+
groups:
71+
npm-minor-patch:
72+
update-types:
73+
- "minor"
74+
- "patch"

.github/workflows/security.yml

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
name: Security
2+
3+
# Security gate: dependency vulnerability scanning (SCA), static analysis
4+
# (CodeQL/SAST), and secret scanning. Runs on every PR to main, on push to
5+
# main, weekly (to catch newly-disclosed CVEs without a code change), and on
6+
# demand. Each job is an independent required check.
7+
8+
on:
9+
push:
10+
branches: [main]
11+
pull_request:
12+
branches: [main]
13+
schedule:
14+
# Mondays 06:00 UTC — surface CVEs disclosed since the last commit.
15+
- cron: "0 6 * * 1"
16+
workflow_dispatch:
17+
18+
permissions:
19+
contents: read
20+
21+
concurrency:
22+
group: security-${{ github.workflow }}-${{ github.ref }}
23+
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
24+
25+
jobs:
26+
# ---- SCA: dependency vulnerability scan -------------------------------
27+
# Audits the PRODUCTION dependency set ([all]) exported from uv.lock. The
28+
# `benchmark` extra is intentionally excluded from [all] (it pulls lm-eval's
29+
# sqlitedict/nltk, which carry unpatchable upstream High CVEs and are never
30+
# installed in production), so this gate fails only on actionable findings.
31+
dependency-audit:
32+
name: Dependency audit (pip-audit)
33+
runs-on: ubuntu-latest
34+
timeout-minutes: 15
35+
steps:
36+
- uses: actions/checkout@v4
37+
38+
- name: Set up Python
39+
uses: actions/setup-python@v5
40+
with:
41+
python-version: "3.12"
42+
43+
- name: Install uv
44+
uses: astral-sh/setup-uv@v5
45+
46+
- name: Export production dependency set from uv.lock
47+
run: |
48+
uv export --frozen --no-dev --no-emit-project --no-hashes \
49+
--extra all --format requirements-txt > requirements-prod.txt
50+
echo "Production dependencies audited:"
51+
wc -l requirements-prod.txt
52+
53+
- name: Audit dependencies (pip-audit)
54+
uses: pypa/gh-action-pip-audit@v1.1.0
55+
with:
56+
inputs: requirements-prod.txt
57+
58+
# ---- SAST: CodeQL static analysis ------------------------------------
59+
codeql:
60+
name: CodeQL (${{ matrix.language }})
61+
runs-on: ubuntu-latest
62+
timeout-minutes: 30
63+
permissions:
64+
contents: read
65+
security-events: write
66+
actions: read
67+
strategy:
68+
fail-fast: false
69+
matrix:
70+
language: [python, javascript-typescript]
71+
steps:
72+
- uses: actions/checkout@v4
73+
74+
- name: Initialize CodeQL
75+
uses: github/codeql-action/init@v3
76+
with:
77+
languages: ${{ matrix.language }}
78+
queries: security-extended
79+
80+
- name: Perform CodeQL analysis
81+
uses: github/codeql-action/analyze@v3
82+
with:
83+
category: "/language:${{ matrix.language }}"
84+
85+
# ---- Secret scanning -------------------------------------------------
86+
# Uses the gitleaks BINARY (MIT-licensed, no key) instead of
87+
# gitleaks-action, which requires a paid GITLEAKS_LICENSE for organization
88+
# repos. On PRs we scan only the PR's commits so pre-existing history can't
89+
# block a PR; on push/schedule we scan the working tree. Config + allowlist
90+
# live in .gitleaks.toml at the repo root (auto-loaded).
91+
secret-scan:
92+
name: Secret scan (gitleaks)
93+
runs-on: ubuntu-latest
94+
timeout-minutes: 10
95+
steps:
96+
- uses: actions/checkout@v4
97+
with:
98+
fetch-depth: 0
99+
100+
- name: Install gitleaks
101+
run: |
102+
version=8.18.4
103+
curl -sSfL \
104+
"https://github.com/gitleaks/gitleaks/releases/download/v${version}/gitleaks_${version}_linux_x64.tar.gz" \
105+
-o /tmp/gitleaks.tar.gz
106+
tar -xzf /tmp/gitleaks.tar.gz -C /tmp gitleaks
107+
sudo install /tmp/gitleaks /usr/local/bin/gitleaks
108+
gitleaks version
109+
110+
- name: Scan for secrets
111+
env:
112+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
113+
run: |
114+
if [ -n "$BASE_SHA" ]; then
115+
echo "Scanning PR commits ${BASE_SHA}..HEAD"
116+
gitleaks detect --source . --log-opts="${BASE_SHA}..HEAD" --redact --no-banner
117+
else
118+
echo "Scanning working tree"
119+
gitleaks detect --source . --no-git --redact --no-banner
120+
fi

.gitleaks.toml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# gitleaks configuration — extends the tuned default ruleset and allowlists
2+
# paths that contain hashes/identifiers (not real secrets) to avoid false
3+
# positives. Used by the Security workflow's secret-scan job.
4+
5+
[extend]
6+
useDefault = true
7+
8+
[allowlist]
9+
description = "Non-secret artifacts: SBOMs, lockfiles, and vendored hashes."
10+
paths = [
11+
'''sbom/.*''',
12+
'''.*\.lock$''',
13+
'''.*package-lock\.json$''',
14+
'''pnpm-lock\.yaml$''',
15+
]

headroom/onnx_runtime.py

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,42 @@
33
from __future__ import annotations
44

55
import ctypes
6+
import os
67
import sys
78
from typing import Any
89

9-
10-
def hf_hub_download_local_first(repo_id: str, filename: str, *, allow_network: bool = True) -> str:
10+
# Pin model artifacts to immutable commit SHAs so a changed or compromised
11+
# upstream HuggingFace repo cannot be pulled silently (supply-chain integrity).
12+
# Repos not listed here fall back to the floating default ref. Set
13+
# HEADROOM_HF_PIN=off to bypass pinning (e.g. when intentionally evaluating a
14+
# newer model revision). To upgrade a model, bump its SHA here deliberately.
15+
_PINNED_REVISIONS: dict[str, str] = {
16+
# chopratejas/kompress-v2-base @ 2026-06-10
17+
"chopratejas/kompress-v2-base": "b1563631b35bfdcee37587ad530147497d820d4c",
18+
"chopratejas/technique-router-onnx": "27b0b4bfa510a1cff66d888072c0b807082721a8",
19+
"chopratejas/siglip-image-encoder-onnx": "d0a9fbd66d4bd8c761bff592d44831f7c2ae184e",
20+
# Third-party repo — pinning matters most here.
21+
"Qdrant/all-MiniLM-L6-v2-onnx": "5f1b8cd78bc4fb444dd171e59b18f3a3af89a079",
22+
}
23+
24+
25+
def _resolve_revision(repo_id: str, revision: str | None) -> str | None:
26+
"""Resolve the HF revision to download: explicit arg wins, else the pinned
27+
SHA for a known repo, else ``None`` (floating ref)."""
28+
if revision is not None:
29+
return revision
30+
if os.environ.get("HEADROOM_HF_PIN", "").strip().lower() in ("off", "0", "false", "no"):
31+
return None
32+
return _PINNED_REVISIONS.get(repo_id)
33+
34+
35+
def hf_hub_download_local_first(
36+
repo_id: str,
37+
filename: str,
38+
*,
39+
allow_network: bool = True,
40+
revision: str | None = None,
41+
) -> str:
1142
"""Download a file from HuggingFace Hub, preferring the local cache.
1243
1344
Tries ``local_files_only=True`` first to avoid a network HEAD request when
@@ -21,6 +52,10 @@ def hf_hub_download_local_first(repo_id: str, filename: str, *, allow_network: b
2152
a cache miss re-raises the local-lookup error. Used by startup
2253
preload so a cold cache cannot block (or, via native crashes in the
2354
download stack, kill) the process before it binds its port.
55+
revision: Explicit git revision (commit SHA / tag / branch). When
56+
``None``, a pinned SHA is applied for known repos (see
57+
``_PINNED_REVISIONS``) for supply-chain integrity; unknown repos use
58+
the floating default ref.
2459
2560
Returns:
2661
Absolute path to the local cached file.
@@ -33,12 +68,14 @@ def hf_hub_download_local_first(repo_id: str, filename: str, *, allow_network: b
3368
from huggingface_hub import hf_hub_download
3469
from huggingface_hub.errors import EntryNotFoundError, LocalEntryNotFoundError
3570

71+
revision = _resolve_revision(repo_id, revision)
72+
3673
try:
37-
return str(hf_hub_download(repo_id, filename, local_files_only=True))
74+
return str(hf_hub_download(repo_id, filename, revision=revision, local_files_only=True))
3875
except (LocalEntryNotFoundError, EntryNotFoundError, OSError):
3976
if not allow_network:
4077
raise
41-
return str(hf_hub_download(repo_id, filename))
78+
return str(hf_hub_download(repo_id, filename, revision=revision))
4279

4380

4481
def create_cpu_session_options(

headroom/paths.py

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,35 @@ def _env(name: str) -> str:
9595
return os.environ.get(name, "").strip()
9696

9797

98+
# ---------------------------------------------------------------------------
99+
# Process-wide stateless flag
100+
# ---------------------------------------------------------------------------
101+
# Stateless mode forbids writes to the workspace. Many persisters are
102+
# module-level singletons reached without a config object, so the proxy records
103+
# the mode here once at startup and writers consult ``process_is_stateless()``.
104+
105+
_PROCESS_STATELESS: bool = False
106+
107+
108+
def set_process_stateless(value: bool) -> None:
109+
"""Record process-wide stateless mode (set once at proxy startup)."""
110+
111+
global _PROCESS_STATELESS
112+
_PROCESS_STATELESS = bool(value)
113+
114+
115+
def process_is_stateless() -> bool:
116+
"""True when the process must not write to the workspace.
117+
118+
True if ``set_process_stateless(True)`` was called OR the ``HEADROOM_STATELESS``
119+
environment variable is set, so non-proxy entrypoints honor it too.
120+
"""
121+
122+
if _PROCESS_STATELESS:
123+
return True
124+
return _env("HEADROOM_STATELESS").lower() in ("1", "true", "yes", "on")
125+
126+
98127
def _resolve(explicit: str | os.PathLike[str] | None, env_var: str, derived: Path) -> Path:
99128
"""Apply the standard precedence: explicit > env > derived.
100129
@@ -364,6 +393,8 @@ def plugin_workspace_dir(plugin_name: str) -> Path:
364393
"HEADROOM_SAVINGS_EVENTS_PATH_ENV",
365394
"HEADROOM_TOIN_PATH_ENV",
366395
"HEADROOM_SUBSCRIPTION_STATE_PATH_ENV",
396+
"set_process_stateless",
397+
"process_is_stateless",
367398
"config_dir",
368399
"workspace_dir",
369400
"ensure_config_dir",
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
"""Shared helpers for diagnostic dumps of upstream-error (>=400) requests.
2+
3+
The dump can contain cleartext prompt / tool / system content, so it is OFF by
4+
default, is never written in stateless mode, and content is redacted unless the
5+
operator explicitly opts in to full content. Used by both the Anthropic and the
6+
OpenAI handlers so the gating stays consistent across providers.
7+
"""
8+
9+
from __future__ import annotations
10+
11+
import os
12+
from typing import Any
13+
14+
15+
def _debug_dump_mode(config: Any) -> str:
16+
"""Return the diagnostic-dump mode for upstream (>=400) errors.
17+
18+
- "off" : nothing written (default, and forced in stateless mode)
19+
- "redacted" : structure, roles, and lengths only — content elided
20+
(``HEADROOM_DEBUG_DUMP=1``/``true``/``on``/``redacted``)
21+
- "full" : everything including content (``HEADROOM_DEBUG_DUMP=full``)
22+
"""
23+
if getattr(config, "stateless", False):
24+
return "off"
25+
raw = os.environ.get("HEADROOM_DEBUG_DUMP", "").strip().lower()
26+
if raw in ("full", "all", "content"):
27+
return "full"
28+
if raw in ("1", "true", "yes", "on", "redacted"):
29+
return "redacted"
30+
return "off"
31+
32+
33+
def _redact_debug_value(value: Any, _max_len: int = 80) -> Any:
34+
"""Recursively elide long strings (likely prompt/tool content) while keeping
35+
structure, roles, type tags, ids, and other short fields for debugging.
36+
37+
Note: short strings (<= ``_max_len``) are preserved, so the redacted tier is
38+
not a guarantee against leaking very short sensitive values — it is a
39+
best-effort convenience. The default ("off") writes nothing at all.
40+
"""
41+
if isinstance(value, str):
42+
return value if len(value) <= _max_len else f"<redacted: {len(value)} chars>"
43+
if isinstance(value, dict):
44+
return {k: _redact_debug_value(v, _max_len) for k, v in value.items()}
45+
if isinstance(value, list):
46+
return [_redact_debug_value(v, _max_len) for v in value]
47+
return value

0 commit comments

Comments
 (0)