feat(linux): linux BPF-LSM mediation filter (closes exec + read bypasses)

[drewmchugh]

Summary

Replaces the seccomp-unotify exec filter from #20 with a BPF-LSM mediation filter that moves the mediation decision into the kernel (closing the TOCTOU race a sibling thread sharing the trapped task's memory could exploit), and adds a file_open LSM hook so the agent cannot read the mediated binary's bytes at all — closing copy-the-binary, ld-linux trick, unprivileged-tmpfs, and shellcode bypasses in addition to the original direct-path exec bypass #20 was scoped to.

This is a fork-and-replace of #20 rather than a follow-on commit: the seccomp-unotify filter, its supervisor handler, and the shebang-chain walker are deleted; BPF-LSM is the sole enforcement path for mediated commands. Sessions where mediation is active hard-fail at startup if BPF-LSM isn't installable, surfacing the specific reason (missing kernel cmdline / caps).

Update (commits a5b6a85, 00063f4, 378f12c). The same BPF-LSM infrastructure now also enforces filesystem subtree deny-within-allow on Linux — matching macOS Seatbelt's (deny file-write* (subpath ...)) semantics for policy.add_deny_access and the required policy-group denies. The kernel side adds eight inode_* LSM hooks plus a dentry parent-chain walk on file_open; the userspace side routes the existing policy.add_deny_access and required policy-group denies into a new protected_roots BPF map. See the new Filesystem subtree deny section below.

How the kernel side decides

Two BPF-LSM hooks share a single (dev, ino) deny map and a cgroup-ancestor scope check:

• bprm_check_security — fires after the kernel has resolved the binary the call will actually load (bprm->file); a -EACCES return atomically aborts the exec syscall, with no race against any user-memory pointer the agent controlled.
• file_open — fires inside do_filp_open for every successful path resolution that yields a file descriptor; denying opens of mediated inodes prevents the agent from reading the binary's bytes through any path (cp / cat / mmap / splice / dynamic linker).

Per-session scoping by cgroup ancestry: the broker creates a per-session cgroup at session start, the agent joins it as its first post-fork action, and the BPF program walks the calling task's cgroup ancestor chain looking for the session cgroup id. Cgroup membership inherits on fork() and is unaffected by reparenting, so daemonized agent descendants stay scoped. Ancestry (not equality) closes a sub-cgroup-escape where an agent with CAP_SYS_ADMIN could mkdir a child cgroup of the session and move into it.

Bypass classes closed

#	Bypass	Closed by
1	Direct path: /usr/bin/gh	bprm_check_security denies the exec
2	TOCTOU on user-memory paths (incl. vfork-bomb)	decision is in-kernel against bprm->file, not user-memory pointers
3	Copy the binary then exec the copy	file_open denies the read of mediated bytes
4	Indirect load: /lib/ld-linux-x86-64.so.2 /usr/bin/gh	file_open denies the linker's open()
5	Unprivileged tmpfs + copy + exec	file_open denies the read at the source
6	Shellcode (read bytes, mmap PROT_EXEC, jump)	file_open denies the source read
7	Hardlink at non-deny-set path	identity is (dev, ino), hardlinks share the inode
8	Daemonize / reparent to break parent-chain identity	scope is cgroup membership, not parent chain
9	Sub-cgroup escape (privileged agent only)	scope check walks ancestor chain
10	Read of a path inside a protected subtree	extended file_open walks dentry parents against protected_roots
11	Write to a file inside a protected subtree	same file_open path (open-for-write)
12	mmap / splice of a protected file	same file_open (mmap setup opens the file)
13	unlink / rmdir / rename of a protected child	new inode_* hooks return -EACCES
14	Create file/dir/symlink/link inside protected	inode_create / inode_mkdir / inode_symlink / inode_link
15	chmod / chown / truncate / utimes on protected	inode_setattr
16	Access via a bind mount of a non-protected source over a protected target	userspace populator inserts both directly-listed inodes AND bind-mount source roots into protected_roots

The vfork-bomb residual #20 left open empirically went from 49/600 bypasses (under the seccomp-unotify filter on this branch with cgroup scoping but no BPF) to 0/600 under BPF-LSM. The pthread variant remains 0/600.

Deployment requirements

Requirement	How
Active LSM stack	lsm=...,bpf in kernel cmdline. The host kernel needs CONFIG_BPF_LSM=y; verify via grep bpf_lsm_bprm_check_security /proc/kallsyms.
Broker capabilities	setcap cap_bpf,cap_sys_admin,cap_dac_override+ep /usr/bin/nono. CAP_BPF for the BPF program load; CAP_SYS_ADMIN for the cgroup-namespace privilege check on mkdir; CAP_DAC_OVERRIDE because cgroup v2 mkdir runs the VFS DAC check before the cgroup-namespace privilege check, and CAP_SYS_ADMIN does not subsume DAC.
Agent invariants	PR_SET_NO_NEW_PRIVS=1 set before agent execve (existing nono behaviour) so file caps don't apply. The broker also verifies CapEff: 0 on the agent post-execve and emits an explicit invariants log line at session start.

If any requirement is unmet, the broker fails session start with an explicit error pointing at the specific fix (kernel cmdline / caps / config). There is no silent partial enforcement.

Filesystem subtree deny (extension)

The first BPF-LSM hook pair gates binaries by inode. The same infrastructure now extends to gating filesystem subtrees — the same problem macOS Seatbelt solves with (deny file-write* (subpath ...)) rules.

Why this exists

Three pre-existing Linux gaps motivated this:

• validate_caps_against_protected_roots hard-rejected on Linux when a profile granted a parent of ~/.nono (e.g. --workdir=$HOME), because Landlock can't express deny-within-allow. macOS handled the same case via Seatbelt deny rules. The user-visible symptom (Jira WRK-2585): claude from $HOME on a workspace fails sandbox init with nono: Sandbox initialization failed: Refusing to grant '/home/bits' (source: Profile) because it overlaps protected nono state root '/home/bits/.nono'.
• policy.add_deny_access (crates/nono-cli/src/profile/mod.rs:98) was a no-op on Linux — add_deny_access_rules short-circuited at if cfg!(target_os = "macos"). shadowfax's Linux profile declares add_deny_access: ["$HOME/.config/password-store/gpg"] and the deny was silently dropped.
• validate_deny_overlaps rejected any session whose allow set covered a required policy-group deny (deny_credentials, deny_keychains_linux, …), forcing Linux profiles to enumerate their allows narrowly enough to dodge the overlap.

All three are now resolved by routing the deny-within-allow enforcement through BPF-LSM at the kernel level.

Kernel side (commit a5b6a85)

A new protected_roots BPF_MAP_TYPE_HASH (capacity 64, keyed by (dev, ino)). dentry_in_protected_subtree walks up to MAX_DENTRY_DEPTH=16 levels via BPF_CORE_READ on d_parent, verifier-friendly with the same #pragma unroll shape as the existing cgroup walker. check_file_open extended to also consult the new map; eight new SEC blocks — inode_unlink, inode_rmdir, inode_rename, inode_create, inode_mkdir, inode_symlink, inode_link, inode_setattr — cover the structural mutations that don't go through file_open. Two new audit reason codes (protected_open_deny, protected_mutate_deny) carry through the existing ringbuf and JSONL pipeline.

bpf_d_path was considered and rejected: it returns mount-namespace-aware paths (the wrong granularity for the bind-mount case below), and several of the new hooks operate on dentry directly with no struct path to hand to it.

Userspace side (commit 00063f4)

• protected_paths.rs: target_os = "macos" gate dropped from validate_requested_path_against_protected_roots. With allow_parent_of_protected: true set on the profile, both platforms admit the parent grant; the OS sandbox layer enforces the subtree at runtime (Seatbelt rules on macOS, BPF-LSM hooks on Linux). New bpf_lsm_protected_roots_for_session(state_roots, add_deny_access, policy_group_denies) merges all three deny sources, runs add_deny_access strings through policy::expand_path so $HOME/... and ~/... placeholders become concrete paths the populator can stat (without the expansion the literal string would never make it into the kernel map — bug caught during end-to-end validation).
• policy.rs: validate_deny_overlaps no-ops on Linux when BPF-LSM is available — the kernel hooks return -EACCES for any in-subtree access regardless of what Landlock allows, so the silent-drop motivation no longer holds. Hosts without BPF-LSM (no lsm=...,bpf in cmdline, no cap_bpf) still hit the existing rejection.
• bpf_lsm.rs: install_mediation_filter takes a protected_paths: &[PathBuf] slice. The populator stats each canonical path, inserts (dev, ino), then parses /proc/self/mountinfo to enumerate bind-mount source roots mounted at-or-under any protected path — needed because the dentry parent walk follows the source filesystem tree, not the mount tree (e.g. shadowfax's ~/.nono/sessions bind-mount). Both the directly-listed inode and the bind-mount source inode go into the same map; the BPF walker doesn't care which.
• exec_strategy.rs: BPF-LSM now installs when EITHER the exec deny set OR the protected_roots set is non-empty (a profile with no mediation.commands but with add_deny_access or any policy-group deny still gets enforcement).
• Plumbing: protected_paths flows through PreparedSandbox → ExecutionFlags → SupervisedRuntimeContext → SupervisorConfig → install_mediation_filter. Audit log dir falls back to ~/.nono/sessions when only protected_paths is active (no exec mediation session).
• 12 new integration tests in crates/nono-cli/tests/bpf_lsm_protected_subtree.rs covering read / write / mmap / unlink / rmdir / rename / create / bind-mount / regression / ~/.nono / add_deny_access / opt-in session start.

Documentation (commit 378f12c)

• docs/linux-bpf-lsm-mediation.md: new ## Filesystem subtree deny section between Performance and Deployment requirements, with hook coverage table, identity model, bind-mount handling, audit, performance numbers, and profile schema. Summary section linked.
• qa-profiles/04-allow-parent-of-protected.json: sample profile granting $HOME with the opt-in plus add_deny_access for $HOME/.config/qa-secret. Used for manual validation.

Performance (microbench)

• 200-iteration cat /etc/passwd inside the sandbox: ~1879 ms (release nono).
• Same loop native: ~254 ms.
• Subtracting ~1.5 s nono one-time setup: ~125 µs added per cat. Each cat does ~50 file opens (loader resolving libs, /etc/passwd itself), so ~2-3 µs per BPF-checked file_open in the busy path.
• 5-run median of claude -p hello: native 1.85 s, sandboxed 4.6 s. The 2.75 s overhead is dominated by the existing nono setup costs that PR feat(linux): linux BPF-LSM mediation filter (closes exec + read bypasses) #26 already incurred; the new BPF code is in the noise.
• Imperceptible for interactive agent workloads. Potentially measurable on tight build loops (e.g. make -j with thousands of forks per second), where the existing nono machinery already dominates the budget.

What changed vs #20

Deleted:

• install_seccomp_exec_filter and its BPF program builder from sandbox/linux.rs
• handle_exec_notification, classify_exec_path, ExecDecision, read_path_at, read_execve_argv_at, count_threads from supervisor_linux.rs
• mediation/shebang.rs (the shebang-chain walker)
• mediation/filter_audit.rs (the seccomp-era audit emitter — replaced by a library-side reader for the BPF ringbuf)
• The seccomp poll branch and exec_notify_fd SCM_RIGHTS plumbing in exec_strategy.rs

Added (original BPF-LSM work):

• crates/nono/src/bpf/mediation.bpf.c — two LSM programs sharing a deny map + scope check, plus a BPF_MAP_TYPE_RINGBUF for audit records
• crates/nono/src/sandbox/bpf_lsm.rs — loader (install_mediation_filter, MediationFilterHandle, SessionCgroup)
• crates/nono/src/sandbox/bpf_audit.rs — userspace ring-buffer reader that appends JSONL events to ~/.nono/sessions/audit.jsonl
• crates/nono/tests/bpf_lsm_smoke.rs — load + attach unit tests
• crates/nono-cli/tests/bpf_lsm_integration.rs — 16 end-to-end tests organised into exec mediation / read mediation / audit / composition groups, with self-skip if the test binary lacks the BPF caps
• make test-integration — build → setcap → cargo-test target so the integration suite actually runs in CI

Added (filesystem subtree deny extension):

• crates/nono/src/bpf/mediation.bpf.c extended — protected_roots map, dentry_in_protected_subtree, file_in_protected_subtree, extended check_file_open, eight new inode_* SEC blocks, two new audit reason codes
• crates/nono/src/sandbox/bpf_lsm.rs extended — MAX_PROTECTED_ROOTS=64, eight new Link fields on MediationFilterHandle, attached_program_count(), protected_roots_map(), collect_protected_root_entries, bind_mount_sources_under, decode_mountinfo_octal, two new BpfLsmError variants
• crates/nono-cli/src/protected_paths.rs extended — bpf_lsm_protected_roots_for_session, target_os = "macos" gate dropped
• crates/nono-cli/src/policy.rs — validate_deny_overlaps no-ops on Linux when BPF-LSM is available
• crates/nono-cli/src/sandbox_prepare.rs + the ExecutionFlags/SupervisedRuntimeContext/SupervisorConfig chain — thread protected_paths end-to-end
• crates/nono-cli/tests/bpf_lsm_protected_subtree.rs — 12 new integration tests
• qa-profiles/04-allow-parent-of-protected.json — sample profile

Schema change (audit.jsonl entries from BPF):

• action_type: "allow_unmediated" / "deny" (no exec_filter_ prefix)
• reason (only on deny): "exec_deny" / "open_deny" / "protected_open_deny" / "protected_mutate_deny"
• interpreter_chain: dropped (kernel resolves shebangs internally; BPF fires for the actually-loaded binary)
• exit_code: 126 on deny, absent on allow

The shim's own mediation::AuditEvent shape is unchanged; both event shapes interleave in the same JSONL file.

Validation

Performed end-to-end on the BPF-LSM workspace AMI (am/bpf-lsm-workspace-ami on dd-source — adds lsm=...,bpf to the kernel cmdline). Each check was re-run after every phase:

• cargo clippy --workspace --all-targets --all-features -- -D warnings -D clippy::unwrap_used clean
• cargo fmt --all -- --check clean
• cargo test -p nono --lib — 628/628 pass
• sudo -E cargo test -p nono --test bpf_lsm_smoke — 7/7 pass (5 original + 2 new for protected_roots)
• make test-integration — 16/16 (original) pass, plus 12/12 new in bpf_lsm_protected_subtree
• Plain cargo test -p nono-cli --test bpf_lsm_integration (no caps) — self-skip cleanly with informative messages
• cargo test -p nono-cli --bins protected_paths — 9/9 unit tests pass (including the rewritten Linux opt-in cases)
• vfork-bomb POC — 0/600 bypasses (was 49/600 with seccomp-unotify alone)
• pthread TOCTOU POC — 0/600 bypasses (regression check)
• sub-cgroup escape POC (privileged-agent threat model) — denied via ancestor walk
• Real claude end-to-end against qa-profiles/04-allow-parent-of-protected.json from $HOME: session starts cleanly, all deny scenarios fire, audit records protected_open_deny and protected_mutate_deny events
• Real claude end-to-end against the shadowfax production Linux profile (DataDog/shadowfax/deployments/linux/profiles/claude.json) extended with allow_parent_of_protected: true: session starts cleanly, claude -p "what is 2+2?" returns "4" (no productivity regression), all deny scenarios fire including ~/.config/password-store/gpg/key from add_deny_access

Documentation

• Design doc: docs/linux-bpf-lsm-mediation.md
• Implementation decisions log (autonomous-session picks, with rationale): docs/linux-exec-filter-bpf-lsm-impl-decisions.md
• Iteration log (vfork residual + the userspace ptrace experiments that didn't land): docs/linux-exec-filter-vfork-decisions.md
• Old seccomp-era plan: archived at docs/archive/linux-exec-filter-plan-seccomp-era.md

Test plan

• Reviewer boots a workspace from am/bpf-lsm-workspace-ami and confirms cat /sys/kernel/security/lsm includes bpf
• make build-release && sudo setcap cap_bpf,cap_sys_admin,cap_dac_override+ep target/release/nono
• sudo -E cargo test -p nono --test bpf_lsm_smoke — expect 7 pass (5 original + 2 new)
• make test-integration — expect 16 (original) + 12 (new bpf_lsm_protected_subtree) = 28 pass
• Run a mediation-active session and confirm ~/.nono/sessions/audit.jsonl receives allow_unmediated and deny/open_deny entries with the expected schema
• Run a session against qa-profiles/04-allow-parent-of-protected.json, verify protected_open_deny and protected_mutate_deny audit events fire when the agent reads/writes $HOME/.config/qa-secret
• Run the kipz POCs (vfork + pthread, 600 attempts each); expect 0 bypasses

🤖 Generated with Claude Code