Commit 8cdc3c4
test(e2e): execute native runtime qualification (#9232)
## Summary
Implement the canonical 24-case native Podman qualification matrix and
preserve exact installer, runtime, lifecycle, recovery, GPU, and cleanup
receipts for the existing B4-F consumer. Candidate execution stays
credential-free and unprivileged with Docker unavailable; production
Podman selection remains disabled and B4-H documentation remains out of
scope.
Commit `1fb09c49f9718f87570d55d90f05581d7375746c` remediated PRA-1 in
code: the secret-bearing native-runtime controller runs only from
trusted `main`, requires the workflow SHA to equal the PR-recorded base
SHA and differ from the candidate SHA, and rejects candidate-workflow
authority in both the trusted planner and the candidate row parser.
Protected run `31984240689`, attempt `1`, at historical commit
`504fcf718a8ece560c021c5ed4656851ef419e84` executed every case
assignable to a runner already authorized for NemoClaw: 15/15 passed
with zero failures, including all six AMD64/ARM64 CPU cases and all nine
AMD64 RTX PRO 6000 GPU cases. The nine ARM64-GPU jobs received no runner
group or runner identity because NemoClaw has no authorized non-Spark
ARM64-GPU runner group. This PR does not claim 24/24 at latest PR commit
`54a3c3b548d86035b752e499b227719f3eb961c2`.
Repository administrator authorization ([administrator
decision](#9232 (comment)))
accepts deferring those nine executions and the complete aggregate
receipt to mandatory pre-activation testing on the physical DGX Spark or
a proper ARM64 NVIDIA-GPU runner. The [exact run
receipt](#9232 (comment))
records every successful job, runner, artifact, digest, and deferred job
identity. Production Podman must remain unselectable and unadvertised
until that pre-activation qualification passes. This exception does not
authorize merge.
## Related Issue
Closes #9144
Related #7744
## Changes
- Add the live 3-agent x 2-architecture x 4-inference-path executor,
including real GPU-backed turns, focused lifecycle/recovery
interactions, rootful-denial coverage for the rootless-only provider,
and exact cleanup.
- Harden installer and per-case receipt handling against symlinks,
unexpected files, credential material, content races, and candidate
processes surviving into trusted evidence handling.
- Bind protected host cleanup to a root-owned run/attempt marker
carrying the account's exact UID/GID, reject pre-existing account or
group authorization, verify subordinate-ID revocation, and grant Ubuntu
user namespaces only to the pinned qualification Podman binary and the
root-owned, mode-0555 run copy of its source-built `pasta` helper
through ephemeral AppArmor profiles.
- Start and verify the ephemeral account's systemd user
manager/runtime-directory units for rootless Aardvark, restrict
restarted user-unit authority to root-owned OS paths, prove the user-bus
socket type/non-symlink/owner and account-side resolution, and stop both
units before identity cleanup.
- Build the compatible `pasta` helper from an official passt.top commit
snapshot pinned by SHA and archive SHA-256, include it in the
checksummed Podman toolchain, and verify the installed run copy by root
ownership, mode, and matching digest.
- Require native rootless overlay storage on the reviewed Ubuntu hosts,
reject any external overlay mount program, and isolate Netavark to a
root-owned per-run `nftables` configuration before candidate execution.
- Materialize exact architecture-specific GPU probe and inference images
in a runner-only preparation step, delete its registry authentication
file before candidate dependencies or execution, and bind NIM/vLLM to a
pinned public model revision whose exact root-owned files are
revalidated by the unprivileged candidate.
- Prove failed provider-network cleanup with the pinned Podman `network
exists` postcondition and report the original validation error together
with any unproven cleanup outcome.
- Remove run-owned volume snapshots on both success and failure; rethrow
the original case failure when cleanup succeeds, or aggregate it with
separately collected cleanup failures when cleanup cannot be proven.
- Add an aggregate validation job that rejects incomplete or mixed
cohorts and feeds the exact 24-case envelope through the canonical B4-F
consumer.
- Align the live full-uninstall proof with the secure retirement
contract by requiring both empty NemoClaw portable configuration
directories to be absent.
- Exercise nested YAML credential sanitization and scalar-document
rejection through behavior tests so the existing credential boundary
remains above its merged-coverage security floor.
- Keep the Podman workload profile dormant and make no production
provider registration, selection, advertising, support-documentation,
NVIDIA/OpenShell, or Kubernetes/Docker behavior changes.
## Type of Change
- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)
## Quality Gates
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent
nine-category security review passed for latest PR commit `54a3c3b54`.
The review confirmed that trusted-main authority, `NVIDIA_API_KEY`
handling, candidate credential-free execution, cleanup, authorization,
digest checks, and artifact boundaries remain unchanged or clearer. The
final delta changes workflow labels, errors, guidance, and matching
tests without weakening an enforcing control. All 28 review
conversations remain resolved; approval is a separate gate.
- [x] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue: the administrator
[decision](#9232 (comment))
limits the exception to nine unavailable ARM64 GPU executions and the
complete aggregate before activation. Those combinations remain
mandatory on suitable ARM64 NVIDIA GPU hardware before B4-H activation.
No required merge check or executed-case receipt is waived.
## Documentation Writer Review
- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Independent Pi CLI review passed for the complete 34-file
diff at latest PR commit `54a3c3b54`. The guidance now states the
trusted-main controller identities, exact dispatch SHA constraints,
GitHub Actions repository-variable locations, `NVIDIA_API_KEY` lifetime
and non-revocation, credential-free candidate execution, successful and
failed case-artifact behavior, selector boundaries, retention periods,
and empty-only portable configuration cleanup. Changed documentation
includes `docs/manage-sandboxes/uninstall-nemoclaw.mdx`,
`test/e2e/README.md`, and `test/e2e/docs/README.md`. Focused e2e-support
validation passed 116 tests, `npm run docs` passed with 0 errors and 2
existing Fern warnings, and normal hooks passed.
- Agent: Pi CLI
<!-- docs-review-head-sha: 54a3c3b -->
<!-- docs-review-agents-blob-sha: b9fb6a9 -->
## DGX Station Hardware Evidence
- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:
## Verification
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — At latest PR commit `54a3c3b54`,
focused e2e-support validation passed 116 workflow-boundary tests and
`src/lib/security/credential-filter.test.ts` passed 47 tests. `npm run
docs` passed, the independent security review passed all nine
categories, and normal hooks passed. Earlier protected qualification
evidence remains tied to its recorded historical commit.
- [x] Applicable broad gate passed: all five required checks have
passing or expected skipped conclusions at latest PR commit
`54a3c3b548d86035b752e499b227719f3eb961c2`. Full CI run `32021563119`
and codebase growth guardrail run `32021562210` passed. Protected run
`31984240689` remains historical functional evidence for commit
`504fcf718a8ece560c021c5ed4656851ef419e84`.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
---
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Protected Qualification**
- Adds a 24-case rootless Podman qualification matrix covering
inference, GPU validation, onboarding, lifecycle recovery, and cleanup.
- Validates credential-free execution, pinned runtime components,
isolated runners, and complete evidence before publishing a 30-day
aggregate report.
- Strengthens authorization and commit identity checks for candidate
qualification runs.
- **Product Boundary**
- Production Podman selection remains unchanged and is not presented as
a supported runtime.
- **Bug Fixes**
- Improves safe cleanup of portable configuration directories during
uninstall and recovery.
- **Tests**
- Expands coverage for qualification workflows, evidence integrity,
security boundaries, lifecycle behavior, and artifact aggregation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>1 parent 588bb6d commit 8cdc3c4
34 files changed
Lines changed: 7092 additions & 234 deletions
File tree
- .github/workflows
- docs/manage-sandboxes
- scripts/checks
- src/lib
- onboard/runtime-provider
- security
- state
- test/e2e
- docs
- live
- registry
- support
- tools/e2e
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| 183 | + | |
| 184 | + | |
183 | 185 | | |
184 | 186 | | |
185 | 187 | | |
| |||
Lines changed: 64 additions & 26 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
101 | 102 | | |
102 | 103 | | |
103 | 104 | | |
| |||
191 | 192 | | |
192 | 193 | | |
193 | 194 | | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
194 | 198 | | |
195 | 199 | | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
202 | 240 | | |
203 | 241 | | |
204 | 242 | | |
| |||
272 | 310 | | |
273 | 311 | | |
274 | 312 | | |
275 | | - | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
276 | 316 | | |
277 | 317 | | |
278 | 318 | | |
| |||
303 | 343 | | |
304 | 344 | | |
305 | 345 | | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | 346 | | |
317 | 347 | | |
318 | 348 | | |
| |||
336 | 366 | | |
337 | 367 | | |
338 | 368 | | |
339 | | - | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
340 | 372 | | |
341 | | - | |
342 | 373 | | |
343 | 374 | | |
344 | 375 | | |
| |||
366 | 397 | | |
367 | 398 | | |
368 | 399 | | |
369 | | - | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
370 | 403 | | |
371 | 404 | | |
372 | 405 | | |
373 | | - | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
374 | 409 | | |
375 | 410 | | |
376 | 411 | | |
| |||
382 | 417 | | |
383 | 418 | | |
384 | 419 | | |
385 | | - | |
| 420 | + | |
386 | 421 | | |
387 | 422 | | |
388 | | - | |
| 423 | + | |
389 | 424 | | |
390 | 425 | | |
391 | 426 | | |
392 | 427 | | |
393 | | - | |
394 | | - | |
| 428 | + | |
| 429 | + | |
395 | 430 | | |
396 | 431 | | |
397 | 432 | | |
| |||
409 | 444 | | |
410 | 445 | | |
411 | 446 | | |
412 | | - | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
413 | 451 | | |
414 | 452 | | |
415 | 453 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
Lines changed: 43 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
22 | 32 | | |
23 | 33 | | |
24 | 34 | | |
| |||
33 | 43 | | |
34 | 44 | | |
35 | 45 | | |
36 | | - | |
37 | 46 | | |
38 | 47 | | |
39 | 48 | | |
| |||
55 | 64 | | |
56 | 65 | | |
57 | 66 | | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
| 67 | + | |
62 | 68 | | |
63 | 69 | | |
64 | 70 | | |
| |||
98 | 104 | | |
99 | 105 | | |
100 | 106 | | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
| 107 | + | |
109 | 108 | | |
110 | 109 | | |
111 | 110 | | |
| |||
196 | 195 | | |
197 | 196 | | |
198 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
199 | 227 | | |
200 | | - | |
| 228 | + | |
201 | 229 | | |
202 | 230 | | |
203 | 231 | | |
204 | | - | |
| 232 | + | |
205 | 233 | | |
206 | 234 | | |
207 | 235 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
195 | 190 | | |
196 | | - | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
197 | 214 | | |
198 | | - | |
| 215 | + | |
199 | 216 | | |
200 | 217 | | |
201 | 218 | | |
| |||
437 | 454 | | |
438 | 455 | | |
439 | 456 | | |
440 | | - | |
| 457 | + | |
441 | 458 | | |
442 | 459 | | |
443 | 460 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
131 | 141 | | |
132 | 142 | | |
133 | 143 | | |
| |||
140 | 150 | | |
141 | 151 | | |
142 | 152 | | |
143 | | - | |
| 153 | + | |
144 | 154 | | |
145 | 155 | | |
146 | 156 | | |
| |||
0 commit comments