Skip to content

Commit 6141341

Browse files
authored
test(e2e): add native runtime qualification producer (NVIDIA#8064)
Add the protected, credential-free native runtime qualification producer and its maintainer guidance. CodeQL and relevant checks passed; the unrelated sandbox start timeout is documented in the PR after three retries.
1 parent 23d9f8f commit 6141341

13 files changed

Lines changed: 1561 additions & 17 deletions

.agents/skills/nemoclaw-maintainer-e2e/SKILL.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,16 +73,17 @@ Choose exactly one mode:
7373
The run skips `jetson-nvmap-gpu` unless `allow_jetson_dispatch` is `true`.
7474
It skips `llama-cpp-dgx-spark-plan` and `llama-cpp-dgx-spark-qualification` unless their runner-queue flag is `true`.
7575
- For protected managed-image runtime qualification, set `E2E_JOBS=managed-image-protected-runtime`. The exact candidate must contain `ci/protected-managed-image-multiarch-activation-v1.json` and `ci/protected-managed-image-runtime-activation-v1.json`.
76+
- For native-runtime qualification evidence, set `E2E_JOBS=native-runtime-qualification-producer`. Use a same-repository open PR and the first workflow attempt. The trusted workflow runs each case under a credential-free candidate account on a reviewed ephemeral runner. The candidate must contain `test/e2e/live/native-runtime-qualification-case.test.ts` before the selector can pass.
7677

7778
Leave `targets` empty and keep Launchable disabled:
7879

7980
```bash
8081
E2E_JOBS="${E2E_JOBS:-}"
8182
case "$E2E_JOBS" in
82-
"" | managed-image-protected-runtime) ;;
83+
"" | managed-image-protected-runtime | native-runtime-qualification-producer) ;;
8384
*) echo "Unsupported manual PR E2E job selector" >&2; exit 1 ;;
8485
esac
85-
REVIEW_REASON='Reviewed the PR head commit for credentialed E2E.'
86+
REVIEW_REASON='Reviewed the commit under review and selected E2E boundary.'
8687
CORRELATION_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
8788
INFERENCE_MODE=mock
8889
ALLOW_JETSON_DISPATCH=false
@@ -109,6 +110,10 @@ The trusted pre-checkout step requires current `maintain` or `admin` permission.
109110
It validates the actor, open PR, repository, head SHA, base SHA, workflow SHA, review reason, and allowed jobs, targets, and Launchable combination.
110111
A second validation after checkout rejects a changed PR identity before preparation.
111112

113+
The native-runtime producer binds the open PR, candidate commit, base commit, trusted workflow commit, and first workflow attempt. It runs the trusted plan from `main` and passes no GitHub, model-provider, API, or messaging credentials to candidate code. Configure `NATIVE_RUNTIME_EPHEMERAL_RUNNER_POOL=enabled` before dispatch. The ARM64 GPU case also requires `NATIVE_RUNTIME_ARM64_GPU_RUNNER_LABEL`; the workflow provides no fallback runner.
114+
115+
The producer stops Docker, masks its service and socket, removes Docker sockets, and rejects a usable `docker` command before candidate execution. It runs the candidate case under a temporary unprivileged account and uploads one evidence artifact for each planned case. Cleanup terminates processes owned by the candidate account and removes that account. If cleanup fails or the runner becomes unavailable, inspect the host and remove the ephemeral runner from service. Recover or replace the runner before dispatching a new run. Do not rerun the same workflow attempt; the producer rejects attempts after the first.
116+
112117
Find and verify the correlated run with bounded GitHub reads:
113118

114119
```bash

.github/workflows/e2e.yaml

Lines changed: 372 additions & 5 deletions
Large diffs are not rendered by default.

test/e2e/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1040,6 +1040,10 @@ After a failure, inspect the workflow artifacts and remove resources that target
10401040

10411041
For `managed-image-protected-runtime`, the workflow supplies the long-lived `NVIDIA_API_KEY` repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as `NGC_API_KEY` and `NIM_NGC_API_KEY` to the temporary, cohort-owned NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. Before starting NIM or vLLM, the live fixture rejects a pre-existing cohort container name. It records the full container ID, requested image, immutable image ID, cohort owner, and provider label, then removes only that exact container after revalidating every field. Missing, ambiguous, name-reused, drifted, or indeterminate cleanup evidence fails the test, as does any retained exact ID or name. A fail-closed refusal can leave the secret-bearing NIM container alive until runner teardown; inspect the redacted artifacts and remove only the verified container. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Revoke it, or rotate it and disable the old value, in the issuing NVIDIA service. Verify that the exposed key is no longer valid.
10421042

1043+
For `native-runtime-qualification-producer`, use a same-repository open PR and the first workflow attempt. The trusted workflow binds the candidate commit, base commit, workflow commit, repository, PR, and plan from `main`. It passes no GitHub, model-provider, API, or messaging credentials to candidate code. Candidate execution uses `env -i` under a temporary unprivileged account on a reviewed ephemeral runner. Configure `NATIVE_RUNTIME_EPHEMERAL_RUNNER_POOL=enabled` before dispatch. The ARM64 GPU case also requires `NATIVE_RUNTIME_ARM64_GPU_RUNNER_LABEL`; the workflow provides no fallback runner. The candidate must contain `test/e2e/live/native-runtime-qualification-case.test.ts`. Until that executor and the required runner capacity exist, the producer fails closed instead of claiming qualification.
1044+
1045+
Before candidate execution, the producer stops Docker, masks its service and socket, removes Docker sockets, and rejects a usable `docker` command. It uploads one evidence artifact for each planned case. Cleanup terminates processes owned by the candidate account and removes that account. If cleanup fails or the runner becomes unavailable, inspect the host and remove the ephemeral runner from service. Recover or replace the runner before dispatching a new run. Do not rerun the same workflow attempt; the producer rejects attempts after the first. Dispatch a new run after recovery.
1046+
10431047
For a manual PR run, provide the current PR number, lowercase 40-character candidate commit SHA, PR source repository, lowercase 40-character base commit SHA, trusted `main` workflow SHA, and a review reason containing 10 to 500 printable characters.
10441048
Leave `jobs` and `targets` empty and keep `include_staging_brev_launchable=false` to use this PR revision selection.
10451049
Keep `allow_jetson_dispatch=false` and `allow_dgx_spark_runner_queue=false` for the default PR revision selection.
@@ -1049,6 +1053,9 @@ To select the protected managed-image runtime qualification, set `jobs=managed-i
10491053
Leave `targets` empty.
10501054
Keep `include_staging_brev_launchable=false`.
10511055
The exact candidate must contain `ci/protected-managed-image-multiarch-activation-v1.json` and `ci/protected-managed-image-runtime-activation-v1.json`.
1056+
To select native-runtime qualification evidence production, set `jobs=native-runtime-qualification-producer`.
1057+
Leave `targets` empty and keep `include_staging_brev_launchable=false`.
1058+
Confirm that the PR comes from `NVIDIA/NemoClaw`, the required ephemeral runner variables are configured, and the workflow has not been rerun.
10521059
The trusted pre-checkout step requires current `maintain` or `admin` permission and validates the exact open PR and selected mode before candidate code runs.
10531060
A second validation after checkout rejects a changed candidate commit, base commit, or PR source repository before preparation.
10541061

test/e2e/support/e2e-operations-workflow-boundary.test.ts

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -414,12 +414,12 @@ const interpolatedNeeds = \${{ toJSON ( needs ) }};
414414
(step) => step.name === "Authenticate manual PR dispatch",
415415
)!;
416416
authentication.run = authentication.run!.replace(
417-
"Manual PR E2E accepts only empty selectors, inference-routing, managed-image-protected-runtime, or jetson-nvmap-gpu with its dispatch flag",
417+
"Manual PR E2E accepts only empty selectors, inference-routing, managed-image-protected-runtime, native-runtime-qualification-producer, or jetson-nvmap-gpu with its dispatch flag",
418418
"Manual PR E2E accepts arbitrary selectors",
419419
);
420420

421421
expect(validateE2eOperationsWorkflow(workflow)).toContain(
422-
"Manual PR authentication must retain Manual PR E2E accepts only empty selectors, inference-routing, managed-image-protected-runtime, or jetson-nvmap-gpu with its dispatch flag",
422+
"Manual PR authentication must retain Manual PR E2E accepts only empty selectors, inference-routing, managed-image-protected-runtime, native-runtime-qualification-producer, or jetson-nvmap-gpu with its dispatch flag",
423423
);
424424
});
425425

@@ -431,14 +431,15 @@ const interpolatedNeeds = \${{ toJSON ( needs ) }};
431431
authentication.run = authentication.run!.replace("inference-routing::false:false | ", "");
432432

433433
expect(validateE2eOperationsWorkflow(workflow)).toContain(
434-
"Manual PR authentication must retain ::false:false | inference-routing::false:false | managed-image-protected-runtime::false:false | :jetson-nvmap-gpu:false:true) ;;",
434+
"Manual PR authentication must retain ::false:false | inference-routing::false:false | managed-image-protected-runtime::false:false | native-runtime-qualification-producer::false:false | :jetson-nvmap-gpu:false:true) ;;",
435435
);
436436
});
437437

438438
it.each([
439439
["maintain", "", "", "false", 0, ""],
440440
["maintain", "inference-routing", "", "false", 0, ""],
441441
["maintain", "managed-image-protected-runtime", "", "false", 0, ""],
442+
["maintain", "native-runtime-qualification-producer", "", "false", 0, ""],
442443
["maintain", "", "jetson-nvmap-gpu", "true", 0, ""],
443444
["maintain", "", "jetson-nvmap-gpu", "false", 1, "accepts only empty selectors"],
444445
["maintain", "network-policy", "", "false", 1, "accepts only empty selectors"],
@@ -822,6 +823,22 @@ const interpolatedNeeds = \${{ toJSON ( needs ) }};
822823
);
823824
});
824825

826+
it.each([
827+
["a lowercase short write method", "gh api -X post repos/NVIDIA/NemoClaw/issues"],
828+
["a lowercase long write method", "gh api --method patch repos/NVIDIA/NemoClaw/issues/1"],
829+
[
830+
"a GraphQL mutation",
831+
"gh api graphql -f query='mutation { closeIssue(input: {}) { issue { id } } }'",
832+
],
833+
])("rejects %s in the qualification planning job", (_label, mutation) => {
834+
const workflow = readE2eOperationsWorkflow();
835+
workflow.jobs["native-runtime-qualification-producer-plan"].steps!.push({ run: mutation });
836+
837+
expect(validateE2eOperationsWorkflow(workflow)).toContain(
838+
"native-runtime-qualification-producer-plan must limit GitHub API access to the reviewed read-only contract",
839+
);
840+
});
841+
825842
it("reserves pull-request write permission for the validated PR reporter", () => {
826843
const workflow = readE2eOperationsWorkflow();
827844
workflow.permissions = { "pull-requests": "write" };
Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
import { createHash } from "node:crypto";
5+
import fs from "node:fs";
6+
import os from "node:os";
7+
import path from "node:path";
8+
9+
import { afterEach, describe, expect, it } from "vitest";
10+
11+
import { writeNativeRuntimeQualificationProducerEvidence } from "../../../tools/e2e/native-runtime-qualification-producer-evidence.mts";
12+
import {
13+
buildNativeRuntimeQualificationProducerPlan,
14+
NATIVE_RUNTIME_QUALIFICATION_FOCUSED_CASE,
15+
} from "../../../tools/e2e/native-runtime-qualification-producer-plan.mts";
16+
17+
const roots: string[] = [];
18+
const INSTALLER = "#!/usr/bin/env bash\nexit 0\n";
19+
const INSTALLER_SHA256 = createHash("sha256").update(INSTALLER).digest("hex");
20+
21+
function fixture() {
22+
const root = fs.mkdtempSync(path.join(os.tmpdir(), "native-runtime-producer-evidence-"));
23+
roots.push(root);
24+
const row = buildNativeRuntimeQualificationProducerPlan({
25+
source: {
26+
repository: "NVIDIA/NemoClaw",
27+
producerWorkflow: ".github/workflows/e2e.yaml",
28+
pullRequestNumber: 8064,
29+
candidateRepository: "NVIDIA/NemoClaw",
30+
candidateSha: "a".repeat(40),
31+
baseRef: "main",
32+
baseSha: "b".repeat(40),
33+
workflowSha: "b".repeat(40),
34+
producerRunId: "123456789",
35+
producerRunAttempt: 1,
36+
dispatchArtifact: {
37+
id: "42",
38+
name: "e2e-dispatch-123456789-1",
39+
digest: `sha256:${"c".repeat(64)}`,
40+
sizeInBytes: 4096,
41+
},
42+
},
43+
installerSha256: INSTALLER_SHA256,
44+
arm64GpuRunner: "reviewed-native-arm64-gpu-runner",
45+
}).include.find((entry) => entry.id === NATIVE_RUNTIME_QUALIFICATION_FOCUSED_CASE)!;
46+
const installerDirectory = path.join(root, "installer");
47+
const executionPath = path.join(root, "execution.json");
48+
const evidenceDirectory = path.join(root, "evidence");
49+
fs.mkdirSync(installerDirectory);
50+
fs.writeFileSync(path.join(installerDirectory, "installer.sh"), INSTALLER);
51+
fs.writeFileSync(
52+
path.join(installerDirectory, "invocation.json"),
53+
JSON.stringify({
54+
receiptVersion: 1,
55+
script: "scripts/install.sh",
56+
scriptSha256: INSTALLER_SHA256,
57+
candidateSha: row.source.candidateSha,
58+
architecture: row.case.architecture,
59+
}),
60+
);
61+
fs.writeFileSync(
62+
path.join(installerDirectory, "candidate-source.json"),
63+
JSON.stringify({
64+
receiptVersion: 1,
65+
repository: "https://github.com/NVIDIA/NemoClaw.git",
66+
revision: row.source.candidateSha,
67+
installerSha256: INSTALLER_SHA256,
68+
}),
69+
);
70+
fs.writeFileSync(
71+
path.join(installerDirectory, "installed-source.json"),
72+
JSON.stringify({
73+
receiptVersion: 1,
74+
repository: "https://github.com/NVIDIA/NemoClaw.git",
75+
requestedRevision: row.source.candidateSha,
76+
installedRevision: row.source.candidateSha,
77+
installMode: "managed",
78+
installerSha256: INSTALLER_SHA256,
79+
}),
80+
);
81+
fs.writeFileSync(
82+
path.join(installerDirectory, "architecture.json"),
83+
JSON.stringify({ receiptVersion: 1, requested: "amd64", runner: "amd64" }),
84+
);
85+
const dockerPosture = {
86+
dockerCommandGuarded: true,
87+
dockerEnvironmentVariablesUnset: true,
88+
dockerServiceInactive: true,
89+
dockerSocketUnitInactive: true,
90+
dockerdProcessNameAbsent: true,
91+
defaultSocketPathsAbsent: true,
92+
};
93+
fs.writeFileSync(
94+
path.join(installerDirectory, "docker-absence.json"),
95+
JSON.stringify({
96+
receiptVersion: 1,
97+
preExecution: dockerPosture,
98+
postExecution: dockerPosture,
99+
}),
100+
);
101+
const execution = {
102+
schemaVersion: 1,
103+
kind: "nemoclaw-native-runtime-qualification-execution-v1",
104+
caseId: row.id,
105+
candidateSha: row.source.candidateSha,
106+
installerSha256: row.installerSha256,
107+
architecture: row.case.architecture,
108+
acceleration: row.case.acceleration,
109+
agent: row.case.agent,
110+
inference: row.case.inference,
111+
rootModes: row.rootModes,
112+
obligations: row.case.obligations,
113+
focusedOperations: row.focusedOperations,
114+
evidenceKinds: row.case.evidenceKinds,
115+
dockerUnavailable: { beforeCandidate: true, afterCandidate: true },
116+
credentialBoundary: {
117+
githubCredentialsAbsent: true,
118+
modelCredentialsAbsent: true,
119+
isolatedUid: true,
120+
},
121+
result: "passed",
122+
};
123+
fs.writeFileSync(executionPath, JSON.stringify(execution));
124+
return { evidenceDirectory, execution, executionPath, installerDirectory, root, row };
125+
}
126+
127+
afterEach(() => {
128+
for (const root of roots.splice(0)) fs.rmSync(root, { force: true, recursive: true });
129+
});
130+
131+
describe("native runtime qualification producer evidence", () => {
132+
it("emits the bounded case-evidence envelope after validating installer and execution receipts", () => {
133+
const value = fixture();
134+
135+
writeNativeRuntimeQualificationProducerEvidence(
136+
value.row,
137+
value.installerDirectory,
138+
value.executionPath,
139+
value.evidenceDirectory,
140+
);
141+
142+
expect(fs.readdirSync(value.evidenceDirectory)).toEqual(["evidence.json"]);
143+
expect(
144+
JSON.parse(fs.readFileSync(path.join(value.evidenceDirectory, "evidence.json"), "utf8")),
145+
).toEqual({
146+
schemaVersion: 1,
147+
kind: "nemoclaw-native-runtime-qualification-case-evidence-v1",
148+
qualificationId: "podman-protected-host-local-inference",
149+
providerId: "podman",
150+
source: value.row.source,
151+
case: value.row.case,
152+
result: "passed",
153+
});
154+
expect(fs.statSync(path.join(value.evidenceDirectory, "evidence.json")).mode & 0o777).toBe(
155+
0o600,
156+
);
157+
});
158+
159+
it.each([
160+
[
161+
"candidate identity",
162+
(value: ReturnType<typeof fixture>) => ({ ...value.execution, candidateSha: "d".repeat(40) }),
163+
],
164+
[
165+
"rootful focus",
166+
(value: ReturnType<typeof fixture>) => ({ ...value.execution, rootModes: ["rootless"] }),
167+
],
168+
[
169+
"cleanup operation",
170+
(value: ReturnType<typeof fixture>) => ({
171+
...value.execution,
172+
focusedOperations: value.row.focusedOperations.slice(0, -1),
173+
}),
174+
],
175+
[
176+
"credential boundary",
177+
(value: ReturnType<typeof fixture>) => ({
178+
...value.execution,
179+
credentialBoundary: {
180+
...value.execution.credentialBoundary,
181+
githubCredentialsAbsent: false,
182+
},
183+
}),
184+
],
185+
[
186+
"Docker boundary",
187+
(value: ReturnType<typeof fixture>) => ({
188+
...value.execution,
189+
dockerUnavailable: { beforeCandidate: true, afterCandidate: false },
190+
}),
191+
],
192+
])("rejects an invalid %s receipt", (_label, mutate) => {
193+
const value = fixture();
194+
fs.writeFileSync(value.executionPath, JSON.stringify(mutate(value)));
195+
196+
expect(() =>
197+
writeNativeRuntimeQualificationProducerEvidence(
198+
value.row,
199+
value.installerDirectory,
200+
value.executionPath,
201+
value.evidenceDirectory,
202+
),
203+
).toThrow("Native runtime qualification");
204+
});
205+
206+
it("rejects a forged installer receipt", () => {
207+
const value = fixture();
208+
fs.writeFileSync(
209+
path.join(value.installerDirectory, "installer.sh"),
210+
`${INSTALLER}echo forged\n`,
211+
);
212+
213+
expect(() =>
214+
writeNativeRuntimeQualificationProducerEvidence(
215+
value.row,
216+
value.installerDirectory,
217+
value.executionPath,
218+
value.evidenceDirectory,
219+
),
220+
).toThrow("installer receipt is invalid");
221+
});
222+
223+
it("rejects unexpected files in the installer receipt directory", () => {
224+
const value = fixture();
225+
fs.writeFileSync(path.join(value.installerDirectory, "log.txt"), "candidate output");
226+
227+
expect(() =>
228+
writeNativeRuntimeQualificationProducerEvidence(
229+
value.row,
230+
value.installerDirectory,
231+
value.executionPath,
232+
value.evidenceDirectory,
233+
),
234+
).toThrow("receipt files are invalid");
235+
});
236+
237+
it("rejects a symbolic link used as the execution receipt", () => {
238+
const value = fixture();
239+
const target = path.join(value.root, "linked-execution.json");
240+
fs.renameSync(value.executionPath, target);
241+
fs.symlinkSync(target, value.executionPath);
242+
243+
expect(() =>
244+
writeNativeRuntimeQualificationProducerEvidence(
245+
value.row,
246+
value.installerDirectory,
247+
value.executionPath,
248+
value.evidenceDirectory,
249+
),
250+
).toThrow("receipt is missing or invalid");
251+
});
252+
});

0 commit comments

Comments
 (0)