Full branch E2E binds ordinary live jobs to a stale managed-image revision

[ericksoa]

Summary

The complete unfiltered branch E2E run for cutover PR #9323 tested the exact CLI commit but bound ordinary live managed-runtime jobs to the older base_sha managed-image revision. That mixes candidate source/CLI behavior with a stale runtime and causes merged fixes to recur or be bypassed.

This must be fixed without publishing a new mutable image tag or changing dependency versions. The branch run needs an exact, content-addressed candidate runtime contract or must prove that the reused runtime contains every required candidate-owned managed-startup and policy change.

Evidence

• Full run: https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706
• Candidate commit: 774e2363ecdfed90502ebbcffa03a4ecb46fc50e
• Ordinary live jobs received E2E_MANAGED_IMAGE_REVISION=8cdc3c41eadf51cd817df9a0f738656d8c940be5.
• Deep Agents Code failed because the candidate onboarding reference expected the published linux/amd64 platform contract while the base-revision validation path supplied the older multi-architecture index:
  https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95696286247
• Both candidate-built protected startup jobs passed, and the protected GPU/local-inference aggregate passed:
  https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95702206708
• Ordinary live jobs still reproduced behavior addressed by candidate-contained fixes, including CA/runtime-policy failures and the Hermes state-volume rejection.

Representative stale-runtime signatures:

• Three jobs could not read /run/nemoclaw/managed-startup-runtime.env and ended with CORPORATE_CA_PROBE_FAIL.
• Nine Hermes jobs failed with transaction path crosses a nested filesystem mount: /sandbox/.hermes.
• Trusted-private MCP discovery reached zero tools in both OpenClaw and Deep Agents lanes.

Required behavior

1. An unfiltered credential-bearing branch run must test one coherent exact candidate: CLI, managed startup/runtime content, policies, and image contract must share immutable provenance.
2. The run must not silently combine a PR candidate CLI with an older base-revision managed runtime when candidate changes affect that runtime.
3. Candidate qualification must use content-addressed artifacts and digests; it must not require publishing a new mutable tag or dependency version.
4. Mixed or missing provenance must fail before fanout with an actionable error that names both revisions.
5. The repaired path must rerun the affected DCode, CA, MCP, and Hermes-volume lanes, followed by the complete unfiltered matrix.

Security and cleanup requirements

• Keep credential authorization restricted to trusted NVIDIA branch actors.
• Do not expose secret values in provenance, logs, or artifacts.
• Preserve fail-closed policy and runtime-contract validation.
• Preserve exact-owner cleanup for any candidate containers, networks, volumes, and retained rollback resources.

Links

• Cutover issue: [Epic #7744 1/6][B3-E] Activate supported buildless managed onboarding #9140
• Cutover PR: feat(onboard): activate buildless managed onboarding #9323
• Related prior DCode issue: Deep Agents live E2E loses base-resolution metadata after lifecycle checks #9386
• Related managed runtime fixes: fix(openclaw): isolate startup guard output #9371, fix(policy): grant managed startup CA bundle read access #9391, fix(onboard): persist managed Hermes state #9373