Skip to content

Commit 3b0e18f

Browse files
mkceichelbeckMichael Eichelbeck
andauthored
fix(hosted): harden provider delivery contracts (#1016)
## Summary - preserve provider credentials, routing, and execution context through hosted preflight and runtime boundaries - align Codex, Claude, and OMP hosted execution, including writable temp paths, process ownership, terminal failure semantics, and bounded completion results - make shipped GitHub delivery authoritative and resilient to transient merge-state races - harden private candidate/image packaging and add focused regression coverage for the deployed contracts ## Deployed validation Validated through `dev.theopenengine.com` with real provider calls, real work, and real `--ship` delivery to `the-open-engine/zeroshot-matrix-demo-20260812-3ebd`. Every shipped PR merged and its repository CI passed. | Provider / coordinator | RunIntent | Shipped PR | |---|---|---| | Azure OpenAI + OMP / custom | `019ffd84-61e7-7d70-b900-f74eab6a6a5c` | demo PR 65 | | Azure OpenAI + OMP / built-in | `019ffd8d-6159-70a3-ac33-cc3a1eeb049d` | demo PR 67 | | OpenRouter + Claude / custom | `019ffd9c-e8f8-7f30-a6b1-d142a54238b6` | demo PR 72 | | OpenRouter + Claude / built-in | `019ffd9c-ef80-7901-b892-4b1fdaf0d244` | demo PR 73 | | OpenAI + Codex / custom | `019ffd9c-f8e8-7070-9b16-781bba765fab` | demo PR 74 | | OpenAI + Codex / built-in | `019ffd9c-ff78-7791-950c-c003ba111378` | demo PR 75 | The targeted Azure failure rows were proven first; only then were the four previously green rows rerun as regression guards. The accumulated demo repository suite passes 543/543 tests. No delivery fixtures or simulated provider results were used. The full runtime tree was tested at `42c6cfd`; this PR was then recomposed as one conventional commit. The only tree delta in `f3ed3fe` is two test-fixture files (no production/runtime files). Zero-cloud main pins `f3ed3fe`, and focused exact-head deployment run `31757758450` completed successfully with image scan, IaC apply, and public health checks. ## Validation - focused completion-boundary suite: 24 passing - private hosted preflight suite: 7 passing - `npm run check`: green (0 errors; existing warnings only) - Opcore staged gate: 92/92 files covered, clean - repository pre-commit validation: green - branch push hooks: green - exact-head PR CI: green (one unrelated Rust timing flake passed on the single rerun) - zero-cloud exact-main CI: [green](https://github.com/the-open-engine/zero-cloud/actions/runs/31757674291) - zero-cloud exact-main full deployment: [green](https://github.com/the-open-engine/zero-cloud/actions/runs/31757674266) Co-authored-by: Michael Eichelbeck <michael@theopenengine.com>
1 parent 7ee8594 commit 3b0e18f

100 files changed

Lines changed: 7414 additions & 876 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.dockerignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.git
2+
.cache
3+
node_modules
4+
**/node_modules
5+
target
6+
zeroshot-rust/target
7+
*.log

AGENTS.md

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,9 @@ cannot succeed. After provider success, trusted Git delivery verifies the mutati
213213
configuration, and that the retained revision remains an ancestor of the current target before
214214
pushing one deterministic-branch commit. Review delivery succeeds only with a verified open pull
215215
request. Ship delivery succeeds only after an authoritative merge receipt or GitHub acceptance of
216-
merge-method auto-merge; an open pull request alone never succeeds. The backend validates the
216+
merge-method auto-merge; an open pull request alone never succeeds. Transient auto-merge
217+
enablement failures use one bounded retry policy and re-read the exact pull-request authority
218+
before retrying or accepting a lost response. The backend validates the
217219
secret-free, versioned result, and any execution, cleanup, or delivery defect produces a closed
218220
failure. Keep the runtime, binary, image, and manifest private. Provider and harness interpretation
219221
belongs only in the Zeroshot runtime bundle and worker, never in hosting, IaC, or the run-intent
@@ -226,6 +228,9 @@ Structured-output recovery eligibility is also registry-derived: every engine wh
226228
capability is `true` or `experimental` must implement its provider-owned, fail-closed recovery
227229
adapter. Recovery always runs as a fresh nested turn with provider sessions, MCP, approval bypass,
228230
write-capable tools, network tools, and user-defined agents/configuration disabled.
231+
OMP SDK children inherit only the registry-declared non-secret configuration environment plus the
232+
fixed minimal process environment. Credentials remain on the private credential channel; never add
233+
arbitrary ambient passthrough or duplicate provider configuration lists beside the registry.
229234

230235
Cluster Protocol Rust types are the source of truth. Files under
231236
`protocol/openengine-cluster/v1/` are generated projections; update them with
@@ -617,11 +622,13 @@ Cluster preflight validates the selected registry entry's `settingsValidator` wi
617622
Detached provider tasks default to the `detached` execution boundary. Embedding runtimes that
618623
already own process, filesystem, and network isolation must set
619624
`ZEROSHOT_TASK_EXECUTION_CONTEXT=benchmark`; the task runner validates and propagates this
620-
provider-neutral boundary so adapters do not attempt incompatible nested containment.
625+
provider-neutral boundary so adapters do not attempt incompatible nested containment. Task
626+
preflight and provider command preparation must share `src/task-execution-context.js`; availability
627+
probes must receive the same validated boundary that command preparation will use.
621628

622629
OMP's supported version, package identity, and release asset digests are pinned once in `omp-release.ts`; the RPC codec and any registry/version-probing/Docker-build code import it. Never recopy the version string, asset names, or digests elsewhere.
623630

624-
OMP's `rpc-stdio` invoke lane uses one shared lifecycle driver, `runOmpRpcTask` (`omp-rpc-driver.ts`), for both foreground (`contract-invoke.ts`) and detached (`task-lib/rpc-watcher.js`) execution, so the two paths produce identical result semantics. Spawn evidence is persisted (via the caller's `onSpawn` hook) before the first stdin write, and is reported only once the child process is confirmed spawned (the Node `'spawn'` event) — never synthesized from a pre-spawn/undefined pid, which would let ownership-based termination signal an unrelated process. Output is normalized-events-only: raw RPC frames, prompt text, and control payloads are never logged, only `OutputEvent`s (`omp-rpc-events.ts`). The detached watcher's prompt never enters its argv (`ps` and `/proc/<pid>/cmdline` expose argv to every local user for the watcher's whole lifetime); `task-lib/runner.js` hands it over the private, length-prefixed stdin pipe in `src/watcher-prompt-channel.js`, and the watcher fails closed — no OMP spawn, ownership-aware cleanup still runs — when that channel is absent, truncated, over the pinned 1 MiB frame contract, or closed before a complete payload. The per-task OMP config overlay (`omp-config-overlay.ts`) and its cleanup are ownership-checked by the shared `src/command-cleanup-ownership.js` owner used by both cleanup call sites; a failed or unsafe cleanup leaves the task's cleanup receipt intact (durably retryable) instead of silently discarding it. Provider `dockerIsolation`/`worktreeIsolation` capabilities are gated in `orchestrator.js` and `preflight.js` before any container/worktree is created, not after.
631+
OMP's `rpc-stdio` invoke lane uses one shared lifecycle driver, `runOmpRpcTask` (`omp-rpc-driver.ts`), for both foreground (`contract-invoke.ts`) and detached (`task-lib/rpc-watcher.js`) execution, so the two paths produce identical result semantics. Spawn evidence is persisted (via the caller's `onSpawn` hook) before the first stdin write, and is reported only once the child process is confirmed spawned (the Node `'spawn'` event) — never synthesized from a pre-spawn/undefined pid, which would let ownership-based termination signal an unrelated process. Output is normalized-events-only: raw RPC frames, prompt text, and control payloads are never logged, only `OutputEvent`s (`omp-rpc-events.ts`). The detached watcher's prompt never enters its argv (`ps` and `/proc/<pid>/cmdline` expose argv to every local user for the watcher's whole lifetime); `task-lib/runner.js` hands it over the private, length-prefixed stdin pipe in `src/watcher-prompt-channel.js`. Watchers are plain detached Node children with no IPC channel, so wrapper completion has exactly one lifetime owner. The watcher fails closed — no OMP spawn, ownership-aware cleanup still runs — when the prompt channel is absent, truncated, over the pinned 1 MiB frame contract, or closed before a complete payload. The per-task OMP config overlay (`omp-config-overlay.ts`) and its cleanup are ownership-checked by the shared `src/command-cleanup-ownership.js` owner used by both cleanup call sites; a failed or unsafe cleanup leaves the task's cleanup receipt intact (durably retryable) instead of silently discarding it. Provider `dockerIsolation`/`worktreeIsolation` capabilities are gated in `orchestrator.js` and `preflight.js` before any container/worktree is created, not after.
625632

626633
OMP session persistence (issue #866): fresh runs pass `--session-dir <partition>`; verified resume
627634
adds `--resume <partition>/<file>` as the exact absolute path Zeroshot already verified, never a
@@ -777,7 +784,9 @@ before a task row exists. The watcher then compares the **complete** committed t
777784
id, full session file path (never a basename), partition and session-file inode identity, artifact
778785
manifest digest, and an `executionFingerprint` (`src/omp-execution-fingerprint.ts`, with generated CommonJS at the matching `.js` path) binding the
779786
pinned OMP release, the config-overlay content digest, the requested `--model`/`--thinking`/
780-
`--approval-mode` selectors, and the concrete provider/model/thinking level OMP reported. That
787+
`--approval-mode` selectors, and the concrete provider/model/thinking level OMP reported. OMP
788+
catalog aliases such as `openrouter/~anthropic/claude-sonnet-latest` are valid exact selectors;
789+
the `~` prefix is accepted only at the start of the model portion. That
781790
fingerprint has exactly one implementation, `src/omp-execution-fingerprint.ts`; do not add a second
782791
digest helper beside the ownership schema, where only its own unit test would exercise it and it
783792
could silently drift from the contract production uses. Every failed, cancelled, or uncertain
@@ -912,6 +921,28 @@ isolated followers retain a bounded complete-record tail for parsing and diagnos
912921
terminal settlement. Isolated settlement re-reads only a fixed-size file tail; the raw task log is
913922
the sole complete-output authority. Never restore whole-record watcher buffering, cumulative output
914923
strings, whole-log terminal reads, or unbounded provider records/events in control-plane state.
924+
Normal host/detached Codex tasks stay in `workspace-write`. When `cwd` is a linked Git worktree,
925+
task preparation may set `additionalWritableDirectories` to the one resolved external Git common
926+
directory and the adapter maps it to deduplicated `--add-dir` arguments only when Codex advertises
927+
that flag. Host/detached runs enable Codex's explicit workspace-write network capability so normal
928+
API, dependency, and Git journeys work without widening filesystem access. GitHub delivery rewrites
929+
GitHub SSH remotes to HTTPS for the push command and delegates credentials to `gh auth
930+
git-credential`; never embed a token in the prompt or remote. Hosted capsules declare the `docker`
931+
execution context and benchmark containers declare
932+
`benchmark`; both use `danger-full-access` because the container is the security boundary, and they
933+
must not receive redundant host Git-directory grants. Do not enable `danger-full-access` for host or
934+
detached execution, do not add unrelated cache or socket paths, and never carry either permission
935+
into read-only structured-output recovery.
936+
Successful `--pr`/`--ship` auto-cleanup may close and remove the live cluster before a foreground
937+
caller emits its result. The orchestrator must retain a bounded in-process final-run handoff
938+
(settled status, ledger snapshot, and terminal messages) until `close()`; foreground reporting must
939+
consume that handoff rather than reopening or querying the closed ledger.
940+
Claude gateway authentication may use `ANTHROPIC_AUTH_TOKEN` with `ANTHROPIC_BASE_URL` and an
941+
explicitly empty `ANTHROPIC_API_KEY`; keep the token, endpoint, and Claude role-model selectors in
942+
the provider's declared isolation passthrough rather than persisting gateway secrets in settings.
943+
When both gateway token and endpoint are explicit, the Zeroshot-owned per-run `--settings` safety
944+
overlay must set the Bedrock, Vertex, and Foundry backend selectors to `0`; this overrides stale
945+
ambient user backend choices while preserving the user's other settings and repository context.
915946
Provider terminal failures are parsed from the newest typed terminal event before generic status
916947
text. Raw provider diagnostics remain task-log-only; `AGENT_OUTPUT`, `failureInfo`, `AGENT_ERROR`,
917948
and `CLUSTER_FAILED` retain only a synthesized error plus provider/event/category/retryability and

cli/index.js

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,10 @@ const {
7878
resolveEffectiveRunPlan,
7979
} = require('../lib/start-cluster');
8080
const { requirePreflight } = require('../src/preflight');
81+
const { resolveTaskExecutionContext } = require('../src/task-execution-context');
8182
const {
83+
buildForegroundResult,
84+
exitCodeForForegroundResult,
8285
exitCodeForResult,
8386
isForegroundStatusSettled,
8487
writeForegroundResult,
@@ -752,16 +755,25 @@ async function finishForegroundRun({ cluster, orchestrator, clusterId, plan, res
752755
plan,
753756
Boolean(resultPath)
754757
);
755-
if (!resultPath) return;
756-
const receipt = writeForegroundResult({
758+
if (resultPath) {
759+
const receipt = writeForegroundResult({
760+
orchestrator,
761+
cluster,
762+
clusterId,
763+
resultPath,
764+
cancelled: foreground.cancelled,
765+
});
766+
process.exitCode = exitCodeForResult(receipt);
767+
console.log(chalk.dim(`Result ${receipt.outcome} committed to ${resultPath}`));
768+
return;
769+
}
770+
const result = buildForegroundResult({
757771
orchestrator,
758772
cluster,
759773
clusterId,
760-
resultPath,
761774
cancelled: foreground.cancelled,
762775
});
763-
process.exitCode = exitCodeForResult(receipt);
764-
console.log(chalk.dim(`Result ${receipt.outcome} committed to ${resultPath}`));
776+
process.exitCode = exitCodeForForegroundResult(result);
765777
} finally {
766778
orchestrator.close();
767779
}
@@ -3053,6 +3065,7 @@ taskCmd
30533065
requireDocker: false, // Docker not needed for plain tasks
30543066
quiet: false,
30553067
provider: providerOverride,
3068+
executionContext: resolveTaskExecutionContext(),
30563069
});
30573070

30583071
// Dynamically import task command (ESM module)
@@ -3727,6 +3740,7 @@ program
37273740
requireDocker: false,
37283741
quiet: false,
37293742
provider: providerName,
3743+
executionContext: resolveTaskExecutionContext(),
37303744
});
37313745

37323746
// Try resuming as task
@@ -5092,9 +5106,18 @@ function formatToolCall(toolName, input) {
50925106
function formatToolResult(content, isError, toolName, toolInput) {
50935107
if (!content) return isError ? 'error' : 'done';
50945108

5109+
let normalizedContent = content;
5110+
if (typeof content !== 'string') {
5111+
try {
5112+
normalizedContent = JSON.stringify(content) ?? String(content);
5113+
} catch {
5114+
normalizedContent = String(content);
5115+
}
5116+
}
5117+
50955118
// For errors, show full message
50965119
if (isError) {
5097-
const firstLine = content.split('\n')[0].substring(0, 80);
5120+
const firstLine = normalizedContent.split('\n')[0].substring(0, 80);
50985121
return chalk.red(firstLine);
50995122
}
51005123

@@ -5123,7 +5146,7 @@ function formatToolResult(content, isError, toolName, toolInput) {
51235146
}
51245147

51255148
// For success, show summary
5126-
const lines = content.split('\n').filter((l) => l.trim());
5149+
const lines = normalizedContent.split('\n').filter((l) => l.trim());
51275150
if (lines.length === 0) return 'done';
51285151
if (lines.length === 1) {
51295152
const line = lines[0].substring(0, 60);

docker/zeroshot-oecp/.dockerignore

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,7 @@
4242
!lib/cluster-worker/worker-internals.js
4343
!scripts/
4444
!scripts/omp/
45-
!scripts/omp/runtime.js
46-
!scripts/omp/runtime-identities.js
47-
!scripts/omp/runtime-lock.js
48-
!scripts/omp/runtime-release.js
45+
!scripts/omp/**
4946
!src/
5047
!src/**
5148
src/target/

docker/zeroshot-oecp/Dockerfile

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,23 @@ RUN npm ci --omit=dev --ignore-scripts --no-audit --no-fund \
2525
&& node -e "const pkg = require( \
2626
'./node_modules/@earendil-works/pi-coding-agent/node_modules/undici/package.json'); \
2727
if (pkg.version !== '8.9.0') throw new Error('expected undici@8.9.0, got ' + pkg.version)"
28+
COPY scripts/omp/runtime-identities.js ./omp-runtime-identities.js
29+
RUN node -e "const fs = require('fs'); const path = require('path'); \
30+
const { execFileSync } = require('child_process'); \
31+
const ids = require('./omp-runtime-identities'); \
32+
const platformPackage = ids.BUN_PLATFORM_PACKAGES[process.platform + ':' + process.arch]; \
33+
if (!platformPackage) throw new Error('unsupported hosted Bun platform'); \
34+
const platformManifestPath = require.resolve(platformPackage + '/package.json'); \
35+
ids.assertPackageIdentity(platformPackage, ids.BUN_RUNTIME_VERSION, { \
36+
packageJsonPath: platformManifestPath, manifest: require(platformManifestPath) }); \
37+
const bunManifestPath = require.resolve(ids.BUN_PACKAGE_NAME + '/package.json'); \
38+
ids.assertPackageIdentity(ids.BUN_PACKAGE_NAME, ids.BUN_RUNTIME_VERSION, { \
39+
packageJsonPath: bunManifestPath, manifest: require(bunManifestPath) }); \
40+
const source = path.join(path.dirname(platformManifestPath), 'bin', 'bun'); \
41+
const target = path.join(path.dirname(bunManifestPath), 'bin', 'bun.exe'); \
42+
fs.copyFileSync(source, target); fs.chmodSync(target, 0o755); \
43+
if (execFileSync(target, ['--version'], { encoding: 'utf8' }).trim() \
44+
!== ids.BUN_RUNTIME_VERSION) throw new Error('hosted Bun version drift');"
2845

2946
FROM docker.io/instructure/tini:v0.19.0@sha256:0f9ccf2e54d010735ad691ebe670e256a3820fe8b92c28101194c68ee1216092 AS tini
3047

@@ -54,7 +71,7 @@ COPY cluster-templates /opt/zeroshot/cluster-templates
5471
COPY lib /opt/zeroshot/lib
5572
COPY src /opt/zeroshot/src
5673
COPY task-lib /opt/zeroshot/task-lib
57-
COPY scripts/omp/runtime.js scripts/omp/runtime-identities.js scripts/omp/runtime-lock.js scripts/omp/runtime-release.js /opt/zeroshot/scripts/omp/
74+
COPY scripts/omp /opt/zeroshot/scripts/omp
5875
COPY src/command-cleanup-ownership.js src/omp-config-overlay.js src/worktree-claude-config.js src/worktree-tooling-env.js /opt/zeroshot/src/
5976
COPY protocol/openengine-cluster/v1/worker.schema.json /opt/zeroshot/protocol/openengine-cluster/v1/worker.schema.json
6077
COPY zeroshot-rust/hosted-node /opt/zeroshot/zeroshot-rust/hosted-node

docker/zeroshot-oecp/Dockerfile.dockerignore

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,7 @@
4242
!lib/cluster-worker/worker-internals.js
4343
!scripts/
4444
!scripts/omp/
45-
!scripts/omp/runtime.js
46-
!scripts/omp/runtime-identities.js
47-
!scripts/omp/runtime-lock.js
48-
!scripts/omp/runtime-release.js
45+
!scripts/omp/**
4946
!src/
5047
!src/**
5148
src/target/

0 commit comments

Comments
 (0)