Skip to content

Commit 8020b63

Browse files
lpcoxCopilotCopilot
authored
feat: add Docker sbx microVM runtime support (#6101)
* feat: add Docker sbx microVM runtime support - Activate sbx in RUNTIME_REGISTRY (executionModel: microvm) - Gate agent service in compose-generator: skip when !runtimeUsesComposeAgent - New src/sbx-manager.ts: create/exec/wait/rm sandbox lifecycle - Wire sbx path in main-action.ts: infra-only compose, then sbx exec - Add sbx cleanup to buildCleanupFn - Update JSON schemas, spec doc, CLI help with sbx option - Add 4 sbx tests to container-runtime.test.ts (14 total) - Postprocess script injects --container-runtime sbx into lock file Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: add sbx CLI install step to smoke-docker-sbx workflow Adds Docker sbx CLI installation via apt (docker-sbx package) and KVM availability check before the AWF invocation step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: use 'sbx version' instead of 'sbx --version' The sbx CLI uses subcommand style (sbx version) not flag style (sbx --version). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: add sbx login step with Docker PAT authentication sbx requires Docker authentication even for basic sandbox creation. Uses DOCKER_PAT and DOCKER_USERNAME repo secrets. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: strip secret env vars from sbx CLI calls createSandbox() and execInSandbox() were spreading process.env into sbx CLI invocations, which could leak credential-bearing env vars into the sbx sandbox. Add sanitizeEnvForSbx() that strips env vars matching secret patterns (TOKEN, SECRET, PASSWORD, KEY, CREDENTIAL, PAT) before passing to sbx. Also removes redundant DOCKER_PAT env from the lock.yml login step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: address sbx runtime review feedback * fix: pass full env to sbx create, sanitize only sbx exec sbx create is a host-side management operation that needs Docker auth credentials (stored on disk by sbx login). Only sbx exec (which runs commands inside the sandbox) gets the sanitized environment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: add sbx connectivity diagnostics before agent launch Run a diagnostic check inside the sandbox after creation to verify: - Network interfaces and DNS config - Proxy connectivity to Squid - Direct connectivity (should fail if iptables are working) - Environment variables (sans secrets) Also adds logging around the agent command execution to help identify where hangs occur. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: add timeouts and verbose logging to sbx daemon/create/exec The previous run hung for 14 minutes with zero output after 'Configuring sbx daemon proxy'. This adds: - 30s timeouts on all sbx daemon operations (status/stop/start) - 2min timeout on sbx create - Per-step [sbx-daemon] log messages with stdout/stderr capture - Error output capture on sbx create failure - Connectivity diagnostics inside sandbox before agent launch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: remove sbx daemon restart, pass proxy via env to sbx create sbx daemon start runs in foreground (blocks forever), not as a background service launcher. The previous approach of stopping and restarting the daemon was hanging the CI run for 14 minutes. Instead, pass DOCKER_SANDBOXES_PROXY as an environment variable directly to the sbx create command. The daemon is already running (started by sbx login/install), so we just need the proxy env var set when creating the sandbox. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: add auth/daemon diagnostics before sbx create Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: background sbx daemon with nohup+disown for cross-step persistence sbx auth requires a running daemon. The daemon is a foreground process that must be backgrounded with nohup+disown to survive across GitHub Actions steps. Added daemon startup with polling wait, then login, then auth verification. Also replaced diagnostic dumps with a proper auth pre-check that fails fast with an actionable error message. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: replace nonexistent 'sbx auth status' with 'sbx ls' probe sbx has no 'auth status' subcommand. Use 'sbx ls' which requires auth and returns exit 0 when authenticated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: initialize sbx network policy before sandbox creation sbx requires a global network policy to be set before 'sbx create' can succeed. Try multiple policy initialization approaches as the exact syntax may vary by version. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: use correct 'sbx policy init' (no --global flag) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: add sbx help output for policy/create syntax discovery Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: use 'sbx policy init allow-all' for network policy Correct syntax is: sbx policy init <allow-all|balanced|deny-all> Using allow-all since AWF's Squid proxy handles domain filtering. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: pipe 'y' to sbx create stdin for interactive confirmation sbx create prompts for user confirmation. With stdin as /dev/null it fails with 'user cancelled operation'. Pipe 'y' via input option. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: use 'yes | sbx create' to bypass interactive TTY check sbx create checks if stdin is a TTY for confirmation. Even piping 'y' via execa input fails because it's not a TTY. Use 'yes |' via bash shell to provide continuous confirmation stream. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: add docker login for template image pulls in sbx create sbx create shell pulls docker/sandbox-templates:shell-docker from Docker Hub and needs registry credentials in Docker's credential store (separate from sbx login which authenticates with the sbx service). Also dumps sbx create --help for flag reference. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: validate Docker Hub credentials before sbx auth Adds pre-validation step that: 1. Checks secret env vars are non-empty (with length output) 2. Tests docker login first (validates creds work for registry) 3. Then does sbx login (service auth) This ensures registry credentials are in Docker's credential store before sbx create tries to pull docker/sandbox-templates:shell-docker. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: remove DOCKER_SANDBOXES_PROXY from sbx create env The DOCKER_SANDBOXES_PROXY env var was being picked up by the sbx CLI itself, routing its Docker Hub auth/registry requests through Squid. This broke credential lookup ('no default account profile set'). The proxy is for sandbox egress, not for host-side management ops. It's already configured inside the sandbox via buildAgentEnvironment() which sets HTTP_PROXY/HTTPS_PROXY and passes them to sbx exec --env. Security note: process.env (which may contain DOCKER_PAT) is safe to pass to sbx create because it's a host-side CLI command. The sandbox is a KVM microVM with hardware isolation — host env vars never reach the sandbox interior. Only sanitizeEnvForSbx()-filtered vars are passed to sbx exec. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: use printf for sbx login, add sbx diagnose - printf '%s' avoids trailing newline that echo adds to PAT - sbx diagnose gives full system state dump for debugging Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: sbx create --debug logging + pre-agent create test - Log sbx create stdout/stderr at info level (was debug, invisible) - Add --debug flag to sbx create for verbose daemon output - Add DOCKER_CONFIG env var pointing at ~/.docker - Test sbx create directly in auth step to isolate env vs auth issue Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: use workspace dir for sbx create test, dump policy help /tmp was blocked by mount policy. Try workspace dir instead and dump 'sbx policy --help' to see available policy subcommands. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: convert Docker mount format to sbx positional format sbx uses positional workspace paths (host path = VM path) with optional :ro suffix. AWF's --mount flags use Docker-style 'host:container:mode' format. Convert by extracting host path and preserving :ro mode. Also note: sbx cannot remap paths (/host prefix from chroot convention is not applicable in microVM context). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: add fs mount policies and pre-pull template image Two issues discovered: 1. sbx policy init allow-all only covers network, not filesystem mounts. Added sbx policy allow for fs:mount:read/write. 2. AWF's sbx create can't pull template from Docker Hub because the step doesn't have DOCKER_PAT in env (daemon credential lookup fails). Pre-pull the image in the auth step so sbx create uses the cached image. Also dumps 'sbx policy allow --help' and 'sbx policy ls' for debugging the correct policy syntax. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: dump policy profile/inspect/init help for mount policy allow-all sets network + fs read/write but NOT fs:mount:read/write. Need to understand the policy model to enable host path mounts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: inspect policy details, try --clone and bare create Trying multiple approaches: 1. Inspect local-policy to see actual rules 2. List available profiles 3. Try --clone (might bypass host mount policy) 4. Try bare create without workspace path Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: check KVM, try --clone and direct mount with --debug --clone bypasses mount policy but hits 500 error — might be KVM. Adding KVM diagnostics and --debug to both create paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: early exit after auth step to speed up iteration Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: explore mount policy config, daemon logs, sbx setup Investigating: - sbx setup --help (might have mount config) - daemon config files and policy store - daemon log for mount denial details - sbx create with . vs absolute path - sbx run (one-step, might handle mounts differently) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: structured sbx mount policy diagnostics Replaces ad-hoc debugging with structured diagnostic script: 1. Version + initial policy state (--wide, --type filesystem, --include-inactive) 2. Full policy reset + re-init cycle with daemon restart 3. Resulting policy state after reset 4. Workspace path verification 5. Clone-based sandbox test with --debug 6. Direct mount sandbox test with --debug 7. Daemon log for policy evaluation details Uses set -uxo pipefail (no -e) to collect all diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: chmod /dev/kvm for sbx microVM access Mount policy is solved after policy reset cycle. The actual blocker was KVM permission denied (os error 13) — runner user not in kvm group. Fix: sudo chmod 666 /dev/kvm before sbx create. Daemon log confirmed: - mount policy: path allowed (fs:mount:read + fs:mount:write) - failed to create VM: KVM error: Permission denied (os error 13) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: sbx create works! Remove early exit, add exec test sbx create succeeded: ✓ Created sandbox 'test-sandbox-direct' Workspace: /home/runner/work/gh-aw-firewall/gh-aw-firewall (direct mount) Agent: shell / Status: running Fixes: - Move KVM chmod to Install step (before daemon start) - Remove early exit 1 to let full AWF flow run - Add sbx exec test (uname -a) to verify VM is functional - Fix broken pipe false failure (bash -c isolation) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: add credential state diagnostics + fix broken pipe exit code sbx create succeeded in auth step but fails in AWF step with 'no default account profile set: secret not found'. Adding diagnostics: - Dump HOME, credential dir, Docker config dir, daemon socket - Fix broken pipe false failure: check stdout for 'Created sandbox' - Track SBX_EXIT_CODE separately from bash pipeline exit code Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: refresh sbx credentials immediately before AWF execution Docker Hub OAuth tokens from 'sbx login' expire between workflow steps. Add a credential refresh step right before 'Execute GitHub Copilot CLI' to ensure sbx create can authenticate with Docker Hub. Auth step confirmed working: sbx create succeeds there. AWF step fails with 'secret not found' minutes later. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: remove XDG_CONFIG_HOME from sbx create env The AWF execution step sets XDG_CONFIG_HOME=$HOME (/home/runner) for Copilot CLI. This breaks sbx credential lookup — sbx stores encrypted secrets at $XDG_CONFIG_HOME/sandboxes/ which defaults to ~/.config/sandboxes/. When XDG_CONFIG_HOME=/home/runner, sbx looks at /home/runner/sandboxes/ instead of /home/runner/.config/sandboxes/, causing 'secret not found'. Fix: unset XDG_CONFIG_HOME when it equals HOME before calling sbx create. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug: unset XDG_CONFIG_HOME in bash + find credential files XDG_CONFIG_HOME removal via env object didn't fix it. Try unsetting inside bash itself. Also improved diagnostics to find all credential- related files on disk (state, config, docker dirs) to determine where sbx actually stores secrets. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: mount /tmp and /usr/local/bin into sbx, deduplicate mounts, clean up Agent command failed because /usr/local/bin/copilot and /tmp/gh-aw/ (prompts, logs) were not mounted into the microVM. Changes: - Add /tmp and /usr/local/bin as default mounts in createSandbox() - Deduplicate mount paths (same path was passed twice) - Remove debug diagnostics (find credential files, etc.) - Clean up auth step: remove verbose echo/policy dumps, keep essentials - Keep 'Refresh sbx credentials' step as insurance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: mount $HOME into sbx for agent writable dirs (.cache, etc.) Copilot CLI needs writable access to ~/.cache to extract its bundled package. Without $HOME mounted, the microVM has no /home/runner and mkdir fails with EACCES. Progress: binary found ✓, prompt file found ✓, now need HOME writable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * diag: probe sbx network routes and gateway to find host access sbx microVM can't reach Docker Compose containers at 172.30.0.10. Need to find the sbx gateway IP to route through Squid published on host port 3128. Testing: route table, common gateway IPs (192.168.127.1, 10.0.2.2, 172.17.0.1), direct outbound, and host.docker.internal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * diag: test DNS resolution and raw IP connectivity from sbx Direct internet and proxy both time out from sbx microVM. Adding: nslookup, getent, curl with --resolve (raw IP bypass DNS), and gateway IP tests on 172.17.0.0 and 172.17.0.1. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): route proxy through sbx gateway IP + publish api-proxy ports The sbx microVM can't reach Docker-internal IPs (172.30.0.x). Diagnostics confirmed that the sbx gateway IP (172.17.0.0) with Squid's published port 3128 returns HTTP 200. Changes: - Use SBX_GATEWAY_IP (172.17.0.0) for proxy env vars in sbx mode instead of the Docker-internal SQUID_IP (172.30.0.10) - Publish api-proxy ports (10000-10004) to host when agent runs outside compose (microVM runtimes like sbx) - Simplify sbx diagnostics to a quick proxy connectivity check - Add test for api-proxy port publishing with sbx runtime Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * diag: log sbx env vars and test api-proxy port reachability Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): merge credential isolation env vars for api-proxy routing buildAgentEnvironment() does not include the credential isolation env vars (COPILOT_API_URL, COPILOT_PROVIDER_BASE_URL, OPENAI_BASE_URL, etc.) because in Docker mode they are merged by assembleOptionalServices() during compose generation. For sbx, call buildAgentCredentialEnv() directly with the sbx gateway IP so the Copilot CLI enters BYOK/offline mode and routes through the api-proxy sidecar instead of attempting direct PAT validation against api.github.com. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): wait for api-proxy health + dump logs on failure Add health check polling (up to 30s) before launching the agent in sbx mode. In Docker mode, depends_on: service_healthy gates this; for sbx we must poll manually from within the microVM. Also dump api-proxy container logs when the agent command fails to help diagnose the 'connection closed before message completed' issue. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): attach api-proxy to awf-ext network for port publishing In network-isolation mode, awf-net is an internal Docker network with no route to the host. Port publishing on containers only attached to an internal network doesn't work — the host (and thus the sbx microVM) cannot reach them. Fix: when the agent runs in a microVM and network isolation is active, also attach api-proxy to the awf-ext bridge network (same as Squid). This makes published ports (10000-10004) reachable from the sbx gateway. Root cause of 'connection closed before message completed' — the sbx VM could TCP-connect to the published port (Docker accepts the SYN) but the internal-only container never received the forwarded traffic. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * debug(sbx): add port reachability diagnostics from sbx VM Try multiple candidate IPs (172.17.0.0, 10.0.2.2, 172.17.0.1) from inside the sbx to identify which address reaches Docker published ports. Also dumps docker port and network info from host perspective. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): use host.docker.internal for api-proxy from microVM The sbx microVM cannot reach Docker-published ports via the gateway IP (172.17.0.0) for the api-proxy — port publishing isn't effective for containers on an internal Docker network. However, host.docker.internal resolves to the docker0 bridge (172.17.0.1) inside the sbx VM, which CAN route to containers on the awf-ext bridge network. Diagnostics proved: - 172.17.0.0:10000 → FAIL (timeout, no port publishing) - 172.17.0.1:10000 → OK (instant, docker0 bridge routes to awf-ext) - 172.17.0.0:3128 → OK (Squid has native port publishing) Use host.docker.internal for api-proxy URLs (COPILOT_API_URL, COPILOT_PROVIDER_BASE_URL, etc.) while keeping 172.17.0.0 for Squid proxy which has working port publishing. Also simplify health check to use host.docker.internal and remove the multi-IP diagnostic probing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): route MCP gateway through host.docker.internal The MCP gateway runs as a Docker container published on the host. From inside the sbx microVM, Docker container names (awmg-mcpg) are not resolvable, so the Copilot CLI couldn't reach the gateway to call safe-output tools (noop, add-comment, add-labels). Changes: - Use host.docker.internal as the gateway domain in mcp-config.json - Bind MCP gateway to 0.0.0.0 (needed for host.docker.internal access) - Set MCP_GATEWAY_HOST_DOMAIN to host.docker.internal Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(sbx): use host.docker.internal for MCP CLI wrappers The mount_mcp_as_cli step generates CLI wrapper scripts that the Copilot CLI calls to invoke MCP tools (safe-outputs). The wrappers use MCP_GATEWAY_DOMAIN to build the URL. In the Docker agent case, this is the container name (awmg-mcpg) resolvable within the Docker network. In the sbx case, Docker names aren't resolvable from the microVM — host.docker.internal must be used instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: avoid shell-based sbx create command injection path * fix: avoid clear-text logging of sensitive env vars (CodeQL) Use a redactSecret() helper that only reports whether a secret is set and its length, without passing the secret value into the log template literal. This avoids the js/clear-text-logging CodeQL alert. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent 47458f0 commit 8020b63

15 files changed

Lines changed: 855 additions & 36 deletions

.github/workflows/smoke-docker-sbx.lock.yml

Lines changed: 89 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/awf-config-spec.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ AWF settings MAY be supplied via config files, including stdin (`--config -`).
183183
- `container.dockerHostPathPrefix``--docker-host-path-prefix`
184184
- `container.runnerToolCachePath`*(config-only; checked first for optional read-only runner tool cache mount, before `RUNNER_TOOL_CACHE` and `/home/runner/work/_tool` auto-detection)*
185185
- `container.mounts[]``-v, --mount` *(repeatable; each array entry maps to one Docker volume mount in `/host_path:/container_path[:ro|rw]` format (both paths must be absolute; host path must exist); in chroot mode, container paths are automatically prefixed with `/host`)*
186-
- `container.containerRuntime``--container-runtime` *(user-facing runtime name, e.g. `"gvisor"`; AWF translates to the Docker OCI runtime identifier, e.g. `"runsc"`. Only the agent container uses the custom runtime; infrastructure containers always use `runc`. When set, AWF injects `extra_hosts` entries for compose-internal services to work around DNS issues with non-default runtimes. Requires the runtime to be installed and registered with Docker on the host.)*
186+
- `container.containerRuntime``--container-runtime` *(user-facing runtime name: `"gvisor"` for OCI runtime in compose, `"sbx"` for Docker sbx microVM. For gvisor: translates to `"runsc"`, injects `extra_hosts` for DNS workaround. For sbx: agent runs in a hypervisor-isolated microVM, infra stays in compose, sbx proxy chains through AWF's Squid.)*
187187
- `chroot.binariesSourcePath`*(config-only; overlays a runner-side binaries directory at `/usr/local/bin` inside chroot mode)*
188188
- `chroot.identity.home`*(config-only; forwarded as `AWF_CHROOT_IDENTITY_HOME` and applied after chroot pivot)*
189189
- `chroot.identity.user`*(config-only; forwarded as `AWF_CHROOT_IDENTITY_USER` and applied to `USER`/`LOGNAME` after chroot pivot)*

docs/awf-config.schema.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -622,8 +622,8 @@
622622
},
623623
"containerRuntime": {
624624
"type": "string",
625-
"enum": ["gvisor"],
626-
"description": "Container runtime for the agent container. Set to \"gvisor\" to run the agent under gVisor's runsc runtime for additional sandboxing. Only the agent container uses the custom runtime; infrastructure containers (squid-proxy, api-proxy) always use the default runc runtime. When set, AWF automatically injects extra_hosts entries for compose-internal services to work around DNS resolution issues with non-default runtimes. Requires gVisor (runsc) to be installed and registered with Docker on the host."
625+
"enum": ["gvisor", "sbx"],
626+
"description": "Container runtime for the agent container. \"gvisor\" runs the agent under gVisor's runsc runtime (OCI runtime, compose-based). \"sbx\" runs the agent inside a Docker sbx microVM with hypervisor isolation; infrastructure containers (squid-proxy, api-proxy) stay in Docker Compose on the host and the sbx proxy chains upstream through AWF's Squid for domain filtering. Only the agent uses the custom runtime; infrastructure containers always use the default runc runtime."
627627
}
628628
}
629629
},

scripts/ci/postprocess-smoke-workflows.ts

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -75,14 +75,14 @@ for (const workflowPath of codexWorkflowPaths) {
7575
}
7676
}
7777

78-
// ── gVisor workflow: inject --container-runtime gvisor into the AWF command ───
78+
// ── Runtime workflow patching: inject --container-runtime into AWF commands ───
79+
// The compiler doesn't support sandbox.agent.containerRuntime yet, so we inject it here.
80+
const runtimeCmdPattern = /awf --config /g;
81+
7982
const gvisorLockPath = path.join(workflowsDir, 'smoke-gvisor.lock.yml');
8083
try {
81-
let gvisorContent = fs.readFileSync(gvisorLockPath, 'utf-8');
82-
// Insert --container-runtime gvisor before --config on the awf command line.
83-
// The compiler doesn't support sandbox.agent.containerRuntime yet, so we inject it here.
84-
const awfCmdPattern = /awf --config /g;
85-
const replacedContent = gvisorContent.replace(awfCmdPattern, 'awf --container-runtime gvisor --config ');
84+
const gvisorContent = fs.readFileSync(gvisorLockPath, 'utf-8');
85+
const replacedContent = gvisorContent.replace(runtimeCmdPattern, 'awf --container-runtime gvisor --config ');
8686
if (replacedContent !== gvisorContent) {
8787
fs.writeFileSync(gvisorLockPath, replacedContent);
8888
console.log(` Injected --container-runtime gvisor into AWF command`);
@@ -93,3 +93,19 @@ try {
9393
} catch {
9494
console.log(`Skipping ${gvisorLockPath}: file not found.`);
9595
}
96+
97+
const sbxLockPath = path.join(workflowsDir, 'smoke-docker-sbx.lock.yml');
98+
try {
99+
const sbxContent = fs.readFileSync(sbxLockPath, 'utf-8');
100+
runtimeCmdPattern.lastIndex = 0;
101+
const sbxReplacedContent = sbxContent.replace(runtimeCmdPattern, 'awf --container-runtime sbx --config ');
102+
if (sbxReplacedContent !== sbxContent) {
103+
fs.writeFileSync(sbxLockPath, sbxReplacedContent);
104+
console.log(` Injected --container-runtime sbx into AWF command`);
105+
console.log(`Updated ${sbxLockPath}`);
106+
} else {
107+
console.log(`Skipping ${sbxLockPath}: no AWF command found to patch.`);
108+
}
109+
} catch {
110+
console.log(`Skipping ${sbxLockPath}: file not found.`);
111+
}

src/awf-config-schema.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -622,8 +622,8 @@
622622
},
623623
"containerRuntime": {
624624
"type": "string",
625-
"enum": ["gvisor"],
626-
"description": "Container runtime for the agent container. Set to \"gvisor\" to run the agent under gVisor's runsc runtime for additional sandboxing. Only the agent container uses the custom runtime; infrastructure containers (squid-proxy, api-proxy) always use the default runc runtime. When set, AWF automatically injects extra_hosts entries for compose-internal services to work around DNS resolution issues with non-default runtimes. Requires gVisor (runsc) to be installed and registered with Docker on the host."
625+
"enum": ["gvisor", "sbx"],
626+
"description": "Container runtime for the agent container. \"gvisor\" runs the agent under gVisor's runsc runtime (OCI runtime, compose-based). \"sbx\" runs the agent inside a Docker sbx microVM with hypervisor isolation; infrastructure containers (squid-proxy, api-proxy) stay in Docker Compose on the host and the sbx proxy chains upstream through AWF's Squid for domain filtering. Only the agent uses the custom runtime; infrastructure containers always use the default runc runtime."
627627
}
628628
}
629629
},

src/cli-options.ts

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -171,9 +171,10 @@ program
171171
)
172172
.option(
173173
'--container-runtime <runtime>',
174-
'Container runtime for the agent container (e.g. "gvisor" for gVisor sandboxing).\n' +
175-
' AWF translates friendly names to Docker runtime identifiers\n' +
176-
' (gvisor → runsc). Unknown values are passed through as-is.'
174+
'Container runtime for the agent container.\n' +
175+
' "gvisor" — OCI runtime via Docker Compose (translates to runsc).\n' +
176+
' "sbx" — Docker sbx microVM with hypervisor isolation.\n' +
177+
' Unknown values are passed through as raw Docker runtime names.'
177178
)
178179

179180
// -- Container Configuration --

src/commands/main-action.test.ts

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ jest.mock('../dind-bootstrap');
3333
jest.mock('./preflight');
3434
jest.mock('./signal-handler');
3535
jest.mock('./validate-options');
36+
jest.mock('../sbx-manager');
3637

3738
import { logger } from '../logger';
3839
import * as dockerManager from '../docker-manager';
@@ -45,6 +46,7 @@ import * as dindBootstrap from '../dind-bootstrap';
4546
import * as preflight from './preflight';
4647
import * as signalHandler from './signal-handler';
4748
import * as validateOptions from './validate-options';
49+
import * as sbxManager from '../sbx-manager';
4850

4951
const mockedLogger = logger as jest.Mocked<typeof logger>;
5052
const mockedDockerManager = dockerManager as jest.Mocked<typeof dockerManager>;
@@ -57,6 +59,7 @@ const mockedDindBootstrap = dindBootstrap as jest.Mocked<typeof dindBootstrap>;
5759
const mockedPreflight = preflight as jest.Mocked<typeof preflight>;
5860
const mockedSignalHandler = signalHandler as jest.Mocked<typeof signalHandler>;
5961
const mockedValidateOptions = validateOptions as jest.Mocked<typeof validateOptions>;
62+
const mockedSbxManager = sbxManager as jest.Mocked<typeof sbxManager>;
6063

6164
/** Minimal WrapperConfig returned by the validateOptions mock. */
6265
const STUB_CONFIG = {
@@ -108,6 +111,10 @@ describe('createMainAction', () => {
108111
mockedDindBootstrap.runDindBootstrap.mockResolvedValue(undefined);
109112
mockedSignalHandler.registerSignalHandlers.mockImplementation(() => {});
110113
mockedCliWorkflow.runMainWorkflow.mockResolvedValue(0);
114+
mockedSbxManager.isSbxAvailable.mockResolvedValue(true);
115+
mockedSbxManager.createSandbox.mockResolvedValue('awf-agent-test');
116+
mockedSbxManager.execInSandbox.mockResolvedValue({ exitCode: 0 });
117+
mockedSbxManager.removeSandbox.mockResolvedValue(undefined);
111118
});
112119

113120
afterEach(() => {
@@ -292,6 +299,45 @@ describe('createMainAction', () => {
292299
await action(['curl https://example.com'], {});
293300
expect(processExitSpy).toHaveBeenCalledWith(42);
294301
});
302+
303+
describe('sbx runtime wiring', () => {
304+
it('passes configured mounts/workdir/environment into sbx create/exec', async () => {
305+
const sbxConfig = {
306+
...STUB_CONFIG,
307+
containerRuntime: 'sbx',
308+
containerWorkDir: '/home/runner/work/repo/repo',
309+
volumeMounts: ['/tmp/tooling:/tmp/tooling:ro'],
310+
enableApiProxy: true,
311+
tty: true,
312+
} as unknown as import('../types').WrapperConfig;
313+
mockedValidateOptions.validateOptions.mockReturnValue(sbxConfig);
314+
mockedCliWorkflow.runMainWorkflow.mockImplementation(async (_config, deps, _callbacks) => {
315+
await deps.startContainers('/tmp/awf-test', ['github.com']);
316+
const result = await deps.runAgentCommand('/tmp/awf-test', ['github.com'], undefined, 10);
317+
return result.exitCode;
318+
});
319+
320+
const action = createMainAction(getOptionValueSource);
321+
await action(['echo hi'], {});
322+
323+
expect(mockedSbxManager.createSandbox).toHaveBeenCalledWith(expect.objectContaining({
324+
extraMounts: ['/tmp/tooling:/tmp/tooling:ro'],
325+
}));
326+
expect(mockedSbxManager.execInSandbox).toHaveBeenCalledWith(
327+
'awf-agent-test',
328+
'echo hi',
329+
expect.objectContaining({
330+
timeoutMinutes: 10,
331+
workDir: '/home/runner/work/repo/repo',
332+
tty: true,
333+
environment: expect.objectContaining({
334+
HTTPS_PROXY: expect.any(String),
335+
SQUID_PROXY_HOST: expect.any(String),
336+
}),
337+
}),
338+
);
339+
});
340+
});
295341
});
296342

297343
describe('when runMainWorkflow throws', () => {

src/commands/main-action.ts

Lines changed: 163 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,19 @@ import { validateOptions } from './validate-options';
2626
import { probeSplitFilesystem } from '../dind-probe';
2727
import { assertTopologySupported, connectTopologyContainers } from '../topology';
2828
import { runDindBootstrap } from '../dind-bootstrap';
29+
import { runtimeUsesComposeAgent } from '../container-runtime';
30+
import { createSandbox, execInSandbox, removeSandbox, isSbxAvailable, SBX_DEFAULT_NAME } from '../sbx-manager';
2931
import type { WrapperConfig } from '../types';
32+
import { buildAgentEnvironment } from '../services/agent-service';
33+
import { buildAgentCredentialEnv } from '../services/api-proxy-credential-env';
34+
import { DEFAULT_DNS_SERVERS } from '../dns-resolver';
35+
import { AGENT_IP, CLI_PROXY_IP, DOH_PROXY_IP, SQUID_IP } from '../host-iptables-shared';
36+
37+
/** Report whether a secret is set (and its length) without exposing the value. */
38+
function redactSecret(value: string | undefined): string {
39+
if (!value) return '(unset)';
40+
return `(set, len=${value.length})`;
41+
}
3042

3143
const SENSITIVE_CONFIG_KEYS = new Set([
3244
'openaiApiKey',
@@ -89,6 +101,15 @@ function buildCleanupFn(
89101
logger.info(`Received ${signal}, cleaning up...`);
90102
}
91103

104+
// Clean up sbx sandbox if using microVM runtime
105+
if (!runtimeUsesComposeAgent(config.containerRuntime) && !config.keepContainers) {
106+
try {
107+
await removeSandbox(SBX_DEFAULT_NAME);
108+
} catch {
109+
// Sandbox may not exist yet — that's fine
110+
}
111+
}
112+
92113
// Copy iptables audit BEFORE stopping containers (volumes are destroyed by `docker compose down -v`)
93114
if (getContainersStarted()) {
94115
preserveIptablesAudit(config.workDir, config.auditDir);
@@ -231,14 +252,154 @@ export function createMainAction(getOptionValueSource: OptionSourceResolver) {
231252
});
232253

233254
try {
255+
// For sbx (microVM) runtime, wrap startContainers and runAgentCommand
256+
// to launch the agent in a sandbox instead of Docker Compose.
257+
const useSbx = !runtimeUsesComposeAgent(config.containerRuntime);
258+
let sbxName: string | undefined;
259+
let sbxEnvironment: Record<string, string> | undefined;
260+
261+
const sbxStartContainers = useSbx
262+
? async (workDir: string, allowedDomains: string[], proxyLogsDir?: string, skipPull?: boolean, onNetworkReady?: () => Promise<void>) => {
263+
// Start infra-only compose (squid, api-proxy — no agent service)
264+
await startContainers(workDir, allowedDomains, proxyLogsDir, skipPull, onNetworkReady);
265+
266+
// Verify sbx is available
267+
if (!await isSbxAvailable()) {
268+
throw new Error('Docker sbx CLI not found. Install sbx to use --container-runtime sbx.');
269+
}
270+
271+
// For sbx, the microVM can't reach Docker internal IPs (172.30.0.x).
272+
// Published Squid port (3128) is accessible via the sbx gateway IP.
273+
// The api-proxy is on the awf-ext bridge network and reachable from
274+
// inside the sbx via `host.docker.internal` (resolves to the docker0
275+
// bridge IP, typically 172.17.0.1).
276+
const SBX_GATEWAY_IP = '172.17.0.0';
277+
const SBX_HOST_DOCKER_INTERNAL = 'host.docker.internal';
278+
279+
sbxEnvironment = buildAgentEnvironment({
280+
config,
281+
networkConfig: {
282+
subnet: '172.30.0.0/24',
283+
squidIp: SBX_GATEWAY_IP,
284+
agentIp: AGENT_IP,
285+
proxyIp: config.enableApiProxy ? SBX_HOST_DOCKER_INTERNAL : undefined,
286+
dohProxyIp: config.dnsOverHttps ? DOH_PROXY_IP : undefined,
287+
cliProxyIp: config.difcProxyHost ? CLI_PROXY_IP : undefined,
288+
},
289+
dnsServers: config.dnsServers || DEFAULT_DNS_SERVERS,
290+
});
291+
292+
// Merge credential isolation env vars (COPILOT_API_URL, COPILOT_PROVIDER_BASE_URL, etc.)
293+
// In Docker mode these are merged by assembleOptionalServices during compose generation.
294+
// For sbx, we call buildAgentCredentialEnv directly with host.docker.internal
295+
// as the proxy target (the api-proxy is on the awf-ext bridge network).
296+
if (config.enableApiProxy) {
297+
const credentialEnv = buildAgentCredentialEnv({
298+
config,
299+
networkConfig: {
300+
subnet: '172.30.0.0/24',
301+
squidIp: SBX_GATEWAY_IP,
302+
agentIp: AGENT_IP,
303+
proxyIp: SBX_HOST_DOCKER_INTERNAL,
304+
},
305+
});
306+
Object.assign(sbxEnvironment, credentialEnv);
307+
}
308+
309+
// Log critical env vars for debugging auth flow (redact secret values)
310+
logger.info(`[sbx-env] COPILOT_API_URL=${sbxEnvironment.COPILOT_API_URL || '(unset)'}`);
311+
logger.info(`[sbx-env] COPILOT_PROVIDER_BASE_URL=${sbxEnvironment.COPILOT_PROVIDER_BASE_URL || '(unset)'}`);
312+
logger.info(`[sbx-env] COPILOT_GITHUB_TOKEN=${redactSecret(sbxEnvironment.COPILOT_GITHUB_TOKEN)}`);
313+
logger.info(`[sbx-env] COPILOT_API_KEY=${redactSecret(sbxEnvironment.COPILOT_API_KEY)}`);
314+
logger.info(`[sbx-env] HTTPS_PROXY=${sbxEnvironment.HTTPS_PROXY || '(unset)'}`);
315+
logger.info(`[sbx-env] COPILOT_PROVIDER_API_KEY=${redactSecret(sbxEnvironment.COPILOT_PROVIDER_API_KEY)}`);
316+
317+
// Create the sandbox with configured mounts, proxy chaining through Squid
318+
const workspaceDir = process.env.GITHUB_WORKSPACE || process.cwd();
319+
sbxName = await createSandbox({
320+
workspaceDir,
321+
squidIp: SQUID_IP,
322+
extraMounts: config.volumeMounts,
323+
});
324+
325+
// Wait for api-proxy to be healthy before launching agent.
326+
// In Docker mode, depends_on: service_healthy gates this; for sbx we poll
327+
// via host.docker.internal which resolves to the docker0 bridge from the VM.
328+
if (config.enableApiProxy) {
329+
logger.info('[sbx] Polling api-proxy health via host.docker.internal...');
330+
const healthCmd = [
331+
'for i in $(seq 1 30); do',
332+
` if curl -sf --max-time 2 http://${SBX_HOST_DOCKER_INTERNAL}:10000/health >/dev/null 2>&1; then`,
333+
' echo "api-proxy healthy after ${i}s"; exit 0;',
334+
' fi;',
335+
' sleep 1;',
336+
'done;',
337+
'echo "api-proxy health timeout"; exit 1',
338+
].join(' ');
339+
340+
const healthResult = await execInSandbox(sbxName, healthCmd, {
341+
timeoutMinutes: 1,
342+
workDir: config.containerWorkDir,
343+
environment: sbxEnvironment,
344+
});
345+
if (healthResult.exitCode !== 0) {
346+
logger.warn('[sbx] api-proxy health check failed — proceeding anyway');
347+
}
348+
}
349+
350+
// Verify squid proxy is reachable from sandbox
351+
logger.info('[sbx-diag] Verifying squid proxy connectivity...');
352+
const diagCmd = [
353+
`echo -n "squid ${SBX_GATEWAY_IP}:3128 → "`,
354+
`curl -sS --max-time 5 --proxy "http://${SBX_GATEWAY_IP}:3128" -o /dev/null -w "%{http_code}" https://api.github.com/ 2>&1`,
355+
'echo ""',
356+
].join(' && ');
357+
358+
const diagResult = await execInSandbox(sbxName, diagCmd, {
359+
timeoutMinutes: 1,
360+
workDir: config.containerWorkDir,
361+
environment: sbxEnvironment,
362+
});
363+
logger.info(`[sbx-diag] Connectivity check exited with code ${diagResult.exitCode}`);
364+
}
365+
: startContainers;
366+
367+
const sbxRunAgentCommand = useSbx
368+
? async (_workDir: string, _allowedDomains: string[], _proxyLogsDir?: string, agentTimeoutMinutes?: number) => {
369+
if (!sbxName) throw new Error('Sandbox not created');
370+
logger.info(`[sbx] Launching agent command in sandbox "${sbxName}" (timeout: ${agentTimeoutMinutes ?? 'none'} min)`);
371+
logger.debug(`[sbx] Agent command: ${config.agentCommand.substring(0, 200)}...`);
372+
const result = await execInSandbox(sbxName, config.agentCommand, {
373+
timeoutMinutes: agentTimeoutMinutes,
374+
workDir: config.containerWorkDir,
375+
environment: sbxEnvironment,
376+
tty: config.tty,
377+
});
378+
logger.info(`[sbx] Agent command exited with code ${result.exitCode}`);
379+
380+
// Dump api-proxy logs for debugging connection issues
381+
if (config.enableApiProxy && result.exitCode !== 0) {
382+
try {
383+
const { execSync } = await import('child_process');
384+
const proxyLogs = execSync('docker logs --tail 80 awf-api-proxy 2>&1', { encoding: 'utf-8', timeout: 10000 });
385+
logger.info(`[sbx-diag] api-proxy logs:\n${proxyLogs}`);
386+
const healthStatus = execSync('docker inspect --format={{.State.Health.Status}} awf-api-proxy 2>&1', { encoding: 'utf-8', timeout: 5000 });
387+
logger.info(`[sbx-diag] api-proxy health status: ${healthStatus.trim()}`);
388+
} catch { /* ignore diagnostic failures */ }
389+
}
390+
391+
return { exitCode: result.exitCode, blockedDomains: [] as string[] };
392+
}
393+
: runAgentCommand;
394+
234395
exitCode = await runMainWorkflow(
235396
config,
236397
{
237398
ensureFirewallNetwork,
238399
setupHostIptables,
239400
writeConfigs,
240-
startContainers,
241-
runAgentCommand,
401+
startContainers: sbxStartContainers,
402+
runAgentCommand: sbxRunAgentCommand,
242403
collectDiagnosticLogs,
243404
assertTopologySupported,
244405
connectTopologyContainers,

0 commit comments

Comments
 (0)