github
diff --git a/‎.github/agents/self-hosted-runner-doctor.md‎
Lines changed: 26 additions & 2 deletions b/‎.github/agents/self-hosted-runner-doctor.md‎
Lines changed: 26 additions & 2 deletions
@@ -73,22 +73,37 @@ Prefer the narrowest match. Examples:
 - `chroot: failed to run command '/bin/sh'` on a glibc daemon → A13 (empty staging, not A4 musl)
 - `unknown shorthand flag: 'd' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)
 - `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)
+- `node: command not found` on ARC/DinD with `runner.topology: arc-dind` even when binary is correctly installed → A16 (sysroot filter was over-broad and dropped the workspace mount)
 - `EAI_AGAIN <awmg-cli-proxy>` in network-isolation + topology-attach → B5
 - `EACCES` in upload-artifact after sudo:false → B6
 - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/<path>` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files)
+- `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs)
 - `FATAL: http_port: IPv6 is not available` → B3
+- `No CA certificates were loaded from the system` in chroot on RHEL/Fedora/Amazon Linux → B9 (missing /etc/pki/ mount)
 - `none of the git remotes correspond to the GH_HOST environment variable` → C4
 - `400 bad request: Authorization header is badly formatted` → C3
+- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
 - `diagnosis=unknown` (proxy reachable, no connection error) or `reachable-but-api-error` from DIFC probe with `GITHUB_SERVER_URL=*.ghe.com` → C7 (DIFC proxy not enterprise-host-aware)
+- `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS)
 
 ### 4. Check for known gaps and notable fixes
 
 If the best match is one of the known open gaps (gVisor/Kata runtime support, `--enable-dind` cleanup, enterprise header-injection extension points, or the remaining `GH_HOST` leak to user steps), say so explicitly instead of implying there is a shipped fix.
 
 A13 / github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 — ARC/DinD split-fs base-userland staging is **fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. The `sysroot-stage` init container copies the signed `build-tools` image filesystem into a `sysroot` volume mounted at `/host:ro` before the agent starts.
 
+A16 / github/gh-aw-firewall#5739 — ARC/DinD sysroot filter over-broad (drops workspace mounts under `_work/`) is **fixed** in AWF version including github/gh-aw-firewall#5739. Filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through.
+
+A17 / github/gh-aw-firewall#5985 — `build-tools` image cannot be digest-pinned; fix pending in open PR github/gh-aw-firewall#5986. Security-sensitive deployments must wait for the merge.
+
+B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`.
+
+B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`.
+
 C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.
 
+C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.
+
 ## Output Requirements
 
 Produce a structured triage report with these sections:
@@ -149,6 +164,8 @@ Establish these facts before matching a failure mode:
 | A13 | `chroot: failed to run command '/bin/sh': No such file or directory` or `[entrypoint][ERROR] capsh not found on host system` on a **glibc/Debian daemon** (not musl/Alpine) | ARC/DinD split-fs: system-mount source dirs (`/tmp/gh-aw/{usr,bin,lib,...}`) are empty because nothing populates them. The entrypoint "musl/Alpine" warning is **misleading** — it fires because no dynamic loader is found, not because the daemon is musl. | **Fixed in AWF v0.27.15**: set `runner.topology: "arc-dind"` in the AWF config JSON. AWF emits a `sysroot-stage` init container that copies the signed `build-tools` image filesystem (`bash`, `capsh`, `gcc`, dev libs, coreutils) into a named `sysroot` volume mounted at `/host:ro` before the agent starts. Use `runner.sysrootImage` to pin a specific image. | Check `awf --version` ≥ v0.27.15; verify `runner.topology: "arc-dind"` is set; inspect compose output for `sysroot-stage` service and `sysroot` volume | #5541, github/gh-aw-firewall#5693, github/gh-aw-firewall#5696 |
 | A14 | `unknown shorthand flag: 'd' in -d` / `Command failed with exit code 125: docker compose up -d --pull never` | ARC/DinD sidecar image lacks `docker-compose-plugin`; AWF uses `docker compose` (v2 plugin syntax) to orchestrate containers but the DinD sidecar only has legacy standalone Docker or no Compose support | Add `docker-compose-plugin` to the DinD sidecar Dockerfile: `RUN apt-get update && apt-get install -y docker-compose-plugin` (Debian/Ubuntu) or `RUN apk add docker-cli-compose` (Alpine) | `docker compose version` inside the DinD sidecar — v2 output confirms plugin is present; inspect sidecar Dockerfile for `docker-compose-plugin` | github/gh-aw-firewall#5729 |
 | A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 |
+| A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 |
+| A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:<digest>` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Open — not yet fixed** (PR github/gh-aw-firewall#5986 in review): adds `'build-tools'` to `IMAGE_DIGEST_KEYS` and threads `ParsedImageTag` through `SysrootServiceParams`. Until merged, security-sensitive deployments cannot digest-pin the sysroot-stage image. | `awf --image-tag build-tools=sha256:abc ...` — throws "invalid key" if fix not installed | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 |
 
 ## Category B — Self-hosted runners
 
@@ -160,7 +177,9 @@ Establish these facts before matching a failure mode:
 | B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 |
 | B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 |
 | B6 | `EACCES` in `upload-artifact` step after a `sudo: false` (`--network-isolation`) AWF run; firewall log/audit dirs present but unreadable | Sidecars write files as non-runner UIDs (squid → uid 13, cli-proxy → `cliproxy`, agent/iptables-init → root). AWF's `chmod -R a+rX` repair runs as the unprivileged runner and silently fails at `debug` level on files it doesn't own | Resolved in AWF: (a) run Node sidecars as runner UID via compose `user:`; (b) root perm-fixer container at cleanup (daemon-run, mounts log dir, chowns to runner UID, skipped when `--keep-containers`); (c) promote swallowed-`chmod` failure from `debug` to `warn` | `ls -la <firewall-logs-dir>` after run — look for root or uid-13 owned files; check AWF logs for the swallowed `chmod` warning | #5545, #5542 |
-| B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf-<ts>-chroot-home/<path>` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717 |
+| B7 | AWF < v0.27.13: unhandled `EACCES` stack trace shows `unlink ... /tmp/awf-<ts>-chroot-home/<path>` (e.g. `.aws/config`, cloud credentials). AWF ≥ v0.27.13: `removeWorkDirectories()` catches the error and emits `[WARN] Failed to remove chroot home directory after permission repair` instead of crashing | In rootless Docker mode the agent container runs with UID namespace remapping. Files created by the agent inside the `chroot-home` temp directory are owned by remapped UIDs. AWF's `removeWorkDirectories()` runs as the unprivileged host runner and `fs.rmSync` fails on these files. | **Partially fixed in AWF v0.27.13** (repair container with CHOWN/DAC_OVERRIDE/FOWNER capabilities); **further fix merged post-v0.27.15** (#5717): in rootless Docker the repair container's `chown` operates within the user namespace and may not change host-level ownership. The post-v0.27.15 fix adds `chmod -R a+rwX` so the host can delete the directory regardless of ownership. Non-fatal if unfixed — leaves an orphan `/tmp/awf-*-chroot-home` dir. **additional hardening** (github/gh-aw-firewall#5766): changes `chown && chmod` to `chown 2>/dev/null; chmod` so `chmod` always runs as a fallback even when `chown` fails within the rootless UID namespace. | `ls -la /tmp/awf-*-chroot-home/` after a rootless run — files owned by non-runner UIDs confirm the mode; upgrade to AWF ≥ v0.27.13; check AWF logs for `[WARN] Failed to remove chroot home directory after permission repair` | #5653, github/gh-aw-firewall#5708, github/gh-aw-firewall#5717, github/gh-aw-firewall#5766 |
+| B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 |
+| B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 
@@ -173,6 +192,7 @@ Establish these facts before matching a failure mode:
 | C5 | `malformed version:` from `gh --repo` in later user steps | `GH_HOST=localhost:18443` leaked into non-AWF steps via `$GITHUB_ENV` | Primary fix belongs in gh-aw; cli-proxy can mitigate only partially | Check `$GITHUB_ENV` and `curl http://localhost:18443/api/v3/meta` | #3937 |
 | C6 | Safe-outputs post-processing talks to github.com instead of GHES | gh-aw emitted `GH_HOST` to the wrong channel for later jobs | Fix the compiler / environment propagation in gh-aw | Inspect `$GITHUB_OUTPUT` and `$GITHUB_ENV` for `GH_HOST` | #1460, #1566 |
 | C7 | `awf-cli-proxy` DIFC-proxy liveness probe loops retrying; cli-proxy logs show `diagnosis=unknown` (AWF < v0.27.12) or `diagnosis=reachable-but-api-error (HTTP NNN)` with a `*.ghe.com` hint (AWF ≥ v0.27.12); AWF fails to start | DIFC proxy is reachable but the forwarded `gh api rate_limit` call returns an HTTP error because the DIFC proxy is not enterprise-host-aware on data-residency `*.ghe.com` tenants | **Partially mitigated**: upgrade to AWF ≥ v0.27.12 for a targeted `*.ghe.com` hint and HTTP status in cli-proxy logs; root cause (DIFC proxy enterprise-host awareness) is **unresolved** in companion projects (github/gh-aw-mcpg#8202, github/gh-aw#41911) | Check `GITHUB_SERVER_URL` for `*.ghe.com`; inspect cli-proxy logs for `diagnosis=unknown` or `reachable-but-api-error (HTTP NNN)`; confirm AWF ≥ v0.27.12 for the targeted hint | #5615, #5616 |
+| C8 | `400 bad request: Authorization header is badly formatted` on **GHEC (`*.ghe.com`)** runners when `COPILOT_API_TARGET=api.business.githubcopilot.com`; Copilot Business calls receive `Bearer` instead of required `token` prefix. Reproduced on AWF v0.27.13 and v0.27.16. | `copilotTargetRequiresGitHubTokenPrefix()` checked `AWF_PLATFORM_TYPE` guard first. On GHEC, AWF auto-injects `AWF_PLATFORM_TYPE=ghec`, which short-circuited to `false` before querying the `GITHUB_TOKEN_PREFIX_COPILOT_TARGETS` catalog | **Fixed in AWF (PR github/gh-aw-firewall#5872)**: catalog endpoints (`api.enterprise.githubcopilot.com`, `api.business.githubcopilot.com`) are now checked first (always `token`); the platform-type guard now only affects the GHES heuristic for unknown targets. Upgrade to AWF version including github/gh-aw-firewall#5872. | `awf --version`; inspect api-proxy logs for 400 on `api.business.githubcopilot.com`; confirm `AWF_PLATFORM_TYPE=ghec` is set | github/gh-aw-firewall#5871, github/gh-aw-firewall#5872 |
 
 ## Category D — Alternative runtimes and adjacent gaps
 
@@ -197,7 +217,10 @@ Establish these facts before matching a failure mode:
 | `node: command not found` on self-hosted or DinD | A8 or B4 |
 | `none of the git remotes correspond to the GH_HOST environment variable` | C4 |
 | `malformed version:` from `gh --repo` | C5 |
-| `400 bad request: Authorization header is badly formatted` | C3 |
+| `400 bad request: Authorization header is badly formatted` | C3 (general GHES header assembly, AWF ≤ v0.27.1); also C8 if on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` |
+| `EACCES: permission denied, mkdir` on a `/tmp/gh-aw/...` path before containers start (pre-flight) | B8 |
+| `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 |
+| `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 |
 | `ENOENT ... /host/usr/local/bin/copilot` | A8 |
 | `mkdirat ... : read-only file system` during chroot agent startup | A12 |
 | `getaddrinfo EAI_AGAIN <topology-peer>` with `awf-cli-proxy could not connect to the external DIFC proxy` | B5 |
@@ -217,3 +240,4 @@ Flag these explicitly instead of implying there is a complete fix:
 - D4 / #4849 — enterprise header injection extension point
 - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes
 - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911)
+- A17 / github/gh-aw-firewall#5985 — ARC/DinD sysroot-stage `build-tools` image cannot be digest-pinned; `--image-tag build-tools=sha256:...` throws "invalid key" (fix pending in open PR github/gh-aw-firewall#5986)