Skip to content

docs: mark A17 (build-tools digest pinning) as fixed#6040

Merged
lpcox merged 3 commits into
mainfrom
copilot/update-a17-status
Jul 9, 2026
Merged

docs: mark A17 (build-tools digest pinning) as fixed#6040
lpcox merged 3 commits into
mainfrom
copilot/update-a17-status

Conversation

Copilot AI commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

PR #5986 merged 'build-tools' into IMAGE_DIGEST_KEYS and threaded ParsedImageTag through SysrootServiceParams, but the runner doctor catalog still showed A17 as "Open — not yet fixed."

Changes

Files updated:

  • .github/workflows/shared/self-hosted-failure-modes.md
  • .github/workflows/self-hosted-runner-doctor.md
  • .github/agents/self-hosted-runner-doctor.md

Copilot AI changed the title [WIP] Update A17 status in knowledge base docs: mark A17 (build-tools digest pinning) as fixed Jul 9, 2026
Copilot finished work on behalf of lpcox July 9, 2026 02:43
Copilot AI requested a review from lpcox July 9, 2026 02:43
@lpcox lpcox marked this pull request as ready for review July 9, 2026 02:56
Copilot AI review requested due to automatic review settings July 9, 2026 02:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the self-hosted runner “runner doctor” failure-mode documentation to reflect that A17 (ARC/DinD build-tools sysroot-stage image digest pinning) has been fixed in AWF via PR #5986, and removes A17 from the “known unresolved items” lists where applicable.

Changes:

  • Mark A17 as fixed (with implementation details) in the Category A failure-modes table entries.
  • Remove A17 from “Known unresolved items” bullet lists in the shared catalog and portable agent copy.
  • Update the Step 4 “Check for known gaps and notable fixes” A17 entry in both runner-doctor docs.
Show a summary per file
File Description
.github/workflows/shared/self-hosted-failure-modes.md Marks A17 as fixed in the Category A table and removes A17 from the unresolved-items list.
.github/workflows/self-hosted-runner-doctor.md Updates Step 4 “known gaps and notable fixes” A17 entry to reflect the fix in PR #5986.
.github/agents/self-hosted-runner-doctor.md Mirrors the same A17 “fixed” updates (Step 4 + Category A table) and removes A17 from unresolved items.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Low

| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 |
| A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 |
| A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:<digest>` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Open — not yet fixed** (PR github/gh-aw-firewall#5986 in review): adds `'build-tools'` to `IMAGE_DIGEST_KEYS` and threads `ParsedImageTag` through `SysrootServiceParams`. Until merged, security-sensitive deployments cannot digest-pin the sysroot-stage image. | `awf --image-tag build-tools=sha256:abc ...` — throws "invalid key" if fix not installed | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 |
| A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:<digest>` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — throws "invalid key" if fix not installed | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 |
| A15 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)`; squid log files unreadable after ARC/DinD run; `awf logs summary` returns `Failed to load logs: EACCES` | `fixArtifactPermissionsForRootless()` binds the log directory into a repair container but does not apply `dockerHostPathPrefix` translation to the bind mount source path; the DinD daemon cannot resolve the runner-local path, so `chmod` exits non-zero | **Fixed in PR github/gh-aw-firewall#5963**: `fixArtifactPermissionsForRootless()` now calls `applyHostPathPrefixToVolumes()` so the repair container bind mount is correctly translated for the DinD daemon. Upgrade AWF to the version that includes github/gh-aw-firewall#5963. Workaround (older AWF): run `chmod -R a+rX` inside the squid container before `docker compose down`. | `ls -la <proxy-logs-dir>` after run — files owned by uid 13 (squid) confirm the mode; check AWF logs for `[WARN] Rootless artifact permission repair failed` | github/gh-aw-firewall#5816, github/gh-aw-firewall#5817, github/gh-aw-firewall#5963 |
| A16 | ARC/DinD with `runner.topology: arc-dind`: custom `--mount` paths (e.g. `${RUNNER_TEMP}/gh-aw:...`) are silently dropped; agent command fails with `node: command not found` or other binary-not-found errors even when the tool is correctly installed and the mount was confirmed daemon-visible | AWF sysroot mount filter (`filterAgentVolumesForSysroot`) was too aggressive: it dropped any mount whose source or target fell under `effectiveHome` (`/home/runner`), incorrectly including daemon-visible workspace paths such as `${RUNNER_TEMP}/gh-aw` (`/home/runner/_work/_temp/...`) | **Fixed in AWF (PR github/gh-aw-firewall#5739)**: filter now only drops dot-directories and the home root; workspace paths under `_work/` now pass through. Upgrade AWF to the version including github/gh-aw-firewall#5739. | Inspect generated agent compose YAML for the expected `--mount` entry; `docker run --rm -v ${RUNNER_TEMP}:${RUNNER_TEMP}:ro alpine ls ${RUNNER_TEMP}` to confirm daemon-visibility | github/gh-aw-firewall#5739 |
| A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:<digest>` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Open — not yet fixed** (PR github/gh-aw-firewall#5986 in review): adds `'build-tools'` to `IMAGE_DIGEST_KEYS` and threads `ParsedImageTag` through `SysrootServiceParams`. Until merged, security-sensitive deployments cannot digest-pin the sysroot-stage image. | `awf --image-tag build-tools=sha256:abc ...` — throws "invalid key" if fix not installed | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 |
| A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:<digest>` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — throws "invalid key" if fix not installed | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 |
@lpcox

lpcox commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Updated the A17 Probe column in both table files to describe both cases: on a patched version, the command succeeds and the pinned digest is visible in the generated compose YAML; on older AWF without the fix, it throws Error: invalid key 'build-tools'. Commit: docs: update A17 Probe to describe success case when fix is present.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

PR #6040 appears to follow the applicable CONTRIBUTING.md guidelines; no review comment needed.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Result
API Status ✅ PASS
GH Check ✅ PASS
File Status ✅ PASS

Overall Result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Smoke Claude for #6040 · 35.1 AIC · ⊞ 3.3K ·
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: PAT Auth

Test Result
GitHub.com HTTP ✅ 200
File Write/Read ⚠️ N/A (template vars unresolved)
MCP Connectivity

Overall: PASS Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 PAT report filed by Smoke Copilot PAT
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Gemini Smoke Test Results

  • GitHub MCP Testing: ✅
  • GitHub.com Connectivity: ✅
  • File Writing Testing: ✅
  • Bash Tool Testing: ✅

Overall status: PASS

Merged PRs:

  1. fix: use tag-only image ref in rootless permission repair
  2. chore: upgrade gh-aw extension to v0.82.5 pre-release

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Test Results

  • Redis PING: ❌ Network is unreachable
  • PostgreSQL pg_isready: ❌ No response
  • PostgreSQL SELECT 1: ❌ Network is unreachable

Overall: FAILhost.docker.internal (172.17.0.1) is unreachable from this runner. Service containers are not accessible.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Smoke Test Results: Copilot BYOK Direct Mode

Test Result
GitHub MCP Connectivity
GitHub.com HTTP ✅ (HTTP 200)
File Write/Read
BYOK Inference Path

Status: PASS

Running in direct BYOK mode via COPILOT_PROVIDER_API_KEY → api-proxy sidecar → api.githubcopilot.com

@lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🔥 Smoke Test Results

Test Status
MCP Connectivity
GitHub.com HTTP ✅ 200
File Write/Read

Overall: PASS

Author: @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3 ❌ NO
Node.js v24.18.0 v22.23.1 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Result: Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions github-actions Bot mentioned this pull request Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🔍 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Result Notes
Module Loading otel.js loads successfully; exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled + internal helpers
Test Suite 59 tests passed across otel.test.js + otel-fanout.test.js (0 failures)
Env Var Forwarding src/services/api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME
Token Tracker Integration onUsage callback exists in token-tracker-http.js (line 285/343) as OTEL hook point; onSpanEnd callback also present
OTEL Diagnostics Graceful degradation confirmed — falls back to FileSpanExporter (/var/log/api-proxy/otel.jsonl) when no OTLP endpoint configured

All 5 scenarios passed. OTEL integration is fully functional.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Notes:

  • Java: Initial Maven run failed with LocalRepositoryNotAccessibleException (pre-existing /home/runner/.m2 owned by root). Resolved by specifying a writable local repository path via -s flag.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Build Test Suite for #6040 · 65.3 AIC · ⊞ 6.9K ·
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

GitHub MCP connectivity: ✅\nGitHub.com connectivity: ✅\nFile Write/Read test: ✅\nBYOK inference: ✅\nRunning in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra\nOverall: PASS\n@lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@lpcox lpcox merged commit 1521792 into main Jul 9, 2026
78 of 80 checks passed
@lpcox lpcox deleted the copilot/update-a17-status branch July 9, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🩺 Runner Doctor Update🩺 Runner Doctor: update A17 status (build-tools digest pinning merged)

3 participants