CI: control-profile required checks stall/cancel on the self-hosted runner pool

[dpeterson01]

What's happening

Required CI checks on the control runner profile (Detect validation scopes, Static gate, Validate workspace, Runtime summary) consistently end up Cancelled, or stuck Queued, which leaves PRs BLOCKED even after review approval. I've hit it on two approved PRs, #6076 and #6079, and it looks systemic rather than PR-specific.

Why it looks like the control / self-hosted pool, not workflow logic or our branches

• Both allow custom model entry in the onboarding BYOK model picker #6076 and enable model discovery for local Ollama BYOK providers #6079 are APPROVED + MERGEABLE but BLOCKED, and the non-passing checks on both are exactly the control-profile jobs.
• ci.yml's concurrency group is scoped per PR (ci-${{ ...pull_request.number... }}), so one PR's run shouldn't cancel another's. This isn't a cross-PR concurrency collision.
• I rebased enable model discovery for local Ollama BYOK providers #6079 onto current main and re-ran; the control checks still cancelled, so it isn't staleness or a conflict on our side.
• In .github/scripts/runners.py, the control profile in default mode maps to the self-hosted pool ["self-hosted","Linux","X64","od-persistent-ci","od-ci-hot-poc"]. The Queued → Cancelled signature fits that pool being unavailable or overloaded.

Suggested direction (your call, since it's your infra)

Either restore/scale the self-hosted od-persistent-ci pool, or set OD_CI_RUNNER_MODE to a mode that routes control to GitHub-hosted runners (per resolve_contract, non-default modes already do this). I'm happy to help with a resilience change to runners.py, for example letting control fall back to GitHub-hosted when the self-hosted pool is down, if that'd be useful. Flagging first rather than sending unsolicited CI changes.

@lefarcen already raised the blocked-check state internally; opening this so it's publicly trackable and linkable from the affected PRs.

[lefarcen]

Hi @dpeterson01 — the narrowing here is really helpful. I checked the workflow side as well, and your read holds up: in .github/scripts/runners.py, control resolves to ["self-hosted","Linux","X64","od-persistent-ci","od-ci-hot-poc"] when OD_CI_RUNNER_MODE is default, and the corresponding ci.yml jobs (Detect validation scopes, Static gate, Validate workspace, Runtime summary) all run on that control profile.

On PR #6076 and PR #6079, the pattern is consistent on the current runs from July 25–29, 2026: Resolve runner profiles completes successfully, while the control jobs are the ones left CANCELLED / QUEUED. That makes this look much more like runner-pool availability/capacity than a PR-specific workflow regression, so I’ve labeled this as a CI bug and we can use this issue as the public tracker.

If you do end up sending a follow-up PR, the resilience direction you suggested looks like the right place to explore: keeping the self-hosted path when it’s healthy, but letting control fall back to GitHub-hosted when that pool is unavailable. We’ll keep this thread updated once the runner-side fix or fallback plan is clearer. ❤️

[dpeterson01]

Opened #6290 with the least-privilege version of the fallback: a maintainer-flippable OD_CI_CONTROL_FALLBACK repo variable that routes control jobs to GitHub-hosted runners, off by default and needing no elevated workflow-token permissions. Automatic runner-health detection is noted there as an optional follow-up if you'd rather grant administration: read.

[lefarcen]

Hey @dpeterson01, this is a good refinement — thanks for keeping the fallback scoped to control instead of piggybacking on OD_CI_RUNNER_MODE. I pulled up #6290, and the maintainer-flip approach makes sense as a first step here: off by default, no extra workflow-token permissions, and it gives us a concrete escape hatch for the exact runner pool that’s wedging approved PRs.

Let’s track the implementation discussion in #6290 from here. Keeping the automatic runner-health detection as a follow-up rather than widening CI permissions up front also feels like the right tradeoff for this pass. ❤️