CLI and Agentic Workflows not Working in Data residency/ghe.com

[domusicman]

Warning - This is a public repository, please be mindful of sharing sensitive data

Why are you seeing this issue?

This is a support escalation! You can find out more about expectations on communications and response times in the on-call docs. Thank you for doing your part to keep GitHub customers happy!

Severity

Note, for Sev1 issues always page the team first, using the PagerDuty information in the service catalog and the command .pager trigger <rotation> <reason>. Opening an issue for a Sev2 incident can come later (and can be done by the available Escalation Engineer).

• sev1 - Engineering must respond, when paged, within their on-call SLO
• sev2 - Engineering must respond to this issue within one business day
• sev3 - Engineering must respond to this issue within 1 week

Service

AgenticWorkflows

Context

⚠️ Kindly provide all the requested information to ensure a prompt and efficient response. ⚠️

GitHub Support Plan

• Top 20
• Premium Plus
• Premium
• Enterprise
• Free/Pro/Team

GitHub Product(s) Impacted (check appropriate boxes and provide GHES version if applicable)

• GitHub.com (including GHEC)
• GitHub Enterprise Server.
  • Running version: x.y.z
  • Recently upgraded from version: x.y.z (upgrade date: yyyy-mm-dd)
• Proxima (ghe.com)
  • Customer tenant: -

Customer Impact

If applicable, list the impacted entities:

• Enterprise: - Verizon
• Organization: -
• Repository: -
• User: -

Zendesk Ticket(s), support bundles, and other relevant Issues and links

Issue

Verizon reports Copilot CLI and Agentic Workflows fail in their data residency deployment https://verizon.ghe.com/ when using token-based auth via COPILOT_GITHUB_TOKEN. Interactive user credential auth works, and a comparable FG PAT created on github.com in their verizon-emu enterprise works, but a PAT originating from the DR host does not. Prior proxy issues were addressed in a separate ticket, and internal notes initially suggested setting GH_HOST=verizon.ghe.com. Customer tested GH_HOST in the workflow and still sees the same auth failure. Logs show authentication succeeds for one request then later fails with invalid API key on /close. Customer provided workflow file and logs, and the issue remains open and may need engineering review.

feature-flag-cleanup.lock.yml

0_agent.txt

1_activation.txt

Additional context

I'm not sure if this is related, but if so, it is still persisting:

#1330

cc:
@github/gh-aw-firewall

Support Squad(s): @github/support-squad-worktent