[CRCR] Umbrella: Multi-issuer OIDC & Buildkite onboarding

[subinz1]

Summary

Umbrella tracking issue for multi-issuer OIDC support in the CRCR callback relay and Buildkite CI provider onboarding.

The CRCR callback relay originally only supported GitHub Actions OIDC tokens. This work extends jwt_helper to support multiple issuers, enabling external CI providers (Buildkite, GitLab CI, CircleCI) to authenticate and report results to the PyTorch HUD.

RFC: pytorch/rfcs#98

---

Completed

Multi-issuer OIDC support

• verify_oidc_token dispatches on iss claim to issuer-specific JWKS and repo-claim extraction — #8453
• Buildkite identity mapped via immutable UUIDs (organization_id, pipeline_id), not slugs — #8453

Externalized provider config

• config/ci_providers.yml — runtime config for Buildkite org/pipeline UUID → GitHub repo mapping — #8468
• required_claims support (e.g., build_branch: [main]) to constrain which builds can authenticate — #8468

CI-neutral callback reporter

• Standalone report_callback.py and Buildkite wrapper (report_callback_buildkite.sh) — #8303

Nightly/periodic self-report model

• Callback Lambda nightly code path: skip state machine, validate OIDC + SHA, single upsert — #8302
• SHA validator with TTL cache — #8304
• DynamoDB write-once guard to prevent replay of finalized records — #8694

HUD integration

• Nightly dashboard view on hud.pytorch.org/crcr — #8538
• Nightly summary stats computed from matrix data — #8696
• Run-level filtering for nightly dashboard query — #8695
• Hide crcr-test from nightly summary page — #8697

---

Remaining / In Progress

Buildkite production onboarding

• vLLM (vllm-project/vllm) Buildkite pipeline: end-to-end validation with real OIDC tokens
• Production ci_providers.yml entry for vLLM pipeline UUIDs

Future CI providers

• GitLab CI issuer support (claims["project_path"])
• CircleCI issuer support (org-specific iss URL)

Hardening

• Lambda field validation: require delivery_id, event_type, job_name in callback payloads
• Automated staleness alerting: flag backends with >36h since last nightly callback

---

CI Provider Reference

CI Engine	Issuer (iss)	Status
GitHub Actions	https://token.actions.githubusercontent.com	Shipped
Buildkite	https://agent.buildkite.com	Shipped (awaiting production onboarding)
GitLab CI	https://gitlab.com	Future
CircleCI	https://oidc.circleci.com/org/<ID>	Future

Context

• Reporter extraction: [CRCR] Add delivery-id and event-type inputs to callback action (Phase 2) #8303
• Original proposal by @atalman: [CRCR] Add delivery-id and event-type inputs to callback action (Phase 2) #8303 (comment)
• RFC: pytorch/rfcs#98

[jewelkm89]

Generic Multi-CI OIDC Authentication Architecture

This document describes a zero-static-secret, provider-agnostic architecture for sending workflow results from any downstream CI/CD provider (Buildkite, GitLab CI, CircleCI, GitHub Actions, Azure DevOps) to an upstream GitHub repository.

---

1. Overview & Architecture

GitHub Actions and GitHub REST APIs do not natively validate raw OpenID Connect (OIDC) JSON Web Tokens (JWTs) issued by third-party CI systems. To allow any downstream CI engine to notify an upstream GitHub repository without storing long-lived API keys, we introduce a Middleware OIDC Gateway.

┌────────────────────────┐
│  Buildkite Pipeline    ├──────┐
└────────────────────────┘      │
                                │
┌────────────────────────┐      │  1. Send Native OIDC JWT + Results
│  GitLab CI Runner      ├──────┼──────────────────────────────────┐
└────────────────────────┘      │                                  │
                                │                                  ▼
┌────────────────────────┐      │                     ┌──────────────────────────┐
│  CircleCI Pipeline     ├──────┘                     │  Generic OIDC Gateway    │
└────────────────────────┘                            │  (AWS Lambda / Worker)   │
                                                      │                          │
┌────────────────────────┐                            │  2. Inspect 'iss' claim  │
│  GitHub Actions        ├───────────────────────────►│  3. Verify against JWKS  │
└────────────────────────┘                            │  4. Match tenant rules   │
                                                      └────────────┬─────────────┘
                                                                   │
                                                                   │ 5. Trigger event via
                                                                   │    GitHub App Token
                                                                   ▼
                                                      ┌──────────────────────────┐
                                                      │ Upstream GitHub Repo     │
                                                      └──────────────────────────┘

Key Highlights

• Zero Long-Lived Secrets: Downstream pipelines use short-lived, dynamically generated OIDC JWTs.
• Provider Agnostic: Supports any CI engine adhering to standard OpenID Connect (OIDC) specifications.
• Centralized Access Control: Access rules are maintained centrally within the gateway configuration.

---

2. CI Provider Issuer Reference

Every standard OIDC provider publishes its public cryptographic keys via a JSON Web Key Set (JWKS) endpoint.

CI Engine	Issuer Claim (iss)	Public JWKS Endpoint
Buildkite	https://agent.buildkite.com	https://agent.buildkite.com/.well-known/jwks.json
GitHub Actions	https://token.actions.githubusercontent.com	https://token.actions.githubusercontent.com/.well-known/jwks.json
GitLab CI	https://gitlab.com	https://gitlab.com/-/oauth/discovery/keys
CircleCI	https://oidc.circleci.com/org/<ORG_ID>	https://oidc.circleci.com/org/<ORG_ID>/.well-known/jwks.json

---

3. Configuration Schema

The gateway enforces access control using an allowlist mapping verified issuer claims to authorized upstream GitHub repositories:

{
  "allowed_sources": [
    {
      "name": "Buildkite Main Pipeline",
      "issuer": "https://agent.buildkite.com",
      "allowed_subjects": [
        "organization:my-org:pipeline:downstream-pipeline:ref:refs/heads/main:*"
      ],
      "upstream_repo": "my-org/upstream-repo"
    },
    {
      "name": "GitLab Production Runner",
      "issuer": "https://gitlab.com",
      "allowed_subjects": [
        "project_path:my-org/downstream-project:ref_type:branch:ref:main"
      ],
      "upstream_repo": "my-org/upstream-repo"
    },
    {
      "name": "GitHub External Actions",
      "issuer": "https://token.actions.githubusercontent.com",
      "allowed_subjects": [
        "repo:external-org/downstream-repo:ref:refs/heads/main"
      ],
      "upstream_repo": "my-org/upstream-repo"
    }
  ]
}

---

4. Implementation Details

Step 1: Downstream Pipeline Requests (Examples)

Buildkite (pipeline.yml)

steps:
  - label: "Send Results to Upstream Gateway"
    command: |
      # 1. Request short-lived OIDC token with target audience
      JWT=$(buildkite-agent oidc request-token --audience "https://gateway.yourdomain.com")

      # 2. Dispatch payload to Gateway
      curl -X POST https://gateway.yourdomain.com/dispatch         -H "Authorization: Bearer $JWT"         -H "Content-Type: application/json"         -d '{
          "status": "passed",
          "build_id": "'"$BUILDKITE_BUILD_NUMBER"'",
          "metrics": { "coverage": "94.2%" }
        }'

GitLab CI (.gitlab-ci.yml)

send-upstream:
  stage: deploy
  id_tokens:
    GATEWAY_JWT:
      aud: "https://gateway.yourdomain.com"
  script:
    - |
      curl -X POST https://gateway.yourdomain.com/dispatch         -H "Authorization: Bearer $GATEWAY_JWT"         -H "Content-Type: application/json"         -d '{
          "status": "passed",
          "commit": "'"$CI_COMMIT_SHA"'"
        }'

---

Step 2: Gateway Verification Logic (Node.js/TypeScript Reference)

import { jwtVerify, createRemoteJWKSet } from 'jose';

interface ClaimMapping {
  issuer: string;
  allowed_subjects: string[];
  upstream_repo: string;
}

export async function handleOidcDispatch(authHeader: string, bodyPayload: any, config: ClaimMapping[]) {
  const token = authHeader.replace(/^Bearer\s+/i, '');
  
  // 1. Decode header to extract issuer without full verification
  const unverified = jwt.decode(token, { complete: true });
  if (!unverified) throw new Error("Invalid JWT token format");

  const issuer = unverified.payload.iss;

  // 2. Locate configuration matching issuer
  const matchConfig = config.find(c => c.issuer === issuer);
  if (!matchConfig) throw new Error(`Untrusted issuer: ${issuer}`);

  // 3. Dynamically fetch keys and verify signature
  const jwksUrl = new URL('/.well-known/jwks.json', issuer);
  const JWKS = createRemoteJWKSet(jwksUrl);
  
  const { payload } = await jwtVerify(token, JWKS, {
    audience: "https://gateway.yourdomain.com",
    issuer: issuer
  });

  // 4. Validate subject (sub) claim against pattern list
  const isAllowed = matchConfig.allowed_subjects.some(pattern => 
    minimatch(payload.sub as string, pattern)
  );

  if (!isAllowed) throw new Error(`Unauthorized subject: ${payload.sub}`);

  // 5. Trigger Upstream Workflow using GitHub App Token
  const installationToken = await getGitHubAppToken(matchConfig.upstream_repo);
  
  await fetch(`https://api.github.com/repos/${matchConfig.upstream_repo}/dispatches`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${installationToken}`,
      'Accept': 'application/vnd.github+json',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      event_type: 'downstream_results',
      client_payload: bodyPayload
    })
  });

  return { success: true };
}

---

5. Security & Maintenance

1. Short Lifetimes: Set JWT expiry times to the minimum allowed duration (e.g., 5 to 15 minutes).
2. Explicit Audience Checking: Ensure the aud claim matches the gateway URL to prevent token replay attacks against other services.
3. Key Caching: Cache public JWKS keys in memory/Redis to reduce latency on high-throughput pipelines.

[jewelkm89]

@atalman @jathu can you please have a review of this nightly CRCR authentication mechanism.

[subinz1]

Minimal Implementation Plan

Thanks @jewelkm89 for the detailed architecture document. The generic multi-CI OIDC gateway design is well-thought-out, but for the immediate scope of this issue we can get Buildkite support working with a much smaller change surface. Here's a concrete plan using vLLM (vllm-project/vllm) as an example — their CI runs on Buildkite.

---

What changes, and where

3 files touched, ~60 lines added:

1. utils/jwt_helper.py (~30 lines)

Currently hardcoded to GitHub Actions OIDC. Change to multi-issuer:

_ISSUERS = {
    "https://token.actions.githubusercontent.com": {
        "jwks": "https://token.actions.githubusercontent.com/.well-known/jwks",
        "repo_claim": lambda claims: claims["repository"],
    },
    "https://agent.buildkite.com": {
        "jwks": "https://agent.buildkite.com/.well-known/jwks",
        "repo_claim": lambda claims: _buildkite_to_repo(
            claims["organization_slug"], claims["pipeline_slug"]
        ),
    },
}

The function becomes:

1. Strip Bearer prefix (existing)
2. Decode header unverified to read iss claim
3. Look up issuer in _ISSUERS → reject unknown issuers with 401
4. Fetch signing key from the issuer's JWKS endpoint
5. Verify signature + audience (pytorch-cross-repo-ci-relay) + issuer
6. Extract verified_repo using the issuer-specific repo_claim lambda

2. utils/jwt_helper.py — Buildkite identity mapping (~15 lines)

Buildkite OIDC has no repository claim. It provides organization_slug + pipeline_slug. We need a static mapping:

_BUILDKITE_REPO_MAP = {
    ("vllm", "vllm-ci"): "vllm-project/vllm",
    # Add more as backends onboard
}

def _buildkite_to_repo(org_slug: str, pipeline_slug: str) -> str:
    key = (org_slug, pipeline_slug)
    repo = _BUILDKITE_REPO_MAP.get(key)
    if repo is None:
        raise HTTPException(401, f"Unknown Buildkite pipeline: {org_slug}/{pipeline_slug}")
    return repo

Why not put this in the allowlist YAML? The allowlist maps repo → level. This maps (org_slug, pipeline_slug) → repo. They're different directions. Keeping it in jwt_helper.py means the identity resolution is complete before the callback handler ever sees it — everything downstream (callback_handler, allowlist, redis, HUD) stays untouched.

3. callback/lambda_function.py (~5 lines)

Line 55 currently does verified_repo = oidc_claims["repository"]. Since verify_oidc_token will now return normalized claims with repository always set (regardless of issuer), this line stays the same. The only change: verify_oidc_token returns a dict that always contains repository, populated by the issuer-specific extractor.

4. Tests (~15 lines)

• Add a test for Buildkite token verification (mock JWKS)
• Add a test for unknown Buildkite pipeline → 401
• Add a test for unknown issuer → 401

---

What does NOT change

• callback_handler.py — sees verified_repo as before
• allowlist.py — vLLM must still be in the allowlist YAML as vllm-project/vllm at whatever level
• sha_validator.py — unchanged
• redis_helper.py — unchanged
• HUD write path — unchanged
• ClickHouse schema — unchanged

---

Downstream (vLLM Buildkite) usage

# In vLLM's Buildkite pipeline.yml
steps:
  - label: "Report to CRCR"
    command: |
      JWT=$(buildkite-agent oidc request-token \
        --audience "pytorch-cross-repo-ci-relay" \
        --lifetime 300)
      python report_callback.py  # same script from test-infra
    env:
      OIDC_TOKEN: "$$JWT"
      CALLBACK_URL: "https://gwuj7w5coh4y4l66urspge6mnm0pwaxq.lambda-url.us-east-1.on.aws/github/callback/"
      STATUS: "completed"
      CONCLUSION: "success"
      # ... other env vars

Or use report_callback_buildkite.sh from PR #8303 which wraps this.

---

Scaling to more providers later

The _ISSUERS dict is the extension point. Adding GitLab or CircleCI later is:

1. Add their JWKS URL and issuer string to _ISSUERS
2. Add a repo_claim lambda that maps their token claims to a GitHub owner/repo
3. Add entries to whatever static mapping that provider needs

No new config schema, no new service, no infrastructure changes.

---

Summary

Component	Change	LOC
jwt_helper.py	Multi-issuer dispatch + Buildkite mapping	~45
lambda_function.py	None (interface unchanged)	0
callback_handler.py	None	0
Tests	3 new test cases	~15
Total		~60

Happy to pick this up if the approach looks good. @atalman @jewelkm89