|
| 1 | +# IC-ADR-003: Backend credentials — typed, declared, provider-supplied |
| 2 | + |
| 3 | +## Metadata |
| 4 | + |
| 5 | +- **Created By:** Alexandre Boyer |
| 6 | +- **Date:** 2026-07-03 |
| 7 | +- **Status:** Draft |
| 8 | +- **Decision Maker(s):** DIRACGrid maintainers |
| 9 | +- **Stakeholders:** interCEde backend implementers; DIRAC/DiracX SiteDirector and PushJobAgent maintainers; DiracX token-issuance maintainers; site/VO operators |
| 10 | +- **Depends on:** IC-ADR-001 (contract shape, lifecycle, Tier A/B/C surface) |
| 11 | + |
| 12 | +> **Scope and altitude.** Direction-setting, like IC-ADR-001. This ADR decides how *backend* |
| 13 | +> authentication credentials — bearer tokens and X.509 proxies — are **typed, declared, supplied, |
| 14 | +> refreshed, scoped and materialised**. It does not decide token *issuance* (DiracX's), *payload* |
| 15 | +> credential renewal (pilot-side, DIRAC's `_monitorProxy` lineage), or the exact dataclass fields |
| 16 | +> (implementation). Long-lived transport secrets (SSH keys) are configuration, not credentials, and |
| 17 | +> are only delimited here. |
| 18 | +
|
| 19 | +## Abstract |
| 20 | + |
| 21 | +Backend auth is on the critical path for the ARC and HTCondor-CE backends, and DIRAC's incumbent |
| 22 | +model does not survive the move to a stateless, cached, async library: credentials are **mutated |
| 23 | +onto** long-lived CE objects (`setProxy`/`setToken`), freshness policy is **duplicated across |
| 24 | +callers**, token opt-in is a **stringly-typed CS tag**, and audience is a bare attribute. This ADR |
| 25 | +replaces that with three pieces: **typed, immutable credential values** (`BearerToken`, |
| 26 | +`X509Proxy`, grouped in a `CredentialSet`); **backend-declared `CredentialRequirements`** (which |
| 27 | +kinds it accepts, for which audience/scopes — data computed from the backend's resolved config, |
| 28 | +replacing `Tag: Token[:vo]` and `audienceName`); and **provider-based supply** — backends pull a |
| 29 | +fresh `CredentialSet` from a caller-supplied `CredentialProvider` when theirs is missing or near |
| 30 | +expiry, so freshness *mechanics* live in one place while issuance/renewal *policy* stays entirely |
| 31 | +in the consumer (DiracX token machinery, DIRAC's proxy manager, or a static file). Materialisation |
| 32 | +(token/proxy files, env injection for CLI backends) is internal (Tier C). ARC's proxy delegation |
| 33 | +remains backend-internal mechanics consuming a provider-supplied proxy. |
| 34 | + |
| 35 | +## Motivation |
| 36 | + |
| 37 | +What the DIRAC code actually does today (all verified against the current codebase): |
| 38 | + |
| 39 | +1. **Mutating setters on cached objects.** `ComputingElement.setProxy`/`setToken` mutate CE |
| 40 | + instances that `QueueCECache` caches and reuses across agent cycles. Credential state and |
| 41 | + connection state are entangled on the same long-lived object — exactly the hazard for a library |
| 42 | + whose backends are cached and context-managed (IC-ADR-001 §2, lifecycle). |
| 43 | +2. **Caller-side renewal policy, duplicated.** `SiteDirector._setCredentials` inspects |
| 44 | + `ce.proxy.getRemainingSecs()` and re-supplies; `WMSUtilities.setPilotCredentials` reimplements |
| 45 | + the same gate for the pilot-kill/log service path. Two copies of the same freshness loop is the |
| 46 | + documented drift pattern IC-ADR-001 exists to end. |
| 47 | +3. **Stringly-typed opt-in.** A CE accepts tokens iff `"Token"` or `f"Token:{vo}"` appears in its |
| 48 | + CS `Tag` list. A *capability declaration* is encoded in a free-form tag shared with scheduling |
| 49 | + metadata. |
| 50 | +4. **Audience as a bare attribute.** Callers mint tokens with `audience=ce.audienceName` — an |
| 51 | + untyped per-CE string (AREX: `https://<host>:<port>`; HTCondorCE: `<host>:9619`) with no |
| 52 | + accompanying scopes/kind information. |
| 53 | +5. **Per-backend materialisation, ad hoc.** HTCondorCE writes the token to a temp file and injects |
| 54 | + `_CONDOR_*` env vars around each CLI call; the base writes proxies to files and exports |
| 55 | + `X509_USER_PROXY`; Cloud reads a `cloud.auth` ini with a magic `PROXY` secret that dumps the |
| 56 | + pilot proxy. |
| 57 | +6. **ARC needs more than a header.** AREX supports bearer tokens *and* X.509 proxy delegations |
| 58 | + (create a delegation via CSR, sign with the proxy chain, upload, renew) — and can need **both at |
| 59 | + once** (`AlwaysIncludeProxy`). |
| 60 | +7. **HTCondorCE needs both at once — today, in production.** Even when SCITOKENS authenticates the |
| 61 | + channel, DIRAC's submit description *unconditionally* contains `use_x509userproxy = true`, and |
| 62 | + the code says why: *"For now, we still need to include a proxy in the submit file — HTCondor |
| 63 | + extracts VOMS attribute from it for the sites"* — the proxy rides along for **site-side |
| 64 | + consumption** (per-VO attribution in site accounting, i.e. the APEL pipeline), not for channel |
| 65 | + auth. `_executeCondorCommand` even refuses to run with neither token nor proxy, and token mode |
| 66 | + sets `_CONDOR_DELEGATE_JOB_GSI_CREDENTIALS=false` to work around a condor 24.4 delegation bug |
| 67 | + (HTCONDOR-2904). Together with ARC's pairing, any model that assumes "one credential per |
| 68 | + backend" is wrong on day one — twice. |
| 69 | + |
| 70 | +### Drivers |
| 71 | + |
| 72 | +- **Stateless and cache-safe.** Credentials must not be hidden mutable state on cached backends. |
| 73 | +- **Policy with the consumer.** Where credentials *come from* (DiracX token service, proxy |
| 74 | + download, a file) and *when they are renewed* is consumer policy; interCEde owns only the |
| 75 | + mechanics of asking at the right time and using them correctly. |
| 76 | +- **Typed declaration.** A consumer must be able to ask a backend "what do you need?" and get data |
| 77 | + — not parse tags. |
| 78 | +- **One supply path for one and many credentials.** Token-only, proxy-only, and token+proxy |
| 79 | + backends use the same machinery. |
| 80 | +- **Testable in containers.** The model must work with IC-ADR-002's ephemeral per-run credentials |
| 81 | + (arcctl test-CA, IDTOKENS, ssh-keygen). |
| 82 | + |
| 83 | +## Specification |
| 84 | + |
| 85 | +### 1. Credential values (Tier A) |
| 86 | + |
| 87 | +Immutable, typed values with an expiry the machinery can read: |
| 88 | + |
| 89 | +```python |
| 90 | +@dataclass(frozen=True) |
| 91 | +class BearerToken: |
| 92 | + value: str # opaque to interCEde; never logged |
| 93 | + expires_at: datetime | None |
| 94 | + |
| 95 | +@dataclass(frozen=True) |
| 96 | +class X509Proxy: |
| 97 | + pem: bytes # full chain, PEM; opaque to interCEde beyond expiry |
| 98 | + expires_at: datetime | None |
| 99 | + |
| 100 | +Credential = BearerToken | X509Proxy |
| 101 | + |
| 102 | +@dataclass(frozen=True) |
| 103 | +class CredentialSet: # what a provider returns; may satisfy >1 kind |
| 104 | + def get(self, kind: type[Credential]) -> Credential | None: ... |
| 105 | +``` |
| 106 | + |
| 107 | +`CredentialSet` exists because of ARC's token+proxy case: requirements may name several kinds, and |
| 108 | +one provider call returns everything needed, atomically. interCEde never inspects credential |
| 109 | +*contents* (no VOMS parsing, no JWT decoding) beyond expiry bookkeeping. |
| 110 | + |
| 111 | +### 2. Requirements — the backend declares, as data (Tier A) |
| 112 | + |
| 113 | +```python |
| 114 | +@dataclass(frozen=True) |
| 115 | +class CredentialRequirements: |
| 116 | + kinds: frozenset[type[Credential]] # the kinds needed TOGETHER (all-of, not a menu) |
| 117 | + audience: str | None = None # token audience, if kinds include BearerToken |
| 118 | + scopes: frozenset[str] = frozenset() # e.g. compute.create/compute.read (WLCG profile) |
| 119 | +``` |
| 120 | + |
| 121 | +A backend computes its `CredentialRequirements` from its resolved configuration (it knows its |
| 122 | +endpoint, hence its audience) and exposes them as a read-only property. This single piece of data |
| 123 | +replaces both the `Tag: Token[:vo]` opt-in *and* `audienceName`: the consumer reads the |
| 124 | +requirements and mints/fetches accordingly. Configuration may still *narrow* a backend that |
| 125 | +accepts several kinds ("this site: proxy only"); it can never widen beyond what the backend |
| 126 | +declares. |
| 127 | + |
| 128 | +Two semantic rules, both forced by the grid CEs: |
| 129 | + |
| 130 | +- **`kinds` means "all of these, together".** Requirements are a concrete ask for the backend's |
| 131 | + *active configuration*, never a menu of alternatives — both first-party grid CEs need a *set* |
| 132 | + (ARC `AlwaysIncludeProxy`; HTCondorCE token + proxy, Motivation 7). A backend that supports |
| 133 | + alternative modes (token-only *or* proxy-only) exposes the mode as configuration, and its |
| 134 | + requirements reflect the configured mode. |
| 135 | +- **A required kind need not authenticate the channel.** HTCondorCE's proxy is materialised into |
| 136 | + the submit description (`use_x509userproxy`) for *site-side* consumption — VOMS attributes |
| 137 | + feeding the site's accounting (APEL) — while the token authenticates the channel. The provider |
| 138 | + sees no difference (it supplies the set); what each member is *for* is backend mechanics |
| 139 | + (Tier C). |
| 140 | + |
| 141 | +### 3. Supply — a provider the backend pulls from (Tier A) |
| 142 | + |
| 143 | +```python |
| 144 | +@runtime_checkable |
| 145 | +class CredentialProvider(Protocol): |
| 146 | + async def get(self, requirements: CredentialRequirements) -> CredentialSet: ... |
| 147 | +``` |
| 148 | + |
| 149 | +- The provider is part of the backend's construction config (registry request). A trivial |
| 150 | + `StaticCredentials(CredentialSet)` provider covers files/fixed tokens — and the integration |
| 151 | + stacks. |
| 152 | +- **The backend pulls; the consumer implements.** Before an operation, if the backend's current |
| 153 | + set is missing or within a freshness margin of expiry, it awaits `provider.get(...)` — once, in |
| 154 | + one place, inside the library. What the provider *does* (call DiracX's token issuer, download a |
| 155 | + proxy, read a refreshed file) is consumer code. This keeps the mechanism/policy split of |
| 156 | + IC-ADR-001 §9 and deletes the SiteDirector/WMSUtilities duplication by construction. |
| 157 | +- Provider calls are per-backend, not per-job: one credential authenticates the *channel/CE*, not |
| 158 | + each submission. |
| 159 | +- Failures raise the typed auth branch of `InterCEdeError` as whole-operation failures (IC-ADR-001 |
| 160 | + §2 partial-failure rule: auth failure is never a per-id outcome). |
| 161 | + |
| 162 | +### 4. Materialisation (Tier C) |
| 163 | + |
| 164 | +Backends that drive CLIs or need files get internal helpers, never a public contract: secure temp |
| 165 | +files (0600, private dir), env injection (`X509_USER_PROXY`, HTCondor `_CONDOR_*`/SciTokens env), |
| 166 | +scoped to the operation and cleaned up deterministically (the backend's context-manager lifecycle |
| 167 | +from IC-ADR-001 §2 is the natural cleanup boundary). Credential values never appear in logs or |
| 168 | +exception messages. |
| 169 | + |
| 170 | +### 5. Delegation is backend mechanics (Tier C) |
| 171 | + |
| 172 | +ARC proxy delegation (CSR → sign with the provider-supplied `X509Proxy` → upload; renew before |
| 173 | +expiry; reuse across submissions where valid) is `ARCBackend`-internal. The provider supplies the |
| 174 | +proxy; everything downstream — delegation ids, renewal, `AlwaysIncludeProxy`-style pairing with a |
| 175 | +token — is invisible to the consumer. |
| 176 | + |
| 177 | +### 6. Boundaries |
| 178 | + |
| 179 | +- **SSH keys are transport configuration**, not rotating credentials: long-lived, file/agent-based, |
| 180 | + no audience. They stay in `SSHTransport` config; revisit only if key rotation becomes a real |
| 181 | + requirement. |
| 182 | +- **Payload credentials are pilot-side.** Renewing the proxy *of the running payload* (DIRAC's |
| 183 | + `_monitorProxy`, `GENERIC_PILOT` branch) belongs to the severed pilot/runner domain, not here. |
| 184 | +- **Issuance is the consumer's.** interCEde never holds refresh tokens, client secrets, or CA |
| 185 | + material; it asks a provider and uses what it gets. |
| 186 | +- **Cloud provider auth** (Libcloud driver keys, the `cloud.auth` ini) is `CloudBackend` internal |
| 187 | + config; whether it adopts the provider model is deferred with the Cloud backend itself. |
| 188 | + |
| 189 | +### 7. Usage sketches (informative) |
| 190 | + |
| 191 | +**(a) The consumer implements the provider once; every backend pulls from it.** A DiracX |
| 192 | +submission task wires its token/proxy machinery into one object and never polices freshness |
| 193 | +again (this is the code that today exists twice, in `SiteDirector._setCredentials` and |
| 194 | +`WMSUtilities.setPilotCredentials`): |
| 195 | + |
| 196 | +```python |
| 197 | +class DiracXPilotCredentials: # consumer-side; satisfies CredentialProvider |
| 198 | + async def get(self, req: CredentialRequirements) -> CredentialSet: |
| 199 | + creds = [] |
| 200 | + if BearerToken in req.kinds: # mint scoped to what the backend declared |
| 201 | + tok = await token_service.mint(audience=req.audience, scopes=req.scopes) |
| 202 | + creds.append(BearerToken(tok.value, tok.expires_at)) |
| 203 | + if X509Proxy in req.kinds: # gProxyManager lineage, behind the provider |
| 204 | + pem = await proxy_store.download(pilot_dn, pilot_group, lifetime=86400) |
| 205 | + creds.append(X509Proxy(pem, expires_at=proxy_expiry(pem))) |
| 206 | + return CredentialSet(creds) |
| 207 | + |
| 208 | +backend = registry.backend( |
| 209 | + {"type": "htcondor-ce", "endpoint": "ce01.example.org:9619", ...}, |
| 210 | + credentials=DiracXPilotCredentials(), |
| 211 | +) |
| 212 | +async with backend: # lifecycle from IC-ADR-001 §2 |
| 213 | + sub = await backend.submit(spec, count=50) |
| 214 | + # before the operation the backend compared its cached CredentialSet against |
| 215 | + # backend.credential_requirements and awaited provider.get(...) only if stale |
| 216 | +``` |
| 217 | + |
| 218 | +**(b) What the HTCondorCE backend declares, and what it does with the set (Tier C).** The |
| 219 | +requirements make the Motivation-7 pairing explicit and typed; the materialisation reproduces |
| 220 | +DIRAC's mechanics without the caller knowing any of it: |
| 221 | + |
| 222 | +```python |
| 223 | +backend.credential_requirements == CredentialRequirements( |
| 224 | + kinds=frozenset({BearerToken, X509Proxy}), # all-of: token for the channel, |
| 225 | + audience="ce01.example.org:9619", # proxy for site-side VOMS/APEL accounting |
| 226 | +) |
| 227 | +# internally, per operation: |
| 228 | +# BearerToken -> 0600 temp file; _CONDOR_SEC_CLIENT_AUTHENTICATION_METHODS=SCITOKENS, |
| 229 | +# _CONDOR_SCITOKENS_FILE=<file> |
| 230 | +# X509Proxy -> 0600 temp file; X509_USER_PROXY=<file>, referenced by the submit |
| 231 | +# description's `use_x509userproxy = true` |
| 232 | +# both cleaned up at the operation/lifecycle boundary; values never logged |
| 233 | +``` |
| 234 | + |
| 235 | +**(c) Integration stacks and standalone use — static files, no issuance machinery.** The |
| 236 | +IC-ADR-002 stacks mint ephemeral credentials into the shared volume; the test harness (or any |
| 237 | +non-DiracX user) wraps them statically: |
| 238 | + |
| 239 | +```python |
| 240 | +provider = StaticCredentials(CredentialSet([ |
| 241 | + BearerToken(Path("credentials/htcondor/idtoken").read_text().strip(), expires_at=None), |
| 242 | +])) |
| 243 | +backend = registry.backend({"type": "htcondor-ce", ...}, credentials=provider) |
| 244 | +``` |
| 245 | + |
| 246 | +## Rationale |
| 247 | + |
| 248 | +- **Provider-pull over mutating setters.** Setters put hidden state on cached objects and force |
| 249 | + every caller to police freshness (two DIRAC copies prove the cost). A pull model puts the |
| 250 | + *check* in one library-side place while leaving the *source and policy* in consumer code — the |
| 251 | + same mechanism/policy line IC-ADR-001 draws for throttling. |
| 252 | +- **Declared requirements over tag sniffing.** `Tag: Token` conflates scheduling metadata with an |
| 253 | + auth capability and is invisible to type checkers and tooling. A typed declaration is |
| 254 | + enumerable, testable, and lets the conformance suite assert that a backend's declared kinds are |
| 255 | + the ones it actually uses. |
| 256 | +- **`CredentialSet` over single credential.** ARC's token+proxy pairing is a first-party |
| 257 | + requirement, not an edge case; modelling it from day one avoids a v2 of the provider protocol. |
| 258 | +- **Immutability.** Frozen values make "refresh" a *replacement*, never an in-place mutation — |
| 259 | + cache-safe and race-free under concurrent bulk operations. |
| 260 | + |
| 261 | +## Rejected Ideas |
| 262 | + |
| 263 | +- **DIRAC-style `set_credential()` mutators.** Hidden state on cached backends; freshness policy |
| 264 | + smeared across callers; the incumbent model this ADR exists to replace. |
| 265 | +- **Backend-driven issuance/refresh** (backend holds a refresh token or talks to the IdP). |
| 266 | + Couples the library to DiracX/IdP specifics, embeds secrets in the library, and moves policy |
| 267 | + inside — the opposite of the IC-ADR-001 §9 mechanism/policy split. |
| 268 | +- **Credentials in the `SubmissionSpec`** (name provisional — IC-ADR-001 Open Issues). Auth is |
| 269 | + per-backend/channel, not per-job; putting it on |
| 270 | + the spec would force every task (status, fetch, kill) to re-thread it and would leak payload vs |
| 271 | + backend credential confusion back in. |
| 272 | +- **A single `Credential` per backend (no set).** Breaks on ARC's token+proxy pairing. |
| 273 | +- **Keeping the `Tag: Token[:vo]` opt-in.** Stringly-typed, CS-coupled, invisible to types and |
| 274 | + tooling. |
| 275 | +- **An interCEde-owned credential store/daemon.** interCEde is stateless (IC-ADR-001 §1); caching |
| 276 | + beyond the in-memory current set is the consumer's business. |
| 277 | + |
| 278 | +## Open Issues |
| 279 | + |
| 280 | +- **Freshness margin.** Fixed library default vs per-backend/per-provider configuration; ARC |
| 281 | + delegation renewal wants a larger margin than a bearer-token header. |
| 282 | +- **Requirements granularity.** Whether `CredentialRequirements` stays per-backend or needs |
| 283 | + per-operation variance (e.g. a backend whose *fetch* endpoint needs a different scope than |
| 284 | + *submit*). Start per-backend; split only on evidence. |
| 285 | +- **Proxy representation.** Raw PEM bytes (current spec) vs a `cryptography` object; and whether |
| 286 | + delegation needs key material the consumer must supply alongside the chain (DIRAC signs the |
| 287 | + delegation CSR with the proxy's own key — implies the provider hands over key+chain, which |
| 288 | + `X509Proxy.pem` as "full chain" must be explicit about). |
| 289 | +- **Multi-VO backends.** One backend instance serving several VOs would need per-VO credential |
| 290 | + sets; today DIRAC instantiates per-queue/VO CEs, and interCEde's per-backend provider assumes |
| 291 | + the same. Confirm with DiracX's multi-VO design. |
| 292 | +- **DiracX alignment.** The provider implementation on the DiracX side (token service, scopes, |
| 293 | + pilot-credential flows) — tracked with the DiracX transition ADRs, not here. |
| 294 | +- **Proxy-for-accounting lifetime.** HTCondorCE's proxy requirement is a WLCG-transition artifact |
| 295 | + (sites attribute usage via the proxy's VOMS attributes, the APEL pipeline). When sites account |
| 296 | + on token claims instead, the backend's declared requirements shrink to token-only — and because |
| 297 | + requirements are *data*, that is a backend-version/configuration change, not an API break. |
| 298 | + Track the WLCG token-transition timeline before hard-coding the pairing as permanent. |
| 299 | +- **Conformance coverage.** IC-ADR-002 already plans token-vs-proxy configuration axes for the |
| 300 | + ARC stack; extend the conformance suite with a "requirements honesty" check (backend declares X, |
| 301 | + suite verifies it authenticates with exactly X). |
0 commit comments