feat(auth): add workload delegation

Signed-off-by: Ryan S <267728323+ironcommit@users.noreply.github.com>

Author: ironcommit

contrib/auth/README.md

@@ -15,6 +15,15 @@ adaptation:
   group binding, not as internal `service:*` principals
 - document provider-specific setup in a local `README.md`
 
+Managed job OBO tests use the provider for user and controller authentication,
+but the workload-to-submitter binding is NeMo Platform auth state. Jobs receive
+`NMP_WORKLOAD_IDENTITY_TOKEN_FILE`, exchange that subject token through
+`/apis/auth/token`, and receive a NeMo Platform token whose top-level subject
+is the job submitter and whose RFC 8693 `act.sub` is the workload actor. Provider
+manifests may include workload-provider token grants for contract tests, but
+managed Docker job OBO must not depend on provider-specific fields such as
+`jti`.
+
 Open-source providers with `mode: compose-ci` are intended for the shared auth
 matrix. Reference-only providers stay documented and manifest-driven but are
 excluded from the local Compose-backed matrix.

contrib/auth/authentik/README.md

@@ -6,9 +6,10 @@
 This directory contains a local Authentik-backed NeMo Platform example. Use it
 to validate three user-visible flows:
 
-- log in to NeMo with Authentik
-- call NeMo APIs through the Authentik gateway
-- run a NeMo job whose workload exchanges a real Authentik workload subject token
+- log in to NeMo Platform with Authentik
+- call NeMo Platform APIs through the Authentik gateway
+- run a NeMo Platform job whose workload exchanges a managed workload proof token for a
+  delegated NeMo Platform access token
 
 All credentials in this example are for local development only.
 
@@ -88,8 +89,11 @@ The 2-minute CLI access-token lifetime is a local demo/testing setting so token
 refresh is easy to observe. Do not use it as a production default; use a longer
 value such as `hours=1` outside the refresh demonstration.
 
-In the Docker Compose runtime, Authentik issues the demo workload subject token,
-but it does not accept the RFC 8693 token exchange grant directly. The Docker
-backend refreshes the Authentik subject token file, the SDK posts that token to
-the NeMo auth service, and the gateway trusts the NeMo auth service JWKS for
-exchanged workload access tokens.
+In the Docker Compose runtime, Authentik authenticates users and controller
+service principals, but managed Docker job OBO does not depend on Authentik
+password-grant workload tokens or an IdP `jti` claim. The Docker backend writes
+a NeMo Platform-owned workload proof token into the job token file, the SDK
+posts that token to the NeMo Platform auth service, and the gateway trusts the
+NeMo Platform auth service JWKS for exchanged workload access tokens. The
+Authentik workload-provider password grant remains in the example manifest for
+direct provider-token contract tests.

contrib/auth/authentik/compose/implementation-details.md

@@ -23,9 +23,9 @@ the parent directory:
 - `../helm/files/blueprints` as the |product-name| blueprint source.
 - `../.generated` for local generated keys and certificates.
 
-The shared tutorial does not build NeMo images for Compose. It runs
-`${IMAGE_REGISTRY:-my-registry}/nmp-api:${BAKE_TAG:-local}` for both the NeMo
-API service and workload jobs submitted by the tutorial.
+The shared tutorial does not build NeMo Platform images for Compose. It runs
+`${IMAGE_REGISTRY:-my-registry}/nmp-api:${BAKE_TAG:-local}` for both the
+NeMo Platform API service and workload jobs submitted by the tutorial.
 
 ## Services
 
@@ -42,7 +42,7 @@ The stack contains:
 - `authentik-redis`: Redis for Authentik.
 - `authentik-server` and `authentik-worker`: Authentik itself.
 
-`nemo` is only on the internal network. Host and workload traffic reaches NeMo
+`nemo` is only on the internal network. Host and workload traffic reaches NeMo Platform
 through the `gateway` service, which also joins the workload network as
 `nemo-gateway`.
 
@@ -59,15 +59,15 @@ share local keys:
 The workload-token private key is mounted into `nemo` at
 `/var/run/secrets/nemo-platform/workload-token-signing/private-key.pem`.
 `platform-compose-authentik.yaml` points
-`auth.token_signing.private_key_file` at that mounted path. The NeMo auth
-service uses the private key to sign workload-exchange access tokens and Scoped
-Access Key JWTs, and Envoy validates those tokens through the NeMo auth service
-JWKS endpoints.
+`auth.token_signing.private_key_file` at that mounted path. The NeMo Platform
+auth service uses the private key to sign workload-exchange access tokens and
+Scoped Access Key JWTs, and Envoy validates those tokens through the
+NeMo Platform auth service JWKS endpoints.
 
 The gateway TLS files are copied into the `gateway-tls` named volume by
 `gateway-tls-init`. The `gateway` service uses that volume to serve HTTPS, and
 the `nemo` service mounts the same volume read-only so Python HTTP clients
-inside NeMo trust the demo gateway certificate.
+inside NeMo Platform trust the demo gateway certificate.
 
 All generated keys and certificates in this example are for local development
 only.
@@ -94,7 +94,7 @@ The `nemo-setup` service account and app-password in the blueprint exist solely
 for automated auth-idp contract tests. They are not part of the browser login
 flow or the workload identity pattern.
 
-## NeMo Compose Configuration
+## NeMo Platform Compose Configuration
 
 `platform-compose-authentik.yaml` configures NeMo Platform for this topology:
 
@@ -104,8 +104,11 @@ flow or the workload identity pattern.
   the `nemo` container.
 - Host-side CLI login uses the port-forward-like public gateway URL
   `https://127.0.0.1:18080`.
-- Workload subject tokens come from Authentik's workload OIDC provider.
-- Exchanged workload access tokens come from NeMo's `/apis/auth/token` endpoint.
+- Authentik provides user and controller service-principal authentication. The
+  Docker managed-job OBO binding is stored in NeMo Platform auth delegation
+  state, not in Authentik.
+- Exchanged workload access tokens come from NeMo Platform's `/apis/auth/token`
+  endpoint.
 
 The Docker jobs executor mounts the `gateway-tls` volume into workload
 containers and sets `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` so workload code
@@ -115,23 +118,23 @@ trusts the local gateway certificate.
 
 Envoy is the public entrypoint for the Compose example. It routes:
 
-- NeMo paths such as `/.well-known/nemo-platform/`, `/apis/`, `/health/`,
-  `/status`, and `/studio/` to `nemo`.
+- NeMo Platform paths such as `/.well-known/nemo-platform/`, `/apis/`,
+  `/health/`, `/status`, and `/studio/` to `nemo`.
 - `/health/gateway/ready` to an Envoy-owned readiness check that verifies both
-  NeMo and Authentik through their upstream clusters.
+  NeMo Platform and Authentik through their upstream clusters.
 - Authentik paths to `authentik-server`.
 
 Before authentication, Envoy removes incoming `X-NMP-Principal-*` and
 `X-NMP-Scopes` headers so a client cannot spoof identity or scopes. For
-protected `/apis/` requests, Envoy calls NeMo's
+protected `/apis/` requests, Envoy calls NeMo Platform's
 `/apis/auth/authenticate` endpoint with the presented bearer token. The auth
-service validates Authentik OIDC tokens, NeMo workload-exchange access tokens,
-and NeMo Scoped Access Keys, then returns trusted `X-NMP-Principal-*` and
-`X-NMP-Scopes` headers for Envoy to forward upstream.
+service validates Authentik OIDC tokens, NeMo Platform workload-exchange access
+tokens, and NeMo Platform Scoped Access Keys, then returns trusted
+`X-NMP-Principal-*` and `X-NMP-Scopes` headers for Envoy to forward upstream.
 
 The gateway callout is required for dynamic or revocable Scoped Access Keys
 because Envoy JWKS validation can only prove token signature, issuer, audience,
-and time claims. It cannot check NeMo's access-key lifecycle state. Compose
+and time claims. It cannot check NeMo Platform's access-key lifecycle state. Compose
 keeps `auth.access_keys.enabled=true` so Scoped Access Keys can be created and
 validated; Envoy performs the bearer-to-header mapping before the request
 reaches service middleware.
@@ -145,7 +148,7 @@ job request should not include `NMP_WORKLOAD_IDENTITY_TOKEN_FILE`,
 `NEMO_WORKLOAD_TOKEN`, or `NEMO_WORKLOAD_TOKEN_FILE`.
 
 When a managed Docker workload starts, the backend creates a dedicated workload
-identity volume, writes an Authentik subject token to:
+identity volume and writes a NeMo Platform-owned Docker workload proof token to:
 
 ```text
 /var/run/secrets/nemo-platform/workload/token
@@ -158,9 +161,28 @@ NMP_WORKLOAD_IDENTITY_TOKEN_FILE=/var/run/secrets/nemo-platform/workload/token
 ```
 
 The SDK reads that file and sends an RFC 8693 token exchange request to the
-NeMo auth service through the gateway. The NeMo auth service validates the
-Authentik subject token, mints a NeMo-signed access token, and returns it to the
-workload. The workload uses that exchanged token for normal NeMo API calls.
+NeMo Platform auth service through the gateway. The Docker backend registered
+an internal workload delegation row before the container started. The
+NeMo Platform auth service validates the proof token, checks the matching row,
+mints a NeMo Platform-signed delegated access token, and returns it to the
+workload. The access token uses the captured job submitter as the top-level
+subject and the Docker workload as the RFC 8693 `act.sub` actor.
+
+Docker supports two proof-provider modes:
+
+- `jwt`: for embedded standalone deployments where the Docker jobs controller
+  has an in-process auth-owned JWT issuer. The proof JWT uses
+  `token_use="nemo_docker_workload_subject"` and `sub=<delegation-row-name>`.
+- `opaque`: for split deployments where the jobs controller calls remote auth
+  state instead of owning signing keys. The file contains a private opaque proof
+  token type whose secret is stored only as a hash in the delegation row.
+
+The checked-in Compose example uses the default `auto` mode, which selects the
+opaque provider unless an embedded auth-owned JWT issuer is available. Docker
+job OBO therefore does not require Authentik to issue a workload token and does
+not depend on an IdP `jti` claim. The Authentik workload-provider password
+grant in the manifest is retained for direct provider-token contract tests, not
+for the managed Docker job exchange loop.
 
 The useful end-to-end validation is the workload job in the shared tutorial:
 the job uses the exchanged token to call the NeMo Platform API and read the

contrib/auth/authentik/gateway/envoy.yaml

@@ -242,6 +242,9 @@ static_resources:
                               - exact: x-nmp-principal-id
                               - exact: x-nmp-principal-email
                               - exact: x-nmp-principal-groups
+                              - exact: x-nmp-principal-on-behalf-of
+                              - exact: x-nmp-principal-on-behalf-of-email
+                              - exact: x-nmp-principal-on-behalf-of-groups
                               - exact: x-nmp-scopes
                           allowed_client_headers:
                             patterns:

contrib/auth/authentik/helm/templates/_envoy-config.tpl

@@ -250,6 +250,9 @@ static_resources:
                               - exact: x-nmp-principal-id
                               - exact: x-nmp-principal-email
                               - exact: x-nmp-principal-groups
+                              - exact: x-nmp-principal-on-behalf-of
+                              - exact: x-nmp-principal-on-behalf-of-email
+                              - exact: x-nmp-principal-on-behalf-of-groups
                               - exact: x-nmp-scopes
                           allowed_client_headers:
                             patterns:

contrib/auth/authentik/kubernetes/implementation-details.md

@@ -37,11 +37,11 @@ upgrade command.
 The chart creates or reuses these additional local-demo Secrets during Helm
 rendering:
 
-- `shared-postgresql` for the shared PostgreSQL superuser, Authentik, and NeMo
+- `shared-postgresql` for the shared PostgreSQL superuser, Authentik, and NeMo Platform
   database passwords.
 - `shared-postgresql-nemo` for the NeMo Platform external database password.
 - `nemo-platform-envoy-tls` for the demo Envoy TLS certificate and CA.
-- `nemo-workload-token-signing-key` for the NeMo-issued workload access token
+- `nemo-workload-token-signing-key` for the NeMo Platform-issued workload access token
   signing key.
 
 The chart generates `Secret/nemo-platform-envoy-tls` during Helm rendering and
@@ -62,13 +62,14 @@ mounted file.
 Use `--wait --wait-for-jobs` when installing the chart so Helm only returns
 after the blueprint has been applied.
 
-## NeMo Kubernetes Override
+## NeMo Platform Kubernetes Override
 
 The umbrella chart passes the Kubernetes-specific NeMo Platform configuration
 through `nemo-platform.platformConfig` values. It also configures
 `nemo-platform.envoyProxy.configOverride` so the NeMo Platform chart's Envoy
 deployment keeps the Authentik path split and validates both Authentik-issued
-tokens, NeMo workload-exchange tokens, and NeMo Scoped Access Key JWTs.
+tokens, NeMo Platform workload-exchange tokens, and NeMo Platform Scoped
+Access Key JWTs.
 
 Kubernetes projected service account token expiration defaults to `600` seconds
 in the jobs backend. Override it through the NeMo Platform chart values if you
@@ -83,23 +84,36 @@ workload pods and injects:
 NMP_WORKLOAD_IDENTITY_TOKEN_FILE=/var/run/secrets/nemo-platform/workload/token
 ```
 
-The SDK reads that file and sends an RFC 8693 token exchange request to the NeMo
-auth service over HTTPS. The chart mounts `ca.crt` from
+The SDK reads that file and sends an RFC 8693 token exchange request to the
+NeMo Platform auth service over HTTPS. The chart mounts `ca.crt` from
 `Secret/nemo-platform-envoy-tls` into Kubernetes workload pods and sets
 `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` so in-pod Python HTTP clients verify
 the demo Envoy certificate. Host-side `nemo` commands should use
 `NMP_CLIENT_SSL_CERT_FILE` instead so unrelated tools keep their normal trust
 store.
 
-The NeMo auth service validates projected service account tokens with the
-TokenReview API and returns a NeMo-signed JWT trusted by the NeMo Platform
-Envoy. The useful end-to-end validation is the workload job in the tutorial:
-the job pod uses the exchanged token to call the NeMo Platform API and read the
-workspace.
+The NeMo Platform auth service validates projected service account tokens with
+the TokenReview API and returns a NeMo Platform-signed JWT trusted by the
+NeMo Platform Envoy. The TokenReview response must include exactly one
+`authentication.kubernetes.io/pod-uid` value. The jobs controller observes the
+created Pod, registers an internal delegation row keyed by that Pod UID,
+service account subject, and audience, and auth later looks up that row from
+the verified TokenReview metadata. The useful end-to-end validation is the
+workload job in the tutorial: the job pod uses the exchanged token to call the
+NeMo Platform API and read the workspace.
+
+Auth only needs RBAC to create TokenReview requests. It does not need to read
+Pods or Jobs for token exchange, and the chart grants no Pod or Job read
+permissions to the auth/API service account for this path. Kubernetes labels,
+annotations, and owner references help the jobs controller reconcile and clean
+up backend resources, but they are not token-exchange authorization inputs.
 
 The workload job request should not include workload auth environment
 variables. The Kubernetes jobs backend owns `NMP_WORKLOAD_IDENTITY_TOKEN_FILE`;
-users must not set `NEMO_WORKLOAD_TOKEN` or `NEMO_WORKLOAD_TOKEN_FILE`.
+users must not set `NMP_PRINCIPAL`, `NEMO_WORKLOAD_TOKEN`, or
+`NEMO_WORKLOAD_TOKEN_FILE`. When workload OBO is enabled, the workload receives
+only the subject token file path and obtains its delegated NeMo Platform access
+token by calling `/apis/auth/token`.
 
 To inspect the projected token mount for a submitted job:
 
@@ -113,8 +127,8 @@ kubectl --context "${KUBE_CONTEXT}" -n "${NAMESPACE}" describe pod \
 
 ## Shared Token Signing Key
 
-Workload identity token exchange requires the NeMo auth service to sign the
-access token it mints from a Kubernetes projected service account subject
+Workload identity token exchange requires the NeMo Platform auth service to
+sign the access token it mints from a Kubernetes projected service account subject
 token. The chart creates `Secret/nemo-workload-token-signing-key` by default,
 mounts `private-key.pem` into the NeMo Platform API pod at
 `/etc/nmp/workload-token/private-key.pem`, and sets
@@ -123,9 +137,9 @@ configuration is used for workload-exchange access tokens and Scoped Access Key
 JWTs. The matching public keys are served from `/apis/auth/jwks`.
 
 The Helm-rendered Envoy config authenticates protected `/apis/` requests by
-calling `/apis/auth/authenticate` on the NeMo API service. It does not use Envoy
-`claim_to_headers` for Scoped Access Keys. This keeps future revocation and
-dynamic-key checks inside the auth service, where access-key records can be
+calling `/apis/auth/authenticate` on the NeMo Platform API service. It does not
+use Envoy `claim_to_headers` for Scoped Access Keys. This keeps future
+revocation and dynamic-key checks inside the auth service, where access-key records can be
 looked up before Envoy forwards trusted principal headers.
 
 Scoped Access Keys remain disabled in the checked-in chart values by default.

contrib/auth/authentik/manifest.yaml

@@ -71,6 +71,7 @@ test_runtimes:
       - platform_access_keys
       - workspace_rbac
       - workload_job
+      - managed_workload_job_obo
       - device_flow
       - docker_subject_token_refresh
   - id: authentik-kubernetes
@@ -86,5 +87,6 @@ test_runtimes:
       - platform_access_keys
       - workspace_rbac
       - workload_job
+      - managed_workload_job_obo
       - device_flow
       - kubernetes_token_review