|
| 1 | +# Hub kcp-proxy — per-workspace access (membership-gated) |
| 2 | + |
| 3 | +**Status:** Design agreed — ready for implementation (A-1…A-6) |
| 4 | +**Owner:** TBD |
| 5 | +**Last updated:** 2026-06-27 |
| 6 | +**Reads as a delta on:** [organizations.md](./organizations.md) (decision O-10), [provider-connectivity-contract.md](./provider-connectivity-contract.md) |
| 7 | +**Companion:** [app-studio-sandbox-runtime.md](./app-studio-sandbox-runtime.md) (the provider that hit this first) |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## Why this doc exists |
| 12 | + |
| 13 | +The hub's user-facing kcp proxy |
| 14 | +([pkg/server/proxy/proxy.go](../pkg/server/proxy/proxy.go)) forwards a user's |
| 15 | +own bearer token to kcp so the request runs with the user's identity and kcp |
| 16 | +enforces their RBAC natively. Before it forwards, it does a **cluster pre-check**: |
| 17 | +it only lets a request through to the **one** workspace recorded in |
| 18 | +`User.Spec.DefaultCluster`. Every other workspace is rejected with a 403 |
| 19 | +`cluster access denied` — *before* the request ever reaches kcp, and regardless |
| 20 | +of whether the user actually has RBAC there. |
| 21 | + |
| 22 | +That single-workspace funnel is the right default for the simplest case, but it |
| 23 | +breaks any user-facing flow that needs a **non-default** workspace. App Studio |
| 24 | +hit it head-on and had to route around the proxy entirely (see |
| 25 | +[Relationship to App Studio](#relationship-to-app-studio-option-b)). This doc |
| 26 | +proposes the platform-wide fix: **authorize the requested cluster against the |
| 27 | +user's membership, not against a single fixed `DefaultCluster`.** |
| 28 | + |
| 29 | +--- |
| 30 | + |
| 31 | +## Current model |
| 32 | + |
| 33 | +### `DefaultCluster` is a fixed "home" pointer |
| 34 | + |
| 35 | +`User.Spec.DefaultCluster` is written **once** by the organization controller |
| 36 | +([pkg/hub/controllers/organization/controller.go](../pkg/hub/controllers/organization/controller.go), |
| 37 | +"Step J") to the kcp logical-cluster ID of the user's **default** workspace |
| 38 | +(the default child Workspace of their personal Org). It is **not** updated when |
| 39 | +the user switches workspaces in the portal — there is no server-side "current |
| 40 | +workspace" concept. |
| 41 | + |
| 42 | +### The proxy gate |
| 43 | + |
| 44 | +`resolveKCPPath` ([proxy.go](../pkg/server/proxy/proxy.go)) accepts only that |
| 45 | +one cluster (or a mount under it): |
| 46 | + |
| 47 | +```go |
| 48 | +// /clusters/{id}/... — validated against defaultCluster |
| 49 | +if clusterID != defaultCluster && !strings.HasPrefix(clusterID, defaultCluster+":") { |
| 50 | + return "", http.StatusForbidden, `{... "message":"cluster access denied" ...}` |
| 51 | +} |
| 52 | +// bare /api|/apis path → scoped to /clusters/{defaultCluster}/... |
| 53 | +``` |
| 54 | + |
| 55 | +A **second** gate (O-10) refuses any `root:kedge:tenants:*` *path* outright |
| 56 | +(`OrgWorkspaceNotDirectlyAccessible`), steering Org-scoped operations to the hub |
| 57 | +REST surface. |
| 58 | + |
| 59 | +### What this means for multi-workspace users |
| 60 | + |
| 61 | +A user can belong to many Orgs and many Workspaces — the |
| 62 | +`UserMembershipIndex` ([apis/tenancy/v1alpha1/types_user_membership_index.go](../apis/tenancy/v1alpha1/types_user_membership_index.go)) |
| 63 | +lists every `(OrgUUID, WorkspaceUUID)` they hold a Membership in, and the |
| 64 | +organization controller's "Step H-backfill" grants them cluster-admin RBAC in |
| 65 | +each of those workspaces. So **kcp would authorize them** in any of their |
| 66 | +workspaces — but the proxy pre-check funnels user-token traffic to the single |
| 67 | +`DefaultCluster` and 403s the rest. |
| 68 | + |
| 69 | +This limitation is already acknowledged in code, in the comment on the |
| 70 | +provider-enable handler |
| 71 | +([pkg/hub/restapi/providers_enable.go](../pkg/hub/restapi/providers_enable.go)): |
| 72 | + |
| 73 | +> the hub's kcp user-proxy pre-checks the cluster path against |
| 74 | +> `User.Spec.DefaultCluster` and 403s every non-default workspace BEFORE |
| 75 | +> forwarding to kcp — even when commit #220's per-workspace RBAC grants would |
| 76 | +> have allowed it. |
| 77 | +
|
| 78 | +The enable flow worked around it by going through a hub REST handler that uses a |
| 79 | +kcp-admin client instead of the user proxy. |
| 80 | + |
| 81 | +--- |
| 82 | + |
| 83 | +## Proposal (Option A) |
| 84 | + |
| 85 | +Make the proxy authorize the requested cluster against the **caller's |
| 86 | +membership**, and let kcp RBAC remain the real enforcement boundary. |
| 87 | + |
| 88 | +### A-1 — Authorize against `UserMembershipIndex`, not a single `DefaultCluster` |
| 89 | + |
| 90 | +`resolveKCPPath` changes from "is this the default cluster?" to "is this a |
| 91 | +workspace the caller is a member of?" (the SA path is handled separately in |
| 92 | +A-6). Concretely: |
| 93 | + |
| 94 | +- For `/clusters/{id}/...`: allow when `{id}` maps to a workspace in the |
| 95 | + caller's `UserMembershipIndex` (or an **edge** under such a workspace — |
| 96 | + `{id}:{edgeName}`, see A-3). |
| 97 | +- For bare `/api|/apis` paths (no cluster segment): **reject — no default.** The |
| 98 | + proxy no longer silently scopes bare paths to `DefaultCluster`; a request with |
| 99 | + no workspace selector can't be authorized against membership, and silently |
| 100 | + defaulting risks hitting the wrong workspace. Clients always address |
| 101 | + `/clusters/{id}`, resolving the ID via REST (A-5). `DefaultCluster` is then |
| 102 | + only a *landing hint* for the UI/CLI on first use, not a server-side |
| 103 | + request-scoping default. |
| 104 | + |
| 105 | +**Back the authorization with an informer/watch on `UserMembershipIndex`, not a |
| 106 | +TTL cache.** The index is continuously reconciled by the Membership controller |
| 107 | +(O-3: it owns the index and keeps it in sync with every Membership write), so an |
| 108 | +informer-backed local view is as fresh as the controller — authorization reads a |
| 109 | +hot in-memory set with no per-request kcp round-trip and no TTL staleness window. |
| 110 | +The proxy already holds `kedgeClient`; add a shared informer for the index and |
| 111 | +gate off its lister. |
| 112 | + |
| 113 | +### A-2 — Cluster → (org, workspace) topology index |
| 114 | + |
| 115 | +Requests address clusters by **ID**; the membership index keys off Org/Workspace |
| 116 | +**UUIDs** (path components). Rather than push cluster IDs into every membership |
| 117 | +entry — or resolve `LogicalCluster` per request — keep a **separate |
| 118 | +reconciler-maintained topology index**: `clusterID → (orgUUID, wsUUID)` over the |
| 119 | +Org/Workspace tree. |
| 120 | + |
| 121 | +- A small hub reconciler (or an informer-derived index over the kcp `Workspace` |
| 122 | + objects, which carry both `spec.cluster` and their path) maintains |
| 123 | + `map[clusterID] → (org, ws)`. It's tenant-wide, not per-user, and reflects |
| 124 | + workspace create/delete continuously — the same freshness model as A-1. |
| 125 | +- The `LogicalCluster`/`newClusterIDResolver` primitive |
| 126 | + ([pkg/hub/provider_cluster_resolver.go](../pkg/hub/provider_cluster_resolver.go)) |
| 127 | + is the per-entry resolve the reconciler uses to populate the index; it is |
| 128 | + **not** on the request path. |
| 129 | + |
| 130 | +This is the key simplification: with the topology index, **org-scope and |
| 131 | +workspace-scope authorization become the same O(1) check** (see A-3). No cluster |
| 132 | +IDs duplicated into membership entries, no per-request kcp call, no fan-out of |
| 133 | +org-scope memberships into synthetic per-workspace entries. |
| 134 | + |
| 135 | +### A-3 — Authorization check (one rule for both scopes) |
| 136 | + |
| 137 | +For a request to `/clusters/{id}`: |
| 138 | + |
| 139 | +1. **Topology (A-2):** `id → (org, ws)`. If `id` isn't in the index it isn't a |
| 140 | + kedge child workspace → fall through to the existing gates (O-10 / 403). |
| 141 | +2. **Membership (A-1):** the caller's `UserMembershipIndex` covers `(org, ws)` |
| 142 | + when it holds **either** a workspace-scope entry `(org, ws)` **or** an |
| 143 | + org-scope entry `(org, "")`. Org-scope is just the `(org, *)` case of the |
| 144 | + same lookup — no special path. |
| 145 | + |
| 146 | +**Edges** (`/clusters/{id}:{edgeName}`) are authorized by their parent: an edge |
| 147 | +mounted under a workspace the caller may reach is allowed. (kcp calls this a |
| 148 | +"mount"; in kedge the mounted thing is an **edge**, so the terminology and the |
| 149 | +allowance are stated in edge terms — `{id}:{edgeName}`, not `{id}:{mountName}`.) |
| 150 | + |
| 151 | +O-10 (no direct access to **Org** workspaces) stays: a request whose target |
| 152 | +resolves to the Org workspace itself (`root:kedge:tenants:{org}`, no `:{ws}`) |
| 153 | +never matches a child entry and is refused as today. So the relaxation is |
| 154 | +strictly "a member may reach their **child** workspaces"; the Org workspace |
| 155 | +remains hub-mediated. |
| 156 | + |
| 157 | +### A-4 — Drop the "current cluster" idea on the client |
| 158 | + |
| 159 | +With A-1 in place there is no need for a server-side "current workspace". The |
| 160 | +client always addresses `/clusters/{id}` for whichever workspace it's operating |
| 161 | +in (no bare-path fallback — A-1); the proxy authorizes it against membership. |
| 162 | +`DefaultCluster` is reduced to a **landing hint** — the workspace the UI/CLI |
| 163 | +points at on first use — with no request-scoping role. |
| 164 | + |
| 165 | +### A-5 — Clients learn the cluster ID via a hub REST endpoint |
| 166 | + |
| 167 | +The client still needs the cluster **ID** to address `/clusters/{id}`. Expose it |
| 168 | +through the existing membership-gated org/workspace REST surface (O-10), reusing |
| 169 | +the A-2 topology index for the `(org, ws) → clusterID` direction: |
| 170 | + |
| 171 | +- **Resolve one:** `GET /api/orgs/{org}/workspaces/{ws}` returns the workspace's |
| 172 | + `clusterID` (add the field; the CLI plugin resolves a workspace name/UUID → |
| 173 | + ID, then writes a kubeconfig server URL of `<front-proxy>/clusters/{id}`). |
| 174 | +- **List many:** `GET /api/orgs/{org}/workspaces` (and the switcher's |
| 175 | + `UserMembershipIndex`-backed listing) carry `clusterID` per row, so the portal |
| 176 | + retargets its kcp/GraphQL client on a workspace switch without an extra call. |
| 177 | + |
| 178 | +Because these endpoints are already gated by `tenant.Middleware` (the caller |
| 179 | +must hold a Membership in `(org, ws)`), a client can only resolve IDs for |
| 180 | +workspaces it can actually reach — the same authorization the proxy then |
| 181 | +re-checks (A-3), so REST and proxy never disagree. This is the symmetric, |
| 182 | +provider-agnostic equivalent of the `X-Kedge-Cluster` header the backend proxy |
| 183 | +injects for provider HTTP traffic: REST hands the **client** the ID; the header |
| 184 | +hands the **provider** the ID; both come from the one topology index. |
| 185 | + |
| 186 | +### A-6 — ServiceAccount / static-token path: pin to the token's cluster claim |
| 187 | + |
| 188 | +Workspace ServiceAccounts (O-14) are **not** membership-expanded — an SA belongs |
| 189 | +to exactly one workspace and must reach only that one. kcp already carries the |
| 190 | +SA's logical cluster **inside the token**: a bound SA JWT has the cluster in the |
| 191 | +`kubernetes.io.clusterName` claim (legacy tokens: |
| 192 | +`kubernetes.io/serviceaccount/clusterName`), and kcp's |
| 193 | +[`WithInClusterServiceAccountRequestRewrite`](../../kcp-dev/kcp/pkg/server/filters/serviceaccounts.go) |
| 194 | +reads that claim and rewrites the request to `/clusters/<clusterName>/...`. |
| 195 | + |
| 196 | +So the proxy's SA path does the same: parse the SA token, read the |
| 197 | +`kubernetes.io.clusterName` claim, and authorize **only** that cluster (or an |
| 198 | +edge under it). No `UserMembershipIndex` lookup, no topology join — the token |
| 199 | +*is* the authorization scope, self-pinned to the SA's home workspace. A request |
| 200 | +that targets any other `/clusters/{id}` than the claim is refused. This keeps SA |
| 201 | +identities strictly single-workspace while users (A-1…A-3) span their member |
| 202 | +workspaces. |
| 203 | + |
| 204 | +--- |
| 205 | + |
| 206 | +## Security analysis |
| 207 | + |
| 208 | +- **kcp RBAC is unchanged and remains authoritative.** The proxy forwards the |
| 209 | + user's own token; kcp evaluates the user's RBAC in the target workspace. The |
| 210 | + proxy gate is **defense-in-depth**, not the primary control. Today it is |
| 211 | + *too tight* (single cluster); A-1 makes it match reality (the workspaces the |
| 212 | + user is a member of) while still failing closed for everything else. |
| 213 | +- **No new trust in client input.** Authorization keys off the authenticated |
| 214 | + user's `UserMembershipIndex`, which the user cannot forge — exactly the model |
| 215 | + the tenant resolver already uses for the `X-Kedge-Org`/`X-Kedge-Workspace` |
| 216 | + headers ([provider_tenant_resolver.go](../pkg/hub/provider_tenant_resolver.go)). |
| 217 | +- **Org workspaces stay sealed** (A-3 / O-10). |
| 218 | +- **Revocation is reconciler-driven, not time-bounded.** Removing a Membership |
| 219 | + makes the Membership controller delete the matching `UserMembershipIndex` |
| 220 | + entry **and** tear down the per-workspace RBAC grant (the inverse of the |
| 221 | + organization controller's Step-H backfill). The proxy's informer reflects the |
| 222 | + index deletion within its propagation latency, and kcp denies independently |
| 223 | + once the RBAC grant is gone — two reconciler-driven controls, no TTL window to |
| 224 | + reason about. |
| 225 | +- **Failure mode is closed:** unknown cluster, non-member, or index-lookup |
| 226 | + error → 403, same as today. |
| 227 | +- **Blast radius is the most security-sensitive path in the system** (every |
| 228 | + user, every `kubectl`, every portal kcp call). This is the reason to document |
| 229 | + and review the design before implementing, and to land it behind tests that |
| 230 | + assert: member→allowed, non-member→403, Org-workspace→403, cross-Org |
| 231 | + isolation, bare-path→**rejected** (no default), SA→only its claim cluster |
| 232 | + (other cluster→403), and edge-under-member-workspace→allowed. |
| 233 | + |
| 234 | +--- |
| 235 | + |
| 236 | +## Relationship to App Studio (Option B) |
| 237 | + |
| 238 | +App Studio needed per-workspace access *now* and could not wait on a change to |
| 239 | +the shared proxy, so it took **Option B**: route tenant traffic through the |
| 240 | +hub's embedded **GraphQL gateway** (`/graphql/{clusterID}`), which serves any |
| 241 | +workspace the caller has RBAC in and is **not** `DefaultCluster`-gated. That |
| 242 | +work added two pieces this proposal builds on: |
| 243 | + |
| 244 | +- The backend proxy injects **`X-Kedge-Cluster`** — the resolved tenant's |
| 245 | + logical-cluster ID |
| 246 | + ([pkg/hub/provider_cluster_resolver.go](../pkg/hub/provider_cluster_resolver.go), |
| 247 | + wired in [pkg/hub/providers/proxy.go](../pkg/hub/providers/proxy.go)). The |
| 248 | + same resolver is reusable for A-2.2. |
| 249 | +- It demonstrated, in production-shaped local runs, that a user token reaching a |
| 250 | + **non-default** workspace works end-to-end once the addressing is right — i.e. |
| 251 | + kcp authorizes it. That is the empirical basis for A-1. |
| 252 | + |
| 253 | +Option A does **not** replace Option B. GraphQL remains the right surface for |
| 254 | +provider data planes (typed schema, subscriptions, the `*Yaml`/`applyYaml` |
| 255 | +conveniences). Option A is about the **raw kcp proxy** — `kubectl`, the portal's |
| 256 | +direct kcp calls, and any future provider that wants user-identity kcp access |
| 257 | +without standing up a GraphQL client. Once A-1 lands, a provider could choose |
| 258 | +either surface; today the proxy forces non-default workspaces onto GraphQL or |
| 259 | +the hub REST handlers. |
| 260 | + |
| 261 | +--- |
| 262 | + |
| 263 | +## Decided |
| 264 | + |
| 265 | +- **Freshness = informer, not TTL.** Authorization gates off an informer-backed |
| 266 | + lister of `UserMembershipIndex`, kept current by the Membership controller — |
| 267 | + as fresh as the controller, no staleness window (see A-1). |
| 268 | +- **Revocation = reconciler-driven.** Removing the Membership makes the |
| 269 | + controller delete the index entry and tear down the per-workspace RBAC grant; |
| 270 | + both the proxy informer and kcp deny without any time bound (see Security |
| 271 | + analysis). |
| 272 | +- **No feature flag.** Ship the membership gate directly, guarded by the test |
| 273 | + matrix above rather than a runtime flag. |
| 274 | +- **Org-scope authorization = topology index, not membership fan-out.** A |
| 275 | + separate reconciler-maintained `clusterID → (org, ws)` topology index (A-2) |
| 276 | + turns authorization into two O(1) in-memory lookups (topology then |
| 277 | + membership), with org-scope as the `(org, *)` case of the same check (A-3). No |
| 278 | + cluster→org resolve on the request path and no fan-out of org-scope |
| 279 | + memberships into synthetic per-workspace entries. |
| 280 | +- **Client gets the cluster ID from hub REST.** The membership-gated |
| 281 | + org/workspace endpoints return `clusterID` (single + listing), reusing the |
| 282 | + topology index (A-5). CLI plugin and UI resolve workspace → ID there, then |
| 283 | + address `/clusters/{id}`; no client-side kcp resolve. |
| 284 | +- **ServiceAccounts = pin to the token's cluster claim, not membership.** The SA |
| 285 | + path reads the SA JWT's `kubernetes.io.clusterName` claim and authorizes only |
| 286 | + that one cluster (or an edge under it), matching kcp's |
| 287 | + `WithInClusterServiceAccountRequestRewrite` (A-6). SAs stay single-workspace. |
| 288 | +- **No bare-path default.** Bare `/api|/apis` (no cluster selector) is |
| 289 | + **rejected**, not silently scoped to `DefaultCluster` — clients always address |
| 290 | + `/clusters/{id}` via the REST-resolved ID (A-1, A-5). `DefaultCluster` becomes |
| 291 | + a UI/CLI landing hint only. |
| 292 | +- **Edges, not "mounts".** The `{id}:{mountName}` allowance is re-expressed in |
| 293 | + kedge terms as `{id}:{edgeName}`, authorized by the parent workspace's |
| 294 | + membership (A-3). |
| 295 | + |
| 296 | +## Open questions |
| 297 | + |
| 298 | +None outstanding — the design decisions above cover the proposal. Remaining work |
| 299 | +is implementation (A-1…A-6) and the test matrix in [Security analysis](#security-analysis). |
0 commit comments