Skip to content

Grant Config Connector controller perms for per-env SA provisioning#15

Open
pcholakov wants to merge 4 commits into
mainfrom
pavel/gcp-env-identity
Open

Grant Config Connector controller perms for per-env SA provisioning#15
pcholakov wants to merge 4 commits into
mainfrom
pavel/gcp-env-identity

Conversation

@pcholakov

@pcholakov pcholakov commented May 22, 2026

Copy link
Copy Markdown
Contributor

Related: restatedev/restate-cloud#833 (architecture overview, locked decisions, e2e validation summary)


Summary

Grants the Config Connector controller SA the permissions it needs to provision per-environment GCP service accounts on the install project. Companion to:

  • restatedev/restate-cloud PR (spec + impl plan)
  • restatedev/nuon-byoc PR `pavel/gcp-env-identity` (bucket-scoped IAM)

What changed

  • New least-privilege custom role `<nuon_id>_cc_env_sa_admin` over the predefined `roles/iam.serviceAccountAdmin`:
    • `iam.serviceAccounts.create`
    • `iam.serviceAccounts.delete`
    • `iam.serviceAccounts.get`
    • `iam.serviceAccounts.getIamPolicy`
    • `iam.serviceAccounts.setIamPolicy`
  • Project-level binding of that role to the CC controller SA.
  • Existing per-SA `serviceAccountAdmin` grant on the install `restate` SA preserved with a "do not remove" comment (load-bearing for the operator's v2.3.0 WI-binding automation).
  • Exports `config_connector_sa_email` so the bucket component in nuon-byoc can scope its bucket-IAM perms to it.

Validation

  • `terraform init -backend=false -input=false` + `terraform validate` clean.
  • `terraform fmt -check` on touched files clean.

Test plan

End-to-end against the GCP dev canary install after the cloud control-plane code lands (Track 2b). See `restate-cloud/docs/testing/gcp-env-identity-e2e.md`.

@pcholakov
pcholakov marked this pull request as ready for review May 22, 2026 08:41
@pcholakov
pcholakov force-pushed the pavel/gcp-env-identity branch 2 times, most recently from 83e3c74 to 5fbfbc7 Compare May 25, 2026 13:39
pcholakov added 4 commits May 25, 2026 18:28
Without proxy.nativeSidecar=true the linkerd-proxy is injected as a
regular container that never exits. Jobs (notably restate-wi-canary)
get stuck 1/2 NotReady on a failure, so the Job's backoffLimit retry
never fires and a transient first-attempt failure (e.g. GCP IAM
eventual consistency on the Workload Identity binding for a freshly
created env namespace) leaves the env unable to come up.

With native sidecar, the proxy is an init container with restartPolicy:
Always — Kubernetes terminates it when the main container exits, the
pod transitions to Failed/Succeeded, and the Job retries normally.
Moves the four kyverno cluster policies (linkerd-injection annotation
mutate, linkerd authz generate, and the two RBAC ClusterRoles that grant
kyverno permission to create policy.linkerd.io resources) into the
sandbox TF as kubectl_manifest resources.

They were previously declared as Nuon [[policy]] type=kubernetes_cluster
entries in restate-byoc-gcp/policies.toml, but Nuon does not actually
deploy them for this app -- the install config state has no policies
field, and verifying on the canary cluster post-reprovision confirmed
none were applied. Moving them into the sandbox makes them part of the
substrate alongside kyverno itself, so a reprovision installs everything
needed for new env namespaces to be linkerd-meshed correctly.
GKE Autopilot allocates the services CIDR from Google's managed range
(34.118.224.0/20 on the canary cluster), which isn't covered by
linkerd's default clusterNetworks (RFC1918 + IPv6 link-local). Without
this, linkerd-proxy treats traffic to ClusterIP services as off-cluster
egress and routes it through the EgressNetwork, losing identity-aware
in-cluster policy and skewing observability for service-to-service
calls.

Pulled dynamically from the cluster resource so it stays correct on
any GKE Autopilot cluster, not just the current canary.
…ning

Add a project-level binding so the CC controller SA can create and
delete per-environment IAMServiceAccount CRs, plus a sandbox output
that exposes the CC controller SA email so a sibling component can
attach the matching bucket-scoped binding for managing per-env GCS
IAMPolicyMember resources on the install's snapshot bucket.

Preserves the existing serviceAccount-scoped binding required by the
restate-operator's v2.3.0 Workload Identity automation.
@pcholakov
pcholakov force-pushed the pavel/gcp-env-identity branch from 5fbfbc7 to aacba5a Compare May 25, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant