Rework provider bootstrap: self-register CatalogEntry via init container; per-provider release versioning (#320)

* Rework provider bootstrap

* ci: take the Tilt E2E job off automatic CI runs

The multi-shard Tilt stack over-subscribes the 2-vCPU GitHub-hosted runner
(root-kcp won't schedule, shards/proxies crash-loop on probe timeouts), so
the job reliably fails on insufficient resources rather than on the code
under test. Comment out the pull_request/push triggers; it stays runnable on
demand via workflow_dispatch. Re-enable once we have bigger workers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: mjudeikis

.github/workflows/tilt-e2e.yaml

@@ -14,10 +14,17 @@ name: Tilt E2E
 # internal retries) if the stack isn't fully settled.
 
 on:
-  pull_request:
-    branches: [main]
-  push:
-    branches: [main]
+  # DISABLED on CI: the multi-shard Tilt stack over-subscribes the 2-vCPU
+  # GitHub-hosted runner — root-kcp won't schedule and the shards/proxies
+  # crash-loop on probe timeouts, so this job fails on insufficient resources
+  # rather than on the code under test. Re-enable the pull_request/push
+  # triggers once we have bigger (self-hosted / larger-runner) workers with
+  # enough CPU+memory to stand up the full operator topology. Until then it
+  # stays runnable on demand via the Actions "Run workflow" button below.
+  # pull_request:
+  #   branches: [main]
+  # push:
+  #   branches: [main]
   workflow_dispatch:
 
 permissions:

.gitignore

@@ -57,6 +57,8 @@ providers/kuery/portal/dist/
 # Data dirs
 data/
 *.kubeconfig
+# Downloaded kubeconfigs that don't end in .kubeconfig (e.g. code-kubeconfig (2).yaml)
+*kubeconfig*.yaml
 
 # Dev tools and certs
 hack/tools/

Makefile

@@ -1191,9 +1191,9 @@ helm-undeploy-provider-infrastructure: ## (experimental) helm uninstall the infr
 KRO_KIND_NAME ?= kedge-kro
 KRO_KIND_KUBECONFIG ?= $(CURDIR)/.kedge-kro.kubeconfig
 KRO_CHART ?= oci://ghcr.io/faroshq/kro-multicluster/charts/kro/kro
-KRO_CHART_VERSION ?= v0.0.1-mc.6
+KRO_CHART_VERSION ?= v0.0.1-mc.7
 KRO_IMAGE_REPO ?= ghcr.io/faroshq/kro-multicluster/kro
-KRO_IMAGE_TAG ?= v0.0.1-mc.6
+KRO_IMAGE_TAG ?= v0.0.1-mc.7
 KRO_NAMESPACE ?= kro-system
 KRO_SEED_DIR ?= providers/infrastructure/examples/rgds
 

Tiltfile.cluster

@@ -53,7 +53,7 @@ cluster_name = 'kcp-tilt'
 # silently shadow any later edit to the default here. The putenv is one-way:
 # parent (this file) → child (the included Tiltfile.static, which reads the env).
 kcp_image_repo = cfg.get('kcp-image-repo', '')
-kcp_image_tag = cfg.get('kcp-image-tag', 'fb8629abb')
+kcp_image_tag = cfg.get('kcp-image-tag', 'v0.32.0')
 # Always export (even empty) so an explicit '' override clears any inherited env.
 os.putenv('KCP_IMAGE_REPO', kcp_image_repo)
 os.putenv('KCP_IMAGE_TAG', kcp_image_tag)

providers/app-studio/deploy/chart/templates/catalogentry.yaml

@@ -1,36 +1,55 @@
 {{- if .Values.catalogEntry.enabled -}}
-apiVersion: providers.kedge.faros.sh/v1alpha1
-kind: CatalogEntry
+# CatalogEntry registers this provider with the kedge hub for routing + the
+# portal Enable flow. It is a kcp resource (providers.kedge.faros.sh/v1alpha1),
+# so it MUST NOT be applied to the hosting cluster where this chart installs.
+# Instead the chart renders it into a ConfigMap that the init container mounts
+# and applies into the provider workspace via the provider kubeconfig
+# (KEDGE_CATALOGENTRY_FILE -> sdkinstall.Bootstrap). The hub watches
+# CatalogEntries in every workspace bound to providers.kedge.faros.sh, including
+# the provider's own, so a self-registered entry is seen.
+#
+# It only REFERENCES the APIExport (apiExport.name); the APIExport, its
+# APIResourceSchemas, the APIExportEndpointSlice, and the bind grant are created
+# by the same init container. The hub no longer provisions anything from this
+# CatalogEntry.
+apiVersion: v1
+kind: ConfigMap
 metadata:
-  name: app-studio
-  annotations:
-    providers.kedge.faros.sh/builtin: "false"
+  name: {{ include "appstudio.fullname" . }}-catalogentry
   labels:
     {{- include "appstudio.labels" . | nindent 4 }}
-spec:
-  displayName: "App Studio"
-  description: "Persistent AI project workspace for planning, memory, and chat."
-  vendor: "kedge"
-  version: {{ .Chart.AppVersion | quote }}
-  category: "AI"
-  iconURL: "/ui/providers/app-studio/icon.svg"
-  dependencies:
-    - name: code
-  ui:
-    url: {{ default (printf "http://%s.%s.svc.cluster.local:%v" (include "appstudio.fullname" .) .Release.Namespace .Values.service.port) .Values.catalogEntry.uiURL | quote }}
-    indexPath: "/"
-  backend:
-    # The hub backend proxy forwards /services/providers/app-studio/* here,
-    # injecting the verified X-Kedge-Tenant/X-Kedge-User headers and the
-    # caller's bearer token. The provider serves /api/projects/* + /healthz.
-    url: {{ default (printf "http://%s.%s.svc.cluster.local:%v" (include "appstudio.fullname" .) .Release.Namespace .Values.service.port) .Values.catalogEntry.backendURL | quote }}
-    healthPath: "/healthz"
-  # Reference-only: the APIExport, schemas, slice, and bind grant are created by
-  # this chart's init container (provider `init`, via the kedge-provider-sdk).
-  apiExport:
-    name: "ai.kedge.faros.sh"
-    permissionClaims:
-      - resource: secrets
-        verbs: ["get", "list", "watch", "create", "update", "delete"]
-        tenantScoped: true
+data:
+  catalogentry.yaml: |
+    apiVersion: providers.kedge.faros.sh/v1alpha1
+    kind: CatalogEntry
+    metadata:
+      name: app-studio
+      annotations:
+        providers.kedge.faros.sh/builtin: "false"
+    spec:
+      displayName: "App Studio"
+      description: "Persistent AI project workspace for planning, memory, and chat."
+      vendor: "kedge"
+      version: {{ .Chart.AppVersion | quote }}
+      category: "AI"
+      iconURL: "/ui/providers/app-studio/icon.svg"
+      dependencies:
+        - name: code
+      ui:
+        url: {{ default (printf "http://%s.%s.svc.cluster.local:%v" (include "appstudio.fullname" .) .Release.Namespace .Values.service.port) .Values.catalogEntry.uiURL | quote }}
+        indexPath: "/"
+      backend:
+        # The hub backend proxy forwards /services/providers/app-studio/* here,
+        # injecting the verified X-Kedge-Tenant/X-Kedge-User headers and the
+        # caller's bearer token. The provider serves /api/projects/* + /healthz.
+        url: {{ default (printf "http://%s.%s.svc.cluster.local:%v" (include "appstudio.fullname" .) .Release.Namespace .Values.service.port) .Values.catalogEntry.backendURL | quote }}
+        healthPath: "/healthz"
+      # Reference-only: the APIExport, schemas, slice, and bind grant are created by
+      # this chart's init container (provider `init`, via the kedge-provider-sdk).
+      apiExport:
+        name: "ai.kedge.faros.sh"
+        permissionClaims:
+          - resource: secrets
+            verbs: ["get", "list", "watch", "create", "update", "delete"]
+            tenantScoped: true
 {{- end }}

providers/app-studio/deploy/chart/templates/deployment.yaml

@@ -39,10 +39,23 @@ spec:
               value: /var/run/secrets/kedge/kedge-provider-kubeconfig
             - name: KEDGE_SCHEMAS_DIR
               value: /etc/kedge/schemas
+            {{- if .Values.catalogEntry.enabled }}
+            # The CatalogEntry is a kcp resource, so the chart cannot apply it to
+            # the hosting cluster. It is rendered into a ConfigMap (mounted below)
+            # and init self-registers it into the provider workspace via the
+            # provider kubeconfig.
+            - name: KEDGE_CATALOGENTRY_FILE
+              value: /etc/kedge/catalogentry/catalogentry.yaml
+            {{- end }}
           volumeMounts:
             - name: kedge-provider-kubeconfig
               mountPath: /var/run/secrets/kedge
               readOnly: true
+            {{- if .Values.catalogEntry.enabled }}
+            - name: catalogentry
+              mountPath: /etc/kedge/catalogentry
+              readOnly: true
+            {{- end }}
       containers:
         - name: provider
           image: "{{ .Values.image.repository }}:{{ default .Chart.AppVersion .Values.image.tag }}"
@@ -129,6 +142,11 @@ spec:
             items:
               - key: kubeconfig
                 path: kedge-provider-kubeconfig
+        {{- if .Values.catalogEntry.enabled }}
+        - name: catalogentry
+          configMap:
+            name: {{ include "appstudio.fullname" . }}-catalogentry
+        {{- end }}
         - name: project-workspaces
           {{- if .Values.workspace.existingClaim }}
           persistentVolumeClaim:

providers/app-studio/deploy/chart/values.yaml

@@ -16,6 +16,12 @@ service:
   type: ClusterIP
   port: 8081
 
+# When true, the chart renders the CatalogEntry (which registers the provider
+# with the hub) into a ConfigMap that the init container applies into the
+# provider workspace via the provider kubeconfig. The CatalogEntry is a kcp
+# resource, so it is NOT applied to the hosting cluster this chart installs into.
+# Set to false to manage the CatalogEntry separately (e.g. GitOps with a
+# different rollout cadence).
 catalogEntry:
   enabled: true
   uiURL: ""

providers/code/README.md

@@ -106,9 +106,11 @@ Connection — deleting the Connection garbage-collects the Secret.
 
 ## Register with the hub
 
-The CatalogEntry is what makes the hub provision the provider's workspace,
-schemas, APIExport, and runtime kubeconfig. The Helm chart renders it
-(`catalogEntry.enabled=true`), or apply the raw manifest:
+The CatalogEntry registers the provider with the hub for routing + the portal
+Enable flow. It is a kcp resource, so it lives in the provider workspace — not
+the hosting cluster. With `catalogEntry.enabled=true` (default) the chart renders
+it into a ConfigMap and the init container self-registers it into the workspace
+via the provider kubeconfig; alternatively apply the raw manifest yourself:
 
 ```sh
 kubectl --kubeconfig kcp-admin.kubeconfig ws use root:kedge:providers
@@ -130,7 +132,8 @@ docker build -t ghcr.io/faroshq/kedge-code-provider:dev providers/code/
 ## Deploy with Helm
 
 The chart ships the provider Deployment, a ClusterIP Service, the ServiceAccount,
-and (optionally) the CatalogEntry. The runtime kubeconfig the controllers need
+and (optionally) the CatalogEntry ConfigMap the init container applies to kcp.
+The runtime kubeconfig the controllers need
 is **minted by the hub** when it reconciles the CatalogEntry and mounted from the
 `kedge-provider-kubeconfig` Secret — the volume is `optional`, so the pod serves
 portal/MCP/packages reads immediately and the controller manager engages once
@@ -148,8 +151,10 @@ helm install code providers/code/deploy/chart \
 
 ### With "Connect with GitHub" (OAuth)
 
-Create a GitHub OAuth App with callback `https://<provider-host>/oauth/github/callback`,
-store its client secret in a Secret, then:
+Create a GitHub OAuth App, store its client secret in a Secret, then enable the
+`githubOAuth.*` block. The portal probes `/services/providers/code/oauth/github/config`
+through the hub; once OAuth is enabled and the provider backend is reachable, the
+**Connect with GitHub** button appears.
 
 ```sh
 kubectl -n code create secret generic kedge-code-github-oauth \
@@ -162,14 +167,88 @@ helm install code providers/code/deploy/chart \
   --set githubOAuth.enabled=true \
   --set githubOAuth.clientId=<oauth-app-client-id> \
   --set githubOAuth.clientSecretRef.name=kedge-code-github-oauth \
-  --set githubOAuth.redirectURL=https://code.example.com/oauth/github/callback \
-  --set githubOAuth.portalOrigin=https://kedge.example.com
+  --set githubOAuth.redirectURL=https://<hub-host>/services/providers/code/oauth/github/callback \
+  --set githubOAuth.portalOrigin=https://<hub-host>
 ```
 
-The OAuth callback is a top-level redirect from GitHub (no kedge auth), so
-`redirectURL` must point at the provider's **own externally-reachable URL**, not
-the hub `/services` proxy. `portalOrigin` should be the hub origin so the popup
-returns the token only to your portal.
+#### Choosing `redirectURL`
+
+GitHub's callback is a **top-level browser redirect with no kedge auth**, so
+`redirectURL` must be publicly reachable and forward to the provider's HTTP
+backend (`:8083`). It must end in `/callback`; the matching `/start` URL is
+derived automatically by swapping `/callback` → `/start` under the **same host
+and path prefix**. Two options:
+
+1. **Reuse the hub ingress (recommended — no extra ingress object):** point at
+   the hub's existing `/services/providers/code/*` proxy:
+   ```
+   https://<hub-host>/services/providers/code/oauth/github/callback
+   ```
+   The proxy forwards these anonymous requests straight to the provider backend,
+   so the whole flow rides the single hub hostname. Set `portalOrigin` to the
+   same hub origin.
+
+2. **The provider's own external host:** if you expose the provider directly
+   (its own ingress/hostname), use:
+   ```
+   https://code.example.com/oauth/github/callback
+   ```
+
+Whichever you pick, register that **exact** callback URL on the GitHub OAuth App,
+and set `portalOrigin` to the hub origin so the popup returns the token only to
+your portal.
+
+### Full production deployment (hub-routed OAuth)
+
+Provider running in its own namespace, registered against an already-running hub,
+with OAuth routed through the hub ingress (no per-provider ingress). The runtime
+kubeconfig the controllers need is supplied as the `kedge-provider-kubeconfig`
+Secret (its key **must** be `kubeconfig`) — mint it via the admin onboarding flow
+(`/bonkers`).
+
+```sh
+# 1. Namespace.
+kubectl create namespace kedge-prod-provider-code
+
+# 2. Provider kubeconfig Secret (key MUST be "kubeconfig").
+kubectl -n kedge-prod-provider-code create secret generic kedge-provider-kubeconfig \
+  --from-file=kubeconfig=kedge/provider-code.kubeconfig
+
+# 3. GitHub OAuth App client secret.
+kubectl -n kedge-prod-provider-code create secret generic code-github-oauth \
+  --from-literal=clientSecret=<oauth-app-client-secret>
+
+# 4. Install the chart from the published OCI registry.
+helm upgrade --install code oci://ghcr.io/faroshq/charts/kedge-code-provider:0.0.82 \
+  -n kedge-prod-provider-code \
+  --set hub.url=https://kedge-kedge-hub.kedge-prod.svc.cluster.local:9443 \
+  --set hub.insecure=true \
+  --set hub.tokenSecretRef.name="" \
+  --set image.tag=v0.0.82 \
+  --set catalogEntry.enabled=false \
+  --set githubOAuth.enabled=true \
+  --set githubOAuth.clientId=<oauth-app-client-id> \
+  --set githubOAuth.clientSecretRef.name=code-github-oauth \
+  --set githubOAuth.clientSecretRef.key=clientSecret \
+  --set githubOAuth.redirectURL=https://console.faros.sh/services/providers/code/oauth/github/callback \
+  --set githubOAuth.portalOrigin=https://console.faros.sh
+```
+
+Notes:
+- `hub.insecure=true` + `hub.tokenSecretRef.name=""` suit an in-cluster hub with
+  a self-signed cert and no static heartbeat token. For a real heartbeat token,
+  create a Secret and set `hub.tokenSecretRef.name`/`.key` instead.
+- `catalogEntry.enabled=false` means the chart does **not** manage the
+  CatalogEntry — the hub uses whatever `backend.url` the existing CatalogEntry
+  declares. **Make sure that `backend.url` points at this deployment's Service**
+  (`http://code-kedge-code-provider.<namespace>.svc.cluster.local:8083`); a stale
+  namespace there makes the hub→provider proxy return **502** (and the OAuth
+  button stays hidden). Leaving `catalogEntry.enabled=true` lets the init
+  container keep `backend.url` in sync with the release namespace automatically.
+- After install, verify the OAuth probe returns `{"enabled":true}`:
+  ```sh
+  curl -s https://console.faros.sh/services/providers/code/oauth/github/config
+  ```
 
 `values.yaml` documents the full surface — image, replicas, hub URL + token
 Secret, the runtime kubeconfig Secret name, the `githubOAuth.*` block, the
@@ -225,7 +304,7 @@ the connection token's `read:packages` scope.
 | `KEDGE_DEV_ALLOW_TENANT_QUERY` | (unset) | `true` lets `?tenant=`/`?token=` replace identity headers (dev only) |
 | `GITHUB_OAUTH_CLIENT_ID` | (unset → OAuth off) | GitHub OAuth App client ID |
 | `GITHUB_OAUTH_CLIENT_SECRET` | (unset) | GitHub OAuth App client secret |
-| `GITHUB_OAUTH_REDIRECT_URL` | (unset) | Absolute callback URL on the provider's own host |
+| `GITHUB_OAUTH_REDIRECT_URL` | (unset) | Absolute callback URL (must end in `/callback`); either the hub `/services/providers/code/oauth/github/callback` proxy route or the provider's own host. `/start` is derived from it |
 | `GITHUB_OAUTH_PORTAL_ORIGIN` | `*` | postMessage target origin (set to the hub origin in prod) |
 | `GITHUB_OAUTH_SCOPES` | `repo,read:org,admin:public_key,read:packages` | Requested OAuth scopes |