test(e2e): infrastructure provider subprocess suite + CI job for provider suites#417
Merged
Conversation
…ider suites The infrastructure provider had e2e coverage only for its kro side (the kind-based template e2e). Its kcp side — provisioning, init bootstrap, the Template controller, the tenant catalog — ran only in unit tests or against a manually-started tilt stack. And the existing quickstart subprocess suite (make e2e-provider) was wired into no CI workflow at all, so neither suite ever ran automatically. Add suites/infraprovider, modeled on suites/provider: kedge-hub with embedded kcp + the infrastructure provider (init, then serve) as host subprocesses. No kind/Helm/Docker/kro — with KRO_KUBECONFIG unset only the stub backend registers, which is exactly what lets the suite drive the Template controller end-to-end without a runtime cluster: - TestABootstrapSeedsCatalog — init seeded every current template; the platform APIExport exists. - TestBStubTemplateFullReconcile — a backend:stub Template reaches Ready=True with its per-template CRD established and the APIExport resources entry added, and deletion runs the finalize chain back down. - TestCRetiredTemplateIsSwept — a retired platform template applied into the live workspace (the pre-retirement deployment state) is deleted by the controller without operator action (#416's reconciler, now covered against real kcp). - TestDProvidersDTO — the hub lists the provider for a logged-in user. - TestETenantSeesTemplatesCatalog — APIBinding in the static user's workspace; the seeded templates are listable through the binding (the read App Studio's picker and MCP list_templates perform). Wire both subprocess suites into CI as one sequential job (they share the embedded kcp's fixed etcd port 2380) in e2e.yaml, and add the make e2e-infra-provider target with the same port guard the quickstart suite uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds end-to-end coverage for the infrastructure provider’s kcp-side lifecycle by introducing a new embedded-kcp subprocess e2e suite (modeled on the existing quickstart provider suite), wiring it into make, and running provider subprocess suites in CI.
Changes:
- Added
test/e2e/suites/infraproviderwhich bootskedge-hub(embedded kcp) +infrastructure-provider(init, thenserve) as subprocesses and validates bootstrap, controller reconcile/finalize, retired-template enforcement, hub/api/providers, and tenant APIBinding catalog reads. - Added
make e2e-infra-providertarget with port guarding and timeout configuration. - Added a new CI job
e2e-providersto runmake e2e-providerandmake e2e-infra-providersequentially.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
test/e2e/suites/infraprovider/main_test.go |
New suite harness: builds binaries, starts hub + provider subprocesses, provisions workspace, runs provider init, then launches provider serve. |
test/e2e/suites/infraprovider/infraprovider_test.go |
New e2e assertions for seeded templates, stub-template reconcile/finalize chain, retired-template sweep, hub DTO, and tenant APIBinding catalog read. |
Makefile |
Adds e2e-infra-provider target and timeout. |
.github/workflows/e2e.yaml |
Adds e2e-providers job to run provider subprocess suites in CI. |
Comment on lines
+177
to
+185
| provCmd.Env = append(os.Environ(), | ||
| "PORT="+providerPort, | ||
| "KEDGE_HUB_URL="+hubURL, | ||
| "KEDGE_HUB_TOKEN="+staticToken, | ||
| "KEDGE_HUB_INSECURE=true", | ||
| "KEDGE_PROVIDER_NAME=infrastructure", | ||
| "INFRASTRUCTURE_KUBECONFIG="+adminKubeconfig, | ||
| "INFRASTRUCTURE_WORKSPACE_PATH="+workspacePath, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
Good catch — fixed in 131d003: init already mints the workspace-scoped SA kubeconfig at dataDir/infrastructure.kubeconfig, and serve now runs with it (INFRASTRUCTURE_WORKSPACE_PATH retarget dropped too — the minted config is already workspace-scoped), so the suite exercises the RBAC init actually granted, same as the chart's serve container.
Review feedback: running serve with the kcp admin kubeconfig bypasses the init/serve split's RBAC — the suite would mask permission regressions in the controller chain. init already mints the workspace-scoped ServiceAccount kubeconfig; serve now uses it, exactly like the chart's serve container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| } | ||
| keepData := os.Getenv("KEDGE_E2E_KEEP_DATA") == "true" | ||
|
|
||
| hubLog, _ := os.Create(filepath.Join(dataDir, "hub.log")) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — all three log-file creates now handle the error and exit (with subprocess cleanup where they're already running) instead of nil-panicking on .Name().
| // container / `make init-provider-infrastructure`). It also mints the | ||
| // workspace-scoped ServiceAccount kubeconfig serve runs with. | ||
| mintedKubeconfig := filepath.Join(dataDir, "infrastructure.kubeconfig") | ||
| initLog, _ := os.Create(filepath.Join(dataDir, "init.log")) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — all three log-file creates now handle the error and exit (with subprocess cleanup where they're already running) instead of nil-panicking on .Name().
| // KRO_KUBECONFIG). Runs with the SA kubeconfig init minted — NOT the | ||
| // admin one — so the suite exercises the RBAC init actually granted, | ||
| // exactly like the chart's serve container. | ||
| provLog, _ := os.Create(filepath.Join(dataDir, "provider.log")) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — all three log-file creates now handle the error and exit (with subprocess cleanup where they're already running) instead of nil-panicking on .Name().
Comment on lines
+116
to
+118
| overrideURL := "http://localhost:" + providerPort | ||
| _ = unstructured.SetNestedField(obj.Object, overrideURL, "spec", "ui", "url") | ||
| _ = unstructured.SetNestedField(obj.Object, overrideURL, "spec", "backend", "url") |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — both SetNestedField overrides now return a clear error instead of silently running against the committed dev-loop URLs.
Comment on lines
+346
to
+348
| req, _ := http.NewRequest(http.MethodGet, hubURL+"/api/providers", nil) | ||
| req.Header.Set("Authorization", "Bearer "+staticToken) | ||
| resp, err := http.DefaultClient.Do(req) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — the wait loops now use an http.Client with a per-attempt timeout (5s DTO / 10s login) so a stalled request can't wedge the suite past its deadline.
Comment on lines
+466
to
+468
| req, _ := http.NewRequest(http.MethodPost, hubURL+"/auth/token-login", nil) | ||
| req.Header.Set("Authorization", "Bearer "+staticToken) | ||
| resp, err := http.DefaultClient.Do(req) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — the wait loops now use an http.Client with a per-attempt timeout (5s DTO / 10s login) so a stalled request can't wedge the suite past its deadline.
Comment on lines
+240
to
+242
| t.Cleanup(func() { | ||
| _ = cl.Resource(templatesGVR).Delete(context.Background(), name, metav1.DeleteOptions{}) | ||
| }) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — cleanup deletes now run under a 15s bounded context so teardown always terminates.
Comment on lines
+392
to
+394
| t.Cleanup(func() { | ||
| _ = tenant.Resource(apiBindingGVR).Delete(context.Background(), "infrastructure", metav1.DeleteOptions{}) | ||
| }) |
Contributor
Author
There was a problem hiding this comment.
Fixed in eb07099 — cleanup deletes now run under a 15s bounded context so teardown always terminates.
- Handle os.Create errors for hub/init/provider logs (a failed create nil-panicked on .Name() and hid the root cause). - Surface SetNestedField errors when overriding the CatalogEntry URLs. - Per-attempt HTTP timeouts inside waitForCondition loops (an unbounded DefaultClient request could hang the whole suite). - Bounded contexts in t.Cleanup deletes so teardown can't hang the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+30
to
+34
| - name: Run quickstart provider e2e | ||
| run: make e2e-provider | ||
|
|
||
| - name: Run infrastructure provider e2e | ||
| run: make e2e-infra-provider |
First CI run of the provider suites flushed out a startup race the suites' own login helper already documents: the hub reports /readyz before the catalog APIs in the providers workspaces are fully servable, so on a slow runner the first CatalogEntry/Provider Create fails with "the server could not find the requested resource" and cascades through the whole quickstart suite. Retry the applies (90s bound) in both the quickstart and infraprovider suites instead of failing on the first attempt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+111
to
+115
| if !waitForCondition(t, 90*time.Second, func() (bool, string) { | ||
| var err error | ||
| created, err = cl.Resource(gvr).Create(ctxWithTimeout(t, 10*time.Second), obj, metav1.CreateOptions{}) | ||
| if err == nil { | ||
| return true, "" |
Comment on lines
+124
to
+140
| // The hub reports /readyz before the admin/catalog APIs in | ||
| // root:kedge:system:providers are fully servable, so the first | ||
| // Create on a slow runner can 404 ("could not find the requested | ||
| // resource"). Retry until the API is up. | ||
| deadline := time.Now().Add(90 * time.Second) | ||
| for { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) | ||
| _, err = cl.Resource(gvr).Create(ctx, obj, metav1.CreateOptions{}) | ||
| cancel() | ||
| if err == nil || apierrors.IsAlreadyExists(err) { | ||
| break | ||
| } | ||
| if time.Now().After(deadline) { | ||
| return fmt.Errorf("create %s %s: %w", obj.GetKind(), obj.GetName(), err) | ||
| } | ||
| time.Sleep(2 * time.Second) | ||
| } |
Comment on lines
+26
to
+29
| - uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: v1.26.1 | ||
|
|
Its first-ever CI run revealed the quickstart suite predates the provider bootstrap refactor: it applies only the CatalogEntry, into root:kedge:providers, while the current flow applies Provider + CatalogEntry into root:kedge:system:providers and delegates the APIExport/schemas/bind-grant to `quickstart-provider init` — so every test from catalog provisioning to tenant enable fails against today's hub. Modernizing it onto the same bootstrap the infraprovider suite uses is queued as a follow-up; until then the CI job runs the infraprovider suite only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
CI update: the first-ever run of the quickstart suite (make e2e-provider) showed it has bitrotted against the provider bootstrap refactor — it applies only a CatalogEntry into root:kedge:providers, while the current flow applies Provider + CatalogEntry into root:kedge:system:providers and delegates APIExport/schemas/bind-grant to |
Comment on lines
+105
to
+109
| // The hub reports /readyz before the catalog APIs in | ||
| // root:kedge:providers are fully servable (same startup race | ||
| // loginStaticTokenAndGetCluster documents), so on a slow runner the | ||
| // first Create can fail with "the server could not find the requested | ||
| // resource". Retry until the API is up rather than failing the suite. |
Comment on lines
+124
to
+128
| // The hub reports /readyz before the admin/catalog APIs in | ||
| // root:kedge:system:providers are fully servable, so the first | ||
| // Create on a slow runner can 404 ("could not find the requested | ||
| // resource"). Retry until the API is up. | ||
| deadline := time.Now().Add(90 * time.Second) |
Comment on lines
+14
to
+22
| # Subprocess provider suites: hub with embedded kcp + provider binaries as | ||
| # host processes, driven over HTTP + kcp dynamic clients. No kind, no Helm, | ||
| # no Docker — the cheapest full-kcp coverage we have. | ||
| # | ||
| # NOTE: the older quickstart suite (make e2e-provider) predates the | ||
| # provider bootstrap refactor (Provider CR + provider-run init) and fails | ||
| # against the current hub — it is deliberately NOT wired here until it is | ||
| # modernized onto the same bootstrap the infraprovider suite uses. | ||
| e2e-providers: |
The infraprovider e2e suite's first run (serving with the init-minted SA kubeconfig instead of the hub-minted cluster-admin one) exposed a real RBAC gap: the runtime ClusterRole granted no delete verb anywhere, so the retired-template sweep (controller deletes Templates on sight) and the finalize chain's per-template CRD deletion both 403 — the features work only under the cluster-admin hub-minted kubeconfig. Grant delete on templates and customresourcedefinitions; the role is idempotently overwritten on init, so existing deployments pick this up on their next init run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+111
to
+115
| if !waitForCondition(t, 90*time.Second, func() (bool, string) { | ||
| var err error | ||
| created, err = cl.Resource(gvr).Create(ctxWithTimeout(t, 10*time.Second), obj, metav1.CreateOptions{}) | ||
| if err == nil { | ||
| return true, "" |
Comment on lines
+124
to
+140
| // The hub reports /readyz before the admin/catalog APIs in | ||
| // root:kedge:system:providers are fully servable, so the first | ||
| // Create on a slow runner can 404 ("could not find the requested | ||
| // resource"). Retry until the API is up. | ||
| deadline := time.Now().Add(90 * time.Second) | ||
| for { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) | ||
| _, err = cl.Resource(gvr).Create(ctx, obj, metav1.CreateOptions{}) | ||
| cancel() | ||
| if err == nil || apierrors.IsAlreadyExists(err) { | ||
| break | ||
| } | ||
| if time.Now().After(deadline) { | ||
| return fmt.Errorf("create %s %s: %w", obj.GetKind(), obj.GetName(), err) | ||
| } | ||
| time.Sleep(2 * time.Second) | ||
| } |
Comment on lines
+18
to
+21
| # NOTE: the older quickstart suite (make e2e-provider) predates the | ||
| # provider bootstrap refactor (Provider CR + provider-run init) and fails | ||
| # against the current hub — it is deliberately NOT wired here until it is | ||
| # modernized onto the same bootstrap the infraprovider suite uses. |
mjudeikis
added a commit
that referenced
this pull request
Jul 11, 2026
…418) * test(e2e): modernize the quickstart suite onto the current bootstrap The suite predated the provider bootstrap refactor: it applied only a CatalogEntry, into root:kedge:providers, and expected the hub to materialize the APIExport/schemas/bind-grant. The current flow applies Provider + CatalogEntry into root:kedge:system:providers, the hub's Provider controller materializes the sub-workspace + SA + provider-token Secret, and `quickstart-provider init` (run with the minted SA kubeconfig) authors the APIExport, schemas, and bind grant. Move the bootstrap into TestMain, mirroring the infraprovider suite: apply provider.yaml + manifest.yaml (URL-overridden, with the API-servable retry), wait for the provider-token Secret, mint the workspace-scoped runtime kubeconfig around it (exactly what `make init-provider-quickstart` scripts), run init, then serve. TestACatalogProvisioning stops applying manifests and reads the CatalogEntry from root:kedge:system:providers; the sub-workspace artifact assertions are unchanged — the artifacts just come from the Provider controller + init now. Re-enable the suite in the e2e-providers CI job (descoped in #417 when its first-ever CI run exposed the bitrot). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): point quickstart init at the chart schemas dir KEDGE_SCHEMAS_DIR=/nonexistent (copied from the Makefile dev flow) made init author an APIExport with zero resources — no greetings APIResourceSchema, so tenant binds exposed no Greeting resource. Point it at the chart's files/schemas, which is what the init container mounts in a real deployment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
E2E coverage survey findings for the providers:
initbootstrap, the Template controller chain, the tenant catalog read — ran only in unit tests or against a manually-started tilt stack.make e2e-provider) was wired into no CI workflow at all — neither suite ever ran automatically.What
New
suites/infraprovider, modeled onsuites/provider: kedge-hub with embedded kcp + the infrastructure provider (init, then serve) as host subprocesses. No kind/Helm/Docker/kro — withKRO_KUBECONFIGunset only the stub backend registers, which is exactly what lets the suite drive the Template controller end-to-end without a runtime cluster:initseeded every current template; the platform APIExport exists.backend: stubTemplate reaches Ready=True with its per-template CRD established and the APIExportspec.resourcesentry added; deletion runs the finalize chain back down./api/providerslists the provider for a logged-in user.list_templatesperform.CI: new
e2e-providersjob ine2e.yamlrunsmake e2e-providerthenmake e2e-infra-providersequentially (they share the embedded kcp's fixed etcd port 2380). Cheapest full-kcp coverage in the pipeline — no kind, no images.Makefile:
e2e-infra-providertarget with the same port guard the quickstart suite uses.Verification
go veton the suite passes. The suite could not be run on my machine (a live dev kcp holds etcd port 2380 — the exact scenario the port guard exists for), so the new CI job on this PR is the validating run; expect possible first-run iteration.Follow-ups (next PRs)
App-studio and code provider subprocess suites on the same pattern — app-studio co-run with the infrastructure provider to cover the project → template-select → dev-binding vertical; code provider wiring-level (catalog, MCP tools, Connection CR error status without real GitHub credentials).
🤖 Generated with Claude Code