Skip to content

Commit ecc9d80

Browse files
mjudeikisclaude
andcommitted
feat(infrastructure): migrate application exposure from Ingress to Gateway API
The `application` template's exposure layer now emits gateway.networking.k8s.io/v1 HTTPRoutes instead of networking.k8s.io/v1 Ingress objects. - Template: two HTTPRoutes (one always, one /api-only in mode=none) attach to a platform Gateway via parentRefs; status reads from httpRoute.hostnames. - Token seam: ${kedge.ingressClass} -> ${kedge.gatewayName} + ${kedge.gatewayNamespace}; env KEDGE_INGRESS_CLASS -> KEDGE_GATEWAY_NAME / KEDGE_GATEWAY_NAMESPACE. Defaults point at the cfgate Cloudflare Tunnel Gateway (cloudflare-tunnel / cfgate-system). - ApplicationSpec.IngressClass -> ApplicationSpec.Gateway (GatewayRef); regenerated deepcopy + CRDs; chart values/deployment/operator-config; Makefile dev target; tests; provider doc + README. sandbox-runner is unchanged (no Ingress; preview goes via the k8s service proxy). Foundation for issue #351 (Core Kedge Gateway Endpoint broker). Bundles unrelated in-flight working-tree changes: kedge-hub HTTPRoute chart template + values, hub bootstrap tweak, provider portal bundles, and local .claude settings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 49a11a8 commit ecc9d80

25 files changed

Lines changed: 609 additions & 405 deletions

File tree

.claude/settings.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"permissions": {
3+
"additionalDirectories": [
4+
"/Users/mjudeikis/go/src/github.com/kcp-dev/kcp/pkg/reconciler/apis/apibinding"
5+
]
6+
}
7+
}

Makefile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -899,7 +899,8 @@ run-provider-infrastructure: build-infrastructure-provider ## Run the infrastruc
899899
KRO_KUBECONFIG=$${KRO_KUBECONFIG:-$$( [ -f "$(KRO_KIND_KUBECONFIG)" ] && echo "$(KRO_KIND_KUBECONFIG)" )} \
900900
INFRASTRUCTURE_KUBECONFIG=$${INFRASTRUCTURE_KUBECONFIG:-$$( [ -f "$(INFRASTRUCTURE_RUNTIME_KUBECONFIG)" ] && echo "$(INFRASTRUCTURE_RUNTIME_KUBECONFIG)" )} \
901901
KEDGE_APP_BASE_DOMAIN=$${KEDGE_APP_BASE_DOMAIN:-apps.127.0.0.1.sslip.io} \
902-
KEDGE_INGRESS_CLASS=$${KEDGE_INGRESS_CLASS:-nginx} \
902+
KEDGE_GATEWAY_NAME=$${KEDGE_GATEWAY_NAME:-cloudflare-tunnel} \
903+
KEDGE_GATEWAY_NAMESPACE=$${KEDGE_GATEWAY_NAMESPACE:-cfgate-system} \
903904
$(BINDIR)/infrastructure-provider
904905

905906
run-provider-infrastructure-operator: build-infrastructure-provider ## Run the infrastructure provider in OPERATOR mode (bootstrap reconcile + serve from a provider + runtime kubeconfig)
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{{- if .Values.httpRoute.enabled }}
2+
apiVersion: gateway.networking.k8s.io/v1
3+
kind: HTTPRoute
4+
metadata:
5+
name: {{ include "kedge-hub.fullname" . }}
6+
labels:
7+
{{- include "kedge-hub.labels" . | nindent 4 }}
8+
{{- with .Values.httpRoute.annotations }}
9+
annotations:
10+
{{- toYaml . | nindent 4 }}
11+
{{- end }}
12+
spec:
13+
parentRefs:
14+
{{- toYaml .Values.httpRoute.parentRefs | nindent 4 }}
15+
{{- with .Values.httpRoute.hostnames }}
16+
hostnames:
17+
{{- toYaml . | nindent 4 }}
18+
{{- end }}
19+
rules:
20+
- backendRefs:
21+
- name: {{ include "kedge-hub.fullname" . }}
22+
port: {{ .Values.service.hub.port }}
23+
{{- end }}

deploy/charts/kedge-hub/values.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,21 @@ ingress:
132132
# - path: /
133133
# pathType: ImplementationSpecific
134134

135+
# -- HTTPRoute (Gateway API; e.g. for the cfgate Cloudflare Tunnel controller).
136+
# The hub serves HTTPS with a self-signed cert on service.hub.port, so when
137+
# attaching to cfgate set the origin annotations below so cloudflared opens a
138+
# TLS connection to the origin without verifying the cert.
139+
httpRoute:
140+
enabled: false
141+
parentRefs:
142+
- name: cloudflare-tunnel
143+
namespace: cfgate-system
144+
annotations: {}
145+
# cfgate.io/origin-protocol: "https"
146+
# cfgate.io/origin-ssl-verify: "false"
147+
hostnames: []
148+
# - kedge.example.com
149+
135150
nodeSelector: {}
136151
tolerations: []
137152
affinity: {}

docs/application-template-architecture.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,14 @@
11
# Application template: 3-tier app with an OIDC-guarded URL
22

3-
Status: **Design proposal, not implemented.**
3+
Status: **Design proposal.** The exposure layer has since been implemented on
4+
**Gateway API** rather than the plain `Ingress` this proposal describes: the
5+
`application` template emits `gateway.networking.k8s.io/v1 HTTPRoute`s that
6+
attach to a platform Gateway (default the cfgate `cloudflare-tunnel` Gateway in
7+
`cfgate-system`), configured via `${kedge.gatewayName}` / `${kedge.gatewayNamespace}`
8+
(`KEDGE_GATEWAY_NAME` / `KEDGE_GATEWAY_NAMESPACE`). Where this document says
9+
`Ingress` / `ingressClassName` / `${kedge.ingressClass}` below, read HTTPRoute /
10+
`parentRefs` / `${kedge.gatewayName}`. The current operational reference is
11+
[providers/infrastructure/docs/application-template-architecture.md](../providers/infrastructure/docs/application-template-architecture.md).
412
Author: 2026-06-12
513
Related: `docs/infrastructure-architecture.md` (the platform/backend layer this builds on),
614
`providers/infrastructure/install/templates/redis-cache.yaml` (the in-graph secret-generation

pkg/hub/kcp/bootstrap.go

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,20 @@ type Bootstrapper struct {
7979

8080
// NewBootstrapper creates a new bootstrapper.
8181
func NewBootstrapper(config *rest.Config) *Bootstrapper {
82-
return &Bootstrapper{config: config}
82+
// The hub admin client fans out across many kcp workspaces (every org, every
83+
// child workspace, every provider export) and polls during provider Enable.
84+
// client-go's default 5 QPS / 10 burst throttles that fan-out and surfaces as
85+
// "client rate limiter Wait ... would exceed context deadline" mid-Enable
86+
// (e.g. while waiting for a provider's APIExport in exportClaimIdentities).
87+
// Give it generous headroom — matching the kuery controller's 50/100 — and
88+
// force RateLimiter to nil so each per-path client (configForPath copies this
89+
// config) builds its own limiter rather than sharing a single contended
90+
// bucket inherited from e.g. a loopback config.
91+
cfg := rest.CopyConfig(config)
92+
cfg.QPS = 50
93+
cfg.Burst = 100
94+
cfg.RateLimiter = nil
95+
return &Bootstrapper{config: cfg}
8396
}
8497

8598
// WithEnabledProviders sets the subset of builtin providers the

providers/infrastructure/README.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,9 +89,11 @@ Values:
8989
image/appVersion).
9090
- `operator.application.*` — the `application` template's exposure layer:
9191
`baseDomain` (the zone apps are served under; **required to enable app
92-
exposure**) and `ingressClass` (defaults to `cloudflare`). These render into
93-
the CR's `spec.application` and become the serve container's
94-
`KEDGE_APP_BASE_DOMAIN` / `KEDGE_INGRESS_CLASS`. See
92+
exposure**) and `gateway.name` / `gateway.namespace` (the Gateway API parent
93+
the generated HTTPRoutes attach to; default `cloudflare-tunnel` /
94+
`cfgate-system`). These render into the CR's `spec.application` and become the
95+
serve container's `KEDGE_APP_BASE_DOMAIN` / `KEDGE_GATEWAY_NAME` /
96+
`KEDGE_GATEWAY_NAMESPACE`. See
9597
[docs/application-template-architecture.md](docs/application-template-architecture.md).
9698

9799
### Verify

providers/infrastructure/apis/v1alpha1/types_infrastructureprovider.go

Lines changed: 25 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -85,16 +85,17 @@ type InfrastructureProviderSpec struct {
8585
Provider ProviderServeSpec `json:"provider"`
8686

8787
// Application configures the `application` template's exposure layer — the
88-
// public URL apps are served on and which ingress controller fulfils it.
89-
// Maps onto the serve container's KEDGE_APP_BASE_DOMAIN / KEDGE_INGRESS_CLASS
90-
// env vars. Optional; without it app exposure stays off (see ApplicationSpec).
88+
// public URL apps are served on and the Gateway API parent that fulfils it.
89+
// Maps onto the serve container's KEDGE_APP_BASE_DOMAIN / KEDGE_GATEWAY_NAME
90+
// / KEDGE_GATEWAY_NAMESPACE env vars. Optional; without it app exposure stays
91+
// off (see ApplicationSpec).
9192
// +optional
9293
Application ApplicationSpec `json:"application,omitempty"`
9394
}
9495

95-
// ApplicationSpec configures the `application` template exposure layer. The two
96-
// fields map 1:1 to the serve container's KEDGE_APP_BASE_DOMAIN and
97-
// KEDGE_INGRESS_CLASS environment variables. See
96+
// ApplicationSpec configures the `application` template exposure layer. The
97+
// fields map 1:1 to the serve container's KEDGE_APP_BASE_DOMAIN,
98+
// KEDGE_GATEWAY_NAME and KEDGE_GATEWAY_NAMESPACE environment variables. See
9899
// docs/application-template-architecture.md.
99100
type ApplicationSpec struct {
100101
// BaseDomain is the DNS zone apps are served under, e.g. "apps.example.com".
@@ -105,12 +106,25 @@ type ApplicationSpec struct {
105106
// +optional
106107
BaseDomain string `json:"baseDomain,omitempty"`
107108

108-
// IngressClass is the ingress controller that fulfils the generated Ingress
109-
// (its spec.ingressClassName). It is substituted for the
110-
// ${kedge.ingressClass} token in a Template's backendConfig before the kro
111-
// RGD is authored. Empty → "cloudflare" (the in-binary default).
109+
// Gateway identifies the Gateway API parent the generated HTTPRoutes attach
110+
// to. Its name/namespace are substituted for the ${kedge.gatewayName} /
111+
// ${kedge.gatewayNamespace} tokens in a Template's backendConfig before the
112+
// kro RGD is authored. Empty fields fall back to the in-binary default
113+
// ("cloudflare-tunnel" in "cfgate-system").
112114
// +optional
113-
IngressClass string `json:"ingressClass,omitempty"`
115+
Gateway GatewayRef `json:"gateway,omitempty"`
116+
}
117+
118+
// GatewayRef points at the platform Gateway (gateway.networking.k8s.io) the
119+
// `application` template's HTTPRoutes attach to via parentRefs.
120+
type GatewayRef struct {
121+
// Name of the Gateway. Empty → "cloudflare-tunnel" (the in-binary default).
122+
// +optional
123+
Name string `json:"name,omitempty"`
124+
125+
// Namespace of the Gateway. Empty → "cfgate-system" (the in-binary default).
126+
// +optional
127+
Namespace string `json:"namespace,omitempty"`
114128
}
115129

116130
// SecretKeyRef points at one key in a Secret in the operator's namespace.

providers/infrastructure/apis/v1alpha1/zz_generated.deepcopy.go

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

providers/infrastructure/backend/kro/backend.go

Lines changed: 30 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -53,32 +53,44 @@ type Backend struct {
5353
// pointed at by KRO_KUBECONFIG.
5454
runtime dynamic.Interface
5555

56-
// ingressClass is the value substituted for the reserved
57-
// ${kedge.ingressClass} token in a Template's backendConfig before the
58-
// RGD is authored. It's a platform-wide setting (the exposure-layer
59-
// controller, e.g. "cloudflare"), so it belongs on the backend, not in
60-
// per-tenant data. See substituteTokens in rgd.go.
61-
ingressClass string
56+
// gatewayName / gatewayNamespace are the values substituted for the
57+
// reserved ${kedge.gatewayName} / ${kedge.gatewayNamespace} tokens in a
58+
// Template's backendConfig before the RGD is authored. They identify the
59+
// platform-wide Gateway API parent the generated HTTPRoutes attach to
60+
// (the exposure-layer Gateway, e.g. cfgate's "cloudflare-tunnel"), so they
61+
// belong on the backend, not in per-tenant data. See substituteTokens in
62+
// rgd.go.
63+
gatewayName string
64+
gatewayNamespace string
6265
}
6366

6467
var _ backend.Backend = (*Backend)(nil)
6568

66-
// DefaultIngressClass is used when KEDGE_INGRESS_CLASS is unset. Cloudflare
67-
// Tunnel is the exposure layer we ship with (reverse tunnels, edge TLS).
68-
const DefaultIngressClass = "cloudflare"
69+
// DefaultGatewayName / DefaultGatewayNamespace are used when
70+
// KEDGE_GATEWAY_NAME / KEDGE_GATEWAY_NAMESPACE are unset. They point at the
71+
// cfgate Cloudflare Tunnel Gateway we ship with (the Gateway API exposure
72+
// layer: reverse tunnels, edge TLS).
73+
const (
74+
DefaultGatewayName = "cloudflare-tunnel"
75+
DefaultGatewayNamespace = "cfgate-system"
76+
)
6977

7078
// New constructs the kro backend against the runtime cluster's dynamic
7179
// client. The caller (controller_manager) builds it from KRO_KUBECONFIG.
72-
// The exposure-layer ingress class is read from KEDGE_INGRESS_CLASS
73-
// (defaulting to "cloudflare") and substituted into backendConfig at RGD
74-
// build time, so swapping ingress controllers is a config change, not a
75-
// template edit.
80+
// The exposure-layer Gateway parent is read from KEDGE_GATEWAY_NAME /
81+
// KEDGE_GATEWAY_NAMESPACE (defaulting to "cloudflare-tunnel" in
82+
// "cfgate-system") and substituted into backendConfig at RGD build time, so
83+
// pointing apps at a different Gateway is a config change, not a template edit.
7684
func New(runtime dynamic.Interface) *Backend {
77-
ingressClass := os.Getenv("KEDGE_INGRESS_CLASS")
78-
if ingressClass == "" {
79-
ingressClass = DefaultIngressClass
85+
gatewayName := os.Getenv("KEDGE_GATEWAY_NAME")
86+
if gatewayName == "" {
87+
gatewayName = DefaultGatewayName
88+
}
89+
gatewayNamespace := os.Getenv("KEDGE_GATEWAY_NAMESPACE")
90+
if gatewayNamespace == "" {
91+
gatewayNamespace = DefaultGatewayNamespace
8092
}
81-
return &Backend{runtime: runtime, ingressClass: ingressClass}
93+
return &Backend{runtime: runtime, gatewayName: gatewayName, gatewayNamespace: gatewayNamespace}
8294
}
8395

8496
// Name returns "kro".
@@ -89,7 +101,7 @@ func (b *Backend) Name() string { return Name }
89101
// error (malformed schema/backendConfig) is returned so the Template
90102
// controller surfaces BackendError; a successful apply reports Ready=true.
91103
func (b *Backend) SetupTemplate(ctx context.Context, tmpl *infrav1alpha1.Template) (backend.TemplateStatus, error) {
92-
rgd, err := buildRGD(tmpl, b.ingressClass)
104+
rgd, err := buildRGD(tmpl, b.gatewayName, b.gatewayNamespace)
93105
if err != nil {
94106
return backend.TemplateStatus{Ready: false, Message: err.Error()}, err
95107
}

0 commit comments

Comments
 (0)