Skip to content

Commit c2f22c3

Browse files
mjudeikis-botmjudeikis-botmjudeikismjudeikis-bot
authored
test: add AgentJoinKubernetes and AgentHelmInstall e2e test cases (#121)
* test: add AgentJoinKubernetes and AgentHelmInstall e2e test cases Add two new e2e test cases for kubernetes-type agent deployment flows: - AgentJoinKubernetes: runs 'kedge agent join --type kubernetes' as a one-shot install command that deploys the agent as a Kubernetes Deployment in kedge-system, then verifies the Deployment becomes available and the hub edge reaches Ready. - AgentHelmInstall: installs the agent via the local kedge-agent Helm chart using 'helm install --wait', then verifies the hub edge reaches Ready. Cleanup uses 'helm uninstall'. Add WaitForDeploymentAvailable to test/e2e/framework/client.go to poll kubectl until availableReplicas >= 1 or timeout. Both tests skip gracefully when no agent kubeconfig is available and clean up best-effort on teardown. Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * fix: use Docker network IP for in-cluster hub URL in e2e tests - Add HubNodePortURL() helper to framework/cluster.go that resolves the hub's Docker network IP and returns https://<ip>:31443, which is reachable from pods inside the agent kind cluster. - Update AgentJoinKubernetes and AgentHelmInstall to call HubNodePortURL() instead of clusterEnv.HubURL (kedge.localhost does not resolve inside pods). Tests are skipped when the Docker IP cannot be determined. - Add --set agent.hub.insecureSkipTLSVerify=true to the helm install in AgentHelmInstall so the agent pod trusts the self-signed hub certificate. - Fix Helm chart deployment.yaml: change --hub-insecure-skip-tls-verify to --insecure-skip-tls-verify, which is the actual flag name accepted by the kedge-agent binary. Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * fix: correct Docker network IP resolution and add diagnostics in e2e tests - Fix HubNodePortURL() to use specific kind network name (kedge-e2e) instead of ranging over all networks — avoids concatenated IPs (e.g. 172.18.0.2172.17.0.2) when the hub node is connected to multiple Docker networks. Format: {{(index .NetworkSettings.Networks "kedge-e2e").IPAddress}} - Add t.Logf("using pod hub URL: %s") in both AgentJoinKubernetes and AgentHelmInstall setup functions so the resolved URL is visible in CI logs. - Add pod-log and pod-describe diagnostics in both edge_becomes_ready assess steps, collected before t.Fatalf so timeout failures include agent logs. - Fix Helm chart deployment.yaml: --hub-insecure-skip-tls-verify -> --insecure-skip-tls-verify (actual flag name accepted by kedge-agent binary). - Add --set agent.hub.insecureSkipTLSVerify=true to AgentHelmInstall helm install so the in-cluster agent pod trusts the self-signed hub certificate. Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * fix: use PodHubURL to preserve cluster name path in agent pod tests The AgentJoinKubernetes and AgentHelmInstall tests were failing with 'websocket: bad handshake' because the agent pod connected to: wss://172.18.0.2:31443/services/agent-proxy/default/... The cluster name segment 'default' came from calling HubNodePortURL() which returns a bare 'https://172.18.0.2:31443' URL with no path. SplitBaseAndCluster falls back to 'default' when no /clusters/<name> path is present. Add PodHubURL(clusterHubURL string) to the test framework. It keeps the /clusters/<name> path from the runner-side URL (clusterEnv.HubURL) while substituting the host:port with the Docker-network NodePort address reachable from inside the agent pods. Replace both framework.HubNodePortURL() call sites in join_token.go with framework.PodHubURL(clusterEnv.HubURL) so SplitBaseAndCluster correctly extracts 'root:kedge:user-default' instead of 'default'. Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: mjudeikis-bot <ai@faros.sh> * feat: make kedge-agent image configurable via env vars for e2e The kedge-agent container image used in AgentJoinKubernetes and AgentHelmInstall e2e tests was hard-coded to ghcr.io/faroshq/kedge-agent:latest which required a network pull in CI and could use a stale image. Changes: - pkg/cli/cmd/install.go: read KEDGE_AGENT_IMAGE, KEDGE_AGENT_IMAGE_TAG, KEDGE_AGENT_IMAGE_PULL_POLICY env vars in installKubernetes; update agentManifestTemplate to use {{.Image}}:{{.ImageTag}} and {{.ImagePullPolicy}} - pkg/cli/cmd/agent.go: same env vars in agentJoinKubernetes; add imagePullPolicy field to both token-based and kubeconfig-based Deployment manifests - test/e2e/framework/cluster.go: add agentImageEnv, agentImageTagEnv, agentImagePullPolicyEnv constants; add LoadAgentImageIntoCluster helper that runs 'kind load docker-image' when KEDGE_AGENT_IMAGE_PULL_POLICY=Never - test/e2e/cases/join_token.go: call LoadAgentImageIntoCluster in AgentJoinKubernetes and AgentHelmInstall setup; pass image overrides to helm install via --set flags - .github/workflows/e2e.yaml: build agent Docker image in e2e-standalone job; set KEDGE_AGENT_IMAGE/TAG/PULL_POLICY=Never env vars on the e2e run step Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: mjudeikis-bot <ai@faros.sh> * test: replace PodHubURL with PodHubURLFromKubeconfig for correct cluster path PodHubURL(clusterEnv.HubURL) was still broken because clusterEnv.HubURL is ALWAYS "https://kedge.localhost:8443" (no /clusters/<name> path). Substituting the host in that bare URL still produces a URL without a cluster path, causing SplitBaseAndCluster to fall back to "default". The cluster path lives in the hub kubeconfig server URL, e.g.: https://kedge.localhost:8443/clusters/root:kedge:user-default Replace PodHubURL(clusterHubURL string) with PodHubURLFromKubeconfig(kubeconfigPath string) which: 1. Calls HubNodePortURL() to get the Docker-network NodePort address. 2. Reads cfg.Host from the hub kubeconfig (contains the full cluster path). 3. Replaces scheme+host with the NodePort address while preserving the /clusters/... path segment. Update AgentJoinKubernetes and AgentHelmInstall to call PodHubURLFromKubeconfig(clusterEnv.HubKubeconfig) so the --hub-url flag passed to in-cluster agents includes the correct cluster path. Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: mjudeikis-bot <ai@faros.sh> * fix: always mark edge Ready on tunnel open regardless of auth method When an in-cluster agent restarts and reconnects using its saved kubeconfig (loaded from a Secret after the join-token exchange), the hub was not marking the edge as Ready/Connected because markEdgeConnected was gated on authenticatedByJoinToken. The agent's edge_reporter (which was supposed to handle this for kubeconfig connections) also failed with 'access denied' due to RBAC propagation timing. Fix: call markEdgeConnected on every tunnel open. The hub is authoritative for edge connectivity state regardless of how the agent authenticated. This is safe because the join token is only cleared when it was set, and SSH credentials are extracted from headers in both paths. Fixes the AgentJoinKubernetes and AgentHelmInstall e2e tests. Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: mjudeikis-bot <ai@faros.sh> --------- Signed-off-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: mjudeikis-bot <ai@faros.sh> Co-authored-by: mjudeikis-bot <mjudeikis-bot@faros.sh> Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Co-authored-by: mjudeikis-bot <ai@faros.sh>
1 parent fb1ca2e commit c2f22c3

9 files changed

Lines changed: 444 additions & 15 deletions

File tree

.github/workflows/e2e.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,17 @@ jobs:
4444
- name: Build hub Docker image
4545
run: docker build -f deploy/Dockerfile.hub -t ghcr.io/faroshq/kedge-hub:test .
4646

47+
- name: Build agent Docker image
48+
run: docker build -f deploy/Dockerfile.agent -t ghcr.io/faroshq/kedge-agent:test .
49+
4750
- name: Run standalone e2e
4851
env:
4952
KEDGE_HUB_IMAGE: ghcr.io/faroshq/kedge-hub
5053
KEDGE_HUB_IMAGE_PULL_POLICY: Never
5154
KEDGE_HUB_IMAGE_TAG: test
55+
KEDGE_AGENT_IMAGE: ghcr.io/faroshq/kedge-agent
56+
KEDGE_AGENT_IMAGE_TAG: test
57+
KEDGE_AGENT_IMAGE_PULL_POLICY: Never
5258
run: make e2e-standalone E2E_TIMEOUT=50m E2E_FLAGS="-ssh-keepalive-duration=30s -keep-clusters"
5359

5460
- name: Collect kind cluster logs

deploy/charts/kedge-agent/templates/deployment.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ spec:
5050
- --hub-url={{ .Values.agent.hub.url }}
5151
{{- end }}
5252
{{- if .Values.agent.hub.insecureSkipTLSVerify }}
53-
- --hub-insecure-skip-tls-verify
53+
- --insecure-skip-tls-verify
5454
{{- end }}
5555
{{- if .Values.agent.tunnelURL }}
5656
- --tunnel-url={{ .Values.agent.tunnelURL }}

pkg/cli/cmd/agent.go

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -303,7 +303,19 @@ metadata:
303303
return fmt.Errorf("creating ServiceAccount: %w", err)
304304
}
305305

306-
image := "ghcr.io/faroshq/kedge-agent:latest"
306+
agentImage := os.Getenv("KEDGE_AGENT_IMAGE")
307+
if agentImage == "" {
308+
agentImage = "ghcr.io/faroshq/kedge-agent"
309+
}
310+
agentImageTag := os.Getenv("KEDGE_AGENT_IMAGE_TAG")
311+
if agentImageTag == "" {
312+
agentImageTag = "latest"
313+
}
314+
agentImagePullPolicy := os.Getenv("KEDGE_AGENT_IMAGE_PULL_POLICY")
315+
if agentImagePullPolicy == "" {
316+
agentImagePullPolicy = "IfNotPresent"
317+
}
318+
image := agentImage + ":" + agentImageTag
307319

308320
var deployManifest string
309321

@@ -348,10 +360,11 @@ spec:
348360
containers:
349361
- name: agent
350362
image: %s
363+
imagePullPolicy: %s
351364
args: [%s]
352365
`,
353366
opts.EdgeName, opts.EdgeName, opts.EdgeName, opts.EdgeName,
354-
opts.EdgeName, image,
367+
opts.EdgeName, image, agentImagePullPolicy,
355368
formatDeployArgs(deployArgs))
356369
} else {
357370
// Kubeconfig-based: mount a Secret containing the hub kubeconfig.
@@ -409,6 +422,7 @@ spec:
409422
containers:
410423
- name: agent
411424
image: %s
425+
imagePullPolicy: %s
412426
args: [%s]
413427
volumeMounts:
414428
- name: hub-kubeconfig
@@ -420,7 +434,7 @@ spec:
420434
secretName: %s
421435
`,
422436
opts.EdgeName, opts.EdgeName, opts.EdgeName, opts.EdgeName,
423-
opts.EdgeName, image,
437+
opts.EdgeName, image, agentImagePullPolicy,
424438
formatDeployArgs(deployArgs),
425439
secretName)
426440
}

pkg/cli/cmd/install.go

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -251,7 +251,8 @@ spec:
251251
serviceAccountName: kedge-agent-{{ .EdgeName }}
252252
containers:
253253
- name: kedge-agent
254-
image: ghcr.io/faroshq/kedge-agent:latest
254+
image: {{.Image}}:{{.ImageTag}}
255+
imagePullPolicy: {{.ImagePullPolicy}}
255256
args:
256257
- --hub-url={{ .HubURL }}
257258
- --edge-name={{ .EdgeName }}
@@ -279,11 +280,27 @@ func installKubernetes(opts *installOptions) error {
279280
return fmt.Errorf("parsing manifest template: %w", err)
280281
}
281282

283+
agentImage := os.Getenv("KEDGE_AGENT_IMAGE")
284+
if agentImage == "" {
285+
agentImage = "ghcr.io/faroshq/kedge-agent"
286+
}
287+
agentImageTag := os.Getenv("KEDGE_AGENT_IMAGE_TAG")
288+
if agentImageTag == "" {
289+
agentImageTag = "latest"
290+
}
291+
agentImagePullPolicy := os.Getenv("KEDGE_AGENT_IMAGE_PULL_POLICY")
292+
if agentImagePullPolicy == "" {
293+
agentImagePullPolicy = "IfNotPresent"
294+
}
295+
282296
var sb strings.Builder
283297
if err := tmpl.Execute(&sb, map[string]string{
284-
"HubURL": opts.hubURL,
285-
"EdgeName": opts.edgeName,
286-
"Token": opts.token,
298+
"HubURL": opts.hubURL,
299+
"EdgeName": opts.edgeName,
300+
"Token": opts.token,
301+
"Image": agentImage,
302+
"ImageTag": agentImageTag,
303+
"ImagePullPolicy": agentImagePullPolicy,
287304
}); err != nil {
288305
return fmt.Errorf("rendering manifest: %w", err)
289306
}

pkg/virtual/builder/agent_proxy_builder_v2.go

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -161,14 +161,17 @@ func (p *virtualWorkspaces) buildEdgeAgentProxyHandler() http.Handler {
161161
p.edgeConnManager.Store(key, dialer)
162162
p.logger.Info("Edge agent tunnel established", "key", key)
163163

164-
// In the join-token flow the agent's edge_reporter cannot call the kcp
165-
// API directly (the join token is not a valid kcp credential), so the
166-
// hub marks the edge Ready here as soon as the tunnel is up.
164+
// The hub is authoritative for edge connectivity state regardless of how
165+
// the agent authenticated. In the join-token flow the agent's
166+
// edge_reporter cannot reach the kcp API directly (the join token is not
167+
// a valid kcp credential). In the kubeconfig flow (e.g. after an
168+
// in-cluster pod restart where the agent loads its saved kubeconfig from
169+
// a Secret) the edge_reporter may fail due to RBAC propagation lag.
170+
// Marking the edge Ready here on every tunnel open is safe and ensures
171+
// the hub view is always up-to-date.
167172
// SSH credentials are passed via headers for server-type edges.
168-
if authenticatedByJoinToken {
169-
sshCreds := extractSSHCredsFromHeaders(r)
170-
go p.markEdgeConnected(context.Background(), cluster, name, sshCreds)
171-
}
173+
sshCreds := extractSSHCredsFromHeaders(r)
174+
go p.markEdgeConnected(context.Background(), cluster, name, sshCreds)
172175

173176
// Block until the tunnel closes, then clean up the entry so stale
174177
// look-ups don't succeed.

0 commit comments

Comments
 (0)