Skip to content

Commit c1490e9

Browse files
mjudeikis-botOpenClaw Botmjudeikis
authored
feat: per-edge bootstrap join token for agent registration (#106)
* feat: per-edge bootstrap join token for agent registration Adds Edge.Status.JoinToken generated by the hub controller on edge creation. The agent can present this token to authenticate its first WebSocket connection without needing a full kubeconfig. - Edge controller generates a random 32-byte base64url token on creation - Agent proxy builder accepts Bearer join token as auth alternative - kedge edge create prints formatted join command after polling for token - kedge edge join-command <name> reprints/regenerates the command - kedge agent join/run accept --token flag (--token already wired through) Closes #105 Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * chore: run make codegen to update CRDs and kcp schemas for JoinToken field * fix: address PR review comments - CLI: update kubernetes join command to use OCI helm chart format (oci://ghcr.io/faroshq/charts/kedge-agent); also show kedge agent run command as alternative since chart exists in deploy/charts/ - Hub: return X-Kedge-Agent-Token response header after successful join token authentication (token-exchange flow) - Tunnel: thread onAgentToken callback through StartProxyTunnel so agents can receive and persist the durable token - Agent: save returned durable token to ~/.kedge/agent-<name>.json after first successful join; reload on restart if --token omitted Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * test: e2e tests for join token feature (#105) - JoinTokenIsSetAfterEdgeCreation: verifies hub controller generates token - AgentConnectsWithJoinToken: agent bootstrap via join token works - InvalidJoinTokenReturns401: wrong token is rejected Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * feat: kedge install command + kubeconfig exchange on join token - Hub returns X-Kedge-Agent-Kubeconfig on successful join token auth - Agent saves kubeconfig to ~/.kedge/agent-<name>.kubeconfig; reuses on restart - New 'kedge install --type server' installs agent as systemd service - New 'kedge install --type kubernetes' generates/applies agent manifest - kedge edge create output updated to show kedge install command Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * fix: join token agent registration and cluster discovery In join-token mode the Edge resource is pre-provisioned by the admin; the join token is a hub-side bootstrap credential only (not a kcp ServiceAccount token), so the kcp API rejects any Get/Create call with Unauthorized. Two bugs fixed: 1. Skip registerEdge when opts.Token is set — the edge pre-exists and the join token cannot be used to authenticate kcp API calls. 2. Cluster name was unknown at first boot: the join token carries no JWT claims so extractClusterNameFromToken() always returned "default", making the WebSocket path use /services/agent-proxy/default/... which would cause authorizeByJoinToken to search the wrong kcp cluster. Fix: extract the cluster name from the hub kubeconfig (written by kedge login) and pass it via --cluster to kedge agent join in the e2e framework. Test changes: - framework.TokenAgent gains WithCluster() so the e2e test can supply the correct cluster name. - AgentConnectsWithJoinToken now calls ClusterNameFromKubeconfig (already present in the framework) after login and passes it to WithCluster(). Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * fix: hub marks edge Ready/Disconnected for join-token connections In join-token mode the agent's edge_reporter cannot call the kcp API (join token is not a valid kcp credential) so the hub now takes ownership of edge status transitions for that auth path: - edge_status.go: add markEdgeConnected() -- patches edge to Connected=true / Phase=Ready using the hub's own kcp credentials - agent_proxy_builder_v2.go: call markEdgeConnected() immediately after the WebSocket tunnel is established when authenticatedByJoinToken=true - agent.go: skip edge_reporter in join-token mode (hub owns status); drain tunnelState channel to avoid goroutine leak Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> * Fix join token * Fix ssh flow * clean join token if kubeconfig is provided * fix: Docker e2e test uses hub-generated join token, fix goimports SSHDockerServerModeConnect was starting the agent with --token dev-token (a static kcp token) which the agent now treats as a join token (opts.Token != "" → join mode). Static tokens are hub-internal auth and are not valid join tokens, so authorizeByJoinToken rejects the connection and the edge never becomes Ready. Fix: pre-create the Edge resource as admin, wait for the hub's TokenReconciler to populate status.joinToken, then pass that real join token to the Docker container. This exercises the full join-token bootstrap flow end-to-end and removes the static-token dependency from the agent test. Also fix goimports: remove extraneous alignment spaces in main.go introduced by ca5611d (golangci-lint goimports check). Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> --------- Co-authored-by: OpenClaw Bot <bot@openclaw.ai> Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Co-authored-by: mjudeikis-bot <mjudeikis-bot@users.noreply.github.com>
1 parent c64e2d6 commit c1490e9

25 files changed

Lines changed: 1924 additions & 161 deletions

File tree

‎Makefile‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -162,23 +162,24 @@ ifeq ($(TYPE),server)
162162
$(BINDIR)/kedge-agent \
163163
--hub-url=https://localhost:8443 \
164164
--insecure-skip-tls-verify \
165-
--token=$(STATIC_AUTH_TOKEN) \
165+
--token=$(KEDGE_EDGE_JOIN_TOKEN) \
166166
--tunnel-url=https://localhost:8443 \
167167
--edge-name=$(DEV_EDGE_NAME) \
168-
--labels=$(KEDGE_EDGE_LABELS) \
169168
--cluster=$(KEDGE_EDGE_CLUSTER) \
170169
--type=server \
171170
--ssh-proxy-port=2222 \
172171
--ssh-user=kedge \
173172
--ssh-password=password
174173
else
175174
hack/scripts/ensure-kind-cluster.sh
176-
$(BINDIR)/kedge-agent join \
177-
--hub-kubeconfig=$(KEDGE_EDGE_KUBECONFIG) \
178-
--kubeconfig=.kubeconfig-kedge-agent \
175+
$(BINDIR)/kedge-agent \
176+
--hub-url=https://localhost:8443 \
177+
--insecure-skip-tls-verify \
178+
--token=$(KEDGE_EDGE_JOIN_TOKEN) \
179179
--tunnel-url=https://localhost:8443 \
180180
--edge-name=$(DEV_EDGE_NAME) \
181-
--labels=$(KEDGE_EDGE_LABELS) \
181+
--kubeconfig=.kubeconfig-kedge-agent \
182+
--cluster=$(KEDGE_EDGE_CLUSTER) \
182183
--type=kubernetes
183184
endif
184185

@@ -247,7 +248,7 @@ HUB_FLAGS_BASE := \
247248
--serving-cert-file=certs/apiserver.crt \
248249
--serving-key-file=certs/apiserver.key \
249250
--hub-external-url=https://localhost:8443 \
250-
--dev-mode
251+
--dev-mode -v 4
251252

252253
# Auth: OIDC via Dex
253254
HUB_FLAGS_OIDC := \

‎apis/kedge/v1alpha1/types_edge.go‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,24 @@ type ServerEdgeSpec struct {
147147
SSHCredentialsRef *corev1.SecretReference `json:"sshCredentialsRef,omitempty"`
148148
}
149149

150+
// Edge condition types.
151+
const (
152+
// EdgeConditionRegistered indicates the edge agent has completed registration
153+
// via the bootstrap join token. False while the edge is awaiting its first
154+
// agent connection; True after the agent authenticates with the join token
155+
// and receives a durable ServiceAccount credential.
156+
EdgeConditionRegistered = "Registered"
157+
)
158+
150159
// EdgeStatus defines the observed state of an Edge.
151160
type EdgeStatus struct {
161+
// JoinToken is a bootstrap token for agent registration. Generated by the hub
162+
// controller when the Edge is first created. The agent presents this token on
163+
// its first WebSocket connection to authenticate without a full kubeconfig.
164+
// Cleared once the agent completes registration.
165+
// +optional
166+
JoinToken string `json:"joinToken,omitempty"`
167+
152168
// Phase describes the current lifecycle phase of the Edge.
153169
Phase EdgePhase `json:"phase,omitempty"`
154170

@@ -179,6 +195,10 @@ type EdgeStatus struct {
179195
// Used by the hub to verify the agent's sshd identity and prevent MITM attacks.
180196
// +optional
181197
SSHHostKey string `json:"sshHostKey,omitempty"`
198+
199+
// Conditions represent the latest available observations of the edge's state.
200+
// +optional
201+
Conditions []metav1.Condition `json:"conditions,omitempty"`
182202
}
183203

184204
// SSHCredentials holds SSH authentication credentials for connecting to server-type edges.

‎cmd/kedge-agent/main.go‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,22 @@ func main() {
4040
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
4141
defer cancel()
4242

43+
logger := klog.FromContext(ctx)
44+
45+
// Token-exchange: if a saved kubeconfig exists from a previous
46+
// join-token registration, use it instead of the bootstrap token.
47+
if opts.HubKubeconfig == "" && opts.EdgeName != "" {
48+
kubeconfigPath, err := agent.LoadAgentKubeconfig(opts.EdgeName)
49+
if err != nil {
50+
logger.Info("Could not check for saved agent kubeconfig", "err", err)
51+
} else if kubeconfigPath != "" {
52+
logger.Info("Using saved agent kubeconfig from previous registration", "edgeName", opts.EdgeName, "path", kubeconfigPath)
53+
opts.HubKubeconfig = kubeconfigPath
54+
opts.Token = "" // Clear join token; SA kubeconfig takes precedence.
55+
opts.UsingSavedKubeconfig = true
56+
}
57+
}
58+
4359
a, err := agent.New(opts)
4460
if err != nil {
4561
return fmt.Errorf("failed to create agent: %w", err)

‎config/crds/kedge.faros.sh_edges.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,64 @@ spec:
150150
For kubernetes edges, append /k8s/ for K8s API proxy.
151151
For server edges, append /ssh for SSH WebSocket terminal.
152152
type: string
153+
conditions:
154+
description: Conditions represent the latest available observations
155+
of the edge's state.
156+
items:
157+
description: Condition contains details for one aspect of the current
158+
state of this API Resource.
159+
properties:
160+
lastTransitionTime:
161+
description: |-
162+
lastTransitionTime is the last time the condition transitioned from one status to another.
163+
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
164+
format: date-time
165+
type: string
166+
message:
167+
description: |-
168+
message is a human readable message indicating details about the transition.
169+
This may be an empty string.
170+
maxLength: 32768
171+
type: string
172+
observedGeneration:
173+
description: |-
174+
observedGeneration represents the .metadata.generation that the condition was set based upon.
175+
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
176+
with respect to the current state of the instance.
177+
format: int64
178+
minimum: 0
179+
type: integer
180+
reason:
181+
description: |-
182+
reason contains a programmatic identifier indicating the reason for the condition's last transition.
183+
Producers of specific condition types may define expected values and meanings for this field,
184+
and whether the values are considered a guaranteed API.
185+
The value should be a CamelCase string.
186+
This field may not be empty.
187+
maxLength: 1024
188+
minLength: 1
189+
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
190+
type: string
191+
status:
192+
description: status of the condition, one of True, False, Unknown.
193+
enum:
194+
- "True"
195+
- "False"
196+
- Unknown
197+
type: string
198+
type:
199+
description: type of condition in CamelCase or in foo.example.com/CamelCase.
200+
maxLength: 316
201+
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
202+
type: string
203+
required:
204+
- lastTransitionTime
205+
- message
206+
- reason
207+
- status
208+
- type
209+
type: object
210+
type: array
153211
connected:
154212
description: Connected indicates whether the edge agent currently
155213
has an active tunnel.
@@ -158,6 +216,13 @@ spec:
158216
description: Hostname is the hostname reported by the connected edge
159217
agent.
160218
type: string
219+
joinToken:
220+
description: |-
221+
JoinToken is a bootstrap token for agent registration. Generated by the hub
222+
controller when the Edge is first created. The agent presents this token on
223+
its first WebSocket connection to authenticate without a full kubeconfig.
224+
Cleared once the agent completes registration.
225+
type: string
161226
labels:
162227
additionalProperties:
163228
type: string

‎config/kcp/apiexport-kedge.faros.sh.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ spec:
6565
resources:
6666
- group: kedge.faros.sh
6767
name: edges
68-
schema: v260304-d4ea891.edges.kedge.faros.sh
68+
schema: v260309-9f178d5.edges.kedge.faros.sh
6969
storage:
7070
crd: {}
7171
- group: kedge.faros.sh

‎config/kcp/apiresourceschema-edges.kedge.faros.sh.yaml‎

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
apiVersion: apis.kcp.io/v1alpha1
22
kind: APIResourceSchema
33
metadata:
4-
name: v260304-d4ea891.edges.kedge.faros.sh
4+
name: v260309-9f178d5.edges.kedge.faros.sh
55
spec:
66
group: kedge.faros.sh
77
names:
@@ -146,6 +146,64 @@ spec:
146146
For kubernetes edges, append /k8s/ for K8s API proxy.
147147
For server edges, append /ssh for SSH WebSocket terminal.
148148
type: string
149+
conditions:
150+
description: Conditions represent the latest available observations
151+
of the edge's state.
152+
items:
153+
description: Condition contains details for one aspect of the current
154+
state of this API Resource.
155+
properties:
156+
lastTransitionTime:
157+
description: |-
158+
lastTransitionTime is the last time the condition transitioned from one status to another.
159+
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
160+
format: date-time
161+
type: string
162+
message:
163+
description: |-
164+
message is a human readable message indicating details about the transition.
165+
This may be an empty string.
166+
maxLength: 32768
167+
type: string
168+
observedGeneration:
169+
description: |-
170+
observedGeneration represents the .metadata.generation that the condition was set based upon.
171+
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
172+
with respect to the current state of the instance.
173+
format: int64
174+
minimum: 0
175+
type: integer
176+
reason:
177+
description: |-
178+
reason contains a programmatic identifier indicating the reason for the condition's last transition.
179+
Producers of specific condition types may define expected values and meanings for this field,
180+
and whether the values are considered a guaranteed API.
181+
The value should be a CamelCase string.
182+
This field may not be empty.
183+
maxLength: 1024
184+
minLength: 1
185+
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
186+
type: string
187+
status:
188+
description: status of the condition, one of True, False, Unknown.
189+
enum:
190+
- "True"
191+
- "False"
192+
- Unknown
193+
type: string
194+
type:
195+
description: type of condition in CamelCase or in foo.example.com/CamelCase.
196+
maxLength: 316
197+
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
198+
type: string
199+
required:
200+
- lastTransitionTime
201+
- message
202+
- reason
203+
- status
204+
- type
205+
type: object
206+
type: array
149207
connected:
150208
description: Connected indicates whether the edge agent currently has
151209
an active tunnel.
@@ -154,6 +212,13 @@ spec:
154212
description: Hostname is the hostname reported by the connected edge
155213
agent.
156214
type: string
215+
joinToken:
216+
description: |-
217+
JoinToken is a bootstrap token for agent registration. Generated by the hub
218+
controller when the Edge is first created. The agent presents this token on
219+
its first WebSocket connection to authenticate without a full kubeconfig.
220+
Cleared once the agent completes registration.
221+
type: string
157222
labels:
158223
additionalProperties:
159224
type: string

‎hack/scripts/dev-edge-setup.sh‎

Lines changed: 26 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -43,51 +43,40 @@ if [[ "${CLUSTER_URL}" =~ /clusters/([^/]+) ]]; then
4343
fi
4444
echo "Detected kcp cluster path: ${CLUSTER_PATH:-<none>}"
4545

46-
# 3. Write .env.edge for make dev-run-edge.
46+
# 3. Wait for join token (set by the hub token controller on Edge creation).
47+
TIMEOUT=60
48+
echo "Waiting for join token on Edge '${EDGE_NAME}'..."
49+
JOIN_TOKEN=""
50+
elapsed=0
51+
while true; do
52+
JOIN_TOKEN=$(kubectl get edge "${EDGE_NAME}" -o jsonpath='{.status.joinToken}' 2>/dev/null || echo "")
53+
if [[ -n "${JOIN_TOKEN}" ]]; then
54+
break
55+
fi
56+
if [ "${elapsed}" -ge "${TIMEOUT}" ]; then
57+
echo "WARNING: Timed out after ${TIMEOUT}s waiting for join token." >&2
58+
echo " The Edge token controller may still be starting up." >&2
59+
break
60+
fi
61+
sleep 2
62+
elapsed=$((elapsed + 2))
63+
printf " waiting... (%ds)\r" "${elapsed}"
64+
done
65+
echo ""
66+
67+
# 4. Write .env.edge for make dev-run-edge.
4768
cat > .env.edge <<EOF
4869
# Generated by hack/scripts/dev-edge-setup.sh — do not commit
4970
KEDGE_EDGE_NAME=${EDGE_NAME}
5071
KEDGE_EDGE_TYPE=${EDGE_TYPE}
5172
KEDGE_EDGE_LABELS=${LABELS}
5273
KEDGE_EDGE_CLUSTER=${CLUSTER_PATH}
74+
KEDGE_EDGE_JOIN_TOKEN=${JOIN_TOKEN}
5375
EOF
5476

55-
if [[ "${EDGE_TYPE}" == "kubernetes" ]]; then
56-
NAMESPACE="kedge-system"
57-
SECRET_NAME="edge-${EDGE_NAME}-kubeconfig"
58-
OUTPUT_FILE=".edge-kubeconfig"
59-
TIMEOUT=120
60-
61-
echo "Waiting for hub kubeconfig secret ${NAMESPACE}/${SECRET_NAME}..."
62-
elapsed=0
63-
while true; do
64-
if kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" &>/dev/null; then
65-
break
66-
fi
67-
if [ "${elapsed}" -ge "${TIMEOUT}" ]; then
68-
echo "WARNING: Timed out after ${TIMEOUT}s waiting for ${SECRET_NAME}." >&2
69-
echo " The Edge RBAC controller may still be starting up." >&2
70-
echo " You can run 'make dev-run-edge TYPE=kubernetes' anyway and" >&2
71-
echo " the secret will be created once the controller is ready." >&2
72-
break
73-
fi
74-
sleep 2
75-
elapsed=$((elapsed + 2))
76-
printf " waiting... (%ds)\r" "${elapsed}"
77-
done
78-
echo ""
79-
80-
if kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" &>/dev/null; then
81-
echo "Extracting hub kubeconfig to ${OUTPUT_FILE}..."
82-
kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" \
83-
-o jsonpath='{.data.kubeconfig}' | base64 -d > "${OUTPUT_FILE}"
84-
# Append the kubeconfig path to .env.edge so dev-run-edge can pick it up.
85-
cat >> .env.edge <<EOF
86-
KEDGE_EDGE_KUBECONFIG=${OUTPUT_FILE}
87-
EOF
88-
fi
89-
fi
90-
9177
echo "Done!"
9278
echo " .env.edge written"
79+
if [[ -n "${JOIN_TOKEN}" ]]; then
80+
echo " Join token: ${JOIN_TOKEN}"
81+
fi
9382
echo " Run 'make dev-run-edge TYPE=${EDGE_TYPE}' to start the agent"

0 commit comments

Comments
 (0)