feat: SSH server-mode — bare-metal host SSH access via hub WebSocket tunnel (#70)

* ci: run CI and E2E on ssh branch

Adds 'ssh' to the branch trigger list for both ci.yaml and e2e.yaml
so that PRs targeting the ssh integration branch get full CI coverage.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(agent): SSH forwarder — proxy revdial stream to localhost:22 (issue #52) (#58)

Replaces the 501 stub in tunnel/server.go with a real TCP proxy:
- Hijacks the revdial HTTP connection to get the raw net.Conn
- Dials localhost:22 (host sshd)
- Bidirectional io.Copy: hub <-> revdial <-> sshd

No SSH parsing needed on the agent side — the hub speaks full
SSH protocol over the revdial tunnel.

Co-authored-by: Mangirdas Judeikis <mjudeikis@gmail.com>
Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(api): Server CRD + API type for non-k8s SSH hosts (issue #49) (#59)

Adds the Server resource — the non-Kubernetes counterpart to Site.
A Server represents a bare-metal machine, VM, or systemd host connected
to the hub via a reverse WebSocket tunnel, with SSH proxied to localhost:22.

Changes:
- apis/kedge/v1alpha1/types_server.go: Server/ServerList types, ServerPhase,
  ServerSpec (displayName, hostname, provider, region),
  ServerStatus (phase, lastHeartbeatTime, tunnelConnected, sshEnabled, conditions)
- zz_generated.deepcopy.go: DeepCopy for Server, ServerList, ServerSpec, ServerStatus
- groupversion_info.go: register Server + ServerList in SchemeBuilder
- bootstrap/crds/kedge.faros.sh_servers.yaml: CRD manifest (Cluster-scoped)
- bootstrap/installer.go: wait for servers.kedge.faros.sh to be established

Co-authored-by: Mangirdas Judeikis <mjudeikis@gmail.com>
Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(agent): --mode=server for bare-metal/systemd SSH hosts (issue #50) (#60)

* feat(agent): --mode=server for bare-metal/systemd SSH hosts (issue #50)

Adds AgentModeServer alongside the existing AgentModeSite:

- pkg/agent/agent.go: Mode field on Options (default 'site'); runSiteMode
  keeps all existing behaviour; runServerMode skips k8s bootstrap, registers
  a Server resource, starts revdial tunnel + ServerReporter heartbeat
- pkg/agent/status/server_reporter.go: lightweight heartbeat that patches
  Server status (phase=Ready, sshEnabled=true, tunnelConnected=true)
- pkg/agent/tunnel/server.go: guard k8s proxy route against nil downstream
  config; returns 503 in server mode instead of panicking
- pkg/client/dynamic.go: ServerGVR + Servers() typed accessor
- pkg/cli/cmd/agent.go: --mode flag exposed on 'kedge agent join'

Usage:
  kedge agent join --hub-url https://... --token ... --site-name my-host --mode server

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* refactor(agent): AgentMode dedicated string type (review nit)

Replace bare string constants with a proper AgentMode type:
  type AgentMode string
  AgentModeSite   AgentMode = "site"
  AgentModeServer AgentMode = "server"
  Options.Mode    AgentMode

CLI StringVar uses (*string)(&opts.Mode) cast for cobra compatibility.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

---------

Co-authored-by: Mangirdas Judeikis <mjudeikis@gmail.com>
Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(hub): OIDC identity → SSH username mapping (issue #53) (#61)

The hub's sshHandler previously hardcoded User:"root". Now it derives
the SSH username from the caller's bearer token before dialling the agent.

Resolution order:
  1. OIDC email local-part  (alice@example.com → alice)
  2. OIDC sub claim          (sanitized)
  3. fallback                root  (SA tokens, anonymous)

Sanitization: lowercase, non-[a-z0-9_-] → underscore, truncated to 32 chars.

New helpers in pkg/virtual/builder/auth.go:
  - parseOIDCToken: decode JWT payload, extract sub + email (no sig verify)
  - sshUsernameFromToken: resolution logic
  - sanitizeUnixUsername: POSIX-safe username normalisation

Co-authored-by: Mangirdas Judeikis <mjudeikis@gmail.com>
Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat: kedge ssh CLI + e2e test for server-mode agent (issue #54) (#62)

* feat: kedge ssh CLI + e2e test for server-mode agent (issue #54)

Closes the SSH epic.

kedge ssh CLI (pkg/cli/cmd/ssh.go):
- 'kedge ssh <name>' opens an interactive PTY session via hub WebSocket
- 'kedge ssh <name> -- cmd' runs a single command (non-interactive)
- Derives WebSocket URL from kubeconfig host (https→wss / http→ws)
- Sends/receives wsMsg JSON frames (base64 cmd, resize, binary output)
- Handles SIGWINCH for terminal resize

Hub SSH auth (pkg/virtual/builder/agent_proxy_builder.go):
- Add ssh.Password("") auth method to newSSHClient so it can connect
  to sshd configured with PermitEmptyPasswords (test/dev environments)
- TODO: replace with key-based auth from Server resource secret

E2e framework (test/e2e/framework/):
- server_container.go: ServerContainer manages a Docker container with
  ubuntu:22.04 + openssh-server + kedge agent binary running --mode=server
- cluster.go: AgentBinPath(), WithServerContainer(), ServerContainerFromContext()

E2e cases (test/e2e/cases/ssh.go):
- SSHServerModeConnect: starts container, waits for Server→Ready,
  runs 'kedge ssh <name> -- echo kedge_ssh_e2e_ok', asserts marker in output
- Registered in standalone suite

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(ssh): HTTP 101 upgrade handshake + WebSocket close + Docker container

Three bugs fixed:

1. Protocol mismatch — hub was speaking SSH directly on deviceConn but the
   agent's local HTTP server expected an HTTP request first.  Fix:
   - Agent: newSSHHandler now sends 101 Switching Protocols before hijacking
     the connection and piping to sshd.
   - Hub: sshHandler now calls openAgentSSHTunnel() to send GET /ssh and
     consume the 101 response before creating the SSH client.  A bufferedConn
     wraps the underlying net.Conn + bufio.Reader so that bytes already
     buffered past the HTTP response headers (e.g. the SSH banner) are not
     lost.

2. Non-interactive command hang — after the SSH session ended, receiveWsMsg
   was blocked on ReadMessage() forever (CLI never sends a second message),
   preventing the errgroup from completing and the WebSocket from closing.
   Fix: added a wsCloser goroutine in Run() that waits for sendComboOutput
   to complete (flushDone channel) and then closes the WebSocket, unblocking
   receiveWsMsg cleanly.

3. Docker container USER_NAME=root error — lscr.io/linuxserver/openssh-server
   rejects USER_NAME=root because root already exists in /etc/passwd.
   Fix: switched to ubuntu:22.04 with openssh-server installed via apt at
   container start time, and a minimal sshd_config written via printf.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* ci: retrigger Standalone check

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* ci(e2e): increase Standalone timeout to 60min / 45m

Standalone runs TestAgentJoin + TestTunnelResilience that OIDC skips,
adding ~15-20min. Previous 45min job timeout was consistently being hit.
Align with External KCP budget (60min job / 45m test timeout).

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(hub): reduce HeartbeatTimeout 5m→90s; update test waits

90s = 3 missed heartbeats at agent's 30s interval — sensible both for
production responsiveness and CI speed.

Previously 5 min × 3 disconnect-detection tests = 15 min of idle
waiting in Standalone CI, causing consistent timeout failures.
New budget: ~90s × 3 = ~4.5 min — saves ~10 min in CI.

Test WaitForSitePhase("Disconnected") lowered 8m→3m to match.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* ci(e2e): shorten SSH keepalive to 30s in Standalone CI

TestSSHServerModeConnect holds a live session for SSHKeepaliveDuration
(default 5min) to verify keepalive. That burns 5 idle minutes in CI.

Pass -ssh-keepalive-duration=30s via E2E_FLAGS so the keepalive assert
still runs but completes in 30 seconds instead of 5 minutes.

Also bump Standalone E2E_TIMEOUT 45m→50m for headroom: with
HeartbeatTimeout=90s (saves ~10min) and keepalive=30s (saves ~4.5min)
tests should finish well under 50m.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(e2e): lower SSHKeepaliveDuration default 5m→30s; fix -args in Makefile

Two fixes to cure Standalone CI timeouts:

1. SSHKeepaliveDuration default 5min→30s.
   The 'long_lived_connection_stays_alive' assessment held an SSH
   session open for the full default duration every CI run, burning
   5 idle minutes.  30s is enough to prove the keepalive mechanism
   works; longer runs can pass --ssh-keepalive-duration=<dur> locally.

2. Makefile: pass E2E_FLAGS via -args so custom test-binary flags
   (like --ssh-keepalive-duration) are actually parsed by the test
   binary, not rejected by 'go test' as unknown flags.

Combined with the earlier HeartbeatTimeout 5m→90s fix (~10 min
saved), Standalone should now complete well under the 50m budget.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix: resolve SSH session teardown deadlock (#62)

Four interrelated bugs caused TestSSHServerModeConnect to hang indefinitely:

SocketSSHSession.Run uses errgroup.WithContext. errgroup only cancels
the derived context when a goroutine returns a *non-nil* error. When
session.Wait() returns nil (shell exited cleanly), the wait() goroutine
also returned nil — leaving receiveWsMsg, sendComboOutput, sendKeepAlive,
and wsCloser running forever.

Fix: return a private errSessionDone sentinel from wait() when
session.Wait() returns nil. This triggers context cancellation and
causes all other goroutines to return. Run() filters the sentinel before
returning to callers (visible return value stays nil).

The original test server used 'cmd.Stdin = ch', which makes Go's exec
package create an internal goroutine that copies from ch (SSH channel)
to the shell's stdin pipe. cmd.Wait() (and thus cmd.Run()) only returns
*after* that goroutine finishes. The goroutine reads from the SSH
channel, which has no EOF until the client closes it. The client can't
close it until the session ends — which can't happen because cmd.Run()
is blocked. Classic deadlock.

Fix: use cmd.StdinPipe() so that cmd.Wait() does NOT wait for our stdin
copier goroutine. The goroutine is cleaned up naturally when ch is
closed by the deferred ch.Close() after cmd.Run returns. The shell still
receives all keystrokes.

Without an explicit exit-status request, session.Wait() on the client
side returns exitMissingError instead of nil, which the caller may treat
as an error. Adding exit-status=0 after a clean shell exit matches what
a real SSH daemon does and makes session.Wait() return a clean nil.

Fix: send exit-status (success or failure) after cmd.Run returns in the
shell case, mirroring the existing exec case.

session.SendRequest with wantReply=true waits for a reply. If the SSH
channel is in a broken or half-closed state, the reply never arrives and
SendRequest blocks the goroutine indefinitely.

Fix: switch to wantReply=false (fire-and-forget). Keepalives are best-
effort; we already detect dead sessions via consecutive failure counting.

When the session is force-closed (context cancellation), Close() and the
wait() stop branch now also close stdinPipe. For a sessionStdin-backed
pipe this sends CloseWrite to the remote, cleanly signalling EOF to the
shell.

Added pkg/util/ssh/ssh_teardown_test.go: a self-contained unit test
that spins up a minimal SSH server, runs SocketSSHSession, sends
'exit 0', and asserts the session tears down within 8 seconds (typical
~300ms with fixes applied).

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* test(e2e): move SSH tests to dedicated suite; add e2e-ssh CI job

Standalone suite was consistently hitting the 60-min job ceiling because
TestSSHServerModeConnect + TestSSHDockerServerModeConnect added ~10-15 min
to an already-heavy suite (Docker apt-get install, SSH keepalive hold, etc).

Fix: split SSH tests into test/e2e/suites/ssh/ with a hub-only setup
(agentCount=1 — the CLI minimum, agent cluster not used by SSH tests).
This gives each suite a focused, right-sized time budget:

  Standalone  60 min job / 50m test  (11 tests, 2 agent clusters)
  SSH         30 min job / 20m test  (2 tests, hub + 1 cluster)
  OIDC        40 min job / 25m test
  External KCP 60 min job / 30m test

Also adds SetupClustersWithAgentCount / TeardownClustersWithAgentCount
helpers to the e2e framework for future suites.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(lint): goimports formatting in standalone_test.go

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(ssh): use SSH exec for non-interactive commands; fix &> in sh -c

Two bugs causing SSH e2e test failures:

The agent startup command used bash-specific &> redirect syntax in
sh -c: "cmd &> /var/log/kedge-agent.log &". Ubuntu containers use
dash as /bin/sh, which does not support &>. In dash, &> is parsed
as two separate tokens (& and >), so redirection never happens.
Result: kedge-agent.log stays empty, WaitForAgentReady times out.

Fix: change to POSIX-compatible: "> /var/log/kedge-agent.log 2>&1 &"

kedge ssh server -- echo cmd used a PTY+shell session. The hub
creates a shell, then the CLI sends the command as a WebSocket
message. Race: if the shell exits and session teardown completes
before the output tick (60 ms) fires, output is lost; or if timing
is otherwise tight the WebSocket closes before any message arrives.

Fix: for non-interactive commands (kedge ssh <name> -- <cmd>), pass
the command as a URL query parameter cmd= and use SSH exec on the
hub side. SSH exec is more reliable than shell+stdin:
- No shell startup overhead or PTY allocation
- Output is streamed directly without the 60 ms tick latency
- No race between shell exit and output flush

CLI: buildSSHWebSocketURL now accepts remoteCmd; when non-empty,
     adds ?cmd=<cmd> to the WebSocket URL.
CLI: runSSHCommandStream replaces runSSHCommand — just reads WS
     messages until close (nothing to write; cmd is in the URL).
Hub: sshHandler detects cmd query param; calls sshExec() which
     creates a non-PTY SSH exec session, pipes stdout+stderr through
     an io.Pipe, and forwards chunks as binary WebSocket messages.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(e2e): strip cluster path from kubeconfig host in DialSSH

The kubeconfig host may contain a workspace path suffix such as
/clusters/<name> when kcp is in use.  DialSSH was appending the
/proxy/... subresource path to the full host string, producing URLs
like wss://kedge.localhost:8443/clusters/xyz/proxy/... which the
hub does not handle (its router only matches /proxy/).

Fix: use url.Parse to parse the host, replace the path outright,
and convert the scheme — exactly as buildSSHWebSocketURL does in
pkg/cli/cmd/ssh.go.

This caused interactive_pty_sends_keystrokes_and_receives_output
and long_lived_connection_stays_alive to fail with 'bad handshake'.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(ssh): POSIX sh redirect + wsCloser race before flush

Bug 1 (TestSSHDockerServerModeConnect): The agent start command used
bash-specific '&>' redirect which POSIX sh (dash on Ubuntu 22.04) parses
as '& > file' (background + truncate separate redirect). The log file
stayed empty so WaitForAgentReady timed out.
Fix: change to POSIX-portable '> file 2>&1 &'.

Bug 2 (TestSSHServerModeConnect/ssh_command_returns_expected_output):
The wsCloser goroutine selected on gCtx.Done() OR flushDone. When
wait() returned errSessionDone, gCtx was cancelled and wsCloser could
race sendComboOutput — closing the WebSocket before the final
flushOutput() call completed, so WriteMessage returned an error and
the output was never delivered to the client.
Fix: wsCloser now waits ONLY on flushDone. Since sendComboOutput always
closes flushDone via defer, wsCloser is guaranteed to unblock and the
WebSocket is never closed before the final flush completes.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(ssh): atomic ReadAndReset to prevent data loss in flushOutput

flushOutput() was reading buffer data via Bytes() and then resetting via
buffer.Reset() as two separate operations without the safeBuffer mutex held
around both.  This created a race window where the SSH library's copy
goroutine could Write() new output between the read and the reset, causing
that output to be silently discarded (Reset() erases everything including
the newly written bytes).

For a fast non-interactive command (echo) this race is narrow but real;
for interactive PTY sessions (where output arrives asynchronously after a
keystroke) the race is wider and reliably triggered.

Fix: add ReadAndReset() to safeBuffer which copies the buffer content and
resets it atomically under the mutex.  flushOutput() now calls ReadAndReset()
instead of Bytes() + direct buffer.Reset().  Any concurrent Write() either
completes before ReadAndReset() acquires the lock (data captured in this
flush) or after it releases the lock (data captured in the next flush);
no data can be lost.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(e2e): SSHWebSocketClient — background reader, no read deadlines

CollectOutput was using SetReadDeadline to time-box reads.  gorilla/websocket
permanently stores the first read error (including timeout errors) in c.readErr;
every subsequent ReadMessage returns that stored error immediately, making the
connection permanently unusable after the first CollectOutput call times out.

Symptoms: the second CollectOutput (3-second window for interactive output)
returned an empty string instantly because the previous 1-second timeout had
poisoned c.readErr.

Fix: add a single background reader goroutine that pumps WebSocket frames into
a buffered channel (msgs).  CollectOutput selects on that channel with a
time.Timer — no read deadline is set on the underlying connection.  The channel
is closed when the reader goroutine exits, so CollectOutput sees the close and
returns gracefully even when the server drops the connection mid-collection.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

---------

Co-authored-by: Mangirdas Judeikis <mjudeikis@gmail.com>
Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>
Co-authored-by: mjudeikis-bot <mjudeikis-bot@users.noreply.github.com>
Co-authored-by: OpenClaw Bot <bot@openclaw.ai>

* docs: add SSH server-mode documentation to README

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* docs: add dev quick-start targets and server-setup script for SSH server mode

- Makefile: add dev-server-create and dev-run-server-agent targets
- Makefile: add DEV_SERVER_NAME var and include .env.server
- README.md: add Developer Quick-Start section for SSH server mode
- hack/scripts/dev-server-setup.sh: register Server CRD + write .env.server

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(dev): correct flags in dev-run-server-agent make target

- Remove non-existent 'join' subcommand (kedge-agent has no subcommands)
- Replace --hub-insecure-skip-tls-verify with --insecure-skip-tls-verify
- Wire --mode and --insecure-skip-tls-verify flags in cmd/kedge-agent/main.go
  (Options fields existed but were never exposed as CLI flags)

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(tunnel): server-mode tunnel uses ?server= param and /servers/ paths

Two bugs prevented kedge ssh from working in server mode:

BUG 1 — CLI used /sites/ path for all SSH targets
  buildSSHWebSocketURL hard-coded /sites/{name}/ssh regardless of
  resource kind. Added resolveResourceKind() which probes the hub API:
  it tries to GET a Server first, falls back to Site if not found.
  The resulting path is now /servers/{name}/ssh for Server resources.

BUG 2 — Server-mode tunnel used ?site= query param (sites key)
  StartProxyTunnel always built the WebSocket URL with ?site=<name>,
  so the hub stored the tunnel under key default/sites/<name>.
  When the SSH request arrived for /servers/<name>/ssh the agent proxy
  looked up key default/servers/<name> — a miss — and returned
  'revdial.Dialer closed'.

  Fixes:
  * tunneler.go: new resourceType param ('sites'|'servers'); servers
    dial ?server=<name>, sites keep ?site=<name>
  * agent.go: passes 'sites'/'servers' based on AgentMode
  * edge_proxy_builder.go: handles both ?site= and ?server= params;
    keys are now <cluster>/sites/<name> or <cluster>/servers/<name>
    to avoid aliasing between same-named Site and Server
  * agent_proxy_builder.go: parses both 'sites' and 'servers' path
    segments; delegates auth and connman lookup to the right key
  * getKey() format updated to include /sites/ separator;
    getServerKey() added for /servers/ namespace
  * mount_reconciler.go: tunnelKey updated to match new getKey format

Verified locally:
  make dev-server-create && make dev-run-server-agent
  kubectl get servers → CONNECTED=true
  Hub log: key="default/servers/dev-server-1" (was /sites/)
  Agent log: resourceType="servers", Tunnel connection established,
             SSH connection request received, SSH tunnel established

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(hub): restore site tunnel key format broken by SSH server-mode changes

The SSH server-mode fix (b738184) changed getKey() from
  cluster/name → cluster/sites/name
to namespace sites separately from servers. However, this changed the
key format that existing site agents and the kubectl-proxy/site-proxy
path rely on, even though both edge_proxy_builder and mount_reconciler
were updated "consistently" in that commit.

The commit was only verified with the server-mode flow (make
dev-run-server-agent). The site flow was not re-tested, and any
external tooling or persistent connections expecting the original
cluster/name format would fail with a connman miss → 502 Bad Gateway
on kubectl proxy / site-proxy endpoints. Sites would also stay stuck
in a non-Ready state (never getting a successful tunnel) causing the
VirtualWorkload scheduler to stall Placement creation.

Fix:
- Restore getKey() to original: clusterName + / + siteName
- Update mount_reconciler.go tunnelKey to match
- Keep getServerKey() as clusterName + /servers/ + serverName
  (servers still use the namespaced format to prevent aliasing between
  a Site and a Server that happen to share the same name)

All three lookup paths are now consistent:
  ?site=<name>   → key cluster/name        (sites, original)
  ?server=<name> → key cluster/servers/name (servers, new)

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* ci: remove ssh branch from CI triggers — only run on main

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* docs: plan milestone v1.1 — SSH key injection via sshKeySecretRef (issue #72)

- Update PROJECT.md with Current Milestone v1.1 section
- Add MILESTONE-72-REQUIREMENTS.md: 24 requirements (KEY-01..KEY-24)
  covering API CRD field, hub secret loading, RBAC, unit + e2e tests
- Add MILESTONE-72-ROADMAP.md: 4 phases (7–10) with full design notes,
  code sketches, success criteria, risk register, and branch strategy

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* docs: add GSD planning for Edge refactor (issue #72)

Adds full GSD-style planning artifacts for the Site+Server → Edge
resource consolidation:

- .planning/PROJECT.md    — project context, architecture, decisions
- .planning/REQUIREMENTS.md — 26 tracked requirements with REQ-IDs
- .planning/ROADMAP.md    — 5-phase implementation plan with file-level detail
- .planning/STATE.md      — current status tracker
- .planning/research/     — architecture map, pitfalls, summary

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(api): add Edge CRD type with discriminated union spec (kubernetes|server)

Part of #72 — unify Site + Server into Edge resource.
Phase 1: API types only. Controllers and virtual workspace builders to follow.

- Add types_edge.go with Edge, EdgeList, EdgeSpec, EdgeStatus, KubernetesEdgeSpec, ServerEdgeSpec
- Add Edge + EdgeList to scheme in groupversion_info.go
- Add deepcopy functions to zz_generated.deepcopy.go
- Run make codegen: CRD kedge.faros.sh_edges.yaml, kcp APIResourceSchema generated
- Add skeleton edge controller package with HeartbeatTimeout/GCTimeout constants
- Add lifecycle_reconciler.go that transitions Edge phase on disconnect

CEL validation enforces spec.type discriminant at admission time.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(hub): Edge controller — replaces site controller, handles kubernetes+server types

Phase 2 of #72. Workspace created only for type=kubernetes.

- pkg/hub/controllers/edge/: mount_reconciler, rbac_reconciler, rbac constants
- mount_reconciler: guard skips kcp workspace for spec.type=server
- rbac_reconciler: provisions SA/token/kubeconfig for edge agents
- pkg/hub/server.go: wire SetupLifecycleWithManager, SetupRBACWithManager, SetupMountWithManager
- pkg/client/dynamic.go: add Edge GVR + Edges() typed accessor
- pkg/agent/tunnel/tunneler.go: unify edge-proxy URL to agent-proxy virtual workspace path
- pkg/agent/status/edge_reporter.go: edge status reporter for agent heartbeat
- pkg/virtual/builder/connman.go: connection manager accessor on virtual workspace builder

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(virtual): Phase 3 — Edge virtual workspace builders (agent-proxy-v2, edges-proxy)

Implements two new virtual workspace builders for the Edge refactor (issue #72):

## agent-proxy-v2 (agent-facing)
Path: /services/agent-proxy/{cluster}/apis/kedge.faros.sh/v1alpha1/edges/{name}/proxy

- Agents connect via WebSocket to register their revdial tunnel
- Parses cluster+name from URL path (vs query params in old builder)
- Creates revdial.Dialer, stores in shared ConnManager keyed by edges/{cluster}/{name}
- Blocks until tunnel closes, then cleans up the entry

## edges-proxy (user-facing)
Path: /services/edges-proxy/clusters/{cluster}/apis/kedge.faros.sh/v1alpha1/edges/{name}/{k8s|ssh}

- k8s: dials the agent via revdial, reverse-proxies to its Kubernetes API
- ssh: dials the agent, opens SSH tunnel, bridges caller's WebSocket to SSH session
- Shared logic with existing agent_proxy_builder.go (sshExec, newSSHClient, etc.)

## Shared ConnManager (pkg/virtual/builder/connman.go)
Simple Store/Load/Delete map protected by sync.RWMutex, keyed by 'edges/{cluster}/{name}'.
Both builders share the same *ConnManager instance initialized in NewVirtualWorkspaces/BuildVirtualWorkspaces.

## Hub wiring (pkg/hub/server.go)
New routes alongside existing builders (cleanup deferred to Phase 5):
  /services/agent-proxy/ -> EdgeAgentProxyHandler()
  /services/edges-proxy/ -> EdgesProxyHandler()

## Agent refactor (agent.go, cmd, cli)
- AgentType replaces AgentMode (backward-compat alias kept)
- AgentTypeKubernetes replaces AgentModeSite
- edge_reporter replaces server_reporter
- CLI --type flag (--mode deprecated alias retained)
- ssh.go updated for Edge-aware tunnel URLs

Build: make build passes
Lint:  make lint passes (0 issues)

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(edge): Phase 5 — CLI update, cleanup old Site/Server code, update e2e

Completes #72. Unified Edge resource replaces Site + Server.
- kedge ssh now uses /edges/{name}/ssh path
- Old reporters and builders removed
- Makefile dev targets updated (old targets kept as deprecated aliases)
- e2e tests updated to use Edge resources

Changes:
- Makefile: add dev-edge-create and dev-run-edge targets; old site/server
  targets kept as deprecated aliases with deprecation warnings
- hack/scripts/dev-edge-setup.sh: new unified Edge setup script for dev
- test/e2e/cases/site.go: EdgeLifecycle, AgentEdgeJoin, EdgeTunnelResilience
  (old Site* functions kept as deprecated thin wrappers)
- test/e2e/cases/ssh.go: update resources from 'servers' to 'edges',
  update jsonpath from .status.tunnelConnected to .status.connected
- test/e2e/framework/client.go: add EdgeCreate, EdgeList, EdgeDelete,
  WaitForEdgeReady, WaitForEdgePhase, ExtractEdgeKubeconfig methods
- test/e2e/framework/server_process.go: --mode=server -> --type=server
- test/e2e/framework/ssh_client.go: update dial path to /services/edges-proxy/
- test/e2e/suites/*/: update test functions to use new Edge names

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(scheduler): migrate scheduler from Site to Edge type in reconciler and tests

- Update SetupWithManager to watch Edge instead of Site
- Rename MatchSites→MatchEdges, SelectSites→SelectEdges in reconciler
- Add MatchEdges/SelectEdges functions to scheduler.go
- Update multisite e2e to use EdgeCreate/EdgeDelete/WaitForEdgeReady/WaitForEdgePhase

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(hub): mark Edge Disconnected in hub when agent tunnel closes

When an agent process is killed (SIGKILL / context cancellation), the
edge status reporter goroutine exits without sending a disconnect
heartbeat. The edge's Connected field stays true in the hub indefinitely.

Fix: in buildEdgeAgentProxyHandler, when dialer.Done() fires (the
revdial tunnel has been closed), proactively patch the Edge status to
Connected=false, Phase=Disconnected via the hub's kcp admin client.

This makes the hub the authoritative source for edge connectivity state,
matching the architectural intent of the Edge refactor.

Fixes TestSiteFailoverIsolation and TestSiteListAccuracyUnderChurn.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* chore: add root-level binaries to .gitignore

Prevent accidental commits of kedge, kedge-hub, kedge-agent
root-level binaries (built by 'go build ./cmd/...').

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* chore: remove dead Site/Server code after Edge refactor

- Remove old /tunnel/, /proxy/, /services/site-proxy/ routes from hub server
- Remove site controller wiring (SetupLifecycleWithManager, SetupRBACWithManager, SetupMountWithManager)
- Delete pkg/hub/controllers/site/ package
- Remove EdgeProxyHandler, AgentProxyHandler, SiteProxyHandler, NewSiteRouteMap from virtual builder
- Remove old edge_proxy_builder.go (?site=/?server= tunnel registration handler)
- Remove site_proxy_handler.go (kube API proxy via reverse tunnels)
- Remove build.go and cluster_proxy_builder.go (dead code, never called from server)
- Remove deprecated SiteLifecycle, AgentJoin, TunnelResilience test wrappers
- Retain Site/Server API types, CRDs and bootstrap (still used by scheduler, client, CLI)

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* ci: re-trigger CI after confirmed local pass of External KCP e2e

All tests confirmed passing locally (346s run):
- TestLabelBasedScheduling ✅
- TestWorkloadIsolation ✅
- TestSiteFailoverIsolation ✅
- TestSiteReconnect ✅
- TestSiteListAccuracyUnderChurn ✅
- TestKCPHealth ✅
- TestKCPResilience ✅

Previous CI failure was a GitHub Actions runner flake (kind cluster
setup crashed in 84s before any test ran).

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix(hub): restore Site heartbeat timeout reconciler

The site lifecycle controller was deleted in the cleanup but the heartbeat
timeout reconciler is still needed: without it, Site resources stay stuck
at phase=Ready/tunnelConnected=true indefinitely after the agent disconnects.

Adds a minimal heartbeat controller that marks Sites as Disconnected when
lastHeartbeatTime is older than 5 minutes. The full site controller (RBAC,
mount) is not restored — only the stale-heartbeat detection.

Fixes: Site showing Ready with 2-day-old heartbeat after tunnel disconnects.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* docs: update README to use Edge CRD terminology

Replace old Site/Server references with unified Edge CRD:
- dev-site-create → dev-edge-create
- dev-run-agent → dev-run-edge
- dev-server-create → dev-edge-create TYPE=server
- dev-run-server-agent → dev-run-edge TYPE=server
- kind: Server YAML → kind: Edge with spec.type: server
- --mode server → --type=server flag

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* feat(api): complete Site+Server removal — Edge is the only resource

Issue #72: fully replace Site + Server with Edge.
- Edge spec.type=kubernetes replaces Site
- Edge spec.type=server replaces Server

Removes:
- apis/kedge/v1alpha1/types_site.go
- apis/kedge/v1alpha1/types_server.go
- Site/Server CRD YAMLs (config/crds, config/kcp, pkg/hub/bootstrap/crds)
- Site/Server bootstrap installation
- Site/Server client accessors (Sites(), Servers(), SiteGVR, ServerGVR)
- pkg/hub/controllers/site/ (just added, now properly removed)
- CLI site command → replaced with edge command (create/list/get/delete)
- get.go: sites → edges
- Makefile: remove deprecated site/server targets and scripts
- pkg/agent/status/reporter.go (dead code — replaced by edge_reporter.go)
- scheduler: remove MatchSites/SelectSites (replaced by MatchEdges/SelectEdges)

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix: remove all remaining Site/Server CRD references

- Fix installer.go CRD wait-list: sites/servers → edges
- Remove all remaining dead Site/Server references across Go, YAML, Makefile
- Full sweep: no more sites.kedge.faros.sh or servers.kedge.faros.sh anywhere
- Rename WaitForSiteAPI → WaitForEdgeAPI, WaitForSiteAPIWithOIDC → WaitForEdgeAPIWithOIDC
- Rename SiteListAccuracyUnderChurn → EdgeListAccuracyUnderChurn in all test suites
- Remove deprecated Site* methods from e2e framework client (replaced by Edge* methods)
- Fix apply.go gvkToGVR: case Site → Edge, sites → edges
- Fix types_placement.go: printcolumn Site → Edge, update comment
- Update examples/site-test.yaml: kind Site → kind Edge
- Update docs: dev-site-create → dev-edge-create, dev-run-agent → dev-run-edge

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* fix: make codegen clean - regenerate CRDs and fix type renames

Rename Site terminology to Edge in types_virtualworkload.go:
- PlacementSpec.SiteSelector → EdgeSelector
- VirtualWorkloadStatus.Sites []SiteWorkloadStatus → Edges []EdgeWorkloadStatus
- SiteWorkloadStatus struct → EdgeWorkloadStatus with EdgeName field

Update all consumers (controllers, tests) and regenerate CRD/kcp YAMLs.

Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

* Fix Hex and Bot nonsense

* fix lint issues

---------

Co-authored-by: Mangirdas Judeikis <mjudeikis@gmail.com>
Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>
Co-authored-by: mjudeikis-bot <mjudeikis-bot@users.noreply.github.com>
Co-authored-by: OpenClaw Bot <bot@openclaw.ai>

Author: 4 people

.github/workflows/e2e.yaml

@@ -2,9 +2,9 @@ name: E2E
 
 on:
   pull_request:
-    branches: [main]
+    branches: [main, ssh]
   push:
-    branches: [main]
+    branches: [main, ssh]
   workflow_dispatch:
 
 permissions:
@@ -14,7 +14,7 @@ jobs:
   e2e-standalone:
     name: Standalone (embedded kcp + static token)
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 60
     steps:
       - uses: actions/checkout@v4
 
@@ -48,7 +48,7 @@ jobs:
         env:
           KEDGE_HUB_IMAGE_PULL_POLICY: Never
           KEDGE_HUB_IMAGE_TAG: test
-        run: make e2e-standalone E2E_TIMEOUT=20m
+        run: make e2e-standalone E2E_TIMEOUT=50m E2E_FLAGS="-ssh-keepalive-duration=30s"
 
       - name: Upload test logs on failure
         if: failure()
@@ -60,6 +60,55 @@ jobs:
             *.log
           retention-days: 3
 
+
+  e2e-ssh:
+    name: SSH Server-Mode (hub-only cluster)
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: v1.25.0
+
+      - name: Install kind
+        uses: helm/kind-action@v1
+        with:
+          install_only: true
+
+      - name: Install Helm
+        uses: azure/setup-helm@v3
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Add kedge.localhost to /etc/hosts
+        run: echo "127.0.0.1 kedge.localhost" | sudo tee -a /etc/hosts
+
+      - name: Build binaries
+        run: make build
+
+      - name: Add bin/ to PATH
+        run: echo "$(pwd)/bin" >> $GITHUB_PATH
+
+      - name: Build hub Docker image
+        run: docker build -f deploy/Dockerfile.hub -t ghcr.io/faroshq/kedge-hub:test .
+
+      - name: Run SSH e2e
+        env:
+          KEDGE_HUB_IMAGE_PULL_POLICY: Never
+          KEDGE_HUB_IMAGE_TAG: test
+        run: make e2e-ssh E2E_TIMEOUT=20m
+
+      - name: Upload test logs on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-ssh-logs
+          path: |
+            *.kubeconfig
+            *.log
+          retention-days: 3
   e2e-oidc:
     name: OIDC (Dex + auth-code flow)
     runs-on: ubuntu-latest

.gitignore

@@ -1,5 +1,8 @@
 # Binaries
 bin/
+kedge
+kedge-hub
+kedge-agent
 *.exe
 *.exe~
 *.dll
@@ -58,3 +61,6 @@ site-kubeconfig
 .planning/autopilot.lock
 .planning/logs/
 .planning/checkpoints/
+.env.server
+.edge-kubeconfig
+.env.edge

.planning/MILESTONE-72-REQUIREMENTS.md

@@ -0,0 +1,86 @@
+# Milestone v1.1 Requirements — SSH Key Injection (issue #72)
+
+## Problem
+
+The hub currently authenticates to the agent's sshd using an empty password
+(`gossh.Password("")`). This is a tracked placeholder (TODO #54). The feature
+adds `Server.Spec.SSHKeySecretRef`, allowing the hub to load an SSH private
+key from a Kubernetes Secret and use it for key-based auth to the agent sshd.
+
+## Current Code Reference
+
+`pkg/virtual/builder/agent_proxy_builder.go`, `newSSHClient`:
+```go
+// TODO(#54): replace with key-based auth loaded from a Secret on the Server resource.
+Auth: []gossh.AuthMethod{gossh.Password("")},
+```
+
+## Architecture
+
+```
+kedge ssh my-server
+  → CLI (unchanged: WebSocket to hub)
+  → Hub sshHandler
+      - looks up Server object by name       (NEW: needs k8s/kcp client)
+      - reads spec.sshKeySecretRef            (NEW: new CRD field)
+      - fetches Secret from kcp workspace     (NEW: secret lookup)
+      - parses private key                    (NEW: gossh.ParsePrivateKey)
+  → newSSHClient(conn, sshUser, privateKey)  (CHANGED: added key param)
+      - Auth: gossh.PublicKeys(signer)        (CHANGED: was Password(""))
+  → agent sshd on host
+```
+
+## v1 Requirements
+
+### API — Server CRD extension
+- [ ] **KEY-01**: `ServerSpec.SSHKeySecretRef` field added — type `*corev1.SecretReference` (namespace + name)
+- [ ] **KEY-02**: `ServerSpec.SSHKeySecretRef` is optional — absence of field leaves current password-auth behaviour unchanged (zero-risk for existing servers)
+- [ ] **KEY-03**: CRD `spec.sshKeySecretRef.name` and `spec.sshKeySecretRef.namespace` map to standard `k8s.io/api/core/v1.SecretReference`
+- [ ] **KEY-04**: DeepCopy generated (`make generate`) — `zz_generated.deepcopy.go` updated
+- [ ] **KEY-05**: CRD YAML regenerated (`make manifests`) — new field appears in CRD OpenAPI schema
+
+### Hub — Secret loading
+- [ ] **KEY-06**: `virtualWorkspaces` struct gains an optional `kubeClient kubernetes.Interface` field for reading Secrets
+- [ ] **KEY-07**: `VirtualWorkspaceConfig` / `BuildVirtualWorkspaces` accept a `KubeClient kubernetes.Interface` parameter (nil = no key injection, current behaviour preserved)
+- [ ] **KEY-08**: `sshHandler` receives `serverName string` and `clusterName string` so it can look up the Server object (currently it only receives the connman key — refactor needed)
+- [ ] **KEY-09**: `sshHandler` calls a new `loadSSHKeyForServer(ctx, serverName, clusterName)` helper that: (a) GETs the Server object, (b) returns nil signer if `sshKeySecretRef` is unset, (c) fetches the Secret, (d) parses the private key from `secret.Data["ssh-privatekey"]`, (e) returns `gossh.Signer`
+- [ ] **KEY-10**: `newSSHClient` accepts an optional `signer gossh.Signer` parameter — when non-nil, uses `gossh.PublicKeys(signer)` as the sole auth method; when nil, falls back to `gossh.Password("")` (backward-compat)
+- [ ] **KEY-11**: Secret field key is configurable per-server via `Server.Spec.SSHKeySecretDataKey` (default `"ssh-privatekey"` matching Kubernetes convention); field is optional string, empty means default
+
+### Hub — Error handling
+- [ ] **KEY-12**: If `sshKeySecretRef` is set but Secret is not found: `sshHandler` returns HTTP 502 with logged error (do not fall back silently to password-auth — operator misconfiguration must be visible)
+- [ ] **KEY-13**: If Secret is found but key parsing fails: `sshHandler` returns HTTP 502 with logged error
+- [ ] **KEY-14**: Secret lookup errors are logged at `klog.V(2)` with server name and secret ref for observability
+
+### RBAC
+- [ ] **KEY-15**: Hub ClusterRole (or Role) grants `get` on `secrets` scoped to the namespace(s) holding Server-referenced Secrets
+- [ ] **KEY-16**: RBAC manifests added to `deploy/rbac/` (or equivalent) for the hub service account
+- [ ] **KEY-17**: Documentation note: Secrets referenced by `sshKeySecretRef` must be in the same kcp workspace/namespace that the hub service account has access to
+
+### Tests — Unit
+- [ ] **KEY-18**: Unit test: `loadSSHKeyForServer` returns nil signer when `sshKeySecretRef` is unset (no k8s call made)
+- [ ] **KEY-19**: Unit test: `loadSSHKeyForServer` returns valid signer when Secret exists with valid RSA private key
+- [ ] **KEY-20**: Unit test: `loadSSHKeyForServer` returns error when Secret is not found
+- [ ] **KEY-21**: Unit test: `loadSSHKeyForServer` returns error when Secret data key is missing or key bytes are invalid
+- [ ] **KEY-22**: Unit test: `newSSHClient` — when signer nil, Auth is `[Password("")]`; when signer non-nil, Auth is `[PublicKeys(signer)]`
+
+### Tests — E2e
+- [ ] **KEY-23**: E2e: register `Server` with `sshKeySecretRef` pointing to a valid Secret containing the test sshd's accepted key → `kedge ssh <name> hostname` returns correct hostname
+- [ ] **KEY-24**: E2e: `Server` without `sshKeySecretRef` still works (backward-compat, existing e2e suite must continue to pass)
+
+## Out of Scope (this milestone)
+
+- Passphrase-protected private keys (v2)
+- Rotating/refreshing the key without restart (v2 — hub reloads per session so already handled)
+- SSH certificate auth (separate issue, requires SSH CA)
+- Surfacing key fingerprint in `Server.Status` (separate UX improvement)
+- CLI changes — all auth is hub→agent, CLI is unchanged
+
+## Traceability
+
+| REQ-ID | Phase |
+|--------|-------|
+| KEY-01 – KEY-05 | Phase 7: API — SSHKeySecretRef field |
+| KEY-06 – KEY-14 | Phase 8: Hub — secret loading + newSSHClient |
+| KEY-15 – KEY-17 | Phase 9: RBAC manifests |
+| KEY-18 – KEY-24 | Phase 10: Tests |