Skip to content

feat(podman): auto-detect container engine and support Podman#49

Closed
wmeints wants to merge 5 commits into
mainfrom
feat/podman-support
Closed

feat(podman): auto-detect container engine and support Podman#49
wmeints wants to merge 5 commits into
mainfrom
feat/podman-support

Conversation

@wmeints

@wmeints wmeints commented Jul 9, 2026

Copy link
Copy Markdown
Member

Why

Huddle was hardwired to Docker (Docker Desktop on Windows). Teams that standardise on Podman couldn't run it: huddle init failed and devcontainers couldn't reach the Docker socket proxy. This change makes Huddle detect the container engine automatically and adapt its socket mapping and container configuration accordingly, so it starts and runs on Podman as well as Docker. Scope is intentionally limited to Docker + Podman.

What changed

The gateway stays engine-agnostic — it drives the engine through the Docker-compatible REST API on /var/run/docker.sock, which Podman also serves. The real differences are around socket mapping and SELinux, handled at the edges:

  • cli/src/runtime.tsContainerRuntime gains:
    • isRemote — whether the engine runs in a VM (Podman machine / Docker Desktop). Detected via podman info --format {{.Host.ServiceIsRemote}}; for Docker, platform !== 'linux'.
    • securityOpts['label=disable'] for Podman, empty for Docker.
  • cli/src/init.ts
    • Pre-creates the socket dir inside the machine VM for remote Podman (podman machine ssh "mkdir -p /tmp/dc-sockets") — Podman does not auto-create a missing bind source the way Docker Desktop does, and native Linux still mkdirs locally.
    • Adds --security-opt label=disable and -e HUDDLE_RUNTIME=<engine> to the huddle container.
  • gateway/src/docker.ts — when HUDDLE_RUNTIME=podman, adds SecurityOpt: ['label=disable'] to each devcontainer's HostConfig so it may access the SELinux-labeled per-container proxy socket.
  • huddle.ps1 — resolves the engine once (same precedence as the CLI: explicit HUDDLE_RUNTIME, then Docker, then Podman), routes every direct docker call through & $RUNTIME, and exports HUDDLE_RUNTIME so the delegated huddle init uses the same engine. On Windows + Docker Desktop nothing changes.
  • cli/dist/index.js — restore the executable bit on the CLI bin entrypoint (was committed as non-executable, causing "permission denied" after a global install).

Important parts to review

  • gateway/src/docker.tslabel=disable on devcontainers. This relaxes SELinux confinement on devcontainers when on Podman. It is required for them to reach the proxy socket (verified: a confined client gets "Permission denied"). The huddle container itself already gets label=disable from the CLI. Note the child-container guard in socket-proxy.ts still rejects label=disable from containers spawned inside a devcontainer — that path is unchanged.
  • cli/src/init.ts — socket-dir precreation for remote Podman. The podman machine ssh call assumes the default machine. Worth a look for multi-machine setups.
  • cli/src/runtime.tsisRemote detection and the Docker platform !== 'linux' heuristic.
  • huddle.ps1 — runtime resolution + HUDDLE_RUNTIME export, ensuring the script and the delegated CLI agree on the engine when both are installed.

Risks

  • Docker regression risk: low. For Docker the new fields are inert (securityOpts empty, no SecurityOpt injected, existing socket-dir behavior preserved for Windows/Linux). huddle.ps1 picks Docker first, so the Windows/Docker Desktop flow is unchanged.
  • label=disable reduces SELinux isolation for the huddle container and Podman devcontainers. Acceptable given the huddle container already has full engine access via the mounted socket, but it is a real posture change on SELinux-enforcing hosts.
  • Podman-machine assumptions: single default machine; /tmp/dc-sockets lives on VM-local tmpfs (shared correctly between huddle and devcontainers). Untested against remote/rootful Podman over SSH or multi-machine configs.
  • Not exercised live: full devcontainer creation on Podman needs the base images (not available locally); the socket-sharing + label=disable behavior was verified in isolation and via the gateway's own startup socket calls.

Verification

  • huddle init --runtime podman on a Podman machine: huddle container comes up on devcontainer-net,podman, correct SecurityOpt/env/mounts, gateway boots with no socket errors, and a request over /var/run/docker.sock from inside returns HTTP 200.
  • huddle.ps1 end-to-end on Podman: resolves podman, delegates huddle init without error, status shows [ON] Huddle draait.
  • CLI + gateway typecheck/build clean; all 85 gateway unit tests pass.

🤖 Generated with Claude Code

@wmeints wmeints requested a review from insuT0ver July 9, 2026 06:28
@wmeints

wmeints commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Follow-up fix: engine detection through the docker shim

Testing surfaced a detection bug on Podman hosts where docker is a symlink/shim to Podman (podman-docker). Podman emulates even docker --version, so the original name-based auto-detect picked docker and misconfigured the socket path, network, and skipped label=disablehuddle init failed with:

Command failed: docker run ... -e HUDDLE_RUNTIME=docker ... -v /var/run/docker.sock:/var/run/docker.sock ...

Fix (commit 2a836d4): probe the real engine via a Podman-only info field — <cmd> info --format {{.Host.ServiceIsRemote}} returns a value on Podman (even through the shim) and errors on Docker. Applied in both cli/src/runtime.ts (detectEngine) and huddle.ps1 (Get-TrueEngine); auto-detect now looks behind the docker command first, then podman. A real Docker engine still wins when present.

Verified: huddle init with no --runtime on a Podman host that exposes docker→podman now resolves podman, creates the VM socket dir, and starts with SecurityOpt=[label=disable], HUDDLE_RUNTIME=podman.

@wmeints

wmeints commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Follow-up fix: operator locked out of the web UI on Podman

On rootless Podman the web UI returned {"error":"forbidden","reason":"endpoint not allowed from devcontainer network"} for the operator on http://localhost:3000.

Cause: Podman's rootless port forwarder (pasta) delivers host -p traffic into the container via its primary network interface, sourced from that subnet. With devcontainer-net (10.89.0.0/24) as the primary network, the operator's browser arrived as e.g. 10.89.0.8 — inside a subnet the source-IP gate (api.ts) blocks — so the UI was rejected. Docker doesn't hit this: its port forward goes through the docker proxy and is sourced from loopback/the bridge gateway, independent of the primary network.

Fix (commit ed3d824): on Podman, start huddle with the egress network (runtime.defaultNetwork) as primary and attach devcontainer-net afterwards. Host traffic then arrives from the non-blocked egress subnet (10.88.0.0/16), while devcontainers keep arriving from devcontainer-net/dc-net-* and stay blocked. Docker keeps its existing ordering.

Verified on rootless Podman: operator GET / and /api/state200; a container on devcontainer-net hitting huddle:3000/api/state403 (gate intact).

Minor note for reviewers: as with Docker's bridge, other containers the user runs on the default podman network could now reach the API — same posture as the existing Docker setup.

@wmeints wmeints force-pushed the feat/podman-support branch from 0f9fb0e to 77c0b93 Compare July 9, 2026 08:25
@wmeints wmeints requested a review from a team July 9, 2026 08:25
@wmeints wmeints marked this pull request as draft July 9, 2026 10:59
@wmeints

wmeints commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Heads-up: fixed a Podman DNS regression that broke egress (pushed as c0b6e5d)

While testing this branch I hit a 502 from api.anthropic.com inside the devcontainer — Claude Code couldn't reach the API at all. Worth a careful look during review since it touches the new Podman networking.

Root cause

The gateway attaches to the --internal devcontainer networks. Podman adds each of those networks' aardvark-dns (the network gateway, e.g. 10.89.x.1) to the gateway's /etc/resolv.conf. Those servers only know container names and answer NXDOMAIN for external hostnames.

The gateway image is Alpine (musl), and musl's resolver queries all nameservers in parallel and uses the first reply. The on-host aardvark returns its NXDOMAIN almost instantly and wins the race against the slower real egress resolver → every external lookup fails with ENOTFOUND → the proxy's upstreamReq.on('error') returns 502.

Confirmed against the live gateway's own audit log (all 502s had res_headers=null/res_body=null, i.e. the proxy's own error path, not an Anthropic response) and reproduced in isolation. Two subtleties that shaped the fix:

  • Reordering resolv.conf doesn't help — musl ignores order, so the broken servers must be removed.
  • huddle init connects the second network after the gateway process starts, so the pollution lands after the first sanitize (hence the settling window below).

Fix (c0b6e5d)

  • New gateway/src/dns-egress.ts: probes each nameserver and drops the ones that can't resolve externally. Fail-safe — leaves the file untouched if no resolver works.
  • Runs at startup, over a short settling window (catches the post-start network connect), and after every gateway network connect/disconnect (docker.ts).
  • Left cli/src/init.ts on run -d + post-start connect — I tried create/connect/start to attach both nets before startup, but that makes Podman drop the egress resolvers entirely. Comment added explaining why.

Verified end-to-end: api.anthropic.com resolves and the upstream returns 401 (auth expected) instead of 502; stable past the settling window. tsc clean, all 85 gateway tests pass.

This is Docker-safe: Docker's embedded DNS resolves external names on internal networks, so the sanitizer is a no-op there.

@wmeints wmeints marked this pull request as ready for review July 9, 2026 11:24
This was referenced Jul 9, 2026
wmeints and others added 5 commits July 10, 2026 08:55
Detect Docker vs Podman across the CLI, gateway, and PowerShell manager,
and handle Podman's socket mapping and SELinux/bind-source differences so
`huddle init` and devcontainers work on Podman as well as Docker.

- runtime.ts: add isRemote (engine-in-VM) and securityOpts to ContainerRuntime
- init.ts: pre-create the socket dir in the machine VM for Podman (it won't
  auto-create a bind source), pass --security-opt label=disable and
  HUDDLE_RUNTIME to the huddle container
- gateway/docker.ts: add SecurityOpt label=disable to devcontainers when
  HUDDLE_RUNTIME=podman so they can reach the SELinux-labeled proxy socket
- huddle.ps1: resolve the runtime once (mirrors the CLI) and route every
  docker call through it; export HUDDLE_RUNTIME for the delegated `huddle init`
- restore the executable bit on the CLI bin entrypoint (cli/dist/index.js)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Many Podman setups symlink or shim `docker` to Podman (podman-docker), and
Podman emulates `docker --version`, so name-based detection wrongly picked
"docker" — misconfiguring the socket path, network, and skipping
label=disable, which made `huddle init` fail.

Probe the real engine instead: Podman's `info` exposes Host.ServiceIsRemote
while Docker's schema does not. Use it (via `<cmd> info --format
{{.Host.ServiceIsRemote}}`) in both the CLI runtime resolver and huddle.ps1
so the engine behind the `docker` command is identified correctly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under rootless Podman (pasta), host port-forward traffic enters the huddle
container via its PRIMARY interface and is sourced from that subnet. With
devcontainer-net primary, the operator's browser appeared to originate from
the devcontainer subnet, so the source-IP gate rejected the UI with
"endpoint not allowed from devcontainer network".

Start the huddle container on the egress network (runtime.defaultNetwork)
as primary and attach devcontainer-net afterwards, so host traffic arrives
from a non-blocked subnet while devcontainers stay blocked. Podman-only;
Docker forwards via its own proxy (source = loopback/bridge-gw, independent
of the primary network) and keeps the existing ordering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tart

Podman phrases an already-connected network differently than Docker
('network is already connected' vs 'already exists in network'), so the
guard in createAndStartContainer re-threw and aborted the devcontainer
start with a 403. Broaden the guard to match both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…NS works

The gateway attaches to the --internal devcontainer networks; Podman then adds
those networks' aardvark-dns (which answers NXDOMAIN for external names) to the
gateway's /etc/resolv.conf. On Alpine/musl the resolver queries all nameservers
in parallel and uses the fastest reply, so the on-host aardvark's instant
NXDOMAIN wins the race and every external lookup fails with ENOTFOUND —
api.anthropic.com and other upstreams then return 502 through the proxy.

Add dns-egress.ts: probe each nameserver and remove the ones that cannot resolve
externally (reordering does not help with musl's parallel queries). It runs at
startup, over a short settling window (to catch the devcontainer-net connect
that `huddle init` performs after the gateway has started), and after every
gateway network connect/disconnect. Fail-safe: leaves resolv.conf untouched when
no resolver works.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@insuT0ver

Copy link
Copy Markdown
Contributor

Ik sluit deze PR omdat de ideeën uit deze branch zijn samengevoegd met #29, #49 en #51 in één consolidatiebranch gekoppeld aan #47: experiment/47-podman-support-consolidated.

Zou je willen testen of dit nu werkt in jouw setup?

huddle experiment use 47

of

huddle init --experiment 47

Dit installeert de CLI en gateway image van dit experiment, zodat je end to end kunt testen op je eigen Podman host of VM. Zodra je klaar bent, kun je terugschakelen naar de stabiele versie met:

huddle experiment reset

Laat gerust weten als je nog problemen of onverwacht gedrag tegenkomt. Dan lossen we dat op in de experiment branch voordat alles naar main gaat.

@insuT0ver insuT0ver closed this Jul 14, 2026
@insuT0ver insuT0ver mentioned this pull request Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants