Arcade joystick: wire the 8BitDo stick into the actual room walls (browser Gamepad API)

[RonTuretzky]

Summary

The 8BitDo Arcade Stick (model 80fe, Switch mode) works today only as a standalone single-wall demo — it does not drive the real Vibecode Room walls (the Chrome windows opened by run-room.sh). This issue tracks wiring the stick into the actual room UI so it can dwell-to-click the room like the camera/pose cursor and the hands pinch-camera do.

Current state (as of branch RonTuretzky/run-room-with-joystick, == origin/main)

• Arcade support lives entirely in the vendored Python at gesture-wall/gesturewall/arcade.py, invoked via gesture-wall/gesturewall/app.py with --source arcade (i.e. python run.py --source arcade). It is a self-contained pygame window that renders a 2x3 tile grid and does dwell-to-click locally (build_grid + DwellSelector). Added in commit f12a014 ("gesture-wall: sync working arcade-stick input").
• It does not publish anywhere: arcade.py has no WebSocket / ws:// / asyncio / emit code. It never reaches the room's fusion stream.
• The room walls (Chrome) are driven by gesturewall.server over the fusion WS on ws://localhost:8770 (walls open with &gesture=1&fusion=ws://...:8770). server.py has no arcade/joystick/--source input path — the stick cannot feed it.
• The frontend has no browser Gamepad API usage: grepping src/ and web/ for gamepad / getGamepads / joystick returns nothing. So the walls also don't read the stick directly.
• Net: "run the room with the joystick" is currently impossible without new code.

Proposed implementation — browser Gamepad API (no Python bridge)

The stick is a USB/Bluetooth HID connected to the Mac, so each Chrome wall window can read it directly via navigator.getGamepads() and feed a synthetic cursor into the existing dwell system. No new server/process, and it works identically in single- and multi-wall mode.

Integration point: src/ui/gesture/GestureLayer.tsx. That component keeps a cursors: Map<number, CursorState> (see ~lines 110-162) that already ingests (a) the camera fusion stream and (b) a local mouse-test cursor at id -1 (MOUSE_ID). The per-frame loop feeds every cursor into MultiDwell / DwellSelector (src/ui/gesture/core.ts) which does dwell-to-click against all button:not(:disabled), [data-dwell] targets. So the whole feature is: add one more synthetic cursor source.

Concretely:

1. New module e.g. src/ui/gesture/gamepad-source.ts: on each animation frame, poll navigator.getGamepads(), pick a device by name hint, read the lever into a direction vector, velocity-integrate it into a normalized {x,y} in [0,1], and compute an engaged boolean. Expose the current {x, y, engaged, connected}.
2. In GestureLayer.tsx, when a gamepad is present (gate behind a new &joystick=1 URL param, added in src/ui/url-params.ts next to the existing gesture/fusion/hands/dwell parsing), write that cursor into the cursors map under a reserved id (e.g. -2, and make it sticky / exempt from the STALE_SECONDS eviction like the mouse cursor). The existing dwell ring + cursor-dot rendering then works for free.

Port these constants/algorithms verbatim from gesture-wall/gesturewall/arcade.py:

• Device auto-select — PREFERRED_DEVICE_HINTS = ("8bitdo","arcade","80fe","switch pro","pro controller") and pick_joystick_index() (prefer a hint match, else first pad).
• Lever reading — lever_direction(): in Switch mode the 8BitDo reports the lever as the four D-pad buttons DEFAULT_DPAD_BUTTONS = (up=11, down=12, left=13, right=14); fall back to analog axes 0/1 (with apply_deadzone, default deadzone 0.4) or a hat. Note screen convention +y is down (pygame hat +y is up, so negate).
• Velocity integration — integrate_cursor(): pos += direction * speed * dt, clamp to [0,1]; default speed = 0.9 (fraction of the wall per second). There is no absolute position on a stick, so it must be integrated — hold the lever and the cursor glides.
• Engage — is_engaged(): with the default engage_button = -1, any button except the four lever/D-pad buttons engages the pointer (equivalent to raising a hand in pose mode); holding it fills the dwell ring and clicks.

Single-wall mode

Must work in single-wall mode (run-room.sh --single, URL ?live=1&view=... with no wall=). Since each wall window reads the gamepad independently and browsers only deliver gamepad input to the focused document, only the focused wall responds — which naturally prevents a two-wall double-click. Verify the &joystick=1 param flows through the single-wall URL path in run-room.sh (the URL_SINGLE line) as well as URL_A / URL_B.

Device setup notes (from arcade.py docstring, verified on macOS)

• Set the stick's mode switch to S (Switch); macOS then exposes it via SDL/HID as a "Nintendo Switch Pro Controller" and the lever comes through as the D-pad buttons. X/XInput mode is Windows-only and won't enumerate on macOS.
• Browser Gamepad API requires a user gesture (button press) before the pad appears, and the page must be focused/visible.

Acceptance criteria

• With the stick connected and &joystick=1 on a wall URL, moving the lever glides a cursor dot on the real room UI; holding an action button dwell-clicks the control/scene node under it — no pygame window, no camera.
• Works on both walls and in --single.
• Falls back cleanly (no errors) when no gamepad is connected.

References

• gesture-wall/gesturewall/arcade.py — port the lever math (lever_direction, integrate_cursor, is_engaged, pick_joystick_index, the hint/dpad constants).
• src/ui/gesture/GestureLayer.tsx — cursor map + dwell loop (integration point).
• src/ui/gesture/core.ts — DwellSelector (dwell-to-toggle with sticky hysteresis).
• src/ui/url-params.ts — add &joystick=1 parsing.
• run-room.sh — thread &joystick=1 into URL_A / URL_B / URL_SINGLE.