Local control: in-SDK sidecar for user_device environments

[abonneth]

Note

Medium Risk
New subprocess/network loop with API credentials and optional desktop/browser control increases attack and operational surface, but behavior is gated by user_device envs, optional extras, and HAI_AUTO_SIDECAR with broad unit test coverage.

Overview
Adds local control so agents with host: user_device environments can drive this machine’s browser or desktop via an in-process sidecar, instead of relying only on remote sandboxes.

The SDK now rewrites agent/session payloads: user_device envs get a deterministic session_id (from API key + env id + capability) so the cloud can route commands to the right channel. Client / AsyncClient use wrapped agents and sessions clients that apply this wiring on create/update/patch and, when HAI_AUTO_SIDECAR is on (default), start sidecars before create_session (including after fetching a named agent).

New hai_agents.local runs background sidecars that long-poll the API, execute driver methods through hai-drivers (Selenium + optional Chrome launch for browser; desktop extra for local desktop), and post results—with retries, idempotent command_uid caching, and one browser / one desktop per machine (in-process and cross-process lock files).

pip install 'hai-agents[browser]' / [desktop] pull in hai-drivers; the CLI adds hai local browser / hai local desktop for manual sidecars when sessions start elsewhere.

Tests cover wiring, auto-start, protocol, and runtime rollback; integration tests disable auto sidecars by default.

^{Reviewed by Cursor Bugbot for commit edc1765. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Lock omits hai-drivers package

High Severity

Optional browser and desktop extras declare hai-drivers, but uv.lock still resolves direct selenium/pyautogui pins and never installs hai-drivers, while SidecarClient._build_driver imports hai_drivers.

Additional Locations (1)

• uv.lock#L154-L205

 

^{Reviewed by Cursor Bugbot for commit fa3ff61. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b2762ab. Configure here.}

[cursor]

Named agent fetch skips resolve

Medium Severity

When create_session is given a catalog agent name and auto sidecars are enabled, the SDK loads the definition with get_agent but does not pass resolve=true. Unresolved string subagent or environment leaves never expose user_device configs, so collect_sidecar_configs can skip required local sidecars while the session still starts.

Additional Locations (1)

• src/hai_agents/client.py#L107-L110

 

^{Reviewed by Cursor Bugbot for commit b2762ab. Configure here.}