Merge branch 'Th0rgal:master' into master

Author: adrienlacombe

.gitignore

@@ -32,6 +32,14 @@ secrets.json
 .open_agent_calibration/
 /tmp/
 /dashboard/console.log
+/.playwright-console.log
+/automations-*.png
+/control-bubbles.png
+/lifetime-cost-fixed.png
+/msg-feedback-failed.png
+/stats-responsive-*.png
+/thoughts-history-*.png
+/unified-bubbles.png
 
 # Claude Code per-project session state
 .claude/

ASK_ASSISTANT_DESIGN.md

@@ -0,0 +1,194 @@
+# Ask Assistant — Design
+
+> A fast, read-mostly **sidecar assistant** that rides alongside a running
+> mission. It can read the full mission history, run bash in the live workspace,
+> and answer questions — **without interrupting the main agent's turn loop or
+> queue**. Powered by a configurable fast model (default: Cerebras
+> `gpt-oss-120b`).
+
+Status: **M1 backend implemented** (store + client + loop + HTTP routes,
+compiles, storage unit-tested). M2–M5 pending.
+Owner: Thomas
+Related: conversation-anchored snapshot (#477, #482), thinking capture (#481),
+Cerebras title fix (#489), `metadata_llm.rs` provider ladder.
+
+---
+
+## 1. Goals & non-goals
+
+### Goals
+- Let the operator **chat about what a mission is doing** at any time, in a
+  separate lane that never touches the harness lock, the message queue, or the
+  mission transcript fed back to the working agent.
+- Give the assistant **real capability**: full read+write `bash` in the live
+  workspace plus history retrieval.
+- **Persistent, browsable Ask threads** per mission, clearable, **never merged
+  into mission history** (the working agent cannot see the Ask conversation).
+- A **dedicated, configurable Ask model** (separate from the metadata model),
+  optimized for *smart + fast + large context*.
+- One **quiet "operator-note"** bridge: when Ask **writes** to the workspace, the
+  working agent gets a passive heads-up on its next turn.
+- Works across web, iOS, and Android (shared API).
+
+### Non-goals (v1)
+- No worktree / sandbox-copy mode (usage is read-dominant).
+- No destructive-write confirmation gate. Full bash, no friction; a visual
+  "agent idle/working" indicator is informational only.
+- ~~Ask never **starts** a main-agent turn. Strictly non-interrupting.~~
+  **Superseded (v1.1):** the Copilot now has explicit steering tools —
+  `stop_agent` (CancelMission, same as the Stop button) and `send_to_agent`
+  (a real composer-equivalent UserMessage, optionally interrupting the
+  current turn first). The system prompt frames these as interventions to
+  use when the operator asks to stop/steer, or in clearly harmful loops;
+  the default posture stays read-mostly.
+- Not every harness *heeds* the operator-note identically — delivery is
+  harness-agnostic, faithfulness is best-effort (§9).
+
+---
+
+## 2. Concept & lanes
+
+Two lanes share one workspace and one mission identity, otherwise isolated:
+
+- **Driver lane** — the working agent: `mission_events`, history, harness lock,
+  message queue.
+- **Co-pilot lane** — the Ask assistant: `ask_threads` / `ask_messages` (separate
+  `ask.db`), no lock, no queue, its own response path.
+
+Hard rule: **nothing ever reads `ask_*` rows into the working agent's prompt.**
+The only cross-lane bridges are the operator-note (Ask write → working agent,
+passive) and "Send to agent" (manual: Ask answer → real composer).
+
+---
+
+## 3. Data model (implemented)
+
+Separate `ask.db` (rusqlite), keyed by `mission_id` (ownership enforced at the
+HTTP layer via the user's per-user mission store).
+
+- `ask_threads(id, mission_id, title, model, created_at, updated_at)`
+- `ask_messages(id, thread_id, seq, role, content, tool_name, tool_call_id,
+  metadata, created_at)` — `role ∈ {user, assistant, tool_call, tool_result}`,
+  FK → `ask_threads ON DELETE CASCADE`
+- `operator_notes(id, mission_id, body, source_thread_id, created_at,
+  flushed_at)` — the M2 bridge buffer
+
+Store: `src/api/ask/store.rs` (`AskStore`). Global singleton via tokio
+`OnceCell` at `<working_dir>/.sandboxed-sh/missions/ask.db`.
+
+---
+
+## 4. Backend: the Ask agentic loop (implemented)
+
+`src/api/ask/` — net-new, small, in-process:
+- `store.rs` — persistence (above).
+- `client.rs` — OpenAI-compatible chat client **with tool-calling** (reuses
+  `MetadataLlmConfig`; forwards `reasoning_effort` so reasoning models return
+  visible content).
+- `mod.rs` — `run_ask_turn`: persist user msg → seed system prompt (recent
+  events + summary) → tool loop (≤6 iterations) → persist assistant answer.
+- `http.rs` — handlers.
+
+Tools (read-on-demand so the 128K context ceiling never bites):
+- `bash(command)` → `WorkspaceExec::output(work_dir, "/bin/bash", ["-lc", cmd])`
+  — **full** read+write, live workspace, host or nspawn. No gate.
+- `read_history(limit)` → `MissionStore::get_latest_events`.
+- `read_file(path)` → bash `cat`.
+
+---
+
+## 5. Model configuration (implemented)
+
+`metadata_llm.rs`:
+- Lifted `resolve_provider_api_key` to module scope.
+- Added `build_assistant_llm_config()` → prefers Cerebras `gpt-oss-120b`
+  (`reasoning_effort=low`), overridable via `ASK_ASSISTANT_MODEL` env; falls back
+  to the metadata provider ladder.
+
+Cerebras caps context at ~128K regardless of model — seed recent window +
+summary, let the model pull more via grep/`read_history`. (M3 adds a Settings
+picker with a `supports_tools` capability flag.)
+
+---
+
+## 6. HTTP API (implemented)
+
+Routes (protected) in `routes.rs`, handlers in `src/api/ask/http.rs`:
+
+```
+POST   /api/control/missions/:id/ask                 send (returns final answer + messages)
+GET    /api/control/missions/:id/ask/threads         list threads
+GET    /api/control/missions/:id/ask/threads/:tid    thread + messages
+DELETE /api/control/missions/:id/ask/threads/:tid    clear/delete thread
+```
+
+These never call `queue_message` or acquire the harness lock.
+
+> M1 returns the final answer synchronously (JSON). Token-level **SSE streaming**
+> (`AgentEvent::Ask*` variants on a separate lane) is folded into M3 when the web
+> panel needs it.
+
+---
+
+## 7. Context strategy (the 128K ceiling)
+
+Seed: recent mission events + summary + the thread's prior messages. Then the
+model pulls more via `read_history` / `bash grep`. Keeps every request under
+128K and scales to arbitrarily long missions.
+
+---
+
+## 8. The quiet operator-note (M2)
+
+Delivery is harness-agnostic; faithfulness is best-effort.
+
+- **Detect** an Ask write by wrapping `bash`: `git status --porcelain` snapshot
+  before/after (non-git fallback deferred to M5). On change →
+  `enqueue_operator_note`.
+- **Deliver** by prepending a `<operator-note>` block into the working agent's
+  **next** `user_message` (all backends take the next turn as a single string:
+  Claude Code `--` arg, OpenCode prompt file, Codex `send_message_streaming`).
+  Centralized helper `prepend_pending_operator_notes(mission_id, user_message)`.
+- **Invariant:** notes are passive — flushed only when a turn is already about to
+  run. Ask can never wake the working agent.
+- **Audit:** a flushed note records an `operator_note` mission event (the *fact*
+  of a write touches the mission; the *chat* does not).
+
+Compatibility: all backends *see* the note; Claude Code *heeds* `<…>`-tagged
+blocks best.
+
+---
+
+## 9. Frontend — web (M3)
+
+Right-side collapsible **Ask panel**: thread switcher, co-pilot styling (distinct
+from the indigo Bot), tool-call rendering, slim composer, informational
+"agent idle/working" banner, per-item "ask about this" spark, "Send to agent"
+bridge. `Ask` button beside Queue/Stop + a keyboard toggle (not a Cmd+W variant).
+
+## 10. Frontend — iOS / Android (M4)
+
+Same endpoints. iOS: `.sheet` with medium/large detents. Android: bottom sheet.
+
+---
+
+## 11. Milestones
+
+- **M1 — Backend core** ✅ tables + store (unit-tested), Ask loop + tools,
+  `build_assistant_llm_config`, routes. Synchronous JSON responses.
+- **M2 — Operator-note bridge** — git-porcelain detection +
+  `prepend_pending_operator_notes` in all backend turn-prep + audit event.
+- **M3 — Web panel** — Ask store + (optional SSE), drawer, picker, spark, bridge.
+- **M4 — Mobile** — iOS sheet + Android bottom sheet.
+- **M5 — Polish** — per-thread cost, auto-titling, non-git write detection,
+  sandbox-copy mode.
+
+---
+
+## 12. Verification
+
+- M1 storage: `cargo test --lib ask::store` (thread/message roundtrip, cascade
+  delete, operator-note take-once) — passing.
+- M1 end-to-end: deploy to **dev**, then
+  `curl -X POST /api/control/missions/<id>/ask -d '{"content":"what is the agent doing?"}'`
+  with a Cerebras key configured. (Pending live run.)

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "sandboxed_sh"
-version = "0.12.0"
+version = "1.3.0"
 edition = "2021"
 rust-version = "1.91"
 description = "Self-hosted orchestrator for AI coding agents (formerly Open Agent)"
@@ -106,6 +106,10 @@ path = "src/bin/automation_manager_mcp.rs"
 name = "orchestrator-mcp"
 path = "src/bin/orchestrator_mcp.rs"
 
+[[bin]]
+name = "assistant-mcp"
+path = "src/bin/assistant_mcp.rs"
+
 [[bin]]
 name = "fido-agent-proxy"
 path = "src/bin/fido_agent_proxy.rs"

Cargo.toml

@@ -9,12 +9,42 @@
 
 ## Backend Services (Thomas)
 
-Thomas's server runs two instances:
+Thomas's server runs two sandboxed.sh instances plus the Hermes assistant stack:
 
 | Service | Port | Domain | Binary |
 |---------|------|--------|--------|
 | `sandboxed-sh-prod` | 3000 | agent-backend.thomas.md | `/usr/local/bin/sandboxed-sh-prod` |
 | `sandboxed-sh-dev` | 3002 | agent-backend-dev.thomas.md | `/usr/local/bin/sandboxed-sh-dev` |
+| `hermes-assistant` | 8642 (loopback) | agent-backend.thomas.md `/hermes-remote` | `/usr/local/bin/hermes` (gateway) |
+| `hermes-dashboard` | 9130 (loopback) | agent-backend.thomas.md `/hermes-desktop` | `/usr/local/bin/hermes` (dashboard) |
+
+**Hermes endpoints** (nginx config in
+`/etc/nginx/sites-enabled/agent-backend.thomas.md` — NOT tracked in git):
+
+- `/hermes-remote` → the gateway's OpenAI-compatible API server (loopback 8642),
+  re-exposed by the sandboxed.sh backend itself (`hermes_remote_proxy` in
+  `src/api/system.rs`). `/v1/*` + SSE only, no WebSocket. Used for *split mode*
+  (a local Hermes sets `GATEWAY_PROXY_URL`/`GATEWAY_PROXY_KEY` and delegates
+  agent work). Key = `API_SERVER_KEY` in `/etc/sandboxed-sh/hermes-assistant.env`,
+  readable via `GET /api/system/hermes-assistant/remote`.
+- `/hermes-desktop` → the full Hermes dashboard backend (loopback 9130) for the
+  desktop app's **native Remote-gateway mode** (`/api/status` + `wss /api/ws`).
+  Unit: `/etc/systemd/system/hermes-dashboard.service` (`hermes dashboard
+  --no-open --host 127.0.0.1 --port 9130 --skip-build`, same `HERMES_HOME`
+  `/var/lib/hermes-assistant` as the gateway so sessions/memory are shared,
+  auth via `HERMES_DASHBOARD_SESSION_TOKEN` = same key, stub web dist at
+  `/var/lib/hermes-assistant/web_dist`). The nginx location pins
+  `Host 127.0.0.1:9130` (host-header check) and must NOT add `X-Forwarded-For`
+  (the WS gate requires a loopback peer).
+
+Hermes gotchas:
+
+- After a prod deploy, `systemctl restart hermes-assistant` so the gateway
+  respawns the freshly installed `/usr/local/bin/assistant-mcp`.
+- When testing the WS handshake with curl, force `--http1.1` — ALPN negotiates
+  h2, which has no `Upgrade` header, and the resulting 401 is a red herring.
+- When rotating the `hsk_` key, update `API_SERVER_KEY`,
+  `HERMES_DASHBOARD_SESSION_TOKEN`, and the desktop's `connection.json` together.
 
 ```bash
 # Production
@@ -175,8 +205,26 @@ auto-resumes from the backend but loses the in-flight turn's progress.
 Worse, an agent in a retry loop can chainsaw the host (the original
 incident: 5 deploys + 2 rollbacks in 90 minutes).
 
-The orchestrator MCP exposes a `deploy_sandboxed_sh` tool that hits the
-internal `/api/system/deploy` endpoint with the safety rails:
+The orchestrator MCP exposes a `deploy_sandboxed_sh` tool that hits an
+internal `/api/system/deploy` endpoint with safety rails.
+
+**Targeting rule:** `deploy_sandboxed_sh` deploys the backend API it is sent
+to. If `target_environment` is omitted, the tool uses the API that launched the
+mission. A prod mission therefore deploys prod by default; a dev mission deploys
+dev by default. To deploy a different Sandboxed.sh environment, set
+`target_environment` explicitly:
+
+| Target | Tool argument | API URL | Service restarted |
+|--------|---------------|---------|-------------------|
+| Current mission API | omit `target_environment` | mission `API_URL` | current API's service |
+| Production | `"prod"` | `http://127.0.0.1:3000` | `sandboxed-sh-prod.service` |
+| Development | `"dev"` | `http://127.0.0.1:3002` | `sandboxed-sh-dev.service` |
+
+The backend also receives an `expected_service` guard from the MCP and refuses
+with HTTP 409 if the request reaches the wrong service. This prevents a prod
+mission that intends to test dev from accidentally deploying prod.
+
+Safety rails:
 
 - **Self-protection** — refuses by default if the calling mission lives on
   the service being restarted. Override with `force=true` only if you
@@ -193,6 +241,7 @@ From inside an agent that has the orchestrator MCP, the call shape is:
 {
   "tool": "deploy_sandboxed_sh",
   "arguments": {
+    "target_environment": "dev",  // optional; "dev" or "prod"; omit for current API
     "git_ref": "origin/master",   // optional; omit to use current checkout
     "skip_build": false,          // optional; true if you built elsewhere
     "force": false                // override safety rails
@@ -210,9 +259,22 @@ Successful response:
 }
 ```
 
-A refusal (HTTP 409 self-target or 429 debounce) surfaces a message that
-explains the override knob, so an agent can decide whether to retry with
-`force=true` or back off.
+A refusal surfaces a message that explains the next step:
+
+- HTTP 409 target mismatch: the request reached the wrong service/API URL.
+- HTTP 409 self-target: pass `force=true` only if killing the caller is acceptable.
+- HTTP 429 debounce: wait or pass `force=true` for an urgent deploy.
+
+For a dev-only restart without rebuilding, use the explicit service command:
+
+```bash
+ssh -i ~/.ssh/cursor root@95.216.112.253 "systemctl restart sandboxed-sh-dev"
+```
+
+Do not call `deploy_sandboxed_sh` from a prod mission unless you intend to
+deploy a Sandboxed.sh backend. For dev testing from prod, prefer
+`target_environment: "dev"` so the request goes to the dev API and carries the
+`sandboxed-sh-dev.service` guard.
 
 ## Locking down the SSH backdoor
 
@@ -374,15 +436,15 @@ Then restart the backend. Workspace-scoped MCPs must be in PATH for both host wo
 
 **Config profile not being applied:** Check that:
 1. The configs exist in the **correct library path** (`.sandboxed-sh/library/` for production, not `.openagent/library/` which was the old path)
-2. The library has the `configs/<profile>/.opencode/oh-my-opencode.json` file
+2. The library has the expected `configs/<profile>/.opencode/` files
 3. Pull latest library: `cd ~/.sandboxed-sh/library && git pull`
 
 **Missions not using correct settings:**
 
 **Missions stuck:** Look for running CLI processes:
 
 ```bash
-ps aux | grep -E "claude|oh-my-opencode"
+ps aux | grep -E "claude|opencode"
 machinectl list   # For container workspaces
 ```