Merge pull request #14 from freshtechbro/codex/desktop-runtime-sibling-plane

fix: stabilize provider fallback recovery and shopping outputs

Author: freshtechbro

README.md

@@ -243,6 +243,7 @@ Use `--output-format json|stream-json` for automation-friendly output.
 - Direct browser, `/ops`, and provider fallback paths now share one bounded challenge orchestration plane. It can try auth navigation, legitimate session or cookie reuse, non-secret field fill, and bounded interaction exploration before yielding to a human.
 - Workflow and manager callers can set `challengeAutomationMode` to `off`, `browser`, or `browser_with_helper`. Effective precedence is `run > session > config`, and hard gates still apply after resolution.
 - The optional helper bridge is browser-scoped, not a desktop agent. `browser` forces it to stand down, and `browser_with_helper` only evaluates it after the existing helper hard gates pass.
+- Shipped builds also compose an internal sibling desktop observation runtime under separate `desktop.*` config. It is permission-off by default, does not widen `/ops` or `ChallengeRuntimeHandle`, and its shipped non-public composed path routes desktop observation back through browser-owned review.
 - Browser fallback returns explicit transport `disposition` values: `completed`, `challenge_preserved`, `deferred`, or `failed`. When orchestration runs during fallback, decision evidence is recorded under `details.challengeOrchestration`.
 - `ProviderRegistry` is the only durable anti-bot pressure authority. Shared runtime and policy own fallback ordering and resume policy; provider modules only contribute extraction logic and `recoveryHints()`.
 - In scope: preserved sessions, normal browser controls, bounded interaction experimentation, human yield packets for secret or human-authority boundaries, and owned-environment fixtures that use vendor test keys only.
@@ -258,6 +259,7 @@ Use `--output-format json|stream-json` for automation-friendly output.
 - **Canvas token authoring and adapter-plugin validation are now first-class**: the extension token panel edits collections, modes, aliases, and bindings, while `scripts/canvas-competitive-validation.mjs` captures grouped evidence for adapters, token round-trip, inbox delivery, surface parity, and optional live Figma smoke.
 - **Canvas surface governance and skill-pack coverage** now include current `/canvas` inventories, handshake/blocker templates, and feedback-evaluation artifacts.
 - **Challenge automation override control is now first-class** across workflows and manager metadata via `challengeAutomationMode` (`off|browser|browser_with_helper`) with `run > session > config` precedence and a browser-scoped helper boundary.
+- **Desktop observation now ships as an internal sibling runtime** with separate `desktop.*` config, repo-local audit artifacts, a non-public core observation-plus-review path, and no public desktop CLI or `/ops` plane yet.
 - **Release packaging/docs were refreshed for v0.0.17**, including current tarball examples, extension version sync, release evidence, and public/private cutover guidance.
 
 ### v0.0.16
@@ -618,6 +620,15 @@ Optional config file: `~/.config/opencode/opendevbrowser.jsonc`
     }
   },
 
+  // Internal sibling desktop observation runtime (permission-off by default)
+  "desktop": {
+    "permissionLevel": "off",
+    "commandTimeoutMs": 10000,
+    "auditArtifactsDir": ".opendevbrowser/desktop-runtime",
+    "accessibilityMaxDepth": 2,
+    "accessibilityMaxChildren": 25
+  },
+
   // Skills configuration
   "skills": {
     "nudge": {

docs/ARCHITECTURE.md

@@ -73,13 +73,15 @@ Legitimacy boundary:
 - `BrowserManager` and `OpsBrowserManager` remain the only surfaced challenge metadata writers.
 - `meta.challengeOrchestration` and fallback `details.challengeOrchestration` can expose `mode`, `source`, `standDownReason`, and helper eligibility so stand-down decisions stay explicit.
 - The optional helper bridge is browser-scoped, not a desktop agent. `browser` disables it, while `browser_with_helper` only evaluates it when the existing hard gates pass.
+- Shipped builds keep desktop entitlement separate under `desktop.*`; that sibling runtime is never granted by `challengeAutomationMode`.
 - Governed advanced lanes stay separately entitlement-gated and are never granted by `challengeAutomationMode`.
 
 ### Roadmap-only desktop boundary
 
-This section is roadmap-only and non-shipping.
+This section is roadmap-only for any public desktop-agent claim. Shipped builds now include an internal sibling desktop observation runtime plus a top-level automation coordinator, but they remain non-public, observation-only, and permission-off by default.
 
-- A future desktop agent must use a new runtime contract separate from `ChallengeRuntimeHandle`.
+- The shipped internal runtime already uses a separate contract from `ChallengeRuntimeHandle`, `BrowserManagerLike`, and `/ops`; any future public desktop agent must preserve that separation.
+- Core composition creates `desktopRuntime` beside `BrowserManager` and `OpsBrowserManager`, then exposes a non-public `observeDesktopAndVerify` entrypoint that routes desktop observation back through browser-owned review before surfacing completion.
 - Minimum capability bar before any desktop-agent claim is allowed:
   - OS-level input actuation outside the browser
   - cross-window and cross-app focus management
@@ -88,7 +90,7 @@ This section is roadmap-only and non-shipping.
   - bounded workspace and abort controls
   - audit artifacts and replay-safe execution logs
   - a typed failure taxonomy separate from the current helper bridge
-- Until that runtime exists, public docs and surfaces must not describe the current helper bridge as a desktop agent.
+- Until a public desktop plane exists, public docs and surfaces must not describe the current helper bridge as a desktop agent or imply that `/ops` is a desktop control channel.
 
 ---
 
@@ -108,7 +110,8 @@ This section is roadmap-only and non-shipping.
          ▼                  ▼                  ▼                       ▼
 ┌─────────────────────────────────────────────────────────────────┐
 │                    Core Runtime (src/core/)                    │
-│  bootstrap.ts → wires managers, injects ToolDeps              │
+│  bootstrap.ts → wires managers, sibling desktop runtime,      │
+│                   automation coordinator, injects ToolDeps     │
 └────────┬────────────────────────────────────────────────────────┘
          │
     ┌────┴────┬─────────────┬──────────────┬──────────┬────────────┬────────────┬────────────┐

docs/CLI.md

@@ -163,6 +163,7 @@ Canonical inventory document: `docs/SURFACE_REFERENCE.md`.
 - `ProviderRegistry` is the only durable anti-bot pressure authority used by policy, runtime routing, and workflow summaries. Provider modules only contribute extraction logic and optional `recoveryHints()`.
 - Direct browser, `/ops`, and provider fallback flows share one bounded challenge plane. It can try auth navigation, legitimate session or cookie reuse, non-secret field fill, and bounded browser-native interaction experimentation before yielding.
 - The optional helper bridge is browser-scoped, not a desktop agent. `browser` keeps it disabled and `browser_with_helper` only evaluates it after the existing hard gates pass.
+- Separate `desktop.*` config controls the shipped internal sibling desktop observation runtime. It stays permission-off by default, adds no public CLI command family, is never enabled by `challengeAutomationMode`, and only the internal core entrypoint composes observation with browser review.
 - Provider and workflow auto-resume still happen only after manager-owned verification clears the blocker.
 - In scope: preserved sessions, visual observation loops, low-level pointer controls, bounded interaction experimentation, reclaimable human yield packets, and owned-environment fixtures that use vendor test keys only.
 - Out of scope: hidden bypass paths, CAPTCHA-solving services, challenge token harvesting, or autonomous unsandboxed solving of third-party anti-bot systems.
@@ -1628,6 +1629,13 @@ When using `--with-config`, a `opendevbrowser.jsonc` is created with documented
       "ytdlpTimeoutMs": 10000
     }
   },
+  "desktop": {
+    "permissionLevel": "off",
+    "commandTimeoutMs": 10000,
+    "auditArtifactsDir": ".opendevbrowser/desktop-runtime",
+    "accessibilityMaxDepth": 2,
+    "accessibilityMaxChildren": 25
+  },
   "daemonPort": 8788,
   "daemonToken": "auto-generated-on-first-run",
   "flags": [],

docs/SURFACE_REFERENCE.md

@@ -294,6 +294,7 @@ Envelope contract:
 - Effective precedence is `run > session > config`.
 - Shipped config defaults resolve to helper-capable posture: `mode=browser_with_helper` and `optionalComputerUseBridge.enabled=true`.
 - The optional helper bridge stays browser-scoped and is not a desktop agent.
+- Separate `desktop.*` config gates the shipped internal sibling desktop observation runtime, but no public desktop CLI, tool, or `/ops` family is exposed and browser review remains the surfaced truth.
 - Provider browser fallback uses explicit transport `disposition` values: `completed`, `challenge_preserved`, `deferred`, and `failed`, and may include `details.challengeOrchestration` when the shared challenge plane ran during fallback.
 - `ProviderRegistry` is the only durable anti-bot pressure authority. Workflow outputs keep their existing keys while reading registry-backed pressure instead of provider-local durable state.
 

docs/privacy.md

@@ -19,6 +19,7 @@ The extension:
 - Does NOT use analytics or telemetry
 - May access page URLs, titles, and page content locally when you use automation or annotation features
 - May honor a local `challengeAutomationMode` setting (`off`, `browser`, or `browser_with_helper`) so bounded browser challenge handling can stand down or proceed on your machine without sending challenge state to OpenDevBrowser-operated services
+- May, if you explicitly enable `desktop.permissionLevel=observe`, capture local desktop or window screenshots plus accessibility snapshots on-device and write repo-local audit artifacts under `.opendevbrowser/desktop-runtime`
 - May store relay settings and the last user-triggered annotation payload locally on-device so the popup can reconnect and reopen recent annotation results
 - May store screenshot-free annotation payloads in a repo-local shared inbox when you explicitly use popup/canvas/in-page `Send` actions so the active chat for that worktree can consume them, or so the payload can be retrieved later when safe chat scoping is unavailable
 - May keep extension-hosted canvas stage annotation selections, region metadata, and optional local crop references on-device only when you explicitly capture or send them during a canvas session
@@ -37,7 +38,7 @@ The extension operates entirely on your local machine:
 
 5. **Local Storage**: The `storage` permission stores your relay configuration (port, pairing token, pairing toggle) and the last annotation payload metadata locally in Chrome. When you explicitly capture or send annotation results, the extension can also persist a local copy of the last annotation payload without screenshots so the popup can reopen it. If you explicitly use a `Send` action, OpenDevBrowser can also write a screenshot-free copy into `.opendevbrowser/annotate/agent-inbox.jsonl` in the current worktree so the intended active chat can consume it, or so the payload can be retrieved later with `annotate --stored` when safe chat scoping is unavailable. This data stays local to your machine and repository.
 
-Challenge automation evaluation also stays local. The optional helper bridge remains browser-scoped and is not a desktop agent.
+Challenge automation evaluation and the internal desktop observation runtime also stay local. The optional helper bridge remains browser-scoped and is not a desktop agent.
 
 ## Data Flow