feat(pointer): add typed mouse input and animated recording overlays

.changeset/robust-terminal-evidence.md

@@ -0,0 +1,5 @@
+---
+"@kitlangton/terminal-control": patch
+---
+
+Preserve color-query replies when terminal escape prefixes arrive across separate output chunks, exclude invisible text when trimming video startup, and keep the original test failure when automatic failure-artifact capture fails. Simplify shared input encoding, terminal ownership, and recording rendering without changing public interfaces.

.changeset/smooth-pointer-input.md

@@ -0,0 +1,7 @@
+---
+"@kitlangton/terminal-control": minor
+---
+
+Add typed mouse input for hover, clicks, and drags through the CLI, Rust sessions, TypeScript client, and MCP. Encode input using the application's negotiated mouse reporting mode and validate current viewport coordinates.
+
+Add opt-in video pointer overlays with smooth travel, press feedback, fades, and reduced motion. Keep animation aligned through edits and resizes without changing real input timing. New recordings use format v2 for typed mouse evidence; current readers continue to accept v1 recordings.

.github/workflows/ci.yml

@@ -26,6 +26,6 @@ jobs:
       - run: TERMCTRL_TEST_BINARY="$PWD/target/release/termctrl" bun test packages/test/src/index.test.ts
       - run: bun run --cwd packages/opentui typecheck
       - run: bun run --cwd packages/opentui test
-      - run: bun run --cwd packages/opentui release:check
+      - run: bun run validate:opentui
       - run: bun run validate:npm "$PWD/target/release/termctrl"
       - run: cargo package

.gitignore

@@ -1,9 +1,9 @@
 /target
 /node_modules
 /packages/*/node_modules
-/packages/*/bin/cellshot
 /packages/*/dist
 /.cellshot-artifacts
+/.termctrl-artifacts
 /npm-artifacts
 /captures
 /docs/screenshots/*.ansi

AGENTS.md

@@ -23,6 +23,7 @@ cargo build --release
 bun run test:npm
 bun run build:npm
 bun run validate:npm
+bun run validate:opentui
 cargo package --list
 cargo package
 ```

CONTEXT.md

@@ -26,6 +26,21 @@ A timestamped terminal event timeline containing output, client or automatic hos
 
 Agents inspect marker names with `termctrl markers` and inspect exact recording moments with `termctrl show --recording ... --at-marker ...` or `--at-ms ...` before committing to a video edit plan.
 
+### Mouse Input And Pointer Overlay
+
+Mouse input is a real, typed action (`move`, `down`, `up`, or `click`) at zero-based terminal cell
+coordinates. The embedded session validates its current viewport and held button, and Ghostty
+encodes the action using the application's negotiated mouse protocol. A `move` without a held
+button is a hover; `down`/`move`/`up` is a drag. Disabled reporting fails rather than injecting
+escape sequences into an unsuspecting shell. Clicks send press/release together with no added delay.
+
+A pointer overlay is an opt-in video presentation of successfully delivered typed mouse input,
+not a terminal cell or the text cursor. Format v2 recordings store each typed mouse event and its
+actual bytes on the recording clock; readers retain v1 support. Raw input is not reverse-engineered
+into mouse events. Source-time sampling aligns animation with edited video and freezes it during
+holds. Reduced motion keeps opacity feedback without travel or press compression. Ordinary
+screenshots and screen text do not include the overlay.
+
 ### ANSI/VT Stream
 
 Raw terminal output bytes containing text and terminal control sequences. Files commonly use an `.ansi` suffix, but the suffix does not imply a separate container format.

README.md

@@ -47,7 +47,7 @@ filters for bounded server-side discovery.
 | MCP task | Tools |
 | --- | --- |
 | Discover | `list_sessions`, `get_session_status` |
-| Observe and drive | `get_screen`, `save_screen`, `send_input`, `interact`, `resize_session` |
+| Observe and drive | `get_screen`, `save_screen`, `send_input`, `send_mouse`, `interact`, `resize_session` |
 | End processes | `stop_session` |
 
 Then ask for terminal work in ordinary language:
@@ -110,6 +110,7 @@ termctrl stop demo
 ```
 
 - `send` accepts `text:<value>`, named keys (`enter`, `escape`, arrows, `tab`, `shift-tab`, `backspace`, `delete`, `home`, `end`, `page-up`, `page-down`), and `ctrl-a` through `ctrl-z`. Pipe exact bytes with `--stdin`.
+- `mouse demo move 12 4` hovers at zero-based column 12, row 4; `mouse demo click 12 4` clicks. Use `down`, `move`, then `up` to drag; `--button right`, `--shift`, `--alt`, and `--ctrl` are supported. The app must enable mouse reporting (hover needs any-event tracking).
 - `wait` blocks until text is visible; use it instead of sleeping. It defaults to five seconds.
 - `status` reports running/exited state, command, cwd, viewport, and recording path. `list` shows running sessions; use `--state`, `--command`, or `--cwd` to filter discovery, and `--all` to include every retained or unavailable entry.
 - `prune --dry-run` previews retained exited sessions and stale sockets; `prune` removes them without deleting recording artifacts.
@@ -193,6 +194,29 @@ An edit plan selects marker ranges with per-clip speed, captions, and optional e
 
 Without `--edit`, export preserves recorded timing. `--footer` renders captions, timecode, and branding in a bottom bar. `--tail-ms 0` removes the default one-second final hold. Keep speeds low enough for text to stay readable.
 
+### Show Mouse Interactions
+
+Mouse visualization is opt-in at export, independent of the terminal's text cursor:
+
+```bash
+termctrl start pointer-demo --record captures/pointer.termctrl -- my-mouse-enabled-tui
+termctrl wait pointer-demo "Ready"
+termctrl mouse pointer-demo move 12 4
+termctrl mouse pointer-demo click 12 4
+termctrl stop pointer-demo
+termctrl video captures/pointer.termctrl --pointer-overlay --out captures/pointer.mp4
+```
+
+The overlay uses a neutral outlined arrow, smooth 220ms travel arriving at the input instant,
+subtle press compression, and short fades. Pointer exports default to 60 fps (otherwise 20);
+`--fps` overrides either default. `--pointer-reduced-motion` keeps fades without travel or scaling.
+Edits preserve alignment through cuts, speed changes, holds, and resizes. No animation changes
+the real input or adds delays. Only typed `mouse` operations provide pointer evidence; raw bytes
+and human input forwarded by `run` are not inferred. `show` and `save` remain unadorned terminal state.
+
+New recordings use format v2; current readers also accept v1. Older binaries must be updated to
+read v2 recordings. Restart older named sessions before using the new mouse command.
+
 Recordings are JSON Lines files containing terminal output and typed input; they can include prompts or secrets. Treat them as sensitive.
 
 ## Pipes And ANSI Streams

docs/driver-protocol.md

@@ -6,7 +6,7 @@ External agent tooling can keep multiple embedded sessions alive through a versi
 termctrl driver
 ```
 
-The driver writes a `hello` message with protocol and Terminal Control versions, then accepts typed operations: `launch`, `status`, `send`, `waitForText`, `waitForIdle`, `waitForExit`, `capture`, `logs`, `recording`, `resize`, `stop`, and `shutdown`. It is intended for clients such as the TypeScript test client, while the shell-facing flat commands remain convenient for individual workflows.
+The driver writes a `hello` message with protocol and Terminal Control versions, then accepts typed operations: `launch`, `status`, `send`, `mouse`, `waitForText`, `waitForIdle`, `waitForExit`, `capture`, `logs`, `recording`, `resize`, `stop`, and `shutdown`. It is intended for clients such as the TypeScript test client, while the shell-facing flat commands remain convenient for individual workflows.
 
 Each request has a numeric `id`, a `method`, optional `params`, and a `sessionId` for every method
 except `shutdown`. Each line receives exactly one response or error with the same `id`:
@@ -24,7 +24,7 @@ increment `protocolVersion`; clients reject unsupported versions rather than gue
 ## Example Requests
 
 ```json
-{"type":"hello","protocolVersion":2,"terminalControlVersion":"<installed-version>"}
+{"type":"hello","protocolVersion":2,"terminalControlVersion":"<installed-version>","capabilities":["mouse"]}
 {"id":1,"method":"launch","sessionId":"app","params":{"command":["my-terminal-app"],"cols":100,"rows":30,"inheritEnv":false,"env":{"TERM":"xterm-256color"}}}
 {"id":2,"method":"waitForText","sessionId":"app","params":{"text":"Ready","timeoutMs":5000}}
 {"id":3,"method":"send","sessionId":"app","params":{"input":[{"type":"text","value":"help"},{"type":"key","value":"enter"}]}}
@@ -44,3 +44,23 @@ Raw ANSI is omitted by default; request `includeAnsi: true` for retained transcr
 ## Input Semantics
 
 Driver input is intentionally exact: text, raw bytes, known key values, and single-letter control input are supported without claiming unimplemented key chords.
+
+`mouse` is an additive capability in protocol 2. Absent `hello.capabilities` means no advertised
+capabilities; clients must check for `"mouse"` before using it, and request a binary update and
+driver restart if absent. Existing request shapes are unchanged.
+
+```json
+{"id":5,"method":"mouse","sessionId":"app","params":{"action":"move","x":12,"y":4}}
+{"id":6,"method":"mouse","sessionId":"app","params":{"action":"click","x":12,"y":4,"button":"right","shift":true}}
+```
+
+Actions are `move`, `down`, `up`, `click`. Coordinates are integer zero-based cells within the
+current viewport. `button` defaults to `left` (`middle` and `right` also supported); `shift`, `alt`,
+and `ctrl` default to false. A move uses the session's currently held button. Only one button may
+be held; up must match it, and another down/click requires releasing it first. Ghostty respects
+negotiated reporting and encoding, including pixel coordinates. Disabled reporting, unsupported
+hover, invalid coordinates, and exited sessions fail without sending input. Success returns null.
+
+Typed mouse input is retained as a format-v2 `mouse` recording entry, including actual bytes;
+there is no duplicate `input` entry. The video pointer overlay is a later export choice, not a
+driver input option. Ordinary raw byte input is never inferred to be mouse input.

docs/releasing.md

@@ -26,6 +26,11 @@ bypass the package-set checks or publish package formats at different versions.
 
 ## Validate The Release
 
+`bun run validate:opentui` packs the current adapter into a temporary directory and checks its
+public exports from clean consumers against OpenTUI 0.4.1 and 0.4.5. It rejects version mismatch
+without rewriting manifests or attempting publication. To check an already assembled artifact,
+pass its directory: `bun run validate:opentui npm-artifacts`.
+
 Run the complete local validation from `AGENTS.md`, followed by the publishable crate checks:
 
 ```bash

docs/rust-library.md

@@ -35,6 +35,13 @@ session.stop()?;
 
 `session::Session` is the embedded lifecycle interface; the flat named-session CLI commands and the external driver are adapters over the same implementation.
 
+Use `session.mouse(MouseEvent::new(Action::Move, 12, 4))` to hover and `Action::Click` to click,
+with types from `terminal_control::mouse`. Coordinates are zero-based cells; `Down`/`Move`/`Up`
+form a drag. Reporting follows the application's negotiated protocol and fails if disabled.
+Set `recording::VideoOptions::pointer_overlay` to `Some(PointerOptions::default())` to visualize
+typed mouse input at export, or use `PointerOptions { reduced_motion: true }` for fades only.
+The overlay does not modify `Frame` or screen captures.
+
 To request optional application semantics, enable the OpenTUI host profile before launch and read
 the provider with the same embedded session:
 
@@ -58,5 +65,6 @@ No connected provider returns `termctrl-semantic-snapshot-v1` with an empty `nod
 ## Versioned Structured Output
 
 - A `save --format json` capture is a `Frame` object with `version: 2`, described by `schemas/frame-v2.schema.json`.
-- A `.termctrl` recording is JSON Lines: its first line is a versioned header and subsequent lines are timed output, input, resize, or marker entries, each described by `schemas/recording-entry-v1.schema.json`.
+- A `.termctrl` recording is JSON Lines: its first line is a version-2 header and subsequent lines are timed output, input, mouse, resize, or marker entries, described by `schemas/recording-entry-v2.schema.json`. Current readers also accept version 1 (`schemas/recording-entry-v1.schema.json`). Older binaries cannot read v2 recordings.
+- A `mouse` entry contains `at_ms`, the typed `event`, and delivered `bytes`. It is client input, not output, and replay never feeds it into the terminal emulator. Mouse timestamps must be nondecreasing.
 - Recording byte arrays contain the original terminal or input bytes as integers from `0` to `255`; recordings can contain sensitive text or input.

docs/typescript-client.md

@@ -60,6 +60,26 @@ console.log(capture.reason, capture.text, capture.frame)
 
 Keyboard presses are typed as the sequences Terminal Control encodes exactly, such as `"Enter"`, `"ArrowDown"`, or `"Control+C"`. Use `session.keyboard.write(bytes)` when a test deliberately needs exact terminal bytes outside that supported key set.
 
+## Mouse Input
+
+```ts
+await session.mouse({ action: "move", x: 12, y: 4 }) // hover
+await session.mouse({ action: "click", x: 12, y: 4 })
+await session.mouse({ action: "down", x: 12, y: 4 })
+await session.mouse({ action: "move", x: 20, y: 4 }) // drag
+await session.mouse({ action: "up", x: 20, y: 4 })
+```
+
+The exported `MouseEvent` type accepts `button: "left" | "middle" | "right"` (default left)
+and optional `shift`, `alt`, `ctrl` booleans. Positions are zero-based integer cells, validated
+against the current viewport. The app must enable mouse reporting; hover requires any-event
+tracking. A click is an immediate press/release. No visual-animation delays are added to tests.
+An older binary without the driver `mouse` capability produces an actionable update error.
+
+With recording enabled, typed mouse evidence can be visualized later with
+`termctrl video recording.termctrl --pointer-overlay --out demo.mp4`.
+Add `--pointer-reduced-motion` for fades without travel or press scaling. Screenshots stay clean.
+
 ## Vitest Matchers And Failure Evidence
 
 Standard `toMatchSnapshot()` and `toMatchInlineSnapshot()` remain the simplest snapshot format because visible text is reviewable in source control. A screen-aware assertion writes configured artifacts on failure:
@@ -76,6 +96,10 @@ await expect(session.screen.text()).resolves.toMatchInlineSnapshot()
 
 `session.writeArtifacts(name)` and failing `toHaveScreenText(...)` assertions can write `screen.txt`, `screen.json`, `screen.svg`, `logs.txt`, and `metadata.json`. Environment variable values are redacted in metadata. `transcript.ansi` and `recording.termctrl` are opt-in because terminal streams and typed input may contain secrets.
 
+`withArtifactsOnFailure` preserves the original failure if evidence capture itself fails, and
+prints a warning for the secondary artifact error. Direct `writeArtifacts` calls still reject
+when capture or writing fails.
+
 Wrap ordinary snapshot assertions when evidence should be saved on any thrown assertion:
 
 ```ts

package.json

@@ -11,6 +11,7 @@
     "changeset": "changeset",
     "version-packages": "changeset version",
     "validate:npm": "node scripts/validate-npm-packages.mjs",
+    "validate:opentui": "node scripts/validate-opentui-package.mjs",
     "release-packages": "node scripts/release-packages.mjs"
   },
   "devDependencies": {

packages/opentui/package.json

@@ -31,9 +31,7 @@
     "build": "rm -rf dist && tsc -p tsconfig.build.json",
     "test": "bun test src/index.test.ts",
     "typecheck": "tsc --noEmit",
-    "prepack": "npm run build",
-    "release": "node script/release.mjs",
-    "release:check": "node script/release.mjs --check"
+    "prepack": "npm run build"
   },
   "peerDependencies": {
     "@opentui/core": ">=0.4.1 <0.5.0"

packages/opentui/src/index.test.ts

@@ -72,6 +72,25 @@ test("omits controls hidden or clipped by their ancestors", async () => {
   setup.renderer.destroy()
 })
 
+test("keeps nested discovery order and descendant labels attached to their renderable", async () => {
+  const setup = await createTestRenderer({ width: 20, height: 5 })
+  try {
+    const parent = new BoxRenderable(setup.renderer, { id: "parent", width: 12, height: 3, onMouseDown: () => {} })
+    const child = new BoxRenderable(setup.renderer, { id: "child", width: 10, height: 1, focusable: true })
+    Reflect.set(child, "plainText", " Nested   action ")
+    parent.add(child)
+    setup.renderer.root.add(parent)
+    await setup.renderOnce()
+    expect(elements(setup.renderer).map((element) => element.num)).toEqual([parent.num, child.num])
+    expect(semanticSnapshot(setup.renderer).nodes.map(({ element, label }) => ({ element, label }))).toEqual([
+      { element: parent.num, label: "Nested action" },
+      { element: child.num, label: "Nested action" },
+    ])
+  } finally {
+    setup.renderer.destroy()
+  }
+})
+
 test("excludes hidden descendants from labels and reports clipped geometry", async () => {
   const setup = await createTestRenderer({ width: 20, height: 5 })
   const button = new BoxRenderable(setup.renderer, { id: "button", focusable: true, width: 5, height: 1 })