Skip to content

release: v0.4.14 — Android element extraction, rebuilt - #195

Merged
justrach merged 1 commit into
mainfrom
release/0.4.14
Jul 26, 2026
Merged

release: v0.4.14 — Android element extraction, rebuilt#195
justrach merged 1 commit into
mainfrom
release/0.4.14

Conversation

@justrach

Copy link
Copy Markdown
Owner

uitree reported what was in the uiautomator XML. What an agent needs is what can be acted on, named the way it reads on screen.

Prompted by a comparison against CursorTouch/Android-MCP. kuri already had roughly twice the command surface (27 vs 14 — they have no list-apps, uninstall, clear, openurl, keyevent, current-activity, logcat, screen-info, getprop, dumpsys, raw touch, multi-point gesture or batch), but their element handling was better. That gap is what this closes.

Bugs the comparison turned up

  • android tap --label never worked. The tool table advertised the flag and android tools printed it, but the dispatcher only read positional coordinates — every documented invocation failed with missing argument: x y.
  • No cli.zig test had ever run. main.zig aggregates tests through an explicit _ = @import(...) block, and both the Android and iOS dispatchers were missing from it. Their test blocks compiled but never executed; I confirmed it by deliberately breaking an assertion and watching the build still report success. That is how a flag could be advertised and unimplemented at once, and it is why the suite goes 44 → 59 here.
  • Attribute values were never XML-decoded. A row labelled "Network & internet" was reported as Network & internet, so --label "Network & internet" could not match what is on screen. Found by running against a real emulator rather than by unit test — and my first fix was incomplete, because names synthesized from descendant text bypassed it.
  • Unlabelled elements were dropped outright. The filter wanted clickable, or text, or an id, or a content-desc — which discards a scrollable RecyclerView or an unlabelled Switch. There was no way to discover the list could be scrolled.

What's new

android state foreground app + screen size + every actionable element, one call instead of three plus manual filtering
uitree --interactive interactivity detection across focusable/long-clickable/checkable/scrollable/selected/password + a narrow widget-class allowlist — 72 elements → 14 on a Settings screen
descendant naming a clickable row whose label lives in child TextViews is addressable at all; the parser now tracks nesting to do it
selectors --id / --class / --desc / --index on find and tap, AND-ed; resource-ids accepted short or fully qualified
type --clear replaces a field instead of appending to it
notifications reads package/title/text; --open pulls the shade down
wait <ms> runs without a device, so it can't fail a script with "no device attached"

State is reported, not just capability: *checked/*unchecked, *password, *focused, *selected, *scrollable. Whether a switch is on is not something a label can tell you.

Verified live

Headless Android 16 emulator, not just unit tests:

  • state on Settings returned rows named from their children — Network & internet Mobile, Wi‑Fi, hotspot — which previously listed as nameless
  • tap --label Messages moved focus from NexusLauncherActivity to ConversationListActivity
  • type --clear: "bluetooth" → "wifi", not "bluetoothwifi"
  • notifications parsed Keyboard layout set to English (US). Tap to change. — nested parens intact

59/59 kuri-mobile tests (was 44, 8 of which never executed), 476/476 in the main project unchanged, clean ReleaseFast build.

Not included

The annotated / set-of-marks screenshot (their use_annotation) needs a PNG decode-draw-encode path in Zig; kuri has /screenshot/som browser-side but nothing reusable here. --json on state/find/uitree is probably the highest-value follow-up for agent consumers.

macOS assets

This repo still has no APPLE_* secrets, so release.yml will take its signed=false branch. The macOS tarballs will be signed and notarized locally via the codedb-notary profile and uploaded over CI's, as for v0.4.13.

🤖 Generated with Claude Code

https://claude.ai/code/session_017yzwAh6c95Ygb4LYttdVCs

`uitree` reported what was in the uiautomator XML. What an agent needs is
what can be acted on, named the way it reads on screen. Prompted by a
comparison against CursorTouch/Android-MCP, whose element handling was
better than ours even though kuri's command surface is twice the size.

Three bugs came out of the comparison.

`android tap --label` never worked. The tool table advertised the flag and
`android tools` printed it, but the dispatcher only read positional
coordinates, so every documented invocation failed with "missing argument:
x y". It could ship that way because no cli.zig test had ever run: main.zig
aggregates tests through an explicit `_ = @import(...)` block and both the
Android and iOS dispatchers were missing from it, so their test blocks
compiled but never executed — a deliberately failing assertion still
reported success. Both are now in the list, which is what took the suite
from 44 to 59.

Attribute values were never XML-decoded, so a row labelled "Network &
internet" was reported as `Network &amp; internet` and could not be matched
by the string a user can actually read. Named and numeric references are
both handled; an ampersand that is not part of an entity is preserved.
Found by running against a real emulator, not by unit tests — and the first
fix was incomplete, because names synthesized from descendant text bypassed
it.

Elements carrying no label were dropped outright: the filter wanted
clickable, or text, or an id, or a content-desc, which discards a scrollable
RecyclerView and an unlabelled Switch. There was no way to discover the list
could be scrolled.

New on the Android side: `state` (foreground app, screen size and every
actionable element in one call, replacing three plus manual filtering);
interactivity detection across focusable/long-clickable/checkable/scrollable/
selected/password plus a narrow widget-class allowlist, exposed as
`uitree --interactive` (72 elements down to 14 on a Settings screen);
descendant name synthesis, so a clickable row whose label lives in child
TextViews is addressable at all — the parser now tracks nesting to do it;
`--id`/`--class`/`--desc`/`--index` selectors on `find` and `tap`, AND-ed,
with resource-ids accepted short or fully qualified; `type --clear` to
replace a field rather than append to it; `notifications`; and `wait`.

State is reported rather than just capability: *checked/*unchecked,
*password, *focused, *selected, *scrollable. Whether a switch is on is not
something a label can tell you.

Verified live against a headless Android 16 emulator: state, the selectors,
tap --label (launcher to Messages), type --clear (replaces, does not
append), notification parsing including a body with nested parentheses, and
entity decoding on a real Settings screen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017yzwAh6c95Ygb4LYttdVCs
@justrach
justrach merged commit 971f7c3 into main Jul 26, 2026
3 of 4 checks passed
@justrach
justrach deleted the release/0.4.14 branch July 26, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant