release: v0.4.14 — Android element extraction, rebuilt - #195
Merged
Conversation
`uitree` reported what was in the uiautomator XML. What an agent needs is what can be acted on, named the way it reads on screen. Prompted by a comparison against CursorTouch/Android-MCP, whose element handling was better than ours even though kuri's command surface is twice the size. Three bugs came out of the comparison. `android tap --label` never worked. The tool table advertised the flag and `android tools` printed it, but the dispatcher only read positional coordinates, so every documented invocation failed with "missing argument: x y". It could ship that way because no cli.zig test had ever run: main.zig aggregates tests through an explicit `_ = @import(...)` block and both the Android and iOS dispatchers were missing from it, so their test blocks compiled but never executed — a deliberately failing assertion still reported success. Both are now in the list, which is what took the suite from 44 to 59. Attribute values were never XML-decoded, so a row labelled "Network & internet" was reported as `Network & internet` and could not be matched by the string a user can actually read. Named and numeric references are both handled; an ampersand that is not part of an entity is preserved. Found by running against a real emulator, not by unit tests — and the first fix was incomplete, because names synthesized from descendant text bypassed it. Elements carrying no label were dropped outright: the filter wanted clickable, or text, or an id, or a content-desc, which discards a scrollable RecyclerView and an unlabelled Switch. There was no way to discover the list could be scrolled. New on the Android side: `state` (foreground app, screen size and every actionable element in one call, replacing three plus manual filtering); interactivity detection across focusable/long-clickable/checkable/scrollable/ selected/password plus a narrow widget-class allowlist, exposed as `uitree --interactive` (72 elements down to 14 on a Settings screen); descendant name synthesis, so a clickable row whose label lives in child TextViews is addressable at all — the parser now tracks nesting to do it; `--id`/`--class`/`--desc`/`--index` selectors on `find` and `tap`, AND-ed, with resource-ids accepted short or fully qualified; `type --clear` to replace a field rather than append to it; `notifications`; and `wait`. State is reported rather than just capability: *checked/*unchecked, *password, *focused, *selected, *scrollable. Whether a switch is on is not something a label can tell you. Verified live against a headless Android 16 emulator: state, the selectors, tap --label (launcher to Messages), type --clear (replaces, does not append), notification parsing including a body with nested parentheses, and entity decoding on a real Settings screen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017yzwAh6c95Ygb4LYttdVCs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
uitreereported what was in the uiautomator XML. What an agent needs is what can be acted on, named the way it reads on screen.Prompted by a comparison against CursorTouch/Android-MCP. kuri already had roughly twice the command surface (27 vs 14 — they have no
list-apps,uninstall,clear,openurl,keyevent,current-activity,logcat,screen-info,getprop,dumpsys, rawtouch, multi-pointgestureorbatch), but their element handling was better. That gap is what this closes.Bugs the comparison turned up
android tap --labelnever worked. The tool table advertised the flag andandroid toolsprinted it, but the dispatcher only read positional coordinates — every documented invocation failed withmissing argument: x y.cli.zigtest had ever run.main.zigaggregates tests through an explicit_ = @import(...)block, and both the Android and iOS dispatchers were missing from it. Theirtestblocks compiled but never executed; I confirmed it by deliberately breaking an assertion and watching the build still report success. That is how a flag could be advertised and unimplemented at once, and it is why the suite goes 44 → 59 here.Network & internet, so--label "Network & internet"could not match what is on screen. Found by running against a real emulator rather than by unit test — and my first fix was incomplete, because names synthesized from descendant text bypassed it.RecyclerViewor an unlabelledSwitch. There was no way to discover the list could be scrolled.What's new
android stateuitree --interactivefocusable/long-clickable/checkable/scrollable/selected/password+ a narrow widget-class allowlist — 72 elements → 14 on a Settings screenTextViews is addressable at all; the parser now tracks nesting to do it--id/--class/--desc/--indexonfindandtap, AND-ed; resource-ids accepted short or fully qualifiedtype --clearnotifications--openpulls the shade downwait <ms>State is reported, not just capability:
*checked/*unchecked,*password,*focused,*selected,*scrollable. Whether a switch is on is not something a label can tell you.Verified live
Headless Android 16 emulator, not just unit tests:
stateon Settings returned rows named from their children —Network & internet Mobile, Wi‑Fi, hotspot— which previously listed as namelesstap --label Messagesmoved focus fromNexusLauncherActivitytoConversationListActivitytype --clear: "bluetooth" → "wifi", not "bluetoothwifi"notificationsparsedKeyboard layout set to English (US). Tap to change.— nested parens intact59/59 kuri-mobile tests (was 44, 8 of which never executed), 476/476 in the main project unchanged, clean
ReleaseFastbuild.Not included
The annotated / set-of-marks screenshot (their
use_annotation) needs a PNG decode-draw-encode path in Zig; kuri has/screenshot/sombrowser-side but nothing reusable here.--jsononstate/find/uitreeis probably the highest-value follow-up for agent consumers.macOS assets
This repo still has no
APPLE_*secrets, sorelease.ymlwill take itssigned=falsebranch. The macOS tarballs will be signed and notarized locally via thecodedb-notaryprofile and uploaded over CI's, as for v0.4.13.🤖 Generated with Claude Code
https://claude.ai/code/session_017yzwAh6c95Ygb4LYttdVCs