feat: describe-ui --no-raw and skill guidance to keep agent context small#62
Closed
4faramita wants to merge 2 commits into
Closed
feat: describe-ui --no-raw and skill guidance to keep agent context small#624faramita wants to merge 2 commits into
4faramita wants to merge 2 commits into
Conversation
data.raw is the raw accessibility tree passed through for debugging sim-use itself; on real app screens it dominates the encoded envelope (~70% on a LINE chat list, 26.6KB -> 9.7KB with the flag) and larger screens can push the payload past agent tool-output truncation limits mid-JSON. The flag is available on all three surfaces (describe-ui, ios describe-ui, android describe-ui); outline/entries/lists and the default --json shape are unchanged. Signed-off-by: 4faramita <14917258+4faramita@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4faramita
marked this pull request as ready for review
July 23, 2026 08:31
Add a "Keeping output small" section to SKILL.md: prefer the text outline over --json, pair --json with --no-raw, verify via the outline instead of screenshots, reuse the verify read as the next observe, and batch known sequences. Point the cheatsheet's --json note at --no-raw. Measured on a LINE Dev chat screen, these rules cut the context consumed by a 10-step agent task by 39-80% depending on the agent's prior habits (text loop / JSON loop / screenshot-verify). Signed-off-by: 4faramita <14917258+4faramita@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4faramita
force-pushed
the
feat/describe-ui-no-raw
branch
from
July 23, 2026 08:34
ea010f7 to
25771d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes that reduce what an agent pays to read the screen:
describe-ui --no-raw(top-level,ios,android): with--json, omit the raw accessibility tree (data.raw) from the envelope.outline/entries/listsare unchanged.--jsonwith--no-raw, verify via the outline instead of screenshots, reuse the verify read as the next observe, batch known sequences.Default behavior is untouched: bare
describe-uistill prints the text outline, and--jsonwithout--no-rawstill carriesraw.Measured savings
On a real app screen (40 outline entries; tokens estimated at ~4 ASCII chars/token, screenshots per Claude's pixel formula):
--no-rawonui --json--jsonuireads/stepCombined on a 10-step agent task: -39% (agent already on the text loop) to -80% (agent on a
--jsonloop) total context consumption.Before / after
The default text outline (unchanged):
ui --json, before (and still the default):{ "ok": true, "data": { "appLabel": "OrderDemo", "entries": [ /* full labels, frames, states */ ], "lists": [ /* … */ ], "outline": "…", "raw": { /* full AX tree: ~70% of the payload */ }, "screen": { /* … */ } } }ui --json --no-raw:{ "ok": true, "data": { "appLabel": "OrderDemo", "entries": [ /* identical */ ], "lists": [ /* identical */ ], "outline": "…", "screen": { /* identical */ } } }When
--jsonis still used, and why quality is unaffecteddescribe-uidefaults to the text outline; agents reach for--jsonin two cases: coordinate math onentries[].frame/screen, and full untruncatedlabel/value(the outline truncates labels at 60 graphemes, values at 30). Both live inentries, not inraw.With
--no-rawthe envelope is byte-identical minus therawkey.rawis a debugging passthrough with no in-repo consumer (the Viewer included), no skill workflow reads it, and selector resolution (@N,--label) happens CLI-side against the AX snapshot, independent of what the agent read. Dropping it also keeps large screens from pushing the payload past agent-harness output truncation (30,000 chars in Claude Code) mid-JSON, which previously produced unparseable output and retries.Tests
Tests/DescribeUINoRawTests.swift: flag parses on all three surfaces, the top-level forwarder copies it, and a result built without the tree encodes norawkey at all (not"raw": null).make test: 822 green.--jsonoutputs byte-identical to before (modulo the status-bar clock);--no-rawenvelope identical minusraw.