Skip to content

feat: describe-ui --no-raw and skill guidance to keep agent context small#62

Closed
4faramita wants to merge 2 commits into
lycorp-jp:mainfrom
4faramita:feat/describe-ui-no-raw
Closed

feat: describe-ui --no-raw and skill guidance to keep agent context small#62
4faramita wants to merge 2 commits into
lycorp-jp:mainfrom
4faramita:feat/describe-ui-no-raw

Conversation

@4faramita

@4faramita 4faramita commented Jul 23, 2026

Copy link
Copy Markdown

Summary

Two changes that reduce what an agent pays to read the screen:

  1. describe-ui --no-raw (top-level, ios, android): with --json, omit the raw accessibility tree (data.raw) from the envelope. outline / entries / lists are unchanged.
  2. A Keeping output small section in the bundled skill: prefer the text outline, pair --json with --no-raw, verify via the outline instead of screenshots, reuse the verify read as the next observe, batch known sequences.

Default behavior is untouched: bare describe-ui still prints the text outline, and --json without --no-raw still carries raw.

Measured savings

On a real app screen (40 outline entries; tokens estimated at ~4 ASCII chars/token, screenshots per Claude's pixel formula):

Change Before After Saving
--no-raw on ui --json 26.6 KB ≈ 6,600 tokens 9.7 KB ≈ 2,400 tokens -63%
Skill: text outline instead of --json ≈ 6,600 tokens/call ≈ 520 tokens/call -92%
Skill: verify via outline, not screenshot ≈ 1,500 tokens/shot ≈ 520 tokens -65%
Skill: reuse verify read as next observe 2 ui reads/step 1 read/step -45% reads

Combined on a 10-step agent task: -39% (agent already on the text loop) to -80% (agent on a --json loop) total context consumption.

Before / after

The default text outline (unchanged):

App: OrderDemo  402x874

[Top  y<120]
  @1  StaticText  "09:41"  (51,22 46x20)
  @2  RadioButton  "Orders"  (20,66 81x36)  selected
  @3  RadioButton  "Cart"  (101,66 73x36)

[Content  y=120..754]
  @8 #1  Cell  "Order #1024, 2 items"  (0,120 402x76)
  @9 #2  Cell  "Order #1023, 5 items"  (0,196 402x76)

ui --json, before (and still the default):

{
  "ok": true,
  "data": {
    "appLabel": "OrderDemo",
    "entries": [ /* full labels, frames, states */ ],
    "lists":   [ /**/ ],
    "outline": "",
    "raw":     { /* full AX tree: ~70% of the payload */ },
    "screen":  { /**/ }
  }
}

ui --json --no-raw:

{
  "ok": true,
  "data": {
    "appLabel": "OrderDemo",
    "entries": [ /* identical */ ],
    "lists":   [ /* identical */ ],
    "outline": "",
    "screen":  { /* identical */ }
  }
}

When --json is still used, and why quality is unaffected

describe-ui defaults to the text outline; agents reach for --json in two cases: coordinate math on entries[].frame / screen, and full untruncated label / value (the outline truncates labels at 60 graphemes, values at 30). Both live in entries, not in raw.

With --no-raw the envelope is byte-identical minus the raw key. raw is a debugging passthrough with no in-repo consumer (the Viewer included), no skill workflow reads it, and selector resolution (@N, --label) happens CLI-side against the AX snapshot, independent of what the agent read. Dropping it also keeps large screens from pushing the payload past agent-harness output truncation (30,000 chars in Claude Code) mid-JSON, which previously produced unparseable output and retries.

Tests

  • Tests/DescribeUINoRawTests.swift: flag parses on all three surfaces, the top-level forwarder copies it, and a result built without the tree encodes no raw key at all (not "raw": null). make test: 822 green.
  • Live spot check on a booted simulator: default text and default --json outputs byte-identical to before (modulo the status-bar clock); --no-raw envelope identical minus raw.

data.raw is the raw accessibility tree passed through for debugging
sim-use itself; on real app screens it dominates the encoded envelope
(~70% on a LINE chat list, 26.6KB -> 9.7KB with the flag) and larger
screens can push the payload past agent tool-output truncation limits
mid-JSON. The flag is available on all three surfaces (describe-ui,
ios describe-ui, android describe-ui); outline/entries/lists and the
default --json shape are unchanged.

Signed-off-by: 4faramita <14917258+4faramita@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@4faramita 4faramita changed the title Feat/describe UI no raw feat: describe-ui --no-raw and skill guidance to keep agent context small Jul 23, 2026
@4faramita
4faramita marked this pull request as ready for review July 23, 2026 08:31
@4faramita
4faramita requested a review from onevcat July 23, 2026 08:31
Add a "Keeping output small" section to SKILL.md: prefer the text
outline over --json, pair --json with --no-raw, verify via the outline
instead of screenshots, reuse the verify read as the next observe, and
batch known sequences. Point the cheatsheet's --json note at --no-raw.
Measured on a LINE Dev chat screen, these rules cut the context
consumed by a 10-step agent task by 39-80% depending on the agent's
prior habits (text loop / JSON loop / screenshot-verify).

Signed-off-by: 4faramita <14917258+4faramita@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@4faramita
4faramita force-pushed the feat/describe-ui-no-raw branch from ea010f7 to 25771d8 Compare July 23, 2026 08:34
@4faramita 4faramita closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant