Skip to content

Rework the Prompt Inspector and give history trimming a cache-friendly mode - #392

Merged
hydall merged 3 commits into
nightlyfrom
feat/context-trim-modes
Sep 5, 2026
Merged

Rework the Prompt Inspector and give history trimming a cache-friendly mode#392
hydall merged 3 commits into
nightlyfrom
feat/context-trim-modes

Conversation

@hydall

@hydall hydall commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Prompt Inspector

  • Rebuilt every inspector surface on InspectorPlaque — the context card's shell from the chat header (surface @ 94 %, primary hairline, soft shadow). It paints no BackdropFilter, so unlike GlassSurface it is safe per row of a 200-message list.
  • Collapsed the timeline's two top rows into one: coverage of the next request is now a block inside its preview, between the parameters and the messages — the slot a captured request already keeps its own coverage in.
  • A captured request now wears the preview's layout (budget bar → parameters → coverage → filtered messages) through a shared RequestBodyView, so a past and a next request stop looking like two different features.
  • Removed the bare Material the old surfaces hid: Card, LinearProgressIndicator, TextButton, and every Colors.white / red / green / orange literal.
  • Padded the Request/Response tab strip to the same 12 px gutter as the plaques under it.
  • Deleted the now-unreferenced CoveragePanel sheet and updated the widget-file list in the UI→DB characterization test.

History trimming

Cutting history one message at a time moves the start of the prompt on almost every turn, so markStablePrefixCacheControl finds nothing in common with the previous request and a provider's prefix cache misses every turn — precisely for the long chats that need it most.

  • Added ApiConfig.historyTrimMode: sliding (the existing cut, still the default) and stepped, which holds the start of the history still and moves it a block at a time.
  • The stepped cut anchors on the oldest message it kept and honours that anchor for as long as the window it opens still fits the budget. A raised max output tokens, a grown lorebook or a deleted anchor moves it; nothing else does.
  • The anchor is stored as a reserved __historyAnchor session var through the existing atomic updateSessionVarsJson — no new column, no read-modify-write. Only the generation path writes it, so opening a preview never moves it.
  • Two knobs, shown under the mode picker only while stepped is selected: step threshold (85 %) and step size (30 %). Both are per connection.

Connection settings

  • New Context section: trim mode, then max output tokens and context size. Those two used to sit among the sampling knobs, which put the numbers deciding what the model sees next to the ones deciding how it writes.
  • Every sampling parameter is back in Generation Parameters; they were a card of their own behind "Advanced", which hid the controls people reach for most often.
  • MenuSelectorItem now has the same outline, fill and metrics as MenuFieldItem's text field, so the protocol selector stops reading as a different kind of control from the fields around it. MenuRangeItem gains the description both siblings already had.

Keeping the previews honest

  • TokenBreakdownCache is keyed on the trim settings as well as the window, so a breakdown taken under one mode is never served under another.
  • The Context tab and the next-request preview rebuild when the connection's context signature changes, and the save itself invalidates the shared breakdown — so surfaces nobody has open (drawer stats, the chat header card) are not left stale either.

Cleanup

  • The cutoff notice now reads "N messages didn't fit into context size" and carries a gear that opens the connection at its Context section.
  • Removed the manual "hide top N" action, its two global AppSettings knobs and the near-limit banner. Curating a chat is what message selection and hide are for; the cache workaround is a trim mode now.

Message selection

  • The selection toolbar gains two buttons that extend the selection from the last tapped message to the top or the bottom of the chat; tapping again with that run already selected clears it. Authored by @hydall in the working tree, carried here rather than left uncommitted.

Verification

  • New test/history_trim_mode_test.dart — 11 cases covering both modes, anchor hold and move, a deleted anchor, the max-output-tokens interaction, and cache invalidation across a mode or knob change.
  • flutter analyze clean; flutter test 3821 passing.
  • Migration 133 → 135 applied against a real device database: user_version bumped, all three columns present, the existing connection left on sliding so behaviour is unchanged until someone opts in.
  • Installed on an Android emulator and walked the timeline, the next request, a captured request, the coverage block, the Context section and the mode picker.

Known: three chat_input_bar clipboard tests fail on a Windows host — the test helper's Windows branch hands pasteboard a file path the mock never reads back. They are host-dependent and pass on CI's ubuntu-latest; unrelated to this PR's changes.

hydall and others added 3 commits September 5, 2026 19:17
The Prompt Inspector mixed three looks — GlassSurface tiles, Material rows at
3 % onSurface and hand-styled Colors.white containers — and answered "what goes
into the prompt?" twice: once as the next request, once as a coverage row
beside it. Rebuild it on the card the chat header already wears.

- Add InspectorPlaque, the context card's shell (surface @ 94 %, primary
  hairline, soft shadow) with no BackdropFilter, so it is safe per row of a
  200-message list where GlassSurface was not.
- Collapse the timeline's two top rows into one: coverage of the next request
  is now a block inside its preview, between the parameters and the messages,
  exactly where a captured request carries its own.
- Give a captured request the preview's layout via a shared RequestBodyView —
  budget bar, parameters, coverage, filtered messages — so past and next
  requests stop looking like two features.
- Drop the bare Material the surface hid behind: Card, LinearProgressIndicator,
  TextButton, and every Colors.white/red/green/orange literal.
- Delete the now-unreferenced CoveragePanel sheet and update the widget-file
  list in the UI→DB characterization test.

Verified: flutter analyze clean; flutter test (3821 passing); installed on an
emulator and walked the timeline, next request, a captured request and the
coverage block.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pped mode

Trimming history one message at a time moves the start of the prompt on almost
every turn, so markStablePrefixCacheControl finds nothing in common with the
last request and a provider's prefix cache misses every single turn — exactly
for the long chats that need it most. Give the connection a choice.

## Trim modes

- Add ApiConfig.historyTrimMode: `sliding` (the existing cut, still the
  default) and `stepped`, which holds the start of the history still and jumps
  it forward a block at a time.
- Anchor the stepped cut on the oldest message it kept and honour it for as
  long as the window it opens still fits the budget; a raised max output
  tokens, a grown lorebook or a deleted anchor moves it, nothing else does.
- Store the anchor as a reserved `__historyAnchor` session var through the
  existing atomic updateSessionVarsJson, so it needs no new column and no
  read-modify-write. Only the generation path writes it — a preview shows what
  would be sent without moving it.
- Expose the two knobs the mode spends as connection fields, shown under the
  mode picker only when it is on `stepped`: step threshold (85 %) and step size
  (30 %).

## Where the settings live

- Add a Context section to the connection: trim mode, then max output tokens
  and context size, which used to sit among the sampling knobs — the two
  numbers that decide what the model *sees* did not belong next to the ones
  that decide how it writes.
- Put every sampling parameter back into Generation Parameters; they were a
  card of their own behind "Advanced", which hid the controls people reach for
  most often.
- Give MenuSelectorItem the same outline, fill and metrics as MenuFieldItem's
  text field, so a selector stops reading as a different kind of control from
  the fields around it. MenuRangeItem gains the `description` both siblings
  already had.

## Keeping the previews honest

- Key TokenBreakdownCache on the trim settings as well as the window, so a
  breakdown taken under one mode is never served under another.
- Rebuild the Context tab and the next-request preview when the connection's
  context signature changes, and invalidate the shared breakdown from the save
  itself, so surfaces nobody has open (drawer stats, the chat header card) are
  not left stale.

## Cleanup

- Reword the cutoff notice to "N messages didn't fit into context size" and put
  a gear on it that opens the connection at the Context section.
- Remove the manual "hide top N" action, its two global AppSettings knobs and
  the near-limit banner: curating a chat is what message selection and hide are
  for, and the cache workaround is now a trim mode.

Verified: new test/history_trim_mode_test.dart (11 cases) covers both modes,
anchor hold/move, a deleted anchor, the max-output-tokens interaction and cache
invalidation; flutter analyze clean; flutter test 3821 passing; migration
133 -> 135 applied on a real device database (columns present, existing
connection left on `sliding`), and the Context section walked on an emulator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Selecting a long run meant tapping each message. The selection toolbar gains
two buttons that extend the selection from the last tapped message to the top
or the bottom of the chat; tapping again with that run already selected clears
it.

Authored in the working tree by hydall; carried in this PR rather than left
uncommitted.
@hydall
hydall merged commit d73a9a2 into nightly Sep 5, 2026
6 of 9 checks passed
@hydall
hydall deleted the feat/context-trim-modes branch September 6, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant