Design: last-resort for non-tool content overflow — export-then-truncate vs full compression (global T2)

[ranxianglei]

Problem

When a conversation's context is dominated by non-tool content (a large user paste, an assistant dump) and there is no compressible tool result to shrink, the kernel's only hard valve — the emergency tool-result truncate (src/truncate-tools.ts) — is a no-op. It only shrinks old tool-result messages, skips the last preserveRecentMessages (5), skips results < 1000 tok, and never touches user/assistant text. The last user message (the current prompt) is always protected (src/recommend.ts). So a provable overflow exists: a large non-tool message that is neither a tool-result nor compressible, at/over the window, with the advisory nudge ignored (model non-compliance).

This is the last-resort gap the other layers do not cover:

• Preventive (output-headroom — reserves the output budget) does not help when the context itself is already over the window from non-tool content.
• Reactive (overflow self-heal — learns the window + arms the next turn's emergency) still lands on an emergency truncate that is a no-op for non-tool content.

Purpose of the last-resort (design intent)

The truncation is not meant to be a permanent loss. Its purpose is to temporarily truncate / drop some information so that a NEW compression can succeed — i.e. it is a bridge to recovery, not the end state. On that basis a lossy last-resort is worth it.

That intent opens the door to combining it with export: in an emergency, export first (so the truncated information is recoverable), then truncate.

Current state (verified)

Capability	Status
Kernel emergency truncate	Only old tool-results; no-op on user/assistant text; skips last 5 + <1000 tok.
Last user message	Always protected (the current prompt).
Export	Exists only in the proxy as an offline CLI command bili export (src/export.ts:147 exportSession(selector,{dir,output,full}) → renderHandoff): default = folded view via kernel prune (summaries in place of compressed ranges); --full = every original message; markdown; optional file output. Operator-invoked, offline, NOT on the emergency path. The extension has no export.
"Full compression"	Does not exist. No tool/mode (kernel, proxy, or extension) that summarizes everything except the last N messages into a single T2 summary. The kernel has a tiered T1→T2→T3 system (src/compression-rules.ts) but no single "full compress" op.
Session persistence	Incremental — compression state + compressed originals only; uncompressed conversation text is not fully persisted (v3 files have a lastMessages snapshot, v2 does not) → an offline export of an older session may lack the full text.

Candidate approaches

A. Export-then-truncate

On emergency (context ≥ threshold, nothing viable to compress, no tool-result to shrink):

1. Export the full current context to a file (reuse the proxy's exportSession/renderHandoff, but make it live — in the emergency path, using the in-memory context the host already has).
2. Truncate the oldest large non-tool message(s) (skip the last user message + the last N), down to a small prefix+suffix (like the tool truncate).

The truncated information is recoverable from the export. This creates room so the model's next compression succeeds (the bridge).

B. Full compression ("global T2")

A new full_compress op: within the remaining compressible space, have the model summarize everything except the last N messages into a single T2 summary (a global tier-2 distill). More elegant — the model summarizes, so the loss is controlled and it produces a real summary block the tier system can later condense — but it needs a model round-trip (not instant) and a new tool/mode.

Open questions (for discussion)

1. Which approach — A, B, or an A-then-B escalation ladder (instant truncate as the immediate backstop, full-compression as the follow-up)?
2. Placement — truncate/full-compress are kernel capabilities (shared by all hosts); export is a host capability (proxy has it, extension doesn't). Should the kernel emit a "last-resort" signal that the host fulfils (export + optional truncate), or should the kernel do the truncate itself while the host does the export?
3. Making export live — the current export is offline CLI. An in-emergency export needs the full current context, which the host has in memory but does not fully persist (incremental persistence). How do we obtain the full text at emergency time?
4. What to truncate — which non-tool messages (oldest first? skip the last user message + last N?), and how much to keep (prefix+suffix size)? The user-paste loss is the risk being mitigated by the export.
5. Full-compression latency — it needs a model round-trip. Acceptable in an emergency (adds a turn), or must the last-resort be instant (truncate)?
6. Extension parity — the extension has no export. Does it need one for the last-resort?
7. Trigger condition — exactly when does the last-resort fire (e.g. ≥95% of the (headroom-reserved) window + no viable range + no tool-result to shrink), and how does it interact with the existing overflow self-heal (learned window + armed emergency) and output-headroom?

Scope / non-goals

• This is a design-discussion issue (no implementation in this issue).
• It complements the already-landed preventive (output-headroom) and reactive (overflow self-heal) layers in the two downstream hosts.

Related

• billion-context PR #172 (proxy: self-heal + output-headroom)
• billion-context-pi PR #177 (extension: self-heal + output-headroom)
• Kernel: src/truncate-tools.ts (emergency truncate), src/recommend.ts (viability + last-user protection), src/compression-rules.ts (T1→T2→T3 tiers), src/nudge-text.ts (tier distill nudges)
• Proxy: src/export.ts (offline exportSession/renderHandoff), src/persist.ts (incremental persistence, v2/v3)

[ranxianglei]

Code review — the "Current state" table is accurate (2 small corrections)

Verified against kernel master:

• truncateLargeToolOutputs (src/truncate-tools.ts:24) — tool-results only (contentType !== "tool-result" → skip, line 43), minOutputTokens: 1000, marker-idempotent, largest-first, stops at threshold * 0.9. ✓
• The emergency node is src/compress.ts:538-558 (emergency-truncate), fires at usage >= truncate.threshold, and is a silent no-op when there are no candidates (returns io unchanged, no effect/signal — the host cannot distinguish "truncated fine" from "nothing was possible").
• Correction 1: protectRecentMessages kernel default is 3 (src/truncate-tools.ts:21); the deployed value is 5 because hosts pass config.preserveRecentMessages (src/compress.ts:551) and the protect layer assumes 5 (src/recommend.ts:115). "Skips the last 5" is correct for deployed hosts.
• Last-user protection (src/recommend.ts:111-124): last visible user message, protected whenever preserveRecentMessages > 0. ✓
• Export: exportSession (billion-context/src/export.ts:147) loads from the persisted store — it cannot be reused as-is on the live path. The reusable piece is renderHandoff(s, full) (src/export.ts:50), which is pure over a Session (v3 lastMessages + kernel prune for the folded view, line 72). ✓
• "Full compression does not exist" ✓ — compression-rules.ts T2/T3 rules are distill/condense prompts for already-compressed blocks; the kernel ops are processTurn/applyCompression (src/compress.ts:72,92), no one-shot "summarize everything except last N".

The key insight the issue is missing: B alone cannot be the last resort

Full compression (B) needs a model round-trip — i.e. a request carrying the entire already-overflowing context plus the distill prompt. If the context provably exceeds the window, the summarization request itself overflows. B can only run after A has created headroom. Therefore:

Recommended ladder: A is the valve, B is the recovery.

• A (instant, deterministic, no model call): export + last-resort truncate → next request fits.
• B (next turn, via the existing mechanism): once headroom exists, the normal nudge → compress-tool flow produces a real T1→T2 summary of the truncated region; the truncation marker references the export path so the summary can cite what was cut. No new full_compress op is needed for phase 1 — B is a later recovery-quality improvement, not a last-resort.

This answers Q1 (ladder) and Q5 (zero added latency on the emergency path).

Recommendations on the open questions

Q2 — Placement. Kernel owns the truncate; host owns the export.

• Kernel: make the emergency node two-stage. Stage 1 = existing tool-result truncate; stage 2 (only if stage 1 yielded 0 candidates) = non-tool last-resort: eligible user/assistant text, skipping the last user message + last preserveRecentMessages + synthetic/pruned (isSyntheticOrPruned) + protected tools (reuse protected.ts), largest-first, same threshold * 0.9 stop, same 2000+2000 prefix/suffix, distinct marker.
• Export-ordering constraint: the export must capture the pre-truncation state. Cleanest: kernel exports a pure findLastResortCandidates(messages, config); the host evaluates it before invoking the pipeline (it already computes the token count for the usage header), exports, then calls the pipeline which performs the truncate. No new pipeline callback needed.
• Extend TruncateResult with lastResort: boolean + truncated: Array<{ ref, role, tokens }> so the host can embed the export path in the marker text and log/notify.

Q3 — Live export. Not a persistence problem: at emergency time the host has the full context in memory (proxy: prepared request body; extension: in-memory session entries). The incremental-persistence gap only affects offline re-exports. Phase 1 reads from memory; no persistence change.

Q4 — What to truncate. Largest-first (existing convention) with the same stop condition; protect last user message + last N + synthetic/pruned + protected tools. Add a floor: never truncate below ~25% of the window and never below the last user message — below that, B-recovery is unlikely and the true last last-resort is a hard handoff (export + new session seeded from the handoff). Suggest tracking that as open question 8.

Q6 — Extension parity. Yes, minimal: a small export-session.ts rendering in-memory entries to markdown (the extension already has rendering primitives in messages.ts; port renderHandoff's shape). Without it, extension A = lossy truncate with no recovery path, contradicting the stated design intent ("not meant to be a permanent loss").

Q7 — Trigger.

• Fire when: usage >= 0.95 × (window − output-headroom) (top 5% band after headroom reservation) and stage 1 yielded 0 candidates and last-resort candidates exist.
• Interaction with self-heal: armed (force tokenCount ≥ 95%) lands the next pipeline run exactly in this band; learned-window re-centering guarantees the band is computed against the true window. The chain composes: headroom reserves output → armed emergency forces usage into the last-resort band → A fires. No new state.
• Idempotency: the last-resort marker must exclude the message from future candidacy (like TRUNCATION_MARKER), so repeated overflows degrade gracefully instead of re-truncating.

Two things worth fixing anyway (independent of the design choice)

1. The silent no-op (src/compress.ts:549): even without implementing the last-resort, the node should emit an effect like emergencyTruncate: { candidates: 0, reason: "no-tool-results" } so hosts can log/notify "overflow imminent, no valve available". One-line kernel change, immediately observable.
2. Export file hygiene: the emergency export writes the full conversation to disk — session directory, 0600, named <sessionId>-<ts>-overflow.md so repeated overflows don't clobber each other (and nothing lands in CWD).

Suggested phase split

• Phase 1 (kernel + both hosts): findLastResortCandidates + two-stage emergency node + host pre-export + marker with path + notify. Deterministic, testable, no new model behavior.
• Phase 2: full_compress op (B) as a recovery-quality improvement — one-shot distill of everything except last N, usable post-A and as an operator command outside the emergency path.
• Phase 3 (if needed): hard handoff (new session seeded from the export) when the floor is reached.

[ranxianglei]

Code review — every reference in the body + comment verified against master

Reviewed the issue and the design comment against the actual kernel/proxy/extension code. All code references check out (line numbers within ±1):

• truncateLargeToolOutputs (src/truncate-tools.ts:24) — tool-results only (contentType !== "tool-result" → skip), minOutputTokens: 1000, protectRecentMessages kernel default 3 (host passes config.preserveRecentMessages = 5 in production, src/config.ts:31), marker [truncated for context space] idempotent, largest-first, stops at threshold * 0.9. ✓
• Emergency node emergencyTruncateNode (src/compress.ts:537-556) — fires at usage >= truncate.threshold; silent no-op confirmed: with 0 candidates it returns io with only effects.truncatedCount: 0 — no reason field, the host cannot distinguish "truncated fine" from "nothing was possible". ✓
• Last-user protection (src/recommend.ts Rule 3) — protected whenever preserveRecentMessages > 0, scanning raw messages (survives decompressed-tool-result tails). ✓
• renderHandoff (billion-context/src/export.ts:50) is pure over a Session; exportSession (:147) loads from the persisted store — so the reusable piece for the live path is renderHandoff only, as stated. ✓
• "Full compression does not exist" ✓ — kernel ops are processTurn/applyCompression; the T2/T3 rules in compression-rules.ts are distill/condense prompts for already-compressed blocks, not a one-shot "summarize everything except last N".

The ladder (A valve → B recovery) is the only sound ordering

Beyond the round-trip argument in the comment, one mechanical detail worth pinning: the kernel's own compression calls (applyCompression) send the full visible range to the model. So B's distill request carries ≥ the overflowing context — if the context provably exceeds the window, B's request overflows too. A is not just faster, it is a prerequisite: B can only run on a context A has already brought under the window. This also bounds B's design: it never needs to handle >window inputs, so no new chunking/rolling-distill machinery is required.

Verified composition with the landed headroom/self-heal layers (Q7)

Checked against billion-context #172 / billion-context-pi #177 (both reviewed, small fixes pushed):

• The nudge/truncate bands now compute against (window − output headroom), and the proxy's overflow self-heal arms lastInputTokens >= learned window — so the proposed trigger usage ≥ 0.95 × (window − headroom) lands exactly where the armed emergency lands. No new state, as the comment claims. ✓
• One nuance the issue should record: "window − headroom" is only meaningful for OpenAI-family windows. Anthropic's Messages API enforces the input limit independently of max_tokens (separate output budget), so reserving headroom there just shifts the bands down by maxOutput for no safety gain. In review I made the reservation protocol-aware in both hosts (bc #172 shouldReserveOutputHeadroom(protocol), bcp #177 shouldReserveOutputHeadroom(api)). The last-resort band must be computed the same way: against the headroom-reserved window the kernel actually receives, not the raw model window — otherwise the 95% band sits at 95% of the wrong denominator.
• The extension-side armed floor (95% of the possibly-wrong configured window when the error stated no window) can still overshoot a smaller real window on turn 1; it converges on turn 2 (re-overflow → window learned). Acceptable, but worth a sentence in the design so nobody "fixes" it into a bug.

Live export source (Q3) — confirmed, no persistence work

• Proxy: the full current context is the prepared request body in memory (the parsed messages about to be sent) — export reads that, not the incremental store.
• Extension: in-memory session entries + sessionManager.buildContextEntries() (exact messages about to be sent, already the source of the context transform).
• The incremental-persistence gap only affects offline re-export of old sessions, as stated. Phase 1 reads from memory; no persistence change. ✓

Two small additions to Phase 1

1. Make the no-op observable with a reason (extends the comment's item 1): the effect should carry why no valve fired, e.g. lastResort: { candidates: 0, reason: "no-tool-results", nonToolCandidates: N } — the host needs that to decide whether to export (worth exporting only when non-tool candidates exist, i.e. the truncate would actually buy room).
2. Stage-2 marker is a kernel export, not a local const: TRUNCATION_MARKER in truncate-tools.ts is module-private; the non-tool stage needs its own exported marker constant so hosts can (a) render it in the export/handoff and (b) the candidate scan excludes already-cut messages (idempotency across repeated overflows, per the comment).

Everything else in the comment reads as correct and buildable; the phase split is right.