fix(compaction): use symmetric heuristic stats pair and authoritative display

[yangjj-iso]

Summary

Closes #2052.

The compaction stats mixed three incommensurable token estimators (provider-authoritative, per-message heuristic, and char/4 on serialized text), producing negative percentReduction values when the region was small relative to the preserved tail.

Implements option 3 from the issue discussion, as approved by @flora131.

Changes

1. Centralized widening/ratio logic (computeWholeContextStats)

Extracted from the old withWholeContextStats into an exported function in compaction-runner.ts, used by planned, fresh, and extension paths. Both sides of the reported comparison now use the same heuristic: region char/4 + explicit tail estimate.

• tokensBefore = region.tokenEstimate + tailEstimate (when tail kept)
• tokensAfter = regionStats.tokensAfter + tailEstimate (when tail kept)
• When the fresh rung drops the tail, the tail term correctly does not appear on either side.

2. Removed authoritative-minus-heuristic tail fallback

getKeptTailTokenEstimate in compaction-boundary.ts now defaults to 0 instead of preparation.tokensBefore - preparation.region.tokenEstimate (authoritative minus heuristic). The old fallback mixed units and could overshoot the real count.

Exported setKeptTailTokenEstimate so test fixtures can set the tail estimate on preparations constructed directly (not through prepareCompactionBoundary).

3. Authoritative display through adapter

• Added VerbatimCompactionDetails.tokensBefore as an optional field carrying the authoritative count through to the TUI display.
• CompactionBoundaryMessageComponent uses displayTokensBefore (from entry.tokensBefore / result.tokensBefore) instead of stats.tokensBefore for the "Compacted from N tokens" line, projected through the BoundaryView adapter.
• Legacy entries without details.tokensBefore fall back to stats.tokensBefore.
• The authoritative tokensBefore is preserved on CompactionEntry and VerbatimCompactionResult for budgeting and threshold decisions.

4. No clamping

The percentage is not clamped: a genuine expansion may still be negative.

Regression tests

Added test/unit/compaction-stats-symmetry.test.ts (11 tests) covering:

• Large kept tails in planned and extension paths
• Fresh compaction with tail kept and dropped
• preserve_recent: 0
• Live and resumed rendering where display count differs from heuristic stats count
• Legacy entries without details.tokensBefore fallback
• computeWholeContextStats symmetric ratio properties

Updated existing tests in compaction-fresh-window.test.ts to use setKeptTailTokenEstimate instead of the removed mixed-unit fallback.

Test results

All 141 affected tests pass. TypeScript type check and Biome lint/format all pass.

Greptile Summary

This change makes compaction reduction statistics compare like-for-like estimates while preserving the provider-authoritative token count in boundary messages. Focused unit and integration coverage confirms that dropping a protected tail includes that tail in the original-context baseline, and that live and resumed views display the authoritative original token count.

Confidence Score: 5/5

No blocking failure remains; the compaction and boundary-rendering paths exercised by the focused checks behave as intended.

No accepted blocking findings remain after exercising fresh compaction with a dropped tail and boundary rendering with distinct heuristic and authoritative counts.

T-Rex Logs

What T-Rex did

• Ran the focused 11-test compaction regression unit suite and the focused four-test fresh-compaction integration suite on the current checkout; both suites passed.
• Compared the current test outcomes with the parent checkout and observed that the unit and integration runs meet the expected behavior for the targeted dropped-tail scenarios, with all targeted tests passing.
• Documented the validation coverage showing the computeWholeContextStats and related boundary display paths align with the intended token counts and tail handling as described in the test-driven validation.

_{Ran code and verified through T-Rex}

_{Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[flora131]

Thanks for the work on this. It looks like #2449 is the newer, focused version of the same #2052 change, and we’re reviewing that PR now.

Would you be okay with us closing #2407 in favor of #2449? That would keep the review queue clear and avoid us reviewing or commenting on the older implementation by mistake. If #2407 contains any behavior that you intentionally did not carry into #2449, please let us know so we can account for it before closing.