(MOT-4372) fix(database): bound query history stored via state::update - #732
(MOT-4372) fix(database): bound query history stored via state::update#732andersonleal wants to merge 3 commits into
Conversation
Oversized history:primary blobs (~8MB) reset the state worker WebSocket and leave state::* unregistered — state::update echoes the full old and new value per append and bypasses max_value_bytes, and the only trim ran on a read path nothing calls. Record now does a capped read-append-trim-replace through state::set (history_max_entries / history_max_bytes, defaults 200 entries / 256KB, 0 disables, hot-reloaded). A stored value that cannot be read is replaced wholesale on the next write, so a pre-existing oversized blob self-heals without ever round-tripping it.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 33 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughDatabase query history now supports configurable per-database entry and serialized-byte limits. History is trimmed during writes, disabled when configured to zero, and reset after unreadable stored state. ChangesQuery history limits
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant QueryHandler
participant WorkerConfig
participant saved_record
participant StateStorage
QueryHandler->>WorkerConfig: clone current configuration
QueryHandler->>saved_record: record query with configuration
saved_record->>WorkerConfig: read history caps
saved_record->>StateStorage: replace history with capped entries
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ions No state worker runs in the e2e stack, so the harness registers state::get/set/update itself — observing every history write and scripting the failure modes a real state worker cannot safely reproduce. Covers entry/byte rotation (tiny caps seeded in the e2e config), oversized-backlog trim, the unreadable-value blind-write self-heal, and a guard that history never touches state::update.
Verified against the pre-fix worker: cases now fail in ~85ms with 'history wrote via state::update (uncapped append)' rather than waiting out the 5s write timeout.
1 participant
Fixes MOT-4372
Problem
The database worker records every console query into the
stateworker (scopedatabase, keyhistory:{db}) via an uncappedstate::updateappend. On a live stackhistory:primaryreached ~8.4MB.state::updateechoes the full old and new value in its response and emits an event carrying both again (state/src/functions.rs), and it is exempt fromstate.max_value_bytes(which guardsstate::setonly) — so each one-line append re-served the whole blob ~4x over the engine⇄worker WebSocket. Past the transport's per-message cap the connection resets, the state SDK reconnect-loops,state::*stays unregistered, and the engine spamsFunction not found: state::update. The only existing trim ran in thedatabase::historyread handler, which nothing calls (the console keeps its own localStorage history), so it never fired. History shipped this way in #645.Fix
record()now does read → append →trim_to_caps→state::setfull replace, still fire-and-forget: a state failure never fails or delays the user's query, andstate::setis the size-guarded path.WorkerConfigknobs (hot-reloaded, read per write; whichever cap hits first, oldest entries dropped;0disables recording):history_max_entries— default 200history_max_bytes— default 262144 (256KB; worst-case set echo ≈ 3×256KB stays under the transport's 1MiB per-message cap)history:primary, the next history write replaces it.trim_to_capsuses exact compact-JSON byte accounting (no re-serialization loop); a unit test locks the formula againstserde_jsonto the byte. A single entry larger than the byte cap yields an empty list rather than a panic.append_opshelper are deleted;database::historykeeps its behavior (newest first, default 50).history_store_resultsknob: history was already metadata-only (SQL truncated to 4000 chars, verb, timing, row count — never result rows) and stays that way, now documented in the README.Tests
cargo fmt --check,cargo clippy --all-targets --all-features -- -D warnings,cargo testall green (385 tests); e2e schema fixture regenerated.Manual verification
On a live stack under heavy console use:
redis-cli HSTRLEN state:database history:primarystays under 262144; state worker shows no reconnect loop; engine stops loggingFunction not found: state::update. Related engine-side reconnect issue (secondary, after disconnect):engine#1993.Follow-up
acp/src/session.rsappend_historygrowssessions:{id}:historyunbounded via the same pattern — tracked separately (noted in MOT-4372).Summary by CodeRabbit
New Features
Bug Fixes
Documentation