fix(derive): grafema self-analyze at scale — cap-lift + compaction-suppress + plan-guard - #469
Closed
Disentinel wants to merge 1 commit into
Closed
fix(derive): grafema self-analyze at scale — cap-lift + compaction-suppress + plan-guard#469Disentinel wants to merge 1 commit into
Disentinel wants to merge 1 commit into
Conversation
Self-analyze of the grafema monorepo (~714k nodes / 1.33M edges) failed three different ways; all three were conservative limits / a concurrency bug that only bite at self-scale. With these, the full derive phase completes (40 packs, 0 SIGSEGV, 0 plan rejects). 1. cap-lift (E-EXEC-001 sibling): the @materialize batch path used the interactive EvalLimits::default() 100k max_intermediate_results. The deadline was already lifted (RFDB_MATERIALIZE_DEADLINE_SECS) but the intermediate cap was not; batch derive of @stdlib/js_local_refs overflows it. Lift it for the batch path (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE). 2. compaction vs parallel-derive heap corruption (the hard one): the materialize handler holds a db.engine.read() lock but commits derived edges across 40 packs; those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps segments WHILE the same materialize's par_join_rows rayon tasks read them -> use-after-unmap heap corruption -> SIGSEGV (confirmed: TSan named join_derived/par_join_rows; rayon=1 passes; disabling compaction passes). Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize phase; deferred compaction runs at the natural barrier (end_bulk_load / explicit Compact) under the exclusive write lock. Correct phase serialization, not masking. 3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003): static_member in js_property_access_full estimates 11.7M facts > the 10M guard and is rejected — but the ACTUAL output is ~811 edges (a ~14,000x cardinality q-error: the estimator cross-products relation sizes, ignoring the highly-selective file+class+method join keys). Make the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M). Deeper fix = selectivity-aware estimation (follow-up). Self-analyze env for a full run: RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000 Still open (separate): RPC 60s client timeouts + slow clear/drop (wall 3); the estimator q-error; full exit-0 + mcp:tool=27 verification in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Superseded by the consolidated PR #475 (feat/self-analyze-hardening) — all 6 branches merged clean into one. Commits preserved there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal: make
grafema analyzecomplete on grafema's OWN monorepo (~714k nodes / 1.33M edges). It failed three different ways at self-scale; all three fixed here. With this branch the full derive phase completes — 40 packs, 0 SIGSEGV, 0 plan rejects.Fixes (all in
packages/rfdb-server)cap-lift —
@materializebatch path used the interactiveEvalLimits::default()100kmax_intermediate_results(the deadline was already lifted, the cap was overlooked).js_local_refsoverflows it. Lifted for the batch path (default unbounded, envRFDB_MATERIALIZE_MAX_INTERMEDIATE).bin/rfdb_server.rs.compaction vs parallel-derive heap corruption (the hard one). Materialize holds a
db.engine.read()lock but commits derived edges across 40 packs; those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps segments while the same materialize'spar_join_rowsrayon tasks read them → use-after-unmap heap corruption → SIGSEGV. Confirmed: TSan namedjoin_derived/par_join_rows;rayon=1passes; disabling compaction passes. Fix:AutoCompactionSuppressGuardsuppresses auto-compaction for the materialize phase; deferred compaction runs at the natural barrier (end_bulk_load/explicitCompact) under the exclusive write lock. Correct phase serialization, not masking.coordinator.rs+bin/rfdb_server.rs.MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003) —
static_memberinjs_property_access_fullestimates 11.7M facts > the 10M guard and is rejected, but the actual output is ~811 edges (a ~14,000× cardinality q-error: the estimator cross-products relation sizes, ignoring the highly-selective file+class+method join keys). Made the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M).plan.rs.Verify
Open (separate, for follow-up / workers)
packages/rfdb/dist/client.js) + slowclear/drop of a large DB block full exit-0 +mcp:tool=27. The derive is done; the post-derive enrichers time out at scale. Raise/scope the RPC timeout; investigate drop perf.🤖 Generated with Claude Code