Skip to content

fix(derive): grafema self-analyze at scale — cap-lift + compaction-suppress + plan-guard - #469

Closed
Disentinel wants to merge 1 commit into
mainfrom
fix/derive-batch-cap
Closed

fix(derive): grafema self-analyze at scale — cap-lift + compaction-suppress + plan-guard#469
Disentinel wants to merge 1 commit into
mainfrom
fix/derive-batch-cap

Conversation

@Disentinel

Copy link
Copy Markdown
Owner

Goal: make grafema analyze complete on grafema's OWN monorepo (~714k nodes / 1.33M edges). It failed three different ways at self-scale; all three fixed here. With this branch the full derive phase completes — 40 packs, 0 SIGSEGV, 0 plan rejects.

Fixes (all in packages/rfdb-server)

  1. cap-lift@materialize batch path used the interactive EvalLimits::default() 100k max_intermediate_results (the deadline was already lifted, the cap was overlooked). js_local_refs overflows it. Lifted for the batch path (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE). bin/rfdb_server.rs.

  2. compaction vs parallel-derive heap corruption (the hard one). Materialize holds a db.engine.read() lock but commits derived edges across 40 packs; those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps segments while the same materialize's par_join_rows rayon tasks read them → use-after-unmap heap corruption → SIGSEGV. Confirmed: TSan named join_derived/par_join_rows; rayon=1 passes; disabling compaction passes. Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize phase; deferred compaction runs at the natural barrier (end_bulk_load/explicit Compact) under the exclusive write lock. Correct phase serialization, not masking. coordinator.rs + bin/rfdb_server.rs.

  3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003)static_member in js_property_access_full estimates 11.7M facts > the 10M guard and is rejected, but the actual output is ~811 edges (a ~14,000× cardinality q-error: the estimator cross-products relation sizes, ignoring the highly-selective file+class+method join keys). Made the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M). plan.rs.

Verify

RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000 \
  grafema analyze --quickstart --verbose   # derive completes, 40 packs, 0 SIGSEGV

Open (separate, for follow-up / workers)

  • Wall 3 — enricher RPC 60s client timeouts (packages/rfdb/dist/client.js) + slow clear/drop of a large DB block full exit-0 + mcp:tool=27. The derive is done; the post-derive enrichers time out at scale. Raise/scope the RPC timeout; investigate drop perf.
  • CI self-analyze job (non-blocking first) so this never regresses silently (its absence hid all of the above).
  • Estimator q-error — selectivity-aware cardinality estimation so E-PLAN-003 stops spuriously tripping (then the env override is unnecessary).
  • Self-analyze is also unacceptably slow (~14 min); perf pass.

🤖 Generated with Claude Code

Self-analyze of the grafema monorepo (~714k nodes / 1.33M edges) failed three
different ways; all three were conservative limits / a concurrency bug that only
bite at self-scale. With these, the full derive phase completes (40 packs, 0
SIGSEGV, 0 plan rejects).

1. cap-lift (E-EXEC-001 sibling): the @materialize batch path used the interactive
   EvalLimits::default() 100k max_intermediate_results. The deadline was already
   lifted (RFDB_MATERIALIZE_DEADLINE_SECS) but the intermediate cap was not; batch
   derive of @stdlib/js_local_refs overflows it. Lift it for the batch path
   (default unbounded, env RFDB_MATERIALIZE_MAX_INTERMEDIATE).

2. compaction vs parallel-derive heap corruption (the hard one): the materialize
   handler holds a db.engine.read() lock but commits derived edges across 40 packs;
   those commits auto-triggered compaction (rayon, memmap2) which rewrites/unmaps
   segments WHILE the same materialize's par_join_rows rayon tasks read them ->
   use-after-unmap heap corruption -> SIGSEGV (confirmed: TSan named
   join_derived/par_join_rows; rayon=1 passes; disabling compaction passes).
   Fix: AutoCompactionSuppressGuard suppresses auto-compaction for the materialize
   phase; deferred compaction runs at the natural barrier (end_bulk_load / explicit
   Compact) under the exclusive write lock. Correct phase serialization, not masking.

3. MAX_MATERIALIZED_FACTS planner guard (E-PLAN-003): static_member in
   js_property_access_full estimates 11.7M facts > the 10M guard and is rejected —
   but the ACTUAL output is ~811 edges (a ~14,000x cardinality q-error: the estimator
   cross-products relation sizes, ignoring the highly-selective file+class+method join
   keys). Make the guard env-overridable (RFDB_MAX_MATERIALIZED_FACTS, default 10M).
   Deeper fix = selectivity-aware estimation (follow-up).

Self-analyze env for a full run:
  RFDB_MATERIALIZE_DEADLINE_SECS=3600 RFDB_MAX_MATERIALIZED_FACTS=200000000

Still open (separate): RPC 60s client timeouts + slow clear/drop (wall 3); the
estimator q-error; full exit-0 + mcp:tool=27 verification in flight.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Disentinel

Copy link
Copy Markdown
Owner Author

Superseded by the consolidated PR #475 (feat/self-analyze-hardening) — all 6 branches merged clean into one. Commits preserved there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant