Exp 160: incremental view maintenance — the v2 stream engine

[danReynolds]

This is an architectural change, not a tuning experiment. It introduces a second execution strategy for reactive streams — materialized results maintained incrementally from row deltas — with the existing re-query path retained as the universal, fail-closed fallback. It is the largest change to the reactive engine since its creation, and it carries the experiment-protocol evidence trail (exp 160) on top of that.

The architecture

Today's engine answers every invalidation by re-executing the query (hash suppression saves decode/emission, never execution). This PR adds the machinery for the engine to instead understand queries and maintain their results:

preupdate hook ──► bounded row deltas ──► writer reply ──► stream engine
                                                              │
                                       classifier (at registration, fail-closed)
                                          ├─ full maintenance   → patch cached rows, emit
                                          ├─ windowed (LIMIT K) → top-K cache, patch/evict
                                          ├─ skip-only          → prove misses, else re-query
                                          ├─ aggregates         → maintain exact COUNT/SUM/MIN/MAX/AVG
                                          └─ everything else    → today's re-query path, unchanged

New durable infrastructure (each usable beyond this PR)

1. Row-delta channel (native/resqlite.c, lib/src/row_deltas.dart) — the preupdate hook captures per-row old/new values (bounded: 256 rows / 32 cols / 256 KB per cycle; poisoned by savepoint rollback or overflow), ships them in writer replies, decoded lazily. This is a CDC-grade primitive: diff-emitting streams, sync engines, and undo journals can all consume it later without touching native code again.
2. Query classifier (lib/src/stream_ivm.dart) — a strict registration-time grammar (bare columns, aggregates with aliases, AND-ed comparisons, composite ORDER BY with pk tiebreak, LIMIT) producing sealed admission shapes. Fail-closed by construction: anything unparsed or unprovable costs a re-query, never correctness.
3. Maintained state machines — full / windowed / skip-only / aggregate states with clone-on-write patching, complete-set window tracking, and exact integer aggregate maintenance.
4. A new ordering primitive: writer-ordered state builds. All IVM baselines (caches, aggregate snapshots) are built through reads on the writer port, whose FIFO totally orders a snapshot against the write replies that carry deltas — reader results and writer replies have no cross-port happens-before (a defect class this PR discovered, fixed, and recorded as a JOURNAL lesson). Paired invariant: maintained state survives only an unbroken chain of processed cycles; any bypassed delta-bearing cycle drops it for an exact rebuild.
5. Randomized equivalence harness (test/stream_ivm_equivalence_test.dart) — seeded write storms (rowid changes, NULLs, savepoint rollbacks, overflow batches) across every admission mode, every emission checked against a fresh select. It caught two ~1-in-5 ordering races deterministic replays never fired; it is the permanent safety net for all future IVM work.

Architectural properties

• Zero public API change — db.stream() is untouched; admission is an internal execution strategy.
• Fail-closed at every layer — capture overflow, classifier miss, unprovable cell, cache inconsistency, schema drift all degrade to today's path. Zero bails observed across all measured workloads.
• One-line revertible — remove the maintain check in the dirty loop and the system is byte-for-byte today's behavior.
• Emission semantics sharpen: admitted streams deliver each write's patch instead of coalescing behind re-query latency (the semantics of an infinitely fast re-query). The exp 045 microtask-batch pattern is the pre-identified mitigation if a workload prefers coalescing.

Evidence (full record: experiments/160-stream-delta-ivm.md)

evidence	result
Tracelite primary gate	many-streams −18.2%, formal improved verdict — the first cleared gate in this direction's history; reproduced order-flipped (+19.5% main-slower)
App-shaped admission audit	7/9 stream shapes admitted (tier 1 alone: 3/8); 7,643 invalidation decisions resolved with zero reader re-queries, 0 bails; burst wall 132.9 ms vs main ≈200 ms while delivering per-write patches
Exp-147 audit	overlap −19% with 500 vs 27 emissions delivered; keyed-PK −14% (matched pair)
Honest costs	high-cardinality density +3.5–4.8% (both collection orders): O(admitted-streams) main-isolate predicate eval replaces reader-distributed suppression — v3 fix specified (per-table equality-predicate index, O(1) dispatch); keyed-PK neutral for the full stack
Found & fixed en route	the diagnostics×reader NOMUTEX race (landed separately as #156) and the two cross-port ordering defects above

What this is not (yet) — the mapped ladder

Tier 4 one-hop joins, the equality-predicate dispatch index, rebuild-storm coalescing, and diff-carrying emissions are specified in signals.json with evidence trails, deliberately not built here.

Test plan

• 315/315 tests incl. 38 classifier/state-machine units and the equivalence harness; 20/20 clean harness loops post-fix
• dart analyze lib test benchmark clean; finalizer green
• Tracelite gate, two passes (standard + order-flipped), both committed in the writeup
• Release-suite record + profile aggregates committed (markdown only per CI guard)

🤖 Generated with Claude Code

[danReynolds]

Added the admission-rate benchmarking + an encapsulation pass (see latest commit): tier-1 admits 30/62 distinct entries on an app-shaped chat+feed stream mix and eliminates 3,000 re-queries in the burst with 0 bails — but the rejected set holds the highest-churn DESC+LIMIT screens (~3,100 remaining reader replies), which is the quantified case for tier-2 (composite-key ordering + LIMIT K+buffer). Per-cycle delta decode also moved out of the engine into RowDeltaBatch; engine-side IVM state is now just StreamEntry.ivm + the table_info cache. 298/298 tests, analyze clean, finalizer green.

[danReynolds]

Pushed three updates (branch rebased onto main with #153 merged — conflicts in the experiments metadata resolved by union):

1. Fix for a pre-existing data race this PR surfaced (own commit, cherry-pickable if you'd rather land it separately): Database.diagnostics() calls sqlite3_db_status on reader connections guarded by in_use — dead code since exp 030's dedicated-reader assignment — so the main isolate was reading live NOMUTEX connections; SCHEMA_USED measures via the connection's pnBytesFreed dry-run, corrupting a mid-query reader's allocation accounting (flaky reader SEGV, ~1-in-30 stream_test runs once this PR's detached admission reads made readers busy at diagnostics-poll time). Read workers now bracket each request with resqlite_reader_set_busy (two ~ns leaf FFI calls); the sacrifice path clears it before Isolate.exit. Bisection + 100/100 clean stress runs in the experiment doc.
2. Test hardening: the e2e IVM tests used fixed 60ms settles and flaked under full-suite load; now poll-until-count / quiet-window. 8/8 clean full-suite runs (~1-in-3 flaked before).
3. JOURNAL: added the transferable lesson (a dead guard flag is a data race waiting for a traffic pattern).

Full suite 8×, analyze, and finalizer all green post-rebase.

[danReynolds]

v2 tier stack pushed (per the goal to take this as far as it goes — all isolated on this branch). Summary of what's now here beyond tier 1:

Tiers — one fail-closed grammar, three admission modes:

• Full maintenance extended with composite ORDER BY intCol [DESC], pk [DESC] (explicit tiebreak ⇒ exact tie order) and LIMIT K windows (top-K cache, complete-set tracking, departures-from-full-windows fall back); TEXT equality under introspected no-COLLATE tables.
• Skip-only (tier 1.5): proven-miss elision for unmaintainable-result shapes (DESC w/o tiebreak, OFFSET, DISTINCT…) — no cached state at all.
• Aggregates (tier 3): COUNT/SUM/MIN/MAX/AVG … AS alias, exactly seeded and maintained; extremum departures re-seed.

Headline results:

evidence	result
Tracelite gate	many-streams −18.2%, formal improved verdict — first cleared primary gate in this direction's history; reproduced order-flipped (+19.5% main-slower)
App-shaped admission audit	7/9 shapes admitted (3/8 at tier 1); 7,643 invalidation decisions with zero reader re-queries, 0 bails; burst wall 132.9 ms vs main's ~200 ms while delivering per-write patches
Honest costs	high-cardinality +3.5–4.8% both orders (O(admitted-streams) main-isolate eval at 100-streams-one-table density — v3 fix identified: per-table equality-predicate index for O(1) dispatch); keyed-PK neutral for v2 (tier-1 alone won it)

Two ordering defects found and fixed by the new randomized equivalence harness (3 seeds × 12 rounds × 9 streams, every emission vs fresh select; both fired ~1-in-5 under load, never in deterministic replays): reader-built IVM baselines race late writer replies → all state now builds through writer-ordered reads; and maintained state survives only an unbroken chain of processed cycles → any bypassed delta-bearing cycle drops the state (a hash-suppressed re-query validates emissions without re-syncing state). Both are recorded as a JOURNAL lesson (cross-port replies have no happens-before).

315/315 tests, 20/20 equivalence-loop runs, analyze + finalizer green.

[danReynolds]

Review: exp 160 (tiered incremental stream maintenance)

Reviewed the native delta capture, stream_ivm.dart, row_deltas.dart, the two ordering fixes in stream_engine.dart, and the equivalence harness (3 focused passes + reproduction probes). This is genuinely impressive work — the tiered fail-closed design is sound, the evidence is strong (twice-reproduced Tracelite wins, first cleared primary gate in this direction), and most of the safety machinery holds up under scrutiny: savepoint/overflow/OOM poisoning, the structurally fail-closed decoder, affinity handled by construction via apply-time is! int gates, NULL semantics mirroring SQL, real clone-on-write, the -1 hash sentinel, writer-ordered reads for steady-state rebuilds.

But the "zero correctness loss" claim does not currently hold. The review found correctness bugs that emit a wrong result instead of bailing — all fixable (bail-instead-of-emit), but blockers for a correctness-critical feature. Ranked:

Blockers (wrong, non-bailed result)

1. Failed standalone write leaks reliable deltas. A non-tx execute/executeBatch that fails mid-statement (UNIQUE/NOT NULL on row K of a multi-row DML) has already appended deltas for rows 1..K-1 with reliable==1. The C error paths (resqlite_execute, resqlite_run_batch's internal ROLLBACK) don't discard the buffer, and the Dart wrappers throw before drainRowDeltas — so the next successful write drains the rolled-back rows as reliable and patches streams with never-committed rows. (Savepoint rollback is poisoned; this standalone path isn't.) Fix: discard/poison the delta buffer on any write error before drain. Not covered by the harness.
2. SUM/AVG int64 overflow wraps silently (reproduced): IVM emits -9223372036854774809 where a fresh re-query raises integer overflow. IvmAggregateState.contribute accumulates in a Dart int (stream_ivm.dart:1314). Fix: detect overflow → bail+reseed. Harness bounds values to [0,1000) so never hits it.
3. Stale baseline through the async admission window (stream_engine.dart:434-466, _scheduleFullBuild/_scheduleAggregateReseed): the entry is in _tableIndex before admission's writer-read completes, so writes during the await re-query (newer state) while admission then installs the older snapshot — no generation/dirty guard around the assignment (the steady-state bypass path has one; admission doesn't). Same defect class the experiment claims closed, on an uncovered path. Fix: generation-counter guard, discard the build if the entry was invalidated meanwhile.

Should-fix (crash / half-apply instead of bail)

4. Non-UTF-8 TEXT cell throws instead of bailing: utf8.decode defaults to allowMalformed:false (row_deltas.dart:108), so CAST(x'ff' AS TEXT) in any column of a modified row throws a FormatException up through the invalidation pass (half-applied + unhandled async error). Fix: allowMalformed:true or catch→null.
5. colCount allocates before range-check (row_deltas.dart:85,146): a corrupt colCount near INT_MAX triggers a multi-GB List.filled before any cell read bails. Cheap cap restores fail-closed.
6. WITHOUT ROWID tables can be admitted (latent fail-open): PRAGMA table_info reports their INTEGER PK as a rowid alias; keys then track rowid not the PK cell. Couldn't weaponize in this SQLite build, but no guard exists — add an explicit WITHOUT ROWID rejection.

Test gaps

7. The equivalence harness never asserts IVM actually engaged — no check on ivmAdmittedTotal/ivmBailTotal. If admission silently broke (or finding Fix 8 correctness and robustness issues #3 forced perpetual fallback), every stream degrades to re-query and the harness still passes green — validating the wrong path. Add ivmAdmittedTotal>0 && ivmBailTotal==0 assertions.
8. Add coverage for: failed-standalone-then-success (Restore Isolate.exit for zero-copy large result transfer #1), SUM near int64 max (Add write lock and nested transaction support #2), non-UTF-8 TEXT (Fix 8 correctness and robustness issues #3), and ALTER TABLE ADD COLUMN (schema drift). The same-column-count rename hazard should be a documented in-code limitation, not only in the writeup.

Merge-readiness

The branch is 48 commits behind main with conflicts in writer.dart (exp 159/180 rewrote the write path — coalescing pump), write_worker.dart, database.dart, and resqlite.c (the diagnostics×reader fix here also shipped in 0.5.0). A substantial rebase is required, and the rebase needs care precisely because it intersects the coalescing pump.

Recommendation: this is high-value, accept-shaped work — but not mergeable until the three blockers bail correctly, the decode-path hardening lands, and the harness asserts engagement (so it can actually catch regressions in these paths). Then rebase onto the post-180 write path.

— Reviewed via 3 focused correctness passes; the SUM-overflow finding is reproduced, #1 and #3 are high-confidence code traces worth confirming with the suggested tests.