v2.11.0#49
Merged
Merged
Conversation
This document outlines the strategy for improving DecentDB's performance metrics in comparison to SQLite. It includes baseline metrics, execution plans, and completion criteria to ensure that enhancements do not regress durability or correctness. The focus is on achieving competitive performance in key areas such as point lookup latency, range scans, joins, and aggregate operations while maintaining existing strengths in bulk insert throughput and other metrics.
…ecoding, and improve FaultyVfsFile write handling - Changed QueryResult's columns from Vec<String> to Arc<[String]> for better memory efficiency. - Added a new method `with_shared_columns` to create QueryResult with shared columns. - Implemented `decode_projection_with_overflow` in Row to handle projections with overflow support. - Enhanced `decode_value_with_overflow` to decode various value types with overflow handling. - Added tests for row projection decoding to ensure correct materialization of requested columns. - Introduced batched write handling in FaultyVfsFile to honor failpoints and improve error handling. - Updated StatsVfsFile to track write statistics for batched writes. - Added a method in WalIndex to check for dirty pages since demotion. - Modified commit logic in WAL writer to conditionally demote cold versions based on dirty pages. - Updated benchmark summary and metrics documentation to reflect recent performance improvements.
- Added a new command-line argument `--benchmark` to run all scales (smoke, medium, full, huge) in order. - Introduced `BENCHMARK_SCALES` constant to define the order of benchmark scales. - Refactored the `run` function to handle benchmark execution and reporting. - Created `run_benchmark_suite` function to execute benchmarks for each scale and generate reports. - Added helper functions for file size retrieval and WAL path generation. - Updated the report generation logic to include additional metrics such as database and WAL sizes before and after checkpoints. - Added new JSON result files for benchmark runs at different scales (smoke, medium, full, huge) with detailed step metrics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed
indexed equality lookups on large checkpointed tables to degrade to
O(manifest_size) per-row linear scans. The cache is now built for all
tables with btree indexes or row_id alias columns, regardless of row count.
This fixes multi-second indexed lookups observed on tables with millions of
rows (Melodee DDB-002/DDB-003).
and FFI callers continue to read and commit correctly after WAL checkpoint
truncation.
partial-write failures preserve the expected file state.
Added
expanding the fast path beyond exact equality probes.
tables with 300K+ rows, verifying bounded lookup time after checkpoint and
reopen.
design/METRIC_IMPROVEMENTS_PLAN.mdto track DecentDB's currentbenchmark standing against SQLite and the priority order for closing or
exceeding each public-facing metric.
benchmarks/rust-baseline --benchmark, which runssmoke,medium,full, andhugescales in order for the selected engine/profile and thengenerates the HTML report.
Changed
metadata with
Arcand improving fast projection decoding for point-lookups.post-seed state for both engines: DecentDB uses
Db::checkpoint_wal()andSQLite uses
PRAGMA wal_checkpoint(TRUNCATE). The newcheckpoint_after_seedstep records duration plus WAL/database bytes beforeand after checkpointing.