RONDB-1074: Two-level hash scheme for lower latency index scans#987
RONDB-1074: Two-level hash scheme for lower latency index scans#987mronstro wants to merge 10 commits into
Conversation
Add PARTITION_HASH table metadata and DDL parsing for base/detail/fanout partition hashing. Validate unsupported create/alter combinations, preserve explicit comments, add a dedicated InvalidPartitionHash error, and gate the feature by data node version. Propagate compact partition hash metadata through dictionary signals to DBTC and DBSPJ. Use the fanout hash composition for NDB API hash calculation, DBTC/DBSPJ routing, and transaction hints while keeping the full primary-key hash for DBACC. Add DBTC base-key index-scan interval pruning for fanout tables using raw grouped base hashes resolved by DIH. Keep DBSPJ correct by disabling old single-fragment prune patterns for fanout tables until fanout interval pruning is implemented.
Document follow-up work from implementation review: ordered-index tables should persist copied partition hash metadata, direct NDB API and DBDICT paths need lower-layer validation, fanout interval scan pruning needs an unambiguous signal mode, and explicit scan partitioning APIs need defined behavior and tests.
Validate partition hash metadata in NDB API dictionary create/alter and in DBDICT parse so direct NDB API callers cannot bypass the SQL-layer checks, and reject values that would truncate in the compact kernel representation. Reject changing the metadata through alter. Persistently copy base-table partition hash metadata into ordered-index table descriptors when the index inherits base-table partitioning, keeping DBTC/DBSPJ table-record lookup local during ordered-index scans. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 32 bits of ScanTabReq::requestInfo are in use, so extended request-info bits are carried in the high half of storedProcId, which has been a constant 0xFFFF from every NDB API since 2004 and is therefore guaranteed zero from older senders. Bit 31 carries the new distribution key interval flag: distributionKey is a grouped partition-hash base hash and the scan covers the fanout interval of fragments starting at it. Readers check the high half for non-zero before decoding, and DBTC masks the low 16 bits when storing the stored-procedure id. The NDB API sets the flag only from the fanout base-key pruning path, via a new SPS_PARTITION_HASH_INTERVAL pruning state. MRR ranges only stay pruned when both prune kind and prune value match, since a raw distribution hash and a grouped base hash must never be merged. getPruned() reports interval-pruned scans as pruned. The flag is version gated on ndbd_support_partition_hash_fanout(), now 25.10.15+, 26.02.6+, 26.04.2+ and anything above 26.04. DBTC only enters the interval path when the flag is set. On fanout tables a pruned scan without the flag is rejected with new error 2203, since rows sharing a base partition key are spread over the fanout interval and a one-partition scan cannot be trusted to see all rows the application intended. This rejects fanout-unaware clients and explicit scan partitioning (SO_PARTITION_ID, SO_PART_INFO, setPartitionId()) loudly instead of silently returning partial results. An interval flag without the distribution key flag or on a table without fanout fails with a schema-version error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a second extended request-info bit, storedProcId bit 30, stating that ScanTabReq::distributionKey is a distinct fragment id and the scan covers exactly that fragment. The NDB API sets it for SO_PARTITION_ID, setPartitionId() and SO_PART_INFO with PS_USER_DEFINED, gated on ndbd_support_partition_hash_fanout() since old data nodes store storedProcId unmasked. The interval flag and the partition id flag are mutually exclusive. DBTC exempts distinct-fragment-id scans from the error 2203 rejection on tables with partition hash fanout: scanning a distinct fragment enumerates fragments rather than locating rows by key, so it is well defined regardless of fanout. This keeps TTL purge scans, which purge fragment by fragment using setPartitionId(), working on fanout tables without any DDL restriction. Hash-valued explicit scan partitioning (SO_PART_INFO with distribution key values) remains rejected on fanout tables. DBTC resolves the fragment id directly (distr_key_indicator = 1 towards DIH) instead of mapping it through the table distribution as a hash, which also keeps the resolution exact during hash-map changes, and validates the fragment id against the DIH fragment count, rejecting out-of-range ids with a fragment error instead of hash-mapping them to an arbitrary fragment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add mysql-test/suite/ndb/t/ndb_partition_hash_fanout.test (+ recorded result) covering the SQL-visible behavior of Phases 1-4: DDL coverage: - Positive creates: 1:0:1, 1:1:2, 1:2:4 and 2:1:4 (the latter two with explicit PARTITION BY KEY () PARTITIONS 8/4), and TTL combined with PARTITION_HASH (allowed since TTL purge uses the distinct-partition-id scan mode). - SHOW CREATE TABLE round-trip for every positive case, and verification that tables without an explicit PARTITION_HASH comment (plain and READ_BACKUP=1) never show a generated one. - Negative creates, all ER_ILLEGAL_HA_CREATE_OPTION at the SQL layer: malformed values (empty, 1, 1:1, 1:-1:2, 1:1:2:3, abc), invalid values (0:1:2, 1:1:0, key-count mismatch, 1:0:2, missing primary key), partition-count conflicts (fanout larger than or not dividing an explicit partition count), user-defined partitioning and fully replicated tables. A fanout of 70000 with default partitioning passes the SQL layer and is instead rejected below it with NDB error 800 (Invalid partition hash definition), surfaced via ER_CANT_CREATE_TABLE. - Alter coverage: PARTITION_HASH is preserved across an unrelated comment change (free comment text also preserved), a READ_BACKUP change and an online add column; changing PARTITION_HASH with ALGORITHM=INPLACE is rejected while ALGORITHM=COPY (table rebuild) is allowed; repartitioning to a count not divisible by the fanout and fully-replicated conversion are rejected. Data and pruning coverage (fanout 1:2:4 table with 8 partitions vs an identical full-PK-hash twin, 500 rows): - Base-key equality scans, base-key plus detail-range scans and ordered LIMIT scans record Ndb_pruned_scan_count = 1 (interval pruning). - Detail-only equality and two-base-key MRR scans are not pruned; full primary-key equality performs a key lookup with no scan. - Result-set equality with the twin table before and after PK and base-key-wide UPDATE/DELETE traffic. - Physical routing verified via ndbinfo.memory_per_fragment: 100 rows of a single base key occupy exactly fanout (4 of 8) fragments on the fanout table, all 8 on the twin, confirming the grouped base-hash interval placement. - fanout = 1 is verified to be a pure no-op: full-PK hash routing, one base key spreads over all fragments and base-key scans are not pruned (all hash paths gate on fanout > 1). Fix ha_ndbcluster.cc PARTITION_HASH error messages that exceeded the 64-byte limit asserted by Ndb_create_helper::failed_illegal_create_option (debug builds aborted, release builds truncated the message): shortened the malformed-form, key-count-mismatch and fanout-needs-detail-keys messages to fit. Update the plan document: clarify that fanout = 1 keeps current behavior (no interval semantics), and mark Phase 5 (DBSPJ interval pruning) as deferred to a future RonDB version. Add the DBSPJ worklog document preserving the design decisions and prune-machinery study (section ownership, T_CONST_PRUNE/T_PRUNE_PATTERN expansion plan, spare QueryTree wire bit, getNodes hash injection point) for when Phase 5 is picked up again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add five test cases to testPartitioning covering the direct NDB API behavior of partition hash fanout tables. All cases use a dedicated FanoutTest table with primary key (BaseKey, DetailKey), a Result column carrying a per-row checksum value, PARTITION_HASH spec 1:1:<fanout> set through the public NdbDictionary::Table::setPartitionHash() API, and eight explicit partitions (PartitionBalance_Specific). fanout_ddl: - Seven invalid metadata combinations are each rejected with error 800 (Invalid partition hash definition): zero base keys, fanout > 1 without detail keys, key counts not matching the primary key, zero fanout, fanout 70000 overflowing the compact Uint16 metadata, and two partition counts not divisible by the fanout. - A valid 1:1:4 spec survives the dictionary round-trip (getters match after retrieval from DICT). - Changing the spec through alterTable() is rejected with error 741. - A table created without any partition hash metadata is normalized to (distribution_key_count, remaining_keys, 1) and creates successfully, confirming computeAggregates() normalization protects plain direct-API creates from the base_count == 0 validation. fanout_pk_ops: - Per-row transactions hinted with startTransaction(table, Key_part_ptr) under error insert 8050, which crashes a data node if DBTC routes an operation to a fragment whose primary is not on the hinted TC node. Covers insert, locking read with value verification, masked update, write (upsert) and delete for all 512 rows: proves the API composed routing hash ((base_hash/z)*z + detail_hash%z) agrees with DBTC. fanout_interval_scan (no error insert: interval scans legitimately span several nodes): - Base-key-only equality prunes to the fanout interval (getPruned()). - Full primary-key equality bounds still prune as an interval. - Multiple MRR ranges with the same base key stay pruned. - MRR ranges with different base keys fall back to unpruned and return exactly the requested rows. - A prunable range mixed with an open range falls back to unpruned. Every scan verifies row contents against the Result checksum and the requested ranges, plus exact row counts. fanout_explicit_prune: - Hash-valued explicit scan pruning via Ndb::PartitionSpec, both the PS_DISTR_KEY_PART_PTR and PS_DISTR_KEY_RECORD variants, is rejected by DBTC with error 2203 (one-partition hash prune is ambiguous on fanout tables). - ScanOptions::SO_PARTITION_ID keeps its existing restriction on non-UserDefined tables (error 4546 at definition time). fanout_one: - PARTITION_HASH 1:1:1 keeps legacy behavior: hinted reads under error insert 8050 confirm routing uses the full-PK hash, full distribution-key equality scans prune to a single partition (SPS_ONE_PARTITION), and base-key-only equality does not prune since the interval machinery gates on fanout > 1. Update the plan document NDB API test section with coverage status and the gaps left for later: startTransaction(NdbRecord) hints, batch operations under hint verification, and unique-index operations. All five test cases pass against a live cluster. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n gate Audit results for backup/restore and online reorg of PARTITION_HASH fanout tables (documented in the plan): - ndb_restore needs no data-path changes: get_part_id() and explicit setPartitionId() are only used for UserDefined-partitioned tables, which cannot have PARTITION_HASH. Fanout rows are restored through ordinary primary key writes, so DBTC recomputes the composed routing hash and placement is correct by construction; the same holds for backup-log replay. The metadata restore preserves the partition hash spec through the backup's packed dictionary info, and PartitionBalance_Specific keeps the backup's fragment count. - Online reorg is correct by construction: per-row movement decisions flow through the fanout-aware Dbtc::hash() and DIH (the Tuple_header::REORG_MOVE bit is set via normal operations), and prepareHashMap() rejects new fragment counts that are not a multiple of the fanout. Fix one real gap: the direct NDB API createTable() had no data node version gate. Old DICT silently ignores unknown table properties, so an old cluster would create a fanout table as an ordinary table while the API client believes it has fanout routing. NdbDictInterface:: createTable() now fails with error 794 (Schema feature requires data node upgrade) when fanout > 1 and ndbd_support_partition_hash_fanout() is false for the connected data nodes. New test coverage, all passing: - mtr ndb_partition_hash_fanout: online reorg section. A single base key on a fanout-2 table with 4 partitions occupies exactly 2 fragments; ALTER ... ALGORITHM=INPLACE, ADD PARTITION PARTITIONS 4 succeeds with data intact, base-key scans still pruned and the base key still in exactly 2 of the now 8 fragments. Adding partitions to a non-multiple count is rejected: prepareHashMap fails with NDB error 800, surfaced as ER_GET_ERRMSG and documented via SHOW WARNINGS. - mtr ndb_partition_hash_fanout_restore (new): START BACKUP of loaded fanout tables, drop, ndb_restore -m -D / -r / -r -R, then verify restored row counts, base-key scan pruning (Ndb_pruned_scan_count), one base key in exactly fanout (4 of 8) fragments, and that writes to the restored table route correctly. - testPartitioning fanout_node_restart: normal followed by initial restart of one data node, then every row read back through an (unhinted) composed-hash routed primary key read plus the interval scan suite - a placement/routing mismatch shows up as error 626. - testPartitioning fanout_system_restart: graceful restartAll (an aborted shutdown intentionally discards transactions committed after the last durable GCP), then the same verification. The row read-back helper checks the operation-level NdbError after execute() since NdbRecord read failures do not necessarily fail the transaction execute; without this a missing row silently read back as zeroes. Update the plan document with the Phase 6 audit findings, manual mixed-version upgrade test steps (SQL layer message and API error 794), and a suggested performance test setup for fanout 1/2/4/8/16. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add ndb_partition_hash_fanout_upgrade, using the ndbmtd_v2 symlink mechanism from ndb_fk_upgrade_restart: run with MTR_RONDB_V2=1 and a pre-25.10.15 ndbmtd_v2 binary in the bin directory. The cluster starts with all data nodes on the old binary and the test performs a rolling upgrade by repointing the NDBMTD_LINK symlink and restarting one data node at a time. The test self-skips when ndbmtd_v2 is not available. Verified scenarios: - All-old cluster: CREATE with PARTITION_HASH is rejected with "PARTITION_HASH not supported by current data node versions" (the ha_ndbcluster gate on ndbd_support_partition_hash_fanout), while plain tables work normally. - Mixed-version cluster (node 1 upgraded, node 2 old): the create is still rejected, since the gate uses the minimum connected data node version; plain-table traffic keeps working across the rolling restarts. - Fully upgraded cluster: the create succeeds and the feature is immediately functional - base-key equality scans are pruned (Ndb_pruned_scan_count) and a single base key occupies exactly fanout (4 of 8) fragments. Update the plan document: upgrade/version testing is now automated; the only manual check left is the direct-API error 794 gate, which fires before spec validation for any fanout > 1 create against old data nodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
KernelMaker
left a comment
KernelMaker
left a comment
There was a problem hiding this comment.
Went through the routing and pruning logic. A few things inline - the first two can produce wrong results silently.
| error.code = CreateTableRef::InvalidPartitionHash; | ||
| return -1; | ||
| } | ||
| if (partition_count_known && (partition_count % fanout) != 0) { |
There was a problem hiding this comment.
This only checks that the fanout divides the partition count, but the interval scan also needs the fanout to divide the hashmap length. DIH resolves each raw hash as m_map[hashValue % m_cnt] with m_cnt = 3840 by default (240 with the small DefaultHashMapSize), and nothing checks m_cnt % fanout.
Example that passes validation: fanout=7 with 14 partitions and the default 3840-bucket map (map[i] = i % 14). Since 7 doesn't divide 3840, a grouped interval can straddle the end of the map: g % 3840 == 3836 gives buckets {3836..3839, 0, 1, 2} -> fragments {0, 1, 2, 3, 0, 1, 2}. sendDihGetNodeReq() appends all seven locations without any dedup, so fragments 0-2 get scanned twice with the same bounds and matching rows come back twice. Roughly (fanout-1)/3840 of base keys land on the wrap, so it's rare and silent. Rows are never missed (placement uses the same raw hashes), it's duplicates only.
This also bites power-of-two fanouts on a 240-bucket map (32 doesn't divide 240), and custom hashmaps installed via setHashMap() can break the alignment regardless of length.
Options: validate hashmap_length % fanout == 0 here and in Dbdict::handleTabInfoInit (plus something for non-default maps), or dedup the resolved fragment ids in DBTC before starting the fragment scans. Worth a fanout=7 test either way - the current tests all use fanouts that divide 3840, which is why this never showed up.
| * Partition-hash fanout needs one base-key prune key to scan a raw hash | ||
| * interval of fanout fragments. Until DBSPJ expands prune keys into that | ||
| * interval, do not push a prune pattern for fanout tables; the child scan | ||
| * remains correct by scanning all fragments with its normal index bounds. |
There was a problem hiding this comment.
This bail-out only protects clients built with this patch. An older mysqld/API client opens a fanout table as an ordinary table (SimpleProperties::unpack ignores unknown keys), still sees the legacy distribution-key metadata, and will serialize T_CONST_PRUNE / T_PRUNE_PATTERN for a pushed child scan like it always did. DBSPJ has the fanout metadata from TC_SCHVERREQ, but neither prune path looks at it - they go through the legacy computePartitionHash() and route the child scan to a single fragment, so rows sitting in the other fanout fragments silently disappear from the join result.
DBTC handles exactly this case kernel-side with error 2203; SPJ has no equivalent. Until Phase 5 does real interval expansion, rejecting T_CONST_PRUNE/T_PRUNE_PATTERN in parseScanFrag() when m_partition_hash_fanout > 1 would turn silent row loss during rolling upgrades into a loud error.
| { 796, DMEC, SE, "Out of schema transaction memory" }, | ||
| { 798, DMEC, AE, "A disk table must not be specified as no logging" }, | ||
| { 799, HA_WRONG_CREATE_OPTION, SE, "Non default partitioning without partitions" }, | ||
| { 800, HA_WRONG_CREATE_OPTION, SE, "Invalid partition hash definition" }, |
There was a problem hiding this comment.
800 is in the range this file reserves for TUP, and DbtupMeta.cpp already sets terrorCode = 800 in release builds when the table id is outside the configured table-record range. With this entry that config mismatch would now be reported as "Invalid partition hash definition", which will send whoever hits it down the wrong path since both errors show up on schema operations.
There's room in the 12xx block CreateTableRef already uses (1224, 1235, 1236) - probably easiest to move it there.
| if (fully_replicated) | ||
| return "PARTITION_HASH is not supported for fully replicated tables"; | ||
|
|
||
| if (user_defined_partitioning) |
There was a problem hiding this comment.
This rejects RANGE/LIST but lets PARTITION BY KEY(...) through - that path keeps HashMapPartition and just sets distribution keys on the columns, so the combination with PARTITION_HASH passes validation. The fanout code then ignores the declared keys completely (the base key is always the first base_count PK columns in key order), so PARTITION BY KEY(b) plus a PARTITION_HASH comment silently partitions by something other than what the user wrote.
The declared d-key flags also still go into DictTabInfo, which is what pre-fanout clients prune on - with a subset key declared, an old client prunes on equality over that single column, which makes the SPJ prune issue above much easier to hit.
I'd reject PARTITION_HASH when explicit partition-key columns are declared (or require them to match the base prefix).
| regCachePtr->tableref, | ||
| 0, | ||
| use_new_hash_function); | ||
| ndbrequire(ok); |
There was a problem hiding this comment.
This ndbrequire is reachable from a malformed request: handle_partition_hash() returns false when the KEYINFO is shorter than the fixed key length (total_len > srcLen), and keylen isn't validated against the table's key length anywhere before hash() runs - it comes straight from the TCKEYREQ. So on a fixed-key fanout table a buggy API client sending a short key crashes the TC node, where the old code just computed a wrong hash and failed further down with a normal error. The other false conditions look unreachable (the counts are validated by DICT, var-key tables go through handle_special_hash which fails gracefully), but this one is driven by the wire. Better to abort the operation with an error code.
2 participants
No description provided.