Security intern01#22
Open
markusw-ron wants to merge 137 commits into
Open
Conversation
…i-col PK (logicalclocks#908) CompileIndexRanges in rdrs_dal.cpp iterated the index record's attrIds in ascending base-table attrId order via getFirstAttrId/getNextAttrId, but the user-supplied range values in lower/upper.values are in user-specified index-column order. For a composite secondary index on a table whose PK columns are interleaved with the index columns in attrId space (e.g. PK=(client_id, occurred_at, event_id), secondary =(customer_number, occurred_at); cn attrId=3, ts attrId=1), the two orders disagree and each value is memcpy'd at the wrong column's offset. A VARCHAR binary landing in a BIGINT slot spills past m_row_size and corrupts adjacent heap metadata in m_scan_buffer; glibc notices later at NdbScanOperation::close() and aborts rdrs2 with "corrupted size vs. prev_size". Derive curr_attrId from the user-supplied column (node.col->getAttrId()) at each iteration instead of walking the record's attrIds. Symmetric change in both lower and upper loops. Add MTR regression test mysql-test/suite/rdrs2/t/rdrs2_scan_composite_index.test using the minimal 3-row schema that triggers the attrId-order mismatch, with both DESC and ASC ordered scans. Pre-fix it crashes rdrs2; post-fix both return the expected rows. Full Go scan suite (mysql-test/suite/rdrs2-golang/rdrs2-golang_index_scan) also green. Also fix TTLPurger VARBINARY encoding of "API_OK" (ttl_purge.cpp): setValue("message", ...) on the VARBINARY(255) column in mysql.ndb_schema_result reads the first byte as a length prefix, treating 'A' (0x41=65) as length and memcpy'ing 1+65 bytes from a 7-byte literal. Junk on the wire in a normal build; under ASan a fatal global-buffer-overflow that blocked every ASan MTR run and was caught while regression-testing the scan fix. Framed as a proper VARBINARY buffer: 1-byte length (6) + "API_OK".
…i-col PK (logicalclocks#908) (logicalclocks#912) CompileIndexRanges in rdrs_dal.cpp iterated the index record's attrIds in ascending base-table attrId order via getFirstAttrId/getNextAttrId, but the user-supplied range values in lower/upper.values are in user-specified index-column order. For a composite secondary index on a table whose PK columns are interleaved with the index columns in attrId space (e.g. PK=(client_id, occurred_at, event_id), secondary =(customer_number, occurred_at); cn attrId=3, ts attrId=1), the two orders disagree and each value is memcpy'd at the wrong column's offset. A VARCHAR binary landing in a BIGINT slot spills past m_row_size and corrupts adjacent heap metadata in m_scan_buffer; glibc notices later at NdbScanOperation::close() and aborts rdrs2 with "corrupted size vs. prev_size". Derive curr_attrId from the user-supplied column (node.col->getAttrId()) at each iteration instead of walking the record's attrIds. Symmetric change in both lower and upper loops. Add MTR regression test mysql-test/suite/rdrs2/t/rdrs2_scan_composite_index.test using the minimal 3-row schema that triggers the attrId-order mismatch, with both DESC and ASC ordered scans. Pre-fix it crashes rdrs2; post-fix both return the expected rows. Full Go scan suite (mysql-test/suite/rdrs2-golang/rdrs2-golang_index_scan) also green. Also fix TTLPurger VARBINARY encoding of "API_OK" (ttl_purge.cpp): setValue("message", ...) on the VARBINARY(255) column in mysql.ndb_schema_result reads the first byte as a length prefix, treating 'A' (0x41=65) as length and memcpy'ing 1+65 bytes from a 7-byte literal. Junk on the wire in a normal build; under ASan a fatal global-buffer-overflow that blocked every ASan MTR run and was caught while regression-testing the scan fix. Framed as a proper VARBINARY buffer: 1-byte length (6) + "API_OK".
* Ring Buffer Table: CREATE TABLE DDL parsing and metadata plumbing
Propagate 3 new ring buffer metadata fields (ring_buffer_size,
ring_idx_col_no, ring_meta_col_no) through the full NDB signal chain:
MySQL handler -> NdbApi -> DBDICT -> DBLQH -> DBTUP (not DBTC).
- Add DictTabInfo tags 166-168 for signal serialization
- Extend CreateTabReq (NewSignalLengthLDMWithRingBuffer = 6+18)
- Extend AlterTabReq SignalLength 14->17 with change mask bit 18
- Parse COMMENT="NDB_TABLE=RING_BUFFER=size@idx_col@meta_col" syntax
- Validate: idx column must be INT and last PK column, meta column
must be VARBINARY(>=16) and nullable, size 1..2^31-1
- Enforce mutual exclusion with TTL
- Add ALTER TABLE prepare/revert/commit staging in DBLQH and DBTUP
- Add RING_BUFFER preservation during ALTER TABLE COMMENT updates
- Add MTR test ndb_ring_buffer_create with 18 test cases
* Ring Buffer Table: INSERT write path with kernel write guard
Add the ring buffer INSERT handler and kernel-level write protection:
- NDB API: OO_RING_BUFFER_OP / OF_RING_BUFFER_OP operation flags
- Flag chain: NdbAPI -> TcKeyReq bit 29 -> LqhKeyReq bit 9 -> TUP Operationrec
- DBTUP write guard: reject writes to ring-buffer tables without
OO_RING_BUFFER_OP or OP_REPLICA_APPLIER flag (error 940)
- LQH packLqhkeyreqLab: re-set ring_buffer_op after preComputedRequestInfoMask
filtering, ensuring backup replicas receive the flag
- ha_ndbcluster: ndb_ring_buffer_write_row() handler that reads meta row
(ring_idx=0) with exclusive lock, computes next slot, writes data row
via writeTuple, and inserts/updates meta row atomically
- Block REPLACE, IODKU, and user-specified ring_idx/ring_meta at handler level
- Intercept ring buffer inserts before peek_indexed_rows() to avoid false
duplicate key match on meta row
- Require DEFAULT on ring_idx column (DDL validation)
- Row count tracking: ha_write_count and uncommitted_rows with accurate delta
- Fix commit 1 AlterTabReq signal length overflow (14 base + 17 with ring buffer)
- Fix Dbdict::alterTable_toLocal conditional signal length
Tests: ndb_ring_buffer_insert (12 cases), ndb_ring_buffer_create (19 cases)
* Ring Buffer Table: bulk INSERT optimization + ring_size=1 fix + test suite
Bulk INSERT optimization: cache meta row state in memory and batch all
data writes for same-prefix rows, reducing NDB round-trips from N+1 to
2 per PK-prefix group. Three-path design: Path A (batch hit, no execute),
Path B (first row per prefix, meta read + cache), Path C (single-row,
original behavior).
Fix ring_size=1 bug: init_first_insert() hardcoded next_pos=2 without
wrapping, causing the second insert to create an orphan row at slot 2
instead of overwriting slot 1. Fixed by reusing advance() which wraps
correctly for any ring_size.
Move ring buffer tests from ndb suite to dedicated ndb_ring_buffer suite.
* Ring Buffer Table: READ filtering hides meta rows + ndb_ring_buffer_show_meta session variable
* Ring Buffer Table: UPDATE/DELETE restricted operations + test suite
UPDATE: allow user-column changes on data rows, block writes to
ring_idx/ring_meta columns and meta row (ring_idx=0). Replica
applier bypasses all checks.
DELETE: allow WHERE on PK-prefix columns (=, IN, OR, BETWEEN,
range) and bare DELETE (no WHERE) which clears the entire table.
Block WHERE referencing ring_idx or non-PK columns. Validation
runs in both start_bulk_delete() and ndb_delete_row() to catch
zero-match cases. Delete scans include meta rows via
SO_RING_BUFFER_SHOW_META so they are cleaned up with data rows.
36 MTR test cases covering: PK and scan updates, system column
protection, meta row protection, full/partial prefix deletes,
OR/BETWEEN/IN operators, bare DELETE, empty table, wrapped ring
lifecycle, DELETE with LIMIT, and cross-operation integrity.
* Ring Buffer Table: ALTER TABLE grow support + block shrink/disable
ALTER TABLE support for ring buffer tables via both inplace and
copying ALTER paths. Growing ring_size is allowed — empty slots are
filled first, then overwriting resumes. Shrinking and disabling are
blocked to prevent data corruption and shrink-bypass via
disable→re-enable.
Inplace ALTER: add ring buffer COMMENT parsing to
inplace_parse_comment() following TTL pattern. Validates size,
column names, blocks shrink/disable, sets ring buffer metadata on
the new table object for change detection.
Copying ALTER: bypass ring buffer INSERT intercept so rows are
copied as-is. Set OO_RING_BUFFER_OP to satisfy kernel write guard.
Include meta rows in copy scan via SO_RING_BUFFER_SHOW_META.
DblqhProxy: accept conditional AlterTabReq signal length (14 or
17) to prevent data node crash on inplace ring buffer ALTER.
Grow adjustment in INSERT path: when ring grew via ALTER, next_pos
may point to an occupied slot. Detect (!ring_full && next_pos <=
count) and redirect to first empty slot (count + 1).
15 MTR test cases covering: grow, shrink blocked, disable blocked,
same-size no-op, chained grows, ring_size=1 grow, bulk insert
after grow, ADD COLUMN preserves ring, multi-prefix grow, not-full
ring grow, heavily-wrapped ring grow.
* Ring Buffer Table: comprehensive test suite + BLOB/TEXT column support
Add comprehensive MTR test suite (31 cases) covering integration
scenarios not addressed by individual operation tests: joins,
subqueries, aggregations, TRUNCATE, transactions (COMMIT/ROLLBACK),
data types (DATETIME/DECIMAL/TEXT), SELECT patterns, complex
operation sequences, edge values, multi-session show_meta scope,
and VIEW support.
Fix BLOB/TEXT column support for ring-buffer tables. The ring buffer
write path was missing set_blob_values() calls after writeTuple(),
so NdbBlob handles were created but never populated with data.
Additionally, NdbBlob internally creates updateTuple() operations
on the main table to update blob head+inline bytes, but these
operations were missing the OO_RING_BUFFER_OP flag required by the
kernel write guard, causing error 940 on INSERT.
* Ring Buffer Table: backup/restore, replication, node restart support + bug fixes
Add test suites for backup/restore (7 cases), replication (8 cases), and
node restart (4 cases). Fix multiple kernel-level issues discovered during
testing:
- Fix ASSERT_BOOL crash during redo replay: ring_buffer_op/show_meta not
initialized in seizeTcrec() and initReqinfoExecSr()
- Fix LCP row count mismatch: Backup/LCP scans now include meta rows via
setRingBufferShowMetaFragFlag in Backup.cpp and Suma.cpp
- Fix LCP restore blocked by write guard: restore.cpp sets ring_buffer_op
in LqhKeyReq
- Fix node restart copy: set ring_buffer_show_meta=1 on copy scan and
ring_buffer_op=1 on copy TC record for ring buffer tables
- Fix redo replay: set ring_buffer_op=1 for ring buffer tables in
initReqinfoExecSr()
- Fix SIGSEGV on non-ring-buffer tables: replace getRingBufferSize()>0
with isRingBuffer() (RNIL=0xffffff00 is >0, matching all tables)
- Fix replication applier blocked by write guard: set OO_RING_BUFFER_OP
for applier writes to ring buffer tables
- Add set_ring_buffer_op() to NdbOperation for ndb_restore support
* Ring Buffer Table: comprehensive test suite + BLOB/TEXT column support
Fix BLOB DELETE crash: in NdbBlob::getHeadFromRecAttr(), treat absent
blob heads (theNullFlag == -1) as NULL instead of asserting. Ring buffer
meta rows are inserted without BLOB columns, so blob heads are absent
by design.
Add 5 new replication test cases: TRUNCATE replication, ALTER TABLE grow
replication, re-INSERT after TRUNCATE, DELETE+re-INSERT cycle.
Add 5 new test suites for ring buffer tables:
- secondary_index: index scans, UNIQUE index, ALTER TABLE ADD INDEX
- blob_text: BLOB/TEXT column DML
- concurrent: multi-session concurrent operations
- misc_dml: prepared statements, LOAD DATA, INSERT IGNORE, stored procs
- ddl_edge: CREATE LIKE/AS SELECT, RENAME, VARCHAR PK, triggers
* Ring Buffer Table: NOT NULL column support + SUMA meta row fix + DDL guards
Fix TRIX/SUMA index build path: SUMA now hides meta rows for
SingleTableScan subscriptions (index builds) while keeping them
visible for TableEvent subscriptions (replication). This allows
ALTER TABLE ADD UNIQUE INDEX to succeed on ring buffer tables.
Add NOT NULL user column support: meta rows get zero-default values
(0 for INT, '' for VARCHAR) for NOT NULL columns via expanded meta
mask in both bulk and single-row write paths.
Add kernel trigger filtering: DBTUP skips SECONDARY_INDEX and FK
triggers for meta rows (ring_idx=0), preventing duplicate key
collisions and FK violations from zero-default values.
Add DDL guards:
- Block NOT NULL BLOB/TEXT columns (meta mask can't set BLOB zero-defaults)
- Block FOREIGN KEY constraints (wrapping auto-deletes break FK semantics)
Fix non-deterministic test ordering in blob_text.test.
* Ring Buffer Table: block creating index on internal columns
Prevent creating secondary indexes (ordered, unique, composite) on
ring buffer internal columns (ring_idx, ring_meta) during both
CREATE TABLE and ALTER TABLE ADD INDEX.
* Ring Buffer Table: NdbRingBufferWriter NDB API helper + test suite
Add NdbRingBufferWriter, a C++ helper class that encapsulates the ring
buffer INSERT protocol at the NDB API level. This enables non-MySQL
clients (ClusterJ, direct NDB API users) to insert into ring buffer
tables without going through the MySQL handler layer.
The writer manages ring_idx and ring_meta columns automatically:
- Reads the meta row (ring_idx=0) with exclusive lock
- Computes the next data slot from the ring metadata
- Writes the data row with OO_RING_BUFFER_OP flag
- Batches rows with the same PK prefix (single meta read)
- Flushes meta row update on PK prefix change or explicit flush()
Key implementation details:
- Resets transaction commit status after expected 626 (tuple not found)
on first insert for a PK prefix, matching the handler's behavior
- Handles both 1-byte and 2-byte VARCHAR length prefixes (charset-safe)
- Zeros NOT NULL columns in meta rows to satisfy NDB constraints
- Friend access to NdbTransaction for commit status reset after 626
Test suite (8 cases): single insert, ring wraparound, multiple PK
prefixes, batch insert, NOT NULL columns, ring_size=1, non-ring-buffer
table rejection, and multi-transaction meta persistence.
* Ring Buffer Table: concurrent insert race condition test
Add test demonstrating that two concurrent writers for the same PK
prefix, when no meta row exists, race on the meta row insertTuple.
NDB cannot lock a non-existent row, so LM_Exclusive in readMetaRow()
does not protect the first-insert case. NDB error 630 (duplicate
tuple) / MySQL error 1062 (ER_DUP_ENTRY) is the safety net.
Test covers both NDB API (deterministic via barrier) and MySQL
(probabilistic, ~25-50% hit rate over 100 iterations).
* Ring Buffer Table: rename RING_BUFFER to MAX_ROWS_PER_PK + NdbAPI test expansion
Rename the user-facing COMMENT keyword from RING_BUFFER to MAX_ROWS_PER_PK
for clarity — the value describes user intent (max rows per PK prefix),
not the implementation mechanism.
Make the @idx_col@meta_col suffix optional: when omitted, the engine
defaults to columns named ring_idx and ring_meta. Users who want custom
column names can still use MAX_ROWS_PER_PK=SIZE@col@col.
Add 5 new NdbAPI test cases (BLOB/TEXT, delete+re-insert, rollback,
multi-wrap batch, multi-column PK prefix) and integrate the NdbAPI test
binary into the MTR framework as ndb_ring_buffer.ndbapi_ring_buffer_test.
Fix benign thread cleanup warning in both NdbAPI test binaries by scoping
Ndb/Ndb_cluster_connection destruction before ndb_end().
* Ring Buffer Table: fix validation bugs + hardening
- Fix server crash on empty MAX_ROWS_PER_PK= (std::stoll("") threw
unhandled exception) in both CREATE TABLE and inplace ALTER paths
- Fix ring_meta minimum size validation: require >= 32 (RING_META_SIZE)
instead of >= 16, preventing metadata truncation on INSERT
- Fix DELETE condition bypass: recursively validate all FUNC_ITEM
arguments (not just FIELD_ITEM), preventing nested functions like
LENGTH(non_pk_col) from bypassing the PK-prefix-only restriction;
also accept Item::PARAM_ITEM for prepared statement compatibility
- Fix strncpy buffer overflow in update_comment_info when preserving
old MAX_ROWS_PER_PK value during ALTER TABLE
- Add warning log in DBTUP meta row filter when readAttributes fails
- Concurrent test: increase iterations 100->1000 with early break on
first duplicate detected to reduce flakiness while keeping the hard
pass requirement
- Add bugfix_validation regression test suite (3 cases)
- Update create.test cases 2-4: VARBINARY(16)->VARBINARY(64)
* Ring Buffer Table: fix BLOB DELETE + harden test suite
Fix: DELETE with WHERE on ring buffer tables with BLOB/TEXT columns
failed to delete data rows. The implicit blob head read added by
NdbBlob::preExecute() did not inherit ring_buffer flags from the
main operation, causing the kernel's meta-row filter to reject the
read with error 626 on ring_idx=0 rows. This aborted the scan after
only the meta row, leaving all data rows behind.
Propagate OF_RING_BUFFER_OP and OF_RING_BUFFER_SHOW_META from the
main operation to the blob head read operation in NdbBlob::preExecute().
Test changes:
- blob_text Case 6: correct expected result for BLOB DELETE
- blob_text Case 8: use distinguishable value for wrap verification
- update_delete Cases 8,22: add ROW_COUNT() checks for no-op DML
- ddl_edge Case 8: document bare COUNT(*) known limitation
- backup_restore Case 8: fix misleading wrap comment
* Ring Buffer Table: comprehensive ClusterJ test suite
Add 20 test cases covering the full ClusterJ ring buffer API surface:
- Core: single insert, fill+wrap, multiple prefixes, batch, cross-tx continuity
- P0: batch exceeding ring size, mixed/interleaved prefixes in single tx,
transaction rollback, delete and update via ClusterJ API
- P1: exact ring fill boundary, multiple wrap cycles, null user columns,
non-existent slot lookup, meta row visibility, query/scan operations
- P2: session reuse across transactions, empty transaction, multi-wrap batch
Also fix schema.sql (remove stale SELECT test query for ring_buffer_sensor)
and register test+model in clusterj-test CMakeLists.txt.
* Ring Buffer Table: ClusterJ ring buffer insert support
Add transparent ring buffer insert protocol to ClusterJ. When a table
has MAX_ROWS_PER_PK configured, makePersistent() automatically manages
the ring_idx/ring_meta columns and the meta row protocol.
NdbJTIE layer:
- Expose isRingBuffer/getRingBufferSize/getRingIdxColumnNo/getRingMetaColumnNo
on NdbDictionary.Table via JNI
- Add OO_RING_BUFFER_OP and OO_RING_BUFFER_SHOW_META operation flags
- Fix sizeOfOptions parameter (was hardcoded 0) for insert/update/write ops
ClusterJ layer:
- Table interface: isRingBuffer, getRingBufferSize, getRingIdxColumn,
getRingMetaColumn
- TableImpl: detect ring buffer at construction, cache column references
- ClusterTransactionImpl: RingBufferWriter lifecycle, readTupleExplicitLock
for 626-tolerant reads, executeNoCommitDirect for mid-tx execution
- RingBufferWriter: meta row read/write, slot computation with wrap-around,
batch optimization for same PK prefix, resource management
- NdbRecordRingBufferInsertOperationImpl: specialized insert operation
* Ring Buffer Table: harden test suite + fix PK prefix bug + unique index tests
Fix RingBufferWriter PK prefix copy using col.getSize() instead of
ndbRecordImpl.lengths[] (element count vs byte size). Add error 626
transaction state reset after meta row not found. Fix insertTuple call
to pass both key and attribute NdbRecords.
Harden ClusterJ RingBufferTest: add sensor_value assertions to 11 tests,
wire in 2 unreachable concurrent tests, assert meta row not visible via
ClusterJ find(), tighten query scan to exact result count, add timestamp
range validation for concurrent test, normalize concurrent state for
deterministic result file output.
Add 3 unique index test cases (dup within prefix, wrap-time collision,
freed value reuse) and 4 NdbAPI test cases (unique dup, wrap collision,
freed reuse, concurrent same-prefix). Add Session.isRingBufferTable()
API. Bump MaxNoOfTables/MaxNoOfAttributes for ClusterJ schema.
* Ring Buffer Table: fix zeroColumn bug + align ClusterJ with NDBAPI
Cross-implementation review of NdbRingBufferWriter (C++) vs
RingBufferWriter (Java) found three issues:
1. zeroColumn() used ndbRecordImpl.lengths[] (element count, e.g. 1 for
INT) instead of col.getSize() (byte count, e.g. 4 for INT). This
caused NOT NULL fixed-length columns in meta rows to be only
partially zeroed, leaving garbage bytes if the column had a non-zero
default. copyPkPrefix() in the same file already used col.getSize()
correctly.
2. batchActive was set to true inside readMetaRow() before writeDataRow()
ran. If writeDataRow() threw, the writer was left in an inconsistent
state with batchActive=true but no data written. Moved the assignment
to addRow() after writeDataRow() succeeds, matching the C++ ordering.
3. Clarified the ring buffer guard in NdbRecordOperationImpl.insert()
with a comment explaining why it exists: the SmartValueHandler / DTO
cache flow calls insert() directly, bypassing the dedicated
NdbRecordRingBufferInsertOperationImpl.endDefinition() path.
Added testNotNullColumns() to the ClusterJ test suite with a new
ring_buffer_notnull table (VARCHAR(50) NOT NULL, INT NOT NULL,
MAX_ROWS_PER_PK=3) that exercises the zeroColumn fix: verifies data
row values, meta row zero-fill (score=0, name=''), and wrap-around.
* Ring Buffer Table: code review follow-ups (perf + clarity)
Five small fixes from a cross-layer review of the ring-buffer feature:
1. DBTUP read filter: replace per-row readAttributes() interpreter
call with the existing isRingBufferMetaRow() direct-offset helper.
ring_idx is a fixed-size INT in the PK so it's always in main
memory at a compile-time offset. Same semantics, no per-row
interpreter overhead on ring-buffer scans, and consistent with
the trigger code in DbtupTrigger.cpp which already uses the helper.
2. ha_ndbcluster.cc: remove dead/duplicate comment blocks. The
"validated in ndbcluster_push_to_engine" comment in ndb_delete_row
was wrong (push_to_engine is not called for DELETE) and the
matching stub comment in ndbcluster_push_to_engine had no
implementation. Consolidated the surviving comment to accurately
describe the validation flow (start_bulk_delete + ndb_delete_row
fallback, cached in m_ring_buffer_delete_allowed).
3. NdbBlob::getHeadFromRecAttr: narrow the absent-blob-head
relaxation to operations that carry OF_RING_BUFFER_OP or
OF_RING_BUFFER_SHOW_META on the parent op. Previously the
relaxation fired for any non-event read where the blob head was
absent, silently masking what used to be an assertion. Restricting
it to ring-buffer ops preserves the meta-row scan fix while
restoring the pre-ring-buffer assertion for every other absent
blob head.
4. Suma::SyncRecord::nextScan: replace the SingleTableScan/TableEvent
ndbassert with an explicit switch defaulting unknown subscription
types to show_meta=0. The assert added no correctness value
(the ternary already produced safe behavior) and would crash debug
builds if a future subscription type were added.
5. DELETE error message on ring-buffer tables: "DELETE requires full
PK prefix WHERE on ring-buffer table" is misleading -- bare DELETE
(no WHERE) is allowed and the validator does not require all
PK-prefix columns to appear, only that referenced columns are
PK-prefix columns. Replaced both call sites with "DELETE WHERE on
ring-buffer table may only reference PK-prefix columns (excluding
ring_idx)" and re-recorded the affected result files
(update_delete, bugfix_validation).
All 18 ring-buffer MTR tests pass with these changes (one ClusterJ
test skipped due to missing MTR_CLASSPATH; unrelated).
* Ring Buffer Table: ClusterJ DynamicObject test coverage
Add 12 testDynamicObject* methods to RingBufferTest to exercise the
DynamicObject reflection/schema-driven DTO path, which is distinct from
the annotation-interface path that all previous tests used.
- Inner DTOs RingSensorDTO / RingNotNullDTO (only table() overridden).
- Helpers setSensorFields / setNotNullFields walk columnMetadata() and
set user columns by index; ring_idx and ring_meta left with mask bits
clear so RingBufferWriter owns them (same convention as the
annotation-interface tests, which never call setRingIdx()).
- First 8 tests (ColumnMetadata, SingleInsert, FillRing, BatchInsert,
BatchExceedingRingSize, MixedPrefixes, DtoCacheInsert, NotNullColumns)
write via DynamicObject and read back via the RingBufferSensor /
RingBufferNotNull annotation interfaces — any value-buffer layout
asymmetry between the two DTO paths surfaces as a mismatch.
- Last 4 tests (Read, Update, Delete, QueryScan) drive both halves
through DynamicObject to cover the remaining DTO-layer operations
that diverge between paths. QueryBuilder uses SQL column names for
DynamicObject types (confirmed pattern from DynamicObjectTest.java).
* Ring Buffer Table: clusterj baseline for 26.05-main rebase
Re-record clusterj_ring_buffer baseline after rebasing from 25.10-main
to 26.05-main, and sanitize one env-specific print so the baseline is
reproducible across machines.
Baseline absorbs three changes:
- Jar version string 25.10.5 -> 26.02.2 (consequence of the base bump).
- Six new [DO-META] lines from testDynamicObjectColumnMetadata (added
in the previous commit but not yet recorded).
- "[DIAG] Checking meta row from second mysqld" print no longer includes
the mysqld2 JDBC URL. MTR assigns mysqld2 a different port base on
different machines, so embedding the actual port made the recorded
baseline env-specific. The url2 variable is still computed and used
for the DriverManager connection; only the diagnostic print is
shortened.
* Ring Buffer Table: extract write-guard and meta-row-filter predicates
Review follow-up for PR logicalclocks#909. Extract the two inline blocks in
DbtupExecQuery.cpp (ring-buffer write guard in execTUPKEYREQ, meta-row
filter in handleReadReq) into named predicate helpers on Dbtup, placed
next to is_ring_buffer_table / isRingBufferMetaRow:
- is_ring_buffer_write_blocked(...)
- is_ring_buffer_meta_row_hidden(...)
jam/terrorCode/tupkeyErrorLab/return stay at the call site. The write
helper takes a pre-extracted is_replica_applier bool (not the raw LQH
flags) to avoid pulling Dblqh::TcConnectionrec into Dbtup.hpp —
Dblqh.hpp already includes Dbtup.hpp, so the reverse would be circular.
A third review comment suggested caching the ring-buffer-table bit on
the Operationrec to skip the Tablerec load on the hot path. Deferred:
duplicating that state requires setting the new flag correctly at every
operation-setup site (keyop, scan, LCP, backup, nr-copy, redo, etc.),
and any missed site fails open (write through or meta-row leak). Doing
it safely needs an audit of every setup path plus a debug assertion
against Tablerec drift; that is a separate, deliberate change.
* Ring Buffer Table: extract meta-tuple predicate for trigger-fire paths
Four identical sites in DbtupTrigger.cpp (lines 690, 717, 751, 895) used
the same inline block to decide whether to skip secondary-index and FK
triggers on ring-buffer meta rows:
const bool skip_idx_fk =
unlikely(is_ring_buffer_table(regTablePtr)) &&
isRingBufferMetaRow(regTablePtr, req_struct->m_tuple_ptr);
Collapse into a new inline helper is_ring_buffer_meta_tuple() in
Dbtup.hpp, placed next to the other ring-buffer predicates.
Deliberately does not consult Operationrec.ring_buffer_op /
ring_buffer_show_meta — trigger skip is a tuple property, not an
operation property (SUMA triggers still need to fire on the meta row
for replication).
* Ring Buffer Table: extract handler code into ha_ndbcluster_ring_buffer.{h,cc}
Move Ring_meta, the DELETE WHERE Item-tree walker, and the
ndb_ring_buffer_write_row / flush_ring_buffer_batch member-fn bodies
out of ha_ndbcluster.cc. Logic unchanged.
execute_no_commit loses its static-inline qualifier so the moved
bodies can call it from the new TU.
* Ring Buffer Table: collapse call sites via ndb_ring_buffer helpers
Unify the two MAX_ROWS_PER_PK parsers, the 4 scan-path show_meta
checks, and the create_index guard behind namespace helpers.
No behavior change; ndb_ring_buffer MTR 19/19.
…i-col PK (logicalclocks#908) (logicalclocks#912) (logicalclocks#914) CompileIndexRanges in rdrs_dal.cpp iterated the index record's attrIds in ascending base-table attrId order via getFirstAttrId/getNextAttrId, but the user-supplied range values in lower/upper.values are in user-specified index-column order. For a composite secondary index on a table whose PK columns are interleaved with the index columns in attrId space (e.g. PK=(client_id, occurred_at, event_id), secondary =(customer_number, occurred_at); cn attrId=3, ts attrId=1), the two orders disagree and each value is memcpy'd at the wrong column's offset. A VARCHAR binary landing in a BIGINT slot spills past m_row_size and corrupts adjacent heap metadata in m_scan_buffer; glibc notices later at NdbScanOperation::close() and aborts rdrs2 with "corrupted size vs. prev_size". Derive curr_attrId from the user-supplied column (node.col->getAttrId()) at each iteration instead of walking the record's attrIds. Symmetric change in both lower and upper loops. Add MTR regression test mysql-test/suite/rdrs2/t/rdrs2_scan_composite_index.test using the minimal 3-row schema that triggers the attrId-order mismatch, with both DESC and ASC ordered scans. Pre-fix it crashes rdrs2; post-fix both return the expected rows. Full Go scan suite (mysql-test/suite/rdrs2-golang/rdrs2-golang_index_scan) also green. Also fix TTLPurger VARBINARY encoding of "API_OK" (ttl_purge.cpp): setValue("message", ...) on the VARBINARY(255) column in mysql.ndb_schema_result reads the first byte as a length prefix, treating 'A' (0x41=65) as length and memcpy'ing 1+65 bytes from a 7-byte literal. Junk on the wire in a normal build; under ASan a fatal global-buffer-overflow that blocked every ASan MTR run and was caught while regression-testing the scan fix. Framed as a proper VARBINARY buffer: 1-byte length (6) + "API_OK".
* Version bump to 24.10.20 * TTL: allow NdbBlob lock-upgrade scans to route to backup replica (logicalclocks#918) DbtcMain.cpp:16438 ndbrequire was too strict for TTL+BLOB+READ_BACKUP/ FULLY_REPLICATED scans: NdbBlob::atPrepareNdbRecord upgrades the user's LM_CommittedRead to LM_Read for blob-read atomicity and stamps rcb=1. The outer if accepts the scan (rcb=1, op_count=0), but the inner check "!ttl_table || (!lockmode && !holdlock)" then fired on the now-set holdlock and killed the data node with error 2341. Allow rcb=1 through the assertion; mirrors the non-TTL policy already documented at sendDihGetNodeReq ("safe to read from backup replicas"). "Read what you locked, even if expired" is preserved because the outer if still requires op_count==0, so no prior in-txn locks exist when the replica branch fires. * ndb_ttl: regression test for TTL+BLOB+READ_BACKUP scan crash Autocommit SELECT of a BLOB column on a TTL table with READ_BACKUP=1 or FULLY_REPLICATED=1 used to kill the data node picked for the scan with ndbrequire at DbtcMain.cpp sendDihGetNodesLab. This test runs both table shapes and expects the two inserted rows back. --------- Co-authored-by: Salman Niazi <salman@logicalclocks.com>
Adds a new DB-level config ArbitrationRankWait (ms, default 60000) so QMGR waits for a rank-1 candidate (typically ndb_mgmd) before falling back to a rank-2 SQL node when electing an arbitrator. If a rank-1 candidate joins later while a rank-2 fallback is active, QMGR demotes the rank-2 arbitrator and re-elects so the mgmd takes over. The ndb_mgmd startup path now keeps the MGM service reachable for data-node bootstrap while rejecting ordinary MGM client commands until arbitration has settled. This lets data nodes allocate node ids, fetch config, set ports, and establish transporter connections without making readiness probes such as ndb_mgm show succeed too early. When a connected data node lacks support for the new behaviour, the startup gate is capped at 2 s to avoid penalising mixed-version upgrades. MgmtSrvr::wait_until_arbitrator() also folds a cold-start check into the wait loop. If no data node has connected within the cold-start window, the gate returns early so a freshly-started cluster does not idle waiting for an arbitrator role it cannot yet be assigned. TransporterFacade gains the support predicates for connected DB nodes and unsupported DB-node versions. Move the mgmd active-arbitrator marker to the point where the arbitrator thread has entered started state and sent ARBIT_STARTCONF back to Qmgr. The code documents why ARBIT_STARTREQ implies all current data nodes have completed the PREP2 agreement on arbitrator node and ticket, including the two-data-node president/non-president failure ordering. Report rank-2 arbitrator demotion as a state transition via reportArbitEvent so it lands in each data node's local ndbd.log, not only the cluster log. Adds ArbitCode::ApiDemoted and a matching getTextArbitState line. Also mirrors the rank-2 fallback infoEvent with g_eventLogger->info so fallback context is visible locally while mgmd is down. Adds MTR coverage under suite/ndb: ndb_arbitration_rank_wait verifies the full handoff trail (mgmd -> rank-2 -> mgmd), ndb_rolling_restart_mgmd_first walks a rolling restart starting with mgmd, and ndb_arbitration_president_failover stops mgmd, waits for rank-2 takeover, kills the president data-node child, restarts mgmd, and verifies data nodes can be cycled afterward. The warm-restart tests use restart:--config-change to bypass MTR's ndb_mgmd_wait_started --no-contact path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…logicalclocks#916) Switch SF_OrderBy to SF_OrderByFull so NDB injects index key columns into the kernel-side result mask itself, instead of requiring callers to list them in readColumns. The JSON projection is still driven by rdrs2's own read_columns vector, so the response shape is unchanged. Pre-patch, ordered scans with a partial readColumns set hit NDB error 4341. Adds orderby_full_test.go covering secondary composite and primary indexes with zero / partial / full / read-all readColumns, ASC and DESC, plus a multi-fragment merge-sort variant on big_tbl. Side-fix in index_scan_helper.go: SQL ORDER BY now repeats the direction per key column. SQL "ORDER BY a, b DESC" means "a ASC, b DESC", which disagrees with NDB's SF_Descending across all key columns.
RONDB-1061: Bump version 26.02.3
RONDB-1058: Prefer ndb_mgmd as arbitrator during restart
RONDB-1058: Arbitrator fixes merged from 25.10-main
RONDB-1058: Arbitration fixes merged to 26.05-main
RONDB-1036: Outer join pushdown aggregation in MySQL
…KEYREQ RONDB-1051: Performance improvement of LQHKEYREQ
mronstro
force-pushed
the
RONDB-1049-security
branch
2 times, most recently
from
May 21, 2026 18:40
f0255f7 to
f0190eb
Compare
…cks#931) Documents the post-logicalclocks#916 ordered-scan key-column relaxation, the actual x-api-key requirement in production, the accepted-but-ignored dataReturnType on scan, null/empty handling for readColumns/filters/index, the Request-too-large 400, and per-type response value encodings (DECIMAL/BIT/BINARY/CHAR/dates/YEAR; unsupported types emit "Unexpected column type").
Reject limit<0, bound.values longer than the index key, varchar/longvarchar CMP values with non-string JSON kind. Flip filter depth check to `>=` so MAX_FILTER_DEPTH=32 caps at exactly 32 nesting levels.
…features RONDB-1053: Work on Rondis features
RONDB-1049: Security improvements in TCKEYREQ
Bump version to 26.02.5
Step 3 Candidate B of the AggInterpreter / JoinAggInterpreter
unification. Extracts the ~270-line post-prelude of the kOpLoadCol
dispatch arm into a single shared helper on AggInterpreterBase.
Each subclass kOpLoadCol body is now reduced to its per-class
prelude -- the part that reads the column data into m_attr_read_buf
and sets up the `header` / `attrDescriptor` / linked-attr state.
The bulk (TypeSupported check, DECIMAL precision/scale word read,
register init with NULL early-return, big switch on every NDB_TYPE_*
including the string capture + m_attr_read_pos advance) becomes a
single loadColumnTypedFromBuf call on the base.
Per-class prelude differences preserved (the original reason
kOpLoadCol stayed per-class in Step 1.4):
AggInterpreter prelude
Simple block_tup->readSingleAttribute path; attrDescriptor always
non-null; linked_cte_attr always false; never enters the
linked-attr / NULL-injection branches of the shared helper.
JoinAggInterpreter prelude
Four-way branch: (a) linked-attr CTE marker (sets linked_cte_attr,
linked_word0/word1 from the linked buffer), (b) linked-attr normal
column, (c) NULL-extended row injection (m_null_local_columns),
(d) regular readSingleAttribute. attrDescriptor null for the
linked paths; non-null for (d). linked_cte_attr drives the CTE
metadata branch inside the shared helper's string case.
Helper signature lets each subclass pass its own state without
having the helper itself depend on subclass-only fields:
Int32 loadColumnTypedFromBuf(
DataType type, bool is_unsigned, Uint32 reg_index,
AttributeHeader* header,
const Uint32* attrDescriptor, // null for linked-attr
bool linked_cte_attr, // JoinAgg-only true
Uint32 linked_word0, Uint32 linked_word1, // CTE type metadata
Dbtup::KeyReqStruct* req_struct,
Uint32& exec_pos, // DECIMAL operand word
const char* class_name); // warning prefix
AggInterpreter's verbose DEB_AGG / PA_INTERP_TRACE / per-type traces
are preserved as the canonical form (consistent with Step 1.4 of the
unification -- JoinAgg gains them in VM_TRACE builds; production
builds compile both out).
Net: 583 deletions / 461 insertions across the four interpreter
files (about 122 lines absolute reduction; the structural win is
that the kOpLoadCol arm in each subclass is now a one-page prelude
plus a single helper call, not a 300-line type-dispatch switch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The aggregation interpreter's attribute-read scratch buffer is used
only inside ProcessRec -- ProcessRec runs single-threaded per LDM
thread and the buffer's liveness is bounded by one ProcessRec call.
That makes the per-interpreter copy redundant; one scratch buffer per
LDM thread serves any number of concurrent interpreters on that
thread.
Add an inline buffer to Dbtup (one Dbtup instance per LDM thread):
Uint32 m_agg_attr_read_buf[MAX_TUPLE_SIZE_IN_WORDS];
Uint32* getAggAttrReadBuf() { return m_agg_attr_read_buf; }
static constexpr Uint32 AGG_ATTR_READ_BUF_WORD_SIZE =
MAX_TUPLE_SIZE_IN_WORDS;
Sized to MAX_TUPLE_SIZE_IN_WORDS (= 18000 words = 72 KB) so any
single-row projection RonDB can construct fits without overflow.
Previous per-interpreter buffer was ATTR_READ_BUF_WORD_SIZE = 2048
words = 8 KB, which was enough for typical workloads but capped
single-column reads at 8 KB total. The new size covers any column
the data dictionary permits.
AggInterpreterBase::m_attr_read_buf becomes a per-call pointer:
ProcessRec sets it on entry from block_tup->getAggAttrReadBuf().
g_attr_read_buf_len_ initialiser switches from ATTR_READ_BUF_WORD_SIZE
to Dbtup::AGG_ATTR_READ_BUF_WORD_SIZE.
initBufBlock loses the m_attr_read_buf carve -- the per-interpreter
m_buf_block allocation shrinks by 8 KB on every interpreter instance.
JoinAggInterpreter::processNullExtendedRow gains a Dbtup* parameter
so it can pass through to ProcessRec. Previously this path called
ProcessRec(nullptr, ...) because no row data is read; the nullptr
prevented binding m_attr_read_buf. DBLQH callers updated to pass
c_tup.
ATTR_READ_BUF_WORD_SIZE macro removed. Stale decimal-scratch local
variables (decimal_info / precision / scale / dec_ret / dec_buf_ptr
/ dec_val_*) in each subclass ProcessRec removed -- they were
ProcessRec-scoped locals used by the old inline kOpLoadCol body and
became dead in Step 3 Cand-B when loadColumnTypedFromBuf absorbed
them. Compiler picked up the unused-variable warnings.
Memory impact:
Per-Dbtup-instance: +72 KB (one per LDM thread)
Per-interpreter: -8 KB (m_buf_block shrinks)
For a node with 8 LDM threads and N concurrent interpreters across
those threads:
Before: N * 8 KB (per-interpreter buffers)
After: 576 KB fixed (8 * 72 KB Dbtup buffers)
+ (N * 0 KB) [interpreters lost their copy]
Crossover ~ 72 concurrent interpreters; above that, the new design
saves. More importantly, the 72 KB ceiling absorbs the worst-case
single-row projection (RonDB's MAX_TUPLE_SIZE_IN_WORDS) instead of
returning ZAGG_OTHER_ERROR on wide rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Route CTE lookups directly to the owning DBLQH by hashing the lookup key in DBSPJ with the local CTE aggregation interpreter, including normalization of CTE virtual group-by attributes and keeping scalar CTE lookups local. Add an explicit CTE aggregation self-feed error path for lookup and scan requests so invalid reuse of the source CTE aggregation state is reported instead of hanging or corrupting state. Make join aggregation interpreter cleanup bounded across scan and JoinAgg release paths, pass jam buffers through interpreter processing, and add guards for linked-only CTE aggregation feeds.
Record the post-Step-3 work behind commits 86627c3, a9aed46, 6ef2ae2, and 934ef29 in the plan docs: - agg_interpreter_unification_plan.md: new Step 4 section covering 4a (per-LDM attr_read_buf scratch), 4b (Init + kOpLoadCol factoring), 4c (bounded beginTeardown/tearDownChunk teardown via CONTINUEB), and 4d (CTE join-agg routing + self-feed + linked-only hardening) with the multi-node ronsql_cte_partial_key SIGSEGV root-cause writeup. - CLAUDE.md: index summary bumped to "Steps 1, 2, 3a, 3b, and 4 complete" with a 4a-4d recap. - cte_filter_phase_k.md: note that the CTE_LOOKUP_ROUTE_FLAG + routeCteLookup forwarding scheme is superseded by Step 4d's direct owner routing; ANTI_JOIN flag semantics unchanged. Docs only; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DBSPJ now sends CTE_LOOKUP_REQ directly to the computed owner in production without setting CTE_LOOKUP_ROUTE_FLAG. Retain the DBLQH rerouting path under VM_TRACE or ERROR_INSERT so debug builds can still diagnose unexpected lookup ownership mismatches.
Round Robin groups are used to decide which query threads can assist reads for an LDM thread, so the group count should be driven by query threads rather than by every CPU that needs a binding. Keep send-thread CPU coverage out of the RR group count and map extra CPU-map entries back onto existing RR groups. Prefer larger L3-local RR groups where possible, while still merging undersized L3 groups when offline CPUs leave fragments that are too small. Keep the allocation bounded by L3 locality when it can satisfy the query-thread layout, but avoid creating many tiny RR groups that cannot carry useful LDM/query work. Improve LDM placement and validation. Spread LDM assignment to avoid neighbouring LDM instances in the same RR group where possible. Require all substantial RR groups to have an LDM, allow at most one tiny RR group without one, and log per-RR-group thread summaries during startup for diagnostics. Extend NdbHW unit coverage with deterministic RR group validation and randomized synthetic topology tests using the same automatic thread-count calculation as production. Add a compile-time NDBHW_TEST_VERBOSE flag for detailed CPU-map output while keeping normal random-test runs practical.
Fix RONDB-1067, not setting m_low_load properly RONDB-1068: Improve assignment of RR Groups
CTE continuation
Merge 26.05 into 26.04-main
First cherry pick
Cherry pick 2 security system v1.1
…hasution DoS on oversized m_num-comm_sections
…BSPJ reportMaliciousSignal plumbing
markusw-ron
force-pushed
the
security-intern01
branch
from
July 13, 2026 08:58
26fd8a0 to
0d7084f
Compare
…length validation in parseDA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.