Skip to content

chore: update submodule data_substrate for cc request bthread mutex deadlock fix #491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liunyl merged 4 commits into from
Jun 15, 2026
Merged

chore: update submodule data_substrate for cc request bthread mutex deadlock fix #491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b2444b8
chore: update submodule data_substrate for cc request bthread mutex d…
liunyl Jun 12, 2026
a6f9fd4
fix: build eloqkv_to_rdb in the non-cloud (ROCKSDB) config
liunyl Jun 14, 2026
aeb6d7a
chore: bump data_substrate to merged main (tx_service#491)
liunyl Jun 14, 2026
dce49a6
rebase with main
liunyl Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion data_substrate
Open in desktop
Submodule data_substrate updated 54 files
+45 −0 .github/workflows/unit-tests.yml
+62 −0 CLAUDE.md
+6 −0 CMakeLists.txt
+89 −0 docs/01-architecture-overview.md
+57 −0 docs/02-threading-model.md
+375 −0 docs/03-concurrency-control.md
+78 −0 docs/04-transaction-execution.md
+143 −0 docs/05-data-model-and-catalog.md
+266 −0 docs/06-distribution-and-clustering.md
+259 −0 docs/07-durability-and-recovery.md
+145 −0 docs/08-range-and-bucket-management.md
+202 −0 docs/09-store-handler.md
+176 −0 docs/10-log-service.md
+29 −0 docs/README.md
+289 −0 docs/standby_replication_protocol.md
+50 −63 tx_service/include/cc/cc_req_misc.h
+123 −141 tx_service/include/cc/cc_request.h
+9 −2 tx_service/include/cc/cluster_config_cc_map.h
+13 −3 tx_service/include/cc/template_cc_map.h
+3 −5 tx_service/include/sk_generator.h
+0 −388 tx_service/include/store/int_mem_store.h
+28 −1 tx_service/include/tx_key.h
+35 −8 tx_service/src/cc/cc_req_misc.cpp
+69 −2 tx_service/src/cc/cc_request.cpp
+0 −4 tx_service/src/cc/cc_shard.cpp
+6 −0 tx_service/src/cc/local_cc_shards.cpp
+15 −9 tx_service/src/remote/cc_node_service.cpp
+6 −7 tx_service/src/remote/remote_cc_request.cpp
+61 −44 tx_service/src/sk_generator.cpp
+93 −68 tx_service/tests/CMakeLists.txt
+0 −7 tx_service/tests/CcEntry-Test.cpp
+6 −7 tx_service/tests/CcPage-Test.cpp
+207 −0 tx_service/tests/CcRequestWait-Test.cpp
+217 −0 tx_service/tests/ClusterCrossNg-Test.cpp
+43 −25 tx_service/tests/LargeObjLRU-Test.cpp
+605 −0 tx_service/tests/MemDataStore-Test.cpp
+20 −94 tx_service/tests/StartTsCollector-Test.cpp
+81 −0 tx_service/tests/TestNodeSmoke-Test.cpp
+105 −0 tx_service/tests/TxConsistency-Test.cpp
+109 −0 tx_service/tests/cluster/scripted_host_manager.cpp
+84 −0 tx_service/tests/cluster/scripted_host_manager.h
+822 −0 tx_service/tests/cluster/test_cluster.cpp
+181 −0 tx_service/tests/cluster/test_cluster.h
+360 −0 tx_service/tests/cluster/txnode_bringup.cpp
+55 −0 tx_service/tests/cluster/txnode_bringup.h
+572 −0 tx_service/tests/cluster/txnode_main.cpp
+44 −0 tx_service/tests/cluster/txnode_workload.proto
+377 −0 tx_service/tests/harness/mem_data_store.cpp
+63 −0 tx_service/tests/harness/mem_data_store.h
+94 −0 tx_service/tests/harness/mem_data_store_factory.h
+148 −0 tx_service/tests/harness/port_util.h
+482 −0 tx_service/tests/harness/test_node.cpp
+151 −0 tx_service/tests/harness/test_node.h
+30 −7 tx_service/tests/include/mock/mock_catalog_factory.h
121 changes: 63 additions & 58 deletions src/tools/eloqkv2rdb/eloqkv2rdb.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,10 @@
* <http://www.gnu.org/licenses/>.
*
*/
#include <algorithm>
#include <array>
#include <atomic>
#include <cctype>
#include <chrono>
#include <condition_variable>
#include <cstdlib>
Expand Down Expand Up @@ -64,7 +66,6 @@
#include <rocksdb/env.h>
#include <rocksdb/options.h>

#include <algorithm>
#include <sstream>
#endif
extern "C"
Expand Down Expand Up @@ -818,6 +819,66 @@ struct ParseWorker
std::thread thd_;
};

// Parses a human-readable size string ("512KB", "1MB", "2GB", "1TB") into a
// byte count. Shared by both the cloud and non-cloud RDB writers as the
// write-buffer flush threshold, so it must be available regardless of
// ROCKSDB_CLOUD_EXPORT.
inline bool ParseSizeBytes(const std::string &size_str, uint64_t &bytes)
{
if (size_str.size() <= 2)
{
return false;
}

std::string number_part = size_str.substr(0, size_str.size() - 2);
std::string unit_part = size_str.substr(size_str.size() - 2);
std::transform(unit_part.begin(),
unit_part.end(),
unit_part.begin(),
[](unsigned char c) { return std::toupper(c); });

uint64_t multiplier = 0;
if (unit_part == "KB")
{
multiplier = 1024ULL;
}
else if (unit_part == "MB")
{
multiplier = 1024ULL * 1024ULL;
}
else if (unit_part == "GB")
{
multiplier = 1024ULL * 1024ULL * 1024ULL;
}
else if (unit_part == "TB")
{
multiplier = 1024ULL * 1024ULL * 1024ULL * 1024ULL;
}
else
{
return false;
}

if (number_part.empty() ||
!std::all_of(number_part.begin(),
number_part.end(),
[](unsigned char c) { return std::isdigit(c); }))
{
return false;
}

try
{
bytes = std::stoull(number_part) * multiplier;
}
Comment on lines +870 to +873

@coderabbitai coderabbitai Bot Jun 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

find . -name "eloqkv2rdb.cpp" -type f

Repository: eloqdata/eloqkv

Length of output: 97

🏁 Script executed:

wc -l src/tools/eloqkv2rdb/eloqkv2rdb.cpp

Repository: eloqdata/eloqkv

Length of output: 100

🏁 Script executed:

sed -n '820,880p' src/tools/eloqkv2rdb/eloqkv2rdb.cpp

Repository: eloqdata/eloqkv

Length of output: 1561

Add overflow check in ParseSizeBytes multiplication.

At line 872, multiplying std::stoull(number_part) * multiplier can silently overflow. If a very large numeric value (e.g., "18446744073709551615KB") is parsed, std::stoull succeeds and returns a uint64_t within range, but the subsequent multiplication overflows and wraps to a smaller positive value. This wrapped value passes the bytes > 0 check and is returned as valid, silently misconfiguring write-buffer flush thresholds.

Proposed fix 
     try
     {
-        bytes = std::stoull(number_part) * multiplier;
+        const uint64_t value = std::stoull(number_part);
+        if (value > (std::numeric_limits<uint64_t>::max() / multiplier))
+        {
+            return false;
+        }
+        bytes = value * multiplier;
     }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/tools/eloqkv2rdb/eloqkv2rdb.cpp` around lines 870 - 873, The
multiplication operation in the `ParseSizeBytes` function at the line where
`std::stoull(number_part) * multiplier` is calculated can silently overflow when
processing very large input values, causing the result to wrap to an invalid
smaller value that still passes downstream checks. Add an overflow guard before
the multiplication by checking if `number_part > UINT64_MAX / multiplier` would
be true; if so, either cap the value at UINT64_MAX, throw an exception, or
return an error to prevent the silent misconfiguration of size thresholds.

catch (...)
{
return false;
}

return bytes > 0;
}

#if ROCKSDB_CLOUD_EXPORT

struct S3UrlComponents
Expand Down Expand Up @@ -1248,62 +1309,6 @@ class ShardProgressPrinter
size_t last_line_size_{0};
};

inline bool ParseSizeBytes(const std::string &size_str, uint64_t &bytes)
{
if (size_str.size() <= 2)
{
return false;
}

std::string number_part = size_str.substr(0, size_str.size() - 2);
std::string unit_part = size_str.substr(size_str.size() - 2);
std::transform(unit_part.begin(),
unit_part.end(),
unit_part.begin(),
[](unsigned char c) { return std::toupper(c); });

uint64_t multiplier = 0;
if (unit_part == "KB")
{
multiplier = 1024ULL;
}
else if (unit_part == "MB")
{
multiplier = 1024ULL * 1024ULL;
}
else if (unit_part == "GB")
{
multiplier = 1024ULL * 1024ULL * 1024ULL;
}
else if (unit_part == "TB")
{
multiplier = 1024ULL * 1024ULL * 1024ULL * 1024ULL;
}
else
{
return false;
}

if (number_part.empty() ||
!std::all_of(number_part.begin(),
number_part.end(),
[](unsigned char c) { return std::isdigit(c); }))
{
return false;
}

try
{
bytes = std::stoull(number_part) * multiplier;
}
catch (...)
{
return false;
}

return bytes > 0;
}

// DSS value format constants and helpers.
// Value layout: [version_ts(8B, MSB=has_ttl)][ttl(8B optional)][record_data]
constexpr uint64_t kTtlMsb = 1ULL << 63;
Expand Down Expand Up @@ -2343,6 +2348,7 @@ int main(int argc, char *argv[])
return -1;
}

#if ROCKSDB_CLOUD_EXPORT
if (FLAGS_thread_count >
EloqKV::Tools::ShardProgressPrinter::kMaxScanThreads)
{
Expand All @@ -2352,7 +2358,6 @@ int main(int argc, char *argv[])
return -1;
}

#if ROCKSDB_CLOUD_EXPORT
if (!EloqKV::Tools::ValidateRequiredCloudFlags())
{
return -1;
Expand Down
Loading