Skip to content

refactor flush data worker #470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lokax merged 38 commits into from
Mar 25, 2026
Merged

refactor flush data worker #470

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
52efbdf
ckpt
lokax Feb 26, 2026
edfb761
phase1-2
lokax Feb 26, 2026
9bdfca4
phase3
lokax Feb 26, 2026
dbe2316
phase4-5
lokax Feb 26, 2026
93784b8
putall phase1
lokax Feb 26, 2026
21a05d7
phase2
lokax Feb 26, 2026
c9dfe4f
phase4
lokax Feb 26, 2026
d2c27e6
phase6
lokax Feb 26, 2026
5b5a9f4
phase7
lokax Feb 26, 2026
2369b98
phase8
lokax Feb 27, 2026
d95eb70
1 worker and fix build
lokax Feb 27, 2026
ae2497d
yield to
lokax Feb 27, 2026
983bccf
yield queue
lokax Feb 28, 2026
6fcb60b
remove yield queue
lokax Feb 28, 2026
f01465f
yield
lokax Feb 28, 2026
300f1cf
debug log
lokax Feb 28, 2026
3f3b8a1
yield update range slices
lokax Feb 28, 2026
d8196ba
map buffer
lokax Feb 28, 2026
96330be
remove debug log
lokax Mar 3, 2026
9d47118
log
lokax Mar 3, 2026
b9751ba
rocksdb handle
lokax Mar 9, 2026
90c191b
fix heap order
lokax Mar 9, 2026
626ee0d
debug log
lokax Mar 11, 2026
8085b55
add ctr dctr
lokax Mar 11, 2026
8db09a8
debug log
lokax Mar 12, 2026
bda648d
debug log
lokax Mar 12, 2026
bf99521
remove range heap
lokax Mar 12, 2026
cf0365d
remove comment
lokax Mar 13, 2026
1cdd8b0
debug log
lokax Mar 13, 2026
2fe44a1
debug log
lokax Mar 13, 2026
0c09d6c
fix log replay service
lokax Mar 16, 2026
ed3fd9a
fix build
lokax Mar 18, 2026
85da27c
remove log and heap
lokax Mar 18, 2026
b284ed6
debug log
lokax Mar 18, 2026
511c84d
fix
lokax Mar 19, 2026
2a98057
fix hash ckpt heap
lokax Mar 19, 2026
616d1ab
remove log
lokax Mar 19, 2026
21b539c
remove debug log
lokax Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions build_tx_service.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,17 @@ find_package(Protobuf REQUIRED)
find_package(GFLAGS REQUIRED)
find_package(MIMALLOC REQUIRED)

# boost_context for FlushDataWorker coroutine refactor (Phase 1)
if(CMAKE_BUILD_TYPE STREQUAL "Debug" AND CMAKE_CXX_FLAGS MATCHES "fsanitize=address")
find_library(Boost_CONTEXT_LIBRARY NAMES boost_context-asan)
else()
find_library(Boost_CONTEXT_LIBRARY NAMES boost_context)
endif()
if(NOT Boost_CONTEXT_LIBRARY)
message(FATAL_ERROR "libboost_context not found")
endif()
find_package(Boost 1.70 REQUIRED)

Comment on lines +33 to +43

@coderabbitai coderabbitai Bot Mar 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify how ASAN is configured in CMake and whether repo tooling expects boost_context-asan.
rg -n --glob 'CMakeLists.txt' --glob '*.cmake' \
  'fsanitize=address|CMAKE_CXX_FLAGS_DEBUG|CMAKE_BUILD_TYPE|CMAKE_CONFIGURATION_TYPES|boost_context-asan|find_package\(Boost|find_library\(Boost_CONTEXT_LIBRARY'

Repository: eloqdata/tx_service

Length of output: 2752

🏁 Script executed:

# Check the target_link_libraries context mentioned in the comment (line 225-227)
sed -n '220,230p' build_tx_service.cmake

Repository: eloqdata/tx_service

Length of output: 458

🏁 Script executed:

# Search for where ASAN flags might be added to CMAKE_CXX_FLAGS in tx_service context
rg -n 'add_compile_options|add_link_options|CMAKE_CXX_FLAGS.*fsanitize' build_tx_service.cmake tx_service/CMakeLists.txt

Repository: eloqdata/tx_service

Length of output: 174

🏁 Script executed:

# Check if boost_context-asan is actually built/provisioned anywhere in the repo
rg -n 'boost_context-asan' --glob '*.cmake' --glob 'CMakeLists.txt'

Repository: eloqdata/tx_service

Length of output: 393

🏁 Script executed:

# Check if there are any CI/build scripts that explicitly enable ASAN for tx_service
rg -n 'ASAN|asan|sanitize.*address' --glob '*.sh' --glob '*.yml' --glob '*.yaml' --glob 'Dockerfile*'

Repository: eloqdata/tx_service

Length of output: 45

🏁 Script executed:

# Check the context around build_tx_service.cmake to see if ASAN is configured elsewhere
head -50 build_tx_service.cmake

Repository: eloqdata/tx_service

Length of output: 1722

🏁 Script executed:

# Compare how eloq_store enables ASAN vs how tx_service should
sed -n '20,40p' store_handler/eloq_data_store_service/build_eloq_store.cmake

Repository: eloqdata/tx_service

Length of output: 790

Remove dead ASAN detection code or implement proper ASAN support.

Lines 34–37 check CMAKE_BUILD_TYPE and CMAKE_CXX_FLAGS for ASAN, but CMAKE_CXX_FLAGS never receives -fsanitize=address in this build (only -DFAULT_INJECTOR is set in CMAKE_CXX_FLAGS_DEBUG). This means the boost_context-asan branch is unreachable dead code. Either remove it, or properly implement ASAN support using an option with add_compile_options() and add_compile_definitions() like eloq_store does (see build_eloq_store.cmake for the pattern).

Additionally, running find_library() before find_package(Boost) (line 42) risks using libraries from the wrong install if Boost_LIBRARY_DIRS is not yet set. Reorder to call find_package(Boost) first, then use the discovered paths as hints for find_library().

The same single-value ${Boost_CONTEXT_LIBRARY} at line 226 cannot adapt to multi-config generators anyway; if conditional linking becomes necessary in the future, use generator expressions.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@build_tx_service.cmake` around lines 33 - 43, The ASAN detection branch using
CMAKE_CXX_FLAGS and the conditional find_library(NAMES boost_context-asan) is
dead and should be removed or replaced with an explicit ASAN option; implement
ASAN support by adding a boolean option (e.g., USE_ASAN), then apply
add_compile_options(-fsanitize=address) / add_compile_definitions(...) as done
in build_eloq_store.cmake and conditionally pick the ASAN library when USE_ASAN
is ON; also reorder the logic so find_package(Boost 1.70 REQUIRED) runs before
calling find_library(Boost_CONTEXT_LIBRARY) and pass Boost_LIBRARY_DIRS (or
Boost::boost targets) as hints to find_library, and avoid a single plain
${Boost_CONTEXT_LIBRARY} for multi-config generators (use generator expressions
if conditional linking is needed later).

find_path(GFLAGS_INCLUDE_PATH gflags/gflags.h)
find_library(GFLAGS_LIBRARY NAMES gflags libgflags)

Expand Down Expand Up @@ -211,9 +222,9 @@ set(ELOQ_SOURCES ${ELOQ_SOURCES} ${PROTO_CC_FILES})

ADD_LIBRARY(txservice ${ELOQ_SOURCES})

target_include_directories(txservice PUBLIC ${INCLUDE_DIR})
target_include_directories(txservice PUBLIC ${INCLUDE_DIR} ${Boost_INCLUDE_DIRS})

target_link_libraries(txservice PUBLIC ${LINK_LIB} ${PROTOBUF_LIBRARIES})
target_link_libraries(txservice PUBLIC ${LINK_LIB} ${PROTOBUF_LIBRARIES} ${Boost_CONTEXT_LIBRARY})

if(WITH_JEMALLOC)
target_link_libraries(txservice PUBLIC jemalloc_cfg)
Expand Down
26 changes: 22 additions & 4 deletions store_handler/bigtable_handler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -613,7 +613,9 @@ void EloqDS::BigTableHandler::FetchTableRanges(
LocalCcShards *shards = Sharder::Instance().GetLocalCcShards();
std::unique_lock<std::mutex> heap_lk(
shards->table_ranges_heap_mux_);
mi_override_thread(shards->GetTableRangesHeapThreadId());
bool is_override_thd = mi_is_override_thread();
mi_threadid_t prev_thd =
mi_override_thread(shards->GetTableRangesHeapThreadId());
mi_heap_t *prev_heap =
mi_heap_set_default(shards->GetTableRangesHeap());

Expand All @@ -622,7 +624,14 @@ void EloqDS::BigTableHandler::FetchTableRanges(
mono_key.size());

mi_heap_set_default(prev_heap);
mi_restore_default_thread_id();
if (is_override_thd)
{
mi_override_thread(prev_thd);
}
else
{
mi_restore_default_thread_id();
}
}

int32_t partition_id =
Expand Down Expand Up @@ -758,7 +767,9 @@ void EloqDS::BigTableHandler::OnFetchRangeSlices(
LocalCcShards *shards = Sharder::Instance().GetLocalCcShards();
std::unique_lock<std::mutex> heap_lk(
shards->table_ranges_heap_mux_);
mi_override_thread(shards->GetTableRangesHeapThreadId());
bool is_override_thd = mi_is_override_thread();
mi_threadid_t prev_thd =
mi_override_thread(shards->GetTableRangesHeapThreadId());
mi_heap_t *prev_heap =
mi_heap_set_default(shards->GetTableRangesHeap());

Expand All @@ -773,7 +784,14 @@ void EloqDS::BigTableHandler::OnFetchRangeSlices(
}

mi_heap_set_default(prev_heap);
mi_restore_default_thread_id();
if (is_override_thd)
{
mi_override_thread(prev_thd);
}
else
{
mi_restore_default_thread_id();
}
heap_lk.unlock();
}

Expand Down
Loading
Loading