DeadLockCheck stops running once the system quiesces: edge-triggered re-arm, failed rounds discarded, unbounded local wait

[liunyl]

Summary

DeadLockCheck detects and breaks lock cycles correctly while a workload is actively producing new lock waits, but it stops running once the system settles into a fully blocked state — precisely when it is needed. A cluster that deadlocks therefore stays deadlocked indefinitely, even though the cycle is of a shape the detector recognises.

Three separate defects contribute. The first is the one that caused an observed permanent hang; the other two make failures silent or self-inflicted.

Defect 1 — detection is edge-triggered and is never re-armed

GatherLockDependancy() only runs when requested_check_ is set:

https://github.com/eloqdata/tx_service/blob/master/tx_service/src/dead_lock_check.cpp#L559-L571

if (!requested_check_.load(std::memory_order_acquire))      { continue; }
else if (LocalCcShards::ClockTs() - last_check_time_ < time_interval_) { continue; }

// Reset the check flag before gather lock dependancy
requested_check_.store(false, std::memory_order_release);
lk.unlock();
GatherLockDependancy();

requested_check_ is set by DeadLockCheck::RequestCheck(), which is called from the paths where a transaction newly begins waiting on a lock (tx_operation.cpp, range_cc_map.h).

Once every participant is already blocked, no new blocking events occur, so nothing calls RequestCheck() again. The flag stays false and the detector idles in its 1s poll forever.

Observed in a wedged cluster:

• 12:20:18.594 — 19 deadlocks detected and aborted (rounds running normally under load)
• 12:20:28.600 — one more detected and aborted; last successful round
• 12:26:03.584 — a single round fires and fails (defect 2 below)
• 12:41 — still nothing; the cluster had been fully stalled for 20+ minutes

A direct test: with the cluster wedged, issuing one fresh createIndex (a genuinely new blocking event) produced no detector round within 40 seconds. Thread stacks confirmed the detector threads on two nodes were parked at dead_lock_check.cpp:526 — the 1s poll in Run() — not stuck inside GatherLockDependancy().

Suggested direction: run a round periodically whenever any cc shard has requests parked in a blocking_queue_, rather than relying solely on the edge trigger. The time_interval_ gate already bounds the cost.

Defect 2 — a failed round is silently discarded and never retried

requested_check_ is cleared before GatherLockDependancy() runs, and the early-return path does not restore it:

https://github.com/eloqdata/tx_service/blob/master/tx_service/src/dead_lock_check.cpp#L243-L249

if (node_unfinished_ > 0)
{
    LOG(INFO) << "[Global dead lock detector]: fails to receive lock "
                 "waiting information from node. Failed nodes count: "
              << node_unfinished_;
    return;
}

So a round that times out waiting for a peer consumes the request and produces nothing. Combined with defect 1, the request is never regenerated. In the incident above this is exactly what happened — the single round at 12:26:03 logged Failed nodes count: 1 and that was the last activity of any kind:

[Global dead lock detector]: fails to receive lock waiting information from node. Failed nodes count: 1

(logged simultaneously on two nodes; the non-responding node was the one holding all the wedged DDL transactions).

Fix: re-arm requested_check_ on the early-return paths, or clear it only after a round completes successfully.

Defect 3 — unbounded wait on local shards (latent)

The reply wait gives up on remote nodes after time_interval_ / 2, but loops without bound while local shards are outstanding:

https://github.com/eloqdata/tx_service/blob/master/tx_service/src/dead_lock_check.cpp#L223-L236

do
{
    con_var_.wait_for(lk,
                      std::chrono::microseconds(time_interval_ / 2),
                      [&]()
                      {
                          return local_result.unfinish_count_.load(
                                     std::memory_order_relaxed) == 0 &&
                                     node_unfinished_ == 0 ||
                                 stop_;
                      });
} while (local_result.unfinish_count_.load(std::memory_order_relaxed) !=
             0 &&
         !stop_);

The while condition tests only local_result.unfinish_count_. If a CheckDeadLockCc never completes on some shard, the detector thread blocks here forever holding mutex_, and global deadlock detection stops cluster-wide.

This did not trigger in the observed incident (the detector threads were confirmed idle in Run(), not here), but the asymmetry with the remote path looks unintended.

Why this matters beyond the cycle we hit

The specific cycle observed is a lock-upgrade ordering problem filed separately, and can be prevented in the lock layer. But deadlock detection is the only general mechanism that can recover from cycles that are not preventable by ordering — in particular the mutual-upgrade shape:

• A holds a catalog ReadLock and requests WriteIntent
• B holds WriteIntent on the same entry and requests the WriteLock upgrade
• B's upgrade fails NoReadLockConflict because of A's read lock; A cannot release it while queued

AcquireWriteIntent never inspects read locks, so A and B can legitimately hold ReadLock and WriteIntent on the same entry simultaneously, making this reachable. It cannot be resolved by any queue policy — one participant must be aborted.

That shape is reachable from ordinary MongoDB paths: createIndexes resolves the collection under MODE_IX first (which maps to a catalog read lock) and escalates to a for-write catalog acquisition afterwards, within one command-level transaction. DML can escalate too — an insert that first makes an index multikey rewrites the table metadata — so a transaction cannot reliably take the write lock up front.

The existing abort policy (abort the waiter, chosen by fewest held entries) is adequate for both shapes and does not need to change.

---

Related: #541 (lock-upgrade ordering) — the specific cycle observed in the incident described above.
Related: eloqdata/eloqdoc#490 (stalled DDL is undiagnosable) — the operator-facing surface of the same incident.