Cloud mode livelocks when local_space_limit is small enough to bound disk use

[thweetkomputer]

Summary

In cloud mode, a local_space_limit small enough to actually bound disk usage
livelocks: ReserveCacheSpace reports no evictable files and returns -ENOSPC
indefinitely, while the physical disk still has tens of gigabytes free.

Reproduced on EloqDoc CI at 1d16594, running the MongoDB jstests core suite
against ELOQDSS_ELOQSTORE + ROCKSDB_CLOUD_S3 with MinIO. Setting
eloq_store_local_space_limit=1GB (core_number=2, so 512 MB per shard) made
the run hang for four hours until the harness timeout killed it:

[async_io_manager.cpp:6005] Cannot reserve 8388608 bytes: used=536870912 limit=536870912 no evictable files
[async_io_manager.cpp:1745] open failed eloqdoc_id7a0cd84-....3625 file id Manifest : Out of disk space
[tx_index_operation.cpp:974] Alter table index flush all old sk tuples failed ... DATA_STORE_ERR. Retry flush old sk operation

df at that moment: /dev/root 145G 86G 60G 59%.

Why nothing is evictable

A file becomes evictable only once it is in closed_files_, and the only
runtime path that puts it there is CloudStoreMgr::CloseFile:

KvError err = IouringMgr::CloseFile(fd);
if (file_key.has_value()) {
    EnqueClosedFile(*file_key);
}

At runtime CloseFile is reached only from IouringMgr::EvictFD, whose loop
condition depends solely on the descriptor count:

while (lru_fd_count_ > fd_limit_ ||
       (free_reg_slots_.empty() && alloc_reg_slot_ >= fd_limit_))

Descriptors are reclaimed lazily, so files opened by earlier work stay open with
ref_count_ == 0 for as long as the count stays under fd_limit_. Space
reclamation therefore depends on descriptor pressure, and when the space budget
is exhausted first there is nothing to release it: the LRU is empty,
HasEvictableFile() is false, and ReserveCacheSpace gives up immediately
rather than driving EvictFD itself.

The two budgets are not comparable

fd_limit is derived from the global limit divided by the data file
size (eloq_store.cpp):

uint64_t max_cached_files = opts.local_space_limit / data_file_bytes;
uint64_t max_fd_limit = max_cached_files > 1 ? max_cached_files - 1 : 1;

Space is accounted per shard, and manifests are charged at manifest_limit
(8 MB default), not the data file size:

shard_local_space_limit_ = opts->local_space_limit / opts->num_threads;
...
int res = ReserveCacheSpace(options_->manifest_limit);

So for a manifest-heavy working set the ceilings diverge by
(manifest_limit / data_file_bytes) * num_threads. With local_space_limit=1GB,
pages_per_file_shift=8 (1 MB data files) and num_threads=2, that is 1023
descriptors against 64 manifests' worth of space — the space ceiling arrives 16x
earlier, so EvictFD never runs.

Setting pages_per_file_shift=11 so data files match manifest_limit narrows it
to the num_threads factor, but the ordering is unchanged: space still runs out
first, so this only postpones the livelock.

Suggestions

1. Have ReserveCacheSpace drive descriptor eviction when it finds nothing
   evictable, instead of returning -ENOSPC on the first try. Space pressure is
   currently unable to trigger the only mechanism that can relieve it.
2. Fail loudly rather than livelocking if space genuinely cannot be reclaimed —
   the caller retries forever and the only symptom is a hang.
3. Derive fd_limit from the same scope and the larger of the two file sizes so
   the descriptor ceiling is reached before the space ceiling.

Separately: automatic sizing overruns the disk

When local_space_limit is unset, CalculateAutoLocalSpaceLimit takes 80% of
std::filesystem::space().capacity — the partition's total size, not what is
free. On CI the partition is 145 GB but ~65 GB is already occupied by the image,
the third-party prefix and build outputs, so EloqStore sizes itself at ~116 GB
against ~80 GB of real headroom and fills the disk. That is what prompted us to
set an explicit limit in the first place, which is how we hit the livelock above.
Using available rather than total capacity would fix the original problem without
requiring anyone to guess a value.