Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ before 1.0).

### Changed

- **`header.head` open-grow replaces the file:** an undersized single-gen
OA is deleted and recreated at the create target (`.mlt` kept). Crash
after unlink is a missing `header.head`, not a zeroed live table.
[`SCHEMA.md`](SCHEMA.md), [`docs/concurrency.md`](docs/concurrency.md).

- **Confirm Class A kind per batch:** a load/write batch is all need-body
(`plan=Some`) or all already-bodied (`plan=None`). Lookup splits loadq
at `header_txs.has_body`; write drain stops on plan polarity. Mixed
Expand Down
2 changes: 1 addition & 1 deletion OPERATOR.md
Original file line number Diff line number Diff line change
Expand Up @@ -880,7 +880,7 @@ nice -n 10 ionice -c 3 ./target/release/rbitcoin-node \
```

Hash-head in-place rehash is gone. An undersized leftover `header.head` may
rewrite once at open (`store: header.head open-grow`).
be deleted and recreated once at open (`store: header.head open-grow`).

## Consensus notes (historical mainnet)

Expand Down
4 changes: 2 additions & 2 deletions SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ Open-address hash head (see [Hash heads](#hash-heads-headerhead--generic)): key

**Overflow:** `header.head.g1`, `.g2`, … same slot count as create. Probe newest-first. No schema bump — same 24 B OA slot format.

**Open:** leftover `header.head/` directory (old 256-way shards) is **Layout refuse** (wipe `header.head` and `header.body`, reindex). A **single** file smaller than the create target is rewritten on open (no concurrent probes).
**Open:** leftover `header.head/` directory (old 256-way shards) is **Layout refuse** (wipe `header.head` and `header.body`, reindex). A **single** file smaller than the create target is **deleted and recreated** on open at the target slot count (`.mlt` kept; no concurrent probes). A crash after unlink and before recreate is a missing `header.head` (reindex), not an empty live table.

---

Expand Down Expand Up @@ -476,7 +476,7 @@ Not `tx.head` — see [`docs/heads.md`](./docs/heads.md).
- Packed value: sole fk (high bit clear), or `MULTI_BIT | list_fk` → sibling `.mlt` (`create_fk:u64 | next:u64`, newest first).
- Multi-list: 16 B prefix collisions and BIP30-style multiples.
- Identity: `get_all` candidates + **body verify**.
- Insert past **7/8** is full: `header.head` rolls a sibling generation at the same slot count. Occupied tables are never rewritten while serving. Undersized **single-gen** files rewrite to the create target **on open**. Leftover 256-way `header.head/` is **Layout refuse**.
- Insert past **7/8** is full: `header.head` rolls a sibling generation at the same slot count. Occupied tables are never rewritten while serving. Undersized **single-gen** files are deleted and recreated at the create target **on open**. Leftover 256-way `header.head/` is **Layout refuse**.

**Not** used for `tx.head` (keyless address) or for scripthash **create lists** (slabs; megakey page chains).

Expand Down
3 changes: 2 additions & 1 deletion SCHEMA_HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,8 @@ log a one-line warn, migrate or refuse with a clear message — do not silently
- Rehash at load **7/8** (earlier eras rehashed more aggressively, e.g. ~1/2).
Current writer: insert past 7/8 is full; `header.head` rolls `header.head.gN`
instead of rewriting occupied slots. Open-grow of an undersized single gen
is the only occupied rewrite (exclusive, no concurrent probes).
deletes the OA file and recreates it at the create target (exclusive, no
concurrent probes; `.mlt` kept).

### Why

Expand Down
83 changes: 65 additions & 18 deletions crates/rbitcoin-store/src/hashhead.rs
Original file line number Diff line number Diff line change
Expand Up @@ -650,18 +650,33 @@ impl HashHead {
Ok(())
}

/// Occupied rewrite to a larger power-of-two. **Only** from
/// `HeaderHead::open` (no concurrent probes).
pub(crate) fn rewrite_to_slots(&self, new_slots: u64) -> Result<(), StoreError> {
/// Create a new OA file at `path`, reopening an existing `.mlt` if present.
fn create_replaced_oa(path: &Path, slots: u64) -> Result<Self, StoreError> {
let slots = slots.max(2).next_power_of_two();
let file = TableFile::create(path, TableKind::HashHead)?;
let multi = MultiList::open(path)?;
let body_bytes = SLOT_SIZE as u64 * slots;
let need = FILE_HEADER_LEN as u64 + body_bytes;
file.ensure_capacity(need)?;
file.set_logical_len(need)?;
file.zero_range(FILE_HEADER_LEN as u64, body_bytes)?;
Ok(Self {
file,
multi,
state: Mutex::new(HashState { slots, occupied: 0 }),
})
}

/// Replace the OA file with a larger power-of-two. Open-only (no concurrent probes).
pub(crate) fn rewrite_to_slots(self, new_slots: u64) -> Result<Self, StoreError> {
let new_slots = new_slots.max(2).next_power_of_two();
let (old_slots, occupied) = {
let state = self.state.lock().unwrap();
(state.slots, state.occupied)
};
if new_slots <= old_slots {
return Ok(());
return Ok(self);
}
let new_bytes = SLOT_SIZE as u64 * new_slots;
let mut entries: Vec<(HeadKey, u64)> = Vec::new();
entries
.try_reserve_exact(occupied as usize)
Expand All @@ -688,19 +703,14 @@ impl HashHead {
slot += n as u64;
}

let need = FILE_HEADER_LEN as u64 + new_bytes;
self.file.ensure_capacity(need)?;
self.file.set_logical_len(need)?;
self.file.zero_range(FILE_HEADER_LEN as u64, new_bytes)?;
{
let mut state = self.state.lock().unwrap();
state.slots = new_slots;
state.occupied = 0;
}
let path = self.file.path().to_path_buf();
drop(self);
std::fs::remove_file(&path).map_err(|e| StoreError::io(&path, e))?;
let fresh = Self::create_replaced_oa(&path, new_slots)?;

entries.sort_unstable_by_key(|(k, _)| Self::hash_slot(k, new_slots));
let n_entries = entries.len() as u64;
let mut cache = SlotPageCache::new(self, new_slots);
let mut cache = SlotPageCache::new(&fresh, new_slots);
for (k, packed) in entries {
match cache.try_place_raw(&k, packed)? {
InsertResult::Done { .. } => {}
Expand All @@ -711,15 +721,16 @@ impl HashHead {
}
}
cache.flush()?;
self.state.lock().unwrap().occupied = n_entries;
drop(cache);
fresh.state.lock().unwrap().occupied = n_entries;
rbitcoin_log::warn!(
"store: header.head open-grow path={} {}→{} slots occupied={}",
self.file.path().display(),
fresh.file.path().display(),
old_slots,
new_slots,
n_entries
);
Ok(())
Ok(fresh)
}

pub fn flush(&self) -> Result<(), StoreError> {
Expand Down Expand Up @@ -1236,6 +1247,42 @@ mod tests {
assert!(running_as_cargo_test_binary() || cfg!(test));
}

#[test]
fn rewrite_to_slots_replaces_file_keeps_keys() {
let path = tmp_path();
let h = HashHead::create_with_slots(&path, 32).unwrap();
let mut k_multi = [0u8; 32];
k_multi[0] = 0x11;
h.insert(&k_multi, Fk(1)).unwrap();
h.insert(&k_multi, Fk(2)).unwrap();
let mut k_sole = [0u8; 32];
k_sole[0] = 0x22;
h.insert(&k_sole, Fk(3)).unwrap();
h.flush().unwrap();
#[cfg(unix)]
let old = std::fs::File::open(&path).unwrap();
#[cfg(unix)]
let old_ino = {
use std::os::unix::fs::MetadataExt;
old.metadata().unwrap().ino()
};
let h = h.rewrite_to_slots(64).unwrap();
assert_eq!(h.slots(), 64);
assert_eq!(h.get_all(&k_multi).unwrap(), vec![Fk(2), Fk(1)]);
assert_eq!(h.get(&k_sole).unwrap(), Some(Fk(3)));
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
assert_ne!(
std::fs::metadata(&path).unwrap().ino(),
old_ino,
"open-grow must replace the OA file, not zero the live inode"
);
drop(old);
}
cleanup_hh(&path);
}

#[test]
fn hashhead_empty_insert_get_miss_flush_and_reopen_multi() {
let path = tmp_path();
Expand Down
20 changes: 19 additions & 1 deletion crates/rbitcoin-store/src/header_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,8 @@ impl HeaderHead {
i += 1;
}
if gens.len() == 1 && gens[0].slots() < target_slots {
gens[0].rewrite_to_slots(target_slots)?;
let g = gens.remove(0);
gens.push(g.rewrite_to_slots(target_slots)?);
}
Ok(Self {
base,
Expand Down Expand Up @@ -533,7 +534,24 @@ mod tests {
h.flush().unwrap();
assert_eq!(h.slots(), 32);
}
#[cfg(unix)]
let old = std::fs::File::open(dir.join("header.head")).unwrap();
#[cfg(unix)]
let old_ino = {
use std::os::unix::fs::MetadataExt;
old.metadata().unwrap().ino()
};
let t = HeaderTable::open(&dir).unwrap();
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
assert_ne!(
std::fs::metadata(dir.join("header.head")).unwrap().ino(),
old_ino,
"HeaderTable::open must replace undersized header.head, not punch it"
);
drop(old);
}
for hash in &hashes {
assert_eq!(t.get_by_hash(hash).unwrap().unwrap().1.hash, *hash);
}
Expand Down
4 changes: 2 additions & 2 deletions docs/concurrency.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ spend annotations.
| Role exclusivity | One appender, one annotator — not a global store mutex |
| `tx.head` insert | **Sole writer**: page-coalesced `pwrite` + `published_len` Release (no CAS, no CPU fence). Role exclusivity — not multi-inserter safe |
| `tx.head` segment seal | Roll opens the next OA immediately; BDZ+fuse8 runs on a sidecar. Lookup probes every unsealed OA until publish. Write joins the sidecar only on the *next* roll, `flush`, or `Drop` (not on the rolling insert). |
| `header.head` overflow | Insert past 7/8 rolls `header.head.gN` (new empty file). Occupied rewrite is open-only on an undersized single gen. |
| `header.head` overflow | Insert past 7/8 rolls `header.head.gN` (new empty file). Occupied rewrite is open-only: undersized single gen deletes the OA file and recreates it. |
| `ChainHub::confirmed` | `RwLock<HashSet>` for O(1) `has_block` (IBD assign path) |

There is **no** global “pause queries during confirm write.” Tip-as-commit +
Expand Down Expand Up @@ -123,7 +123,7 @@ API tokens: [`COMPAT.md`](../COMPAT.md) (Esplora headers, Electrum JSON-RPC extr

Single Class A writer is intentional. Multi‑GiB **FdOnly grow** is fallocate-only
(no remap). Hash heads do not rewrite occupied tables while serving; a leftover
undersized `header.head` may rewrite **once on open**. Class C tip tables use L2
undersized `header.head` may be **deleted and recreated** once on open. Class C tip tables use L2
write-behind (`flush_class_c_tip` before BQ dequeue); large tables stay L0.
See **[io-modality.md](./io-modality.md)** for operator IO levers.

Expand Down
Loading