Keep unique linux assets based on content hash

[tarkah]

Resolves #64

This computes blake3 over all installable kernel files & ensures that "conflicts" (same file name / different hash) across entries get written out & linked so that we don't clobber a previous entry's assets w/ assets that have fundamentally changed within the same kernel version.

To ensure all conflicts get written out, we append a conflict id suffix to each file ONLY if conflicts exist. If no conflicts exist, the file has no suffix / looks as it did before.

The suffix is just a deterministic conflict id based on the stable sort of each conflicting assets blake3 hash -> its index into that conflict array. So rerunning will always produce the same suffix & linkage from the entry without leaking any implementation detail.

Test

I've clobbered my current state initrd with echo "1" >> /usr/lib/kernel/7.1.5-46.stable/50-default.initrd. I have two states sharing this kernel version, so this should now produce 2 distinct assets & each entry should link to the correct one:

Looks good!

[tarkah]

Converting to draft as we finalize the exact convention for shipping conflicts as separate files.

[ermo]

Loving the snafu contexts.

Have a few questions related to the method that appears to be the load-bearing functionality, though. =)

[ermo]

Suggested change

	fn get_with_context<'b>(
	/// (explain why this is a clever way of doing things)
	fn get_with_context<'b>(

[tarkah]

Will do 👍

[ermo]

My brain is too stupid to understand what this does. Help?

[tarkah]

It's a catch-all pattern match which binds the value to that var. So anything not 0 or 1 will get caught by this as len var.

[bhh32]

@tarkah, this looks good to me except for one thing. Looking at what changed_files() does, it looks like it might be introduction a bit of a performance hit through rehashing on the copy detection pass that already existed.

Since you're hashing them all up front now, when the copy detection pass runs through changed_files() via install()

blsforme/blsforme/src/bootloader/systemd_boot/mod.rs

Lines 277 to 283 in 63f8d31

	// build up the total changeset
	let mut changeset = vec![(sysroot.join(&entry.kernel.image), vmlinuz.clone())];
	changeset.extend(initrds);
	// Determine which need copying now.
	let needs_writing = changed_files(changeset.as_slice());
	log::trace!("requires update: {needs_writing:?}");

it matches on files_identical():

blsforme/blsforme/src/file_utils.rs

Lines 70 to 75 in 63f8d31

	pub fn changed_files(files: &[(PathBuf, PathBuf)]) -> Vec<(&PathBuf, &PathBuf)> {
	let mut hasher = blake3::Hasher::new();
	files
	.iter()
	.filter_map(|(source, dest)| match files_identical(&mut hasher, source, dest) {

files_identical() then rehashes the source file that was already hashed up front by the PR code:

blsforme/blsforme/src/file_utils.rs

Lines 41 to 60 in 63f8d31

	/// Compare two files with blake3 to see if they differ
	fn files_identical(hasher: &mut blake3::Hasher, a: &Path, b: &Path) -> io::Result<bool> {
	let fi_a = File::open(a)?;
	let fi_b = File::open(b)?;
	let fi_a_m = fi_a.metadata()?;
	let fi_b_m = fi_b.metadata()?;
	if fi_a_m.size() != fi_b_m.size() || fi_a_m.file_type() != fi_b_m.file_type() {
	Ok(false)
	} else {
	hasher.update_mmap_rayon(a)?;
	let result_a = hasher.finalize();
	hasher.reset();
	hasher.update_mmap_rayon(b)?;
	let result_b = hasher.finalize();
	hasher.reset();
	Ok(result_a == result_b)
	}
	}

I'm not sure if we're super worried about this, but that doubles the source I/O per boot sync, which depending on the system and how many states there are (the system I'm writing this on has 204 states), it could be negligible to several seconds anytime a successful moss sync -u or moss it <pkg-name> command is ran.

[tarkah]

@tarkah, this looks good to me except for one thing. Looking at what changed_files() does, it looks like it might be introduction a bit of a performance hit through rehashing on the copy detection pass that already existed.

Yup, I am aware of this but I've purposely tried to minimize any alterations to the existing flows & logic. I think we can clean this up in a future wider refactor.

The good news is we only ever pass in up to 5 states from moss and blake3 be pretty quick so I don't see this as much of a blocker currently.