[Reproducible Builds] lwk non-determinism

[xrviv]

While verifying the reproducibility of Blockstream Green desktop (green_qt 3.4.0 AppImage) for WalletScrutiny, we found that liblwk.so does not build deterministically: two consecutive cargo build --locked --release -p lwk_bindings runs from the same source tree, same toolchain, same container, and same target path produce binaries differing by 2,511,912 bytes (of ~32 MB unstripped). This prevents deterministic rebuild verification of the bundled usr/lib/liblwk.so using the published recipe.

Reproducer

Environment (green_qt CI-equivalent container):

• Base: Ubuntu 22.04.5 LTS (jammy), x86_64
• rustc 1.85.0 (4d91de4e48198da2e33413efdcd9cd2cc0c46688, 2025-02-17), host x86_64-unknown-linux-gnu, LLVM 19.1.7
• cargo 1.85.0 (d73d2caf9, 2024-12-31)
• cc: gcc (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0; linker: GNU ld (GNU Binutils for Ubuntu) 2.38

git clone https://github.com/Blockstream/lwk /lwk-test/src
cd /lwk-test/src
git checkout bdbf0904466c51985f262d8ad2d5436951d80a3e   # wasm_0.17.1-87-gbdbf0904 (pinned by green_qt 3.4.0)
git submodule sync && git submodule update --init --recursive

CARGO_TARGET_DIR=/lwk-test/t5 cargo build --locked --release -p lwk_bindings
cp /lwk-test/t5/release/liblwk.so /lwk-test/t5-first.so
cargo clean --target-dir /lwk-test/t5
CARGO_TARGET_DIR=/lwk-test/t5 cargo build --locked --release -p lwk_bindings
cmp /lwk-test/t5-first.so /lwk-test/t5/release/liblwk.so   # differs

Note the second build uses the identical target path, eliminating CARGO_TARGET_DIR/path-embedding as a cause (we tested cross-path pairs too — same result).

Evidence

• GNU Build IDs: 3ce1ba644a3045a52414db4a028975bd6902cfab vs e946e9d012caf4f58d3257fdb69ab161c28c6c24 (GNU ld --build-id sha1 — a content hash, so the divergence is real content, not just the note section).
• 2,511,912 differing bytes in 49 clusters, mapping (readelf -SW + cmp -l) to: .dynsym, .text, .rodata, .eh_frame_hdr, .eh_frame, .gcc_except_table, .symtab, .strtab — i.e. symbol tables, code, and unwind data shift together.
• The string-level diff is the tell: 94 of 95 differing strings are the same Rust symbols with different .llvm.<decimal> suffixes, e.g.:
  • build 1: _ZN42_..Debug$GT$3fmt17hc04d7db5617ccf91E.llvm.6939166226008123532
  • build 2: same symbols with .llvm.7473467968691498770 / .llvm.15652946576325426135
• .llvm.<N> suffixes are the promoted-local-symbol module hashes added by ThinLTO. lwk_bindings' release profile sets no explicit lto key, so cargo defaults to thin-local LTO.
• LTO-off control confirms the mechanism: with CARGO_PROFILE_RELEASE_LTO=off (same-path pair), the diff collapses from 2,511,912 bytes to 3,361 bytes, the .llvm.<N> suffixes disappear from the binary entirely, and the string-level diff drops to zero. ThinLTO accounts for 99.87% of the divergence; a small residual (a few .dynsym/.symtab entries and isolated code/data spots) remains.
• codegen-units = 1 alone (default LTO, same-path pair): also still differs — no .llvm.<N> suffixes are observed in these binaries either, yet the two builds differ in size by 24 bytes (25,122,696 vs 25,122,720), and the only string-level diff is again GCC_except_table<N> label renumbering (one build has an extra label).
• LTO=off + codegen-units=1 combined: still differs by ~6 KB (21 clusters). The string-level diff here is suggestive: the only differing strings are GCC_except_table<N> local labels with shifted numbers (e.g. GCC_except_table2142 vs GCC_except_table2145). This is consistent with unstable LLVM/backend symbol numbering or emission order; the residual cause remains unidentified and is independent of both the LTO mode and the codegen-unit count.

Why it matters

green_qt bundles liblwk.so in the official AppImage. Everything else in the 3.4.0 AppImage payload that differed in our rebuild traces to timestamps or packaging-tool versions (separate report/issue), but liblwk's diff is build-nondeterminism: repeated controlled builds produced different binaries, so the recipe cannot currently provide deterministic verification. A deterministic liblwk would make the Green desktop AppImage realistically verifiable end-to-end.

Suggested next steps

1. As a diagnostic mitigation (not a complete fix — see point 3): lto = "off" in the release profile removes 99.87% of the divergence (the ThinLTO module-hash churn). Note the cargo profile docs: lto = false still enables thin-local LTO; only lto = "off" disables LTO entirely — which is why the default release profile (no lto key) exhibits the .llvm.<hash> churn.
2. Investigate the ThinLTO module-hash instability under rustc 1.85.0 / LLVM 19.1.7 (we can provide all binary pairs and full logs).
3. A small residual divergence (visible as GCC_except_table<N> renumbering, consistent with unstable symbol numbering/emission order — in one configuration it even changes the binary size by 24 bytes) survives every combination we tested: lto=off alone, codegen-units=1 alone, and both together. Its cause remains unidentified — possible directions: build parallelism scheduling, a build script or proc-macro with nondeterministic iteration order, or a vendored C/C++ dependency.

We're happy to share all six .so files (both builds of each test pair), full build logs, and the diff analysis.

[xrviv]

Small correction and evidence manifest: since drafting this issue we ran a fourth test pair (t8: codegen-units = 1 alone, default LTO — same result, see the Evidence section), so we hold eight .so files (four test pairs), not six. SHA256 manifest of the retained binaries, for integrity verification if any pair is requested:

bcc488d8f141cd4f707500c4105ad2b096780fbd83655c8fc3bb93b9aca5430d  t5-build1.so  (default release profile)
7eb5639d965381e3425d0b203924a256438b2a9dbf52810d5caaad963d99dd9c  t5-build2.so
2f7963417d1d4899737910bfb1e27eded6ddab519afc2db328f55e001c807e7f  t6-build1.so  (LTO=off)
769a83be10ea28815add43ea4853611b20fb6d4a82d2788f8208dd316ca912b4  t6-build2.so
7f257fdecface83e3aa92c260779cdc37b7dd3ff32f0c88044d517c545151434  t7-build1.so  (LTO=off + codegen-units=1)
324d3d46149628db483f87d8d743f8f7392950b61c9cb50d9f22df32e1fe31c1  t7-build2.so
e79519406fe619b1e0f00fa8bf653fc23396a80a11a8d1627c5ecd1aea94d92b  t8-build1.so  (codegen-units=1, default LTO; 25,122,696 bytes)
92801dd362878c7776911e29beaa5c7b099c523a8b7e2f058ba422e29e37d024  t8-build2.so  (codegen-units=1, default LTO; 25,122,720 bytes)

Within each pair, build1/build2 are consecutive builds of the same tree at the same target path. Happy to provide any pair (xz-compressed) plus full build logs on request.