diff --git a/CHANGELOG.md b/CHANGELOG.md index 031d7c0..13093e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,118 @@ All notable changes to `gobin` are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.5.0] + +Go 1.27 verified against the released toolchain, plus a performance pass. +Layouts were checked field-by-field against the `go1.27.1` and `go1.26.8` +source trees: the V5 moduledata support from 0.3.0 is **correct as released**, +so what needed fixing was the code paths it shares with older versions. + +### Fixed + +- **The V5 layout could be chosen for a pre-1.27 binary, silently emptying it.** + It was selected from a *negative* signal — no `.typelink` section — which PE + never has at any version, nor wasm, nor RELRO ELF. With no version string to + disagree, a Go 1.20-1.26 binary decoded as V5 and every field past `types` + shifted: on a version-scrubbed `basic_go124_windows_amd64.exe`, types went + 494 → 0, itabs 16 → 0, init tasks 17 → 0 and `has_main` true → false, with no + error raised. `Moduledata::parse` now parses both candidate layouts and keeps + the one that is internally self-consistent (`layout_self_consistent`). +- **`abi.MapType` was read with the Go 1.27 layout on every Go version.** The + map descriptor has changed shape six times and had its flag bits renumbered + once more; gobin implemented one. On Go ≤ 1.26 it over-read the record (24 + bytes on 1.24-1.26, 48 on 1.14-1.23), so every `TypeDetail::Map` field past + `Hasher` was garbage, and `descriptor_size` mislocated the trailing + `UncommonType`. `MapLayout` now models all six eras, selected from the Go + version or — when scrubbed — from the moduledata version and pclntab magic, + with a structural probe for the one window those cannot separate. + `MapFlags` normalizes the three flag encodings. +- **`text_va()` returned `Some(0)` on Go 1.26+ without a moduledata.** Go 1.26 + stopped writing `textStart` into the pcHeader but left the slot, so every + `entry_va` collapsed to a raw `entry_off`. Zero now reads as absent, and the + executable section base is used as a real fallback. +- **The Go 1.27 type walk ran past its bound**, using `etypes` where + `runtime.moduleTypelinks` uses `types + typedesclen` — a field that was + parsed but never read. Past the bound it surfaced unnamed junk types. +- **One unparseable record ended the whole descriptor walk**, which on Go 1.27 + is the only type-enumeration strategy there is. It now skips and continues + under a bounded budget. `itab_stride` likewise stops rather than guessing a + method count, which desynchronized the inline itab walk. +- **`internal/runtime/*` was not classified as runtime code**, so half the + runtime of a Go 1.24+ binary read as ordinary internal library code. + +### Added + +- `.go.type` / `.go.func` (`__go_type` / `__go_func`) section recognition. Go + 1.27 replaced `.typelink` / `.itablink` with these; their presence is a + positive V5 signal and a Go detection marker + (`ConfidenceSignal::TypeSectionPresent`). +- RELRO section names: `-buildmode=pie` / `c-shared` / `c-archive` prefix these + with `.data.rel.ro`, which the classifier now strips. Previously every PIE + binary looked like it had no type sections. +- `TFLAG_GC_MASK_ON_DEMAND` / `TFLAG_DIRECT_IFACE` / `TFLAG_REGULAR_MEMORY` + with matching `AbiType` accessors. Go 1.27 deleted type-level GC programs and + dropped `abi.MaxPtrmaskBytes` from 2048 to 16, so `Type.GCData` now usually + addresses an empty BSS slot rather than a bitmap; `gc_mask_va` reports those + as absent. +- `benches/extract.rs`, a `divan` suite over the fixture corpus with allocation + profiling and stage-level benches (`stage_context`, `stage_buildinfo`, + `stage_pclntab`). + +### Changed — breaking + +- `Moduledata::parse` takes a `LayoutHints` struct instead of a + `(PclntabVersion, bool, Option)` tail. +- `types::extract_types_iter` takes the `&Moduledata` and a `TypeAbi` bundle + instead of re-deriving both; `type_at_va` and `extract_all_types` follow. +- `TypeDetail::Map` carries `key_va` / `elem_va` / `group_va` plus a boxed + `MapTypeExtra` tagged with the `MapLayout` it was read under. Fields a layout + does not record are `None` rather than `0`. `GoType` shrank 392 → 176 bytes. +- `GoSections` gained `go_type`, `go_func`, `noptrdata`, `data_section` and + `text_section`. +- moduledata discovery moved to `structures::locate::ModuledataLocator`, + replacing three near-identical scan loops. + +### Performance + +Extraction is 47-74% faster and allocates about a fifth as much. `full_sweep` +(parse plus functions, types, itabs, strings, init order and embeds): + +| Fixture | Before | After | | Allocations | Bytes | +|-----------------|----------|---------|------|-------------|-------------------| +| ELF 1.26 | 4.80 ms | 2.56 ms | −47% | 725 → 320 | 416 KB → 49 KB | +| ELF 1.27 | 4.73 ms | 2.53 ms | −47% | 1035 → 292 | 594 KB → 44 KB | +| Mach-O stripped | 3.98 ms | 1.90 ms | −52% | 1126 → 297 | 595 KB → 48 KB | +| PE 1.27 | 10.42 ms | 3.11 ms | −70% | 1176 → 334 | 630 KB → 48 KB | +| wasm 1.27 | 11.99 ms | 3.18 ms | −74% | 961 → 262 | 1.70 MB → 1.28 MB | + +- The build-info magic scan walked the image twice, the "aligned" pass one byte + at a time. Now one sweep, narrowed first to the regions that can hold a + `SBUILDINFO` symbol: PE `parse` 4.10 ms → **160 µs**, wasm → 1.52 ms. +- `types()` / `all_types()` / `type_at()` re-ran moduledata discovery — scan + included — on every call; they now reuse the parsed one: **−88% PE, −90% wasm**. +- `init_order()` indexed all ~1800 functions to resolve a dozen init PCs: + **1.12 ms → 22 µs, 206 KB → 2.1 KB**. +- `embedded_assets()` allocated two `String`s per candidate entry for a + compare-only sort key, re-read each word three times, and swept the + executable section and pclntab: **2.12 ms → 361 µs, 401 allocations → 5**. +- `find_bytes` searches eight bytes at a time (SWAR), backing the build-ID, + build-info and heuristic scans; the moduledata scan compares words, not + slices. + +### Test corpus + +- `go127` fixtures rebuilt with **released go1.27.1**; `build.sh` no longer + special-cases them via `gotip`. `go127` gained the variants it lacked against + the `go126` anchor (`embed`, `cgo`, `minimal`, `_cover`, `_fips`, stripped), + so those surfaces run against a V5 moduledata for the first time. +- New `go111` / `go112` / `go113` / `go123` fixtures complete the `abi.MapType` + coverage; the sweep asserts all six eras are present. +- New `basic_go127_linux_amd64_pie` (RELRO names), + `basic_go124_windows_amd64_noversion.exe` (scrubbed version) and + `embed_go127_wasip1_wasm` (the one case where the address-space view is not + the file) cover the inputs that exposed the bugs above. + ## [0.4.1] ### Fixed diff --git a/Cargo.lock b/Cargo.lock index afd25fb..2d8086a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,10 +2,96 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "clap" +version = "4.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" +dependencies = [ + "clap_builder", +] + +[[package]] +name = "clap_builder" +version = "4.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" +dependencies = [ + "anstyle", + "clap_lex", + "terminal_size", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "condtype" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf0a07a401f374238ab8e2f11a104d2851bf9ce711ec69804834de8af45c7af" + +[[package]] +name = "divan" +version = "0.1.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a405457ec78b8fe08b0e32b4a3570ab5dff6dd16eb9e76a5ee0a9d9cbd898933" +dependencies = [ + "cfg-if", + "clap", + "condtype", + "divan-macros", + "libc", + "regex-lite", +] + +[[package]] +name = "divan-macros" +version = "0.1.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9556bc800956545d6420a640173e5ba7dfa82f38d3ea5a167eb555bc69ac3323" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + [[package]] name = "gobin" -version = "0.4.1" +version = "0.5.0" dependencies = [ + "divan", "goblin", ] @@ -20,6 +106,18 @@ dependencies = [ "scroll", ] +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + [[package]] name = "log" version = "0.4.33" @@ -50,6 +148,25 @@ dependencies = [ "proc-macro2", ] +[[package]] +name = "regex-lite" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + [[package]] name = "scroll" version = "0.13.0" @@ -81,8 +198,97 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "terminal_size" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874" +dependencies = [ + "rustix", + "windows-sys", +] + [[package]] name = "unicode-ident" version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[patch.unused]] +name = "analysir" +version = "0.1.0" + +[[patch.unused]] +name = "analyssa" +version = "0.6.0" + +[[patch.unused]] +name = "autoit-rs" +version = "0.1.1" + +[[patch.unused]] +name = "darwinscope" +version = "0.1.1" + +[[patch.unused]] +name = "dotscope" +version = "0.9.0" + +[[patch.unused]] +name = "innospect" +version = "0.1.3" + +[[patch.unused]] +name = "mallabel" +version = "0.1.0" + +[[patch.unused]] +name = "nimrod" +version = "0.3.1" + +[[patch.unused]] +name = "nsis" +version = "0.4.0" + +[[patch.unused]] +name = "pascalscript" +version = "0.1.2" + +[[patch.unused]] +name = "securs-abi" +version = "0.1.0" + +[[patch.unused]] +name = "securs-fleet" +version = "0.1.0" + +[[patch.unused]] +name = "securs-spec" +version = "0.1.0" + +[[patch.unused]] +name = "securs-wg" +version = "0.1.0" + +[[patch.unused]] +name = "undelphi" +version = "0.3.2" + +[[patch.unused]] +name = "visualbasic" +version = "0.3.1" diff --git a/Cargo.toml b/Cargo.toml index 25ad730..b23f4de 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "gobin" -version = "0.4.1" +version = "0.5.0" edition = "2024" rust-version = "1.88" description = "Static analysis library for Go compiled binaries - identification and metadata extraction" @@ -44,3 +44,8 @@ goblin = { version = "0.10.7", default-features = false, features = [ ] } [dev-dependencies] +divan = "0.1.21" + +[[bench]] +name = "extract" +harness = false diff --git a/benches/extract.rs b/benches/extract.rs new file mode 100644 index 0000000..4ae95fe --- /dev/null +++ b/benches/extract.rs @@ -0,0 +1,221 @@ +//! Extraction benchmarks across the fixture corpus. +//! +//! Two axes matter for gobin's callers, and `divan`'s allocation profiler +//! reports both in one table: +//! +//! - **Wall time** per extraction surface, which is what a triage pipeline +//! pays per sample. +//! - **Allocations and bytes**, which is what a pipeline running thousands of +//! samples in parallel actually feels. +//! +//! Fixtures are chosen to span the layouts that cost differently rather than +//! to cover every version: an ELF with named sections (the cheap path), a PE +//! with none (which forces the moduledata scan), a stripped Mach-O, and a wasm +//! module (whose address space is a reconstructed linear-memory image). +//! +//! Run with `cargo bench`, or a subset with +//! `cargo bench -- types` / `cargo bench -- parse`. + +// Workspace-level `[lints.clippy]` applies to benchmark binaries too, but a +// benchmark harness legitimately panics on a missing fixture (there is nothing +// to measure) and has no rustdoc surface. +#![allow( + missing_docs, + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::type_complexity, + clippy::arithmetic_side_effects, + clippy::indexing_slicing +)] + +use std::sync::OnceLock; + +use divan::{AllocProfiler, Bencher, black_box}; +use gobin::{ + GoBinary, + formats::BinaryContext, + structures::{buildinfo, pclntab}, +}; + +#[global_allocator] +static ALLOC: AllocProfiler = AllocProfiler::system(); + +fn main() { + divan::main(); +} + +/// Short names the benches sweep. Kept short so divan's table stays readable; +/// [`path_of`] maps each to its fixture. +const FIXTURES: &[&str] = &[ + "elf126", + "elf127", + "pe127", + "macho127s", + "wasm127", + "types127", +]; + +/// Fixture path for a short name. +fn path_of(name: &str) -> &'static str { + match name { + "elf126" => "tests/samples/basic_go126_linux_amd64", + "elf127" => "tests/samples/basic_go127_linux_amd64", + "pe127" => "tests/samples/basic_go127_windows_amd64.exe", + "macho127s" => "tests/samples/basic_go127_darwin_arm64_stripped", + "wasm127" => "tests/samples/basic_go127_wasip1_wasm", + "types127" => "tests/samples/types_go127_linux_amd64", + other => panic!("unknown fixture {other}"), + } +} + +/// Read a fixture once per process and hand out a shared borrow. +/// +/// Benchmarks measure gobin, not the filesystem, so the read must not land +/// inside the timed region — and it must not be re-counted by the allocation +/// profiler on every iteration either. +fn fixture(path: &str) -> &'static [u8] { + static CACHE: OnceLock>> = OnceLock::new(); + let cache = CACHE.get_or_init(|| std::sync::Mutex::new(Vec::new())); + let mut guard = cache.lock().expect("fixture cache poisoned"); + if let Some((_, data)) = guard.iter().find(|(p, _)| p == path) { + return data; + } + let data: &'static [u8] = Box::leak( + std::fs::read(path) + .unwrap_or_else(|e| panic!("read {path}: {e}")) + .into_boxed_slice(), + ); + guard.push((path.to_string(), data)); + data +} + +/// Detection + format parse + pclntab + buildinfo + moduledata: what every +/// caller pays before asking for anything. +#[divan::bench(args = FIXTURES)] +fn parse(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + bencher.bench(|| black_box(GoBinary::parse(black_box(data))).is_some()); +} + +/// Function enumeration with names and source files resolved — the most +/// commonly consumed surface. +#[divan::bench(args = FIXTURES)] +fn functions(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.functions().count())); +} + +/// Reflection-visible type descriptors. +#[divan::bench(args = FIXTURES)] +fn types(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.types().count())); +} + +/// Transitive closure over every reachable descriptor. +#[divan::bench(args = FIXTURES)] +fn all_types(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.all_types().len())); +} + +/// Go string-literal recovery — a full pointer-aligned sweep of the image. +#[divan::bench(args = FIXTURES)] +fn strings(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.strings().count())); +} + +/// Interface/concrete-type pairs. +#[divan::bench(args = FIXTURES)] +fn itabs(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.itab_pairs().count())); +} + +/// Inline-tree decoding over every function — the heaviest pclntab surface. +#[divan::bench(args = FIXTURES)] +fn inline_trees(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| { + let Some(pcl) = bin.pclntab() else { return 0 }; + let mut n = 0usize; + for (_, off) in pcl.func_entries() { + if let Some(fd) = pcl.parse_func(off) { + n += bin.inline_tree(&fd).count(); + } + } + black_box(n) + }); +} + +/// Format parse: goblin plus, for wasm, the linear-memory reconstruction. +/// Everything else is measured on top of this. +#[divan::bench(args = FIXTURES)] +fn stage_context(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + bencher.bench(|| { + black_box(BinaryContext::new(black_box(data))) + .sections() + .has_gopclntab + }); +} + +/// Build-info blob location and decode. PE and wasm have no `.go.buildinfo` +/// section, so this is the stage that has to search for its own input. +#[divan::bench(args = FIXTURES)] +fn stage_buildinfo(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let ctx = BinaryContext::new(data); + bencher.bench(|| black_box(buildinfo::extract(black_box(&ctx))).is_some()); +} + +/// pclntab location and header parse. PE has no `.gopclntab` section and falls +/// back to a magic scan. +#[divan::bench(args = FIXTURES)] +fn stage_pclntab(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let ctx = BinaryContext::new(data); + bencher.bench(|| black_box(pclntab::parse(black_box(&ctx))).is_some()); +} + +/// Package initialization order, decoded from `moduledata.inittasks`. +#[divan::bench(args = FIXTURES)] +fn init_order(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.init_order().len())); +} + +/// `//go:embed` payload recovery — a symbol-independent structural search. +#[divan::bench(args = FIXTURES)] +fn embeds(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + let bin = GoBinary::parse(data).expect("fixture parses"); + bencher.bench(|| black_box(bin.embedded_assets().len())); +} + +/// A full metadata sweep — the shape of an actual triage run. +#[divan::bench(args = FIXTURES)] +fn full_sweep(bencher: Bencher, name: &str) { + let data = fixture(path_of(name)); + bencher.bench(|| { + let Some(bin) = GoBinary::parse(black_box(data)) else { + return 0usize; + }; + let mut n = bin.functions().count(); + n += bin.types().count(); + n += bin.itab_pairs().count(); + n += bin.strings().count(); + n += bin.init_order().len(); + n += bin.embedded_assets().len(); + black_box(n) + }); +} diff --git a/examples/dump.rs b/examples/dump.rs index e921063..b2a8242 100644 --- a/examples/dump.rs +++ b/examples/dump.rs @@ -205,6 +205,9 @@ fn print_report(report: &ConfidenceReport) { eprintln!(" [+] buildinfo section present") } ConfidenceSignal::BuildidNotePresent => eprintln!(" [+] build-id note present"), + ConfidenceSignal::TypeSectionPresent { section } => { + eprintln!(" [+] type-metadata section present: {section}"); + } ConfidenceSignal::BuildIdMarkerFound => eprintln!(" [+] build-id raw marker found"), ConfidenceSignal::BuildinfoParsed => eprintln!(" [+] buildinfo blob parsed"), ConfidenceSignal::BuildinfoMissing { reason } => { diff --git a/src/detection.rs b/src/detection.rs index 34ea772..06b6449 100644 --- a/src/detection.rs +++ b/src/detection.rs @@ -131,6 +131,15 @@ pub enum ConfidenceSignal { BuildinfoSectionPresent, /// ELF `.note.go.buildid` (or `Go\0\0` note marker) was present. BuildidNotePresent, + /// A Go type-metadata section was present. These names are unique to the + /// Go linker, so any of them is structural proof on its own — useful on + /// stripped binaries where `.gopclntab` was renamed away. + TypeSectionPresent { + /// Which section matched, in its ELF spelling (`".typelink"`, + /// `".itablink"`, `".go.type"`, `".go.func"`). Mach-O spellings map + /// onto the same values. + section: &'static str, + }, /// Build ID raw marker (`\xff Go build ID:`) was found. BuildIdMarkerFound, /// Build info blob was successfully parsed. @@ -252,8 +261,57 @@ pub fn heuristic_hits(data: &[u8]) -> usize { /// ~100MB, needles of 10-40 bytes), this is fast enough and avoids pulling in a /// heavier substring search dependency. pub(crate) fn find_bytes(haystack: &[u8], needle: &[u8]) -> Option { - if needle.is_empty() || needle.len() > haystack.len() { - return None; + let (first, rest) = needle.split_first()?; + // Anchor on the first byte, then compare the tail only at candidates. + // `windows(n).position` compares every candidate window in full, which is + // the dominant cost of the whole-image searches PE and wasm binaries force + // when there is no section to narrow them to. + let last_start = haystack.len().checked_sub(needle.len())?; + let mut from = 0usize; + while from <= last_start { + let window = haystack.get(from..)?; + let hit = find_byte(window, *first)?; + let start = from.checked_add(hit)?; + if start > last_start { + return None; + } + let tail_start = start.checked_add(1)?; + let tail_end = tail_start.checked_add(rest.len())?; + if haystack.get(tail_start..tail_end) == Some(rest) { + return Some(start); + } + from = tail_start; + } + None +} + +/// Index of the first occurrence of `byte` in `haystack`. +/// +/// Eight bytes are tested per iteration with the classic SWAR zero-byte trick: +/// XOR-ing the word against a broadcast of the target turns "contains `byte`" +/// into "contains a zero byte", which `(x - 0x01..01) & !x & 0x80..80` answers +/// in three instructions. A byte-at-a-time `position` is roughly an order of +/// magnitude slower over the megabyte-scale buffers this crate searches, and +/// pulling in a `memchr` dependency for it is not worth the supply-chain +/// surface on a crate that otherwise depends only on `goblin`. +fn find_byte(haystack: &[u8], byte: u8) -> Option { + const LANES: usize = 8; + const LOW: u64 = 0x0101_0101_0101_0101; + const HIGH: u64 = 0x8080_8080_8080_8080; + + let broadcast = u64::from_ne_bytes([byte; LANES]); + let (words, tail) = haystack.as_chunks::(); + for (i, chunk) in words.iter().enumerate() { + let x = u64::from_ne_bytes(*chunk) ^ broadcast; + if x.wrapping_sub(LOW) & !x & HIGH == 0 { + continue; + } + let base = i.checked_mul(LANES)?; + let hit = chunk.iter().position(|&b| b == byte)?; + return base.checked_add(hit); } - haystack.windows(needle.len()).position(|w| w == needle) + let consumed = haystack.len().checked_sub(tail.len())?; + tail.iter() + .position(|&b| b == byte) + .and_then(|hit| consumed.checked_add(hit)) } diff --git a/src/formats.rs b/src/formats.rs index 16a3a41..7c1fce0 100644 --- a/src/formats.rs +++ b/src/formats.rs @@ -3,14 +3,16 @@ //! Go binaries can be ELF (Linux, FreeBSD, etc.), Mach-O (macOS, iOS), or PE (Windows). //! Each format stores Go metadata in differently-named sections: //! -//! | Structure | ELF Section | Mach-O Section | PE Section | -//! |-------------|----------------------|---------------------|---------------| -//! | pclntab | `.gopclntab` | `__gopclntab` | (in `.rdata`) | -//! | Build info | `.go.buildinfo` | `__go_buildinfo` | (in `.data`) | -//! | Module data | `.go.module` | `__go_module` | (in `.data`) | -//! | Build ID | `.note.go.buildid` | (raw marker) | (raw marker) | -//! | Type links | `.typelink` | (in `__rodata`) | (in `.rdata`) | -//! | Itab links | `.itablink` | (in `__rodata`) | (in `.rdata`) | +//! | Structure | ELF Section | Mach-O Section | PE Section | +//! |-------------------|----------------------|---------------------|---------------| +//! | pclntab | `.gopclntab` | `__gopclntab` | (in `.rdata`) | +//! | Build info | `.go.buildinfo` | `__go_buildinfo` | (in `.data`) | +//! | Module data | `.go.module` | `__go_module` | (in `.data`) | +//! | Build ID | `.note.go.buildid` | (raw marker) | (raw marker) | +//! | Type links ≤1.26 | `.typelink` | `__typelink` | (in `.rdata`) | +//! | Itab links ≤1.26 | `.itablink` | `__itablink` | (in `.rdata`) | +//! | Type region 1.27+ | `.go.type` | `__go_type` | (in `.rdata`) | +//! | `go:funcdesc` 1.27+ | `.go.func` | `__go_func` | (in `.rdata`) | //! //! For PE binaries, Go does not create dedicated section names. Instead, the pclntab //! lives inside `.rdata` and the build info lives inside `.data`. Detection falls back @@ -48,6 +50,14 @@ use crate::{ }, }; +/// Wasm section id of the Code section, which holds the module's function +/// bodies — the wasm equivalent of `.text`. +const WASM_CODE_SECTION_ID: u8 = 10; + +/// Wasm section id of the Data section, which holds every initialized byte of +/// a module's linear memory. +const WASM_DATA_SECTION_ID: u8 = 11; + /// The executable format of the binary being analyzed. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum BinaryFormat { @@ -109,13 +119,50 @@ pub struct GoSections { pub go_buildinfo: Option, /// File byte range of the moduledata section. pub go_module: Option, - /// File byte range of the typelink section (ELF only). + /// File byte range of the typelink section (ELF / Mach-O, Go ≤ 1.26). + /// Removed by Go 1.27, which replaced the typelink array with a walk over + /// the type-descriptor region — see [`Self::go_type`]. pub typelink: Option, - /// File byte range of the itablink section (ELF only). + /// File byte range of the itablink section (ELF / Mach-O, Go ≤ 1.26). + /// Removed by Go 1.27, which stores itabs inline in the types region. pub itablink: Option, + /// File byte range of the type-descriptor section (`.go.type` / + /// `__go_type`, Go 1.27+). Spans exactly `[moduledata.types, + /// moduledata.etypes)`, so its presence is a positive V5-layout signal and + /// its bounds locate the type region without a moduledata. Absent on PE + /// (which keeps everything in `.rdata`) and on wasm. + pub go_type: Option, + /// File byte range of the `go:funcdesc` section (`.go.func` / `__go_func`, + /// Go 1.27+) — the `·f` function-value descriptors that used to sit in + /// `.rodata`. Recorded as a Go 1.27 structural marker. + pub go_func: Option, /// File byte range of the FIPS-140 info section (`.go.fipsinfo` / /// `__go_fipsinfo`, Go 1.24+). Present only in FIPS-mode builds. pub fipsinfo: Option, + /// File byte range of the non-pointer initialized data section + /// (`.noptrdata` / `__noptrdata`), where the Go linker emits + /// `runtime.firstmoduledata` before the dedicated `.go.module` section + /// existed. Searched first by [`crate::structures::locate`]'s scan, which + /// is otherwise a whole-image sweep. + pub noptrdata: Option, + /// File byte range of the initialized data section (`.data` / `__data`), + /// or — for wasm, which has no named sections — of the Data section + /// payload. PE merges every Go data symbol into `.data`, so it is the PE + /// equivalent of `noptrdata` for moduledata discovery. + /// + /// Like every range in this struct this is a **file** offset. For wasm + /// that is *not* an offset into + /// [`BinaryContext::structure_search_data`], which presents the + /// reconstructed linear-memory image instead; callers that search through + /// that view must not use this range to narrow it. + pub data_section: Option, + /// File byte range of the executable text section (`.text` / `__text`). + /// + /// Not a Go-specific section, but the Go linker places `runtime.text` at + /// its start on every format, which makes it the last-resort source for + /// [`crate::GoBinary::text_va`] on binaries whose moduledata cannot be + /// located. + pub text_section: Option, } /// A contiguous byte range within the binary file, with its virtual address. @@ -201,7 +248,12 @@ impl<'a> BinaryContext<'a> { go_module: None, typelink: None, itablink: None, + go_type: None, + go_func: None, fipsinfo: None, + noptrdata: None, + data_section: None, + text_section: None, }; let mut segments = Vec::new(); let mut elf_note_segments = Vec::new(); @@ -347,6 +399,30 @@ impl<'a> BinaryContext<'a> { // payload bytes inside the section. sections.has_go_buildid_note = true; } + if sec.id == WASM_CODE_SECTION_ID && sec.payload_size > 0 { + // The Code section is wasm's executable region. Recorded + // under `text_section` so the structural searches skip it + // exactly as they skip `.text` elsewhere — it is the + // largest part of a Go wasm module and can hold none of + // the data structures they look for. + sections.text_section = Some(SectionRange { + offset: sec.payload_offset, + size: sec.payload_size, + va: 0, + }); + } + if sec.id == WASM_DATA_SECTION_ID && sec.payload_size > 0 { + // Every initialized byte a Go wasm module has lives in the + // Data section, so its file range bounds the searches that + // would otherwise sweep the whole module. Recorded with + // `va: 0` because wasm addresses are linear-memory offsets + // that this file range does not carry. + sections.data_section = Some(SectionRange { + offset: sec.payload_offset, + size: sec.payload_size, + va: 0, + }); + } } // Reconstruct the linear-memory image. Cap at 256 MB so an // adversarial wasm with absurd offsets can't blow our memory. @@ -390,6 +466,65 @@ impl<'a> BinaryContext<'a> { } } + /// Byte ranges of [`Self::data`] — i.e. **file** offsets — that can hold + /// Go data structures, in order and non-overlapping. + /// + /// Callers that search through [`Self::structure_search_data`] want + /// [`Self::search_regions`] instead; for wasm the two views share no + /// offsets. + /// + /// Several extraction surfaces have no symbol to look up and must search + /// memory structurally — the build-info blob, `embed.FS` file arrays, the + /// moduledata. All of them are *data*, so the two largest regions of a Go + /// binary can be excluded outright: the executable section (`.text`, + /// `__text`, or a wasm Code section) and the pclntab, which is a + /// self-contained table in its own encoding. Together those are typically + /// more than half the image. + /// + /// Returns the whole range when the section table names neither, so a + /// stripped or unusual binary loses no coverage. + pub fn data_regions(&self) -> Vec<(usize, usize)> { + let len = self.data.len(); + let mut skip: Vec<(usize, usize)> = [self.sections.text_section, self.sections.gopclntab] + .into_iter() + .flatten() + .map(|r| (r.offset.min(len), r.offset.saturating_add(r.size).min(len))) + .filter(|(a, b)| b > a) + .collect(); + skip.sort_unstable(); + + let mut regions = Vec::new(); + let mut cursor = 0usize; + for (from, to) in skip { + if from > cursor { + regions.push((cursor, from)); + } + cursor = cursor.max(to); + } + if cursor < len { + regions.push((cursor, len)); + } + regions + } + + /// The same idea as [`Self::data_regions`], but as ranges into + /// [`Self::structure_search_data`] — the view every structural parser + /// actually reads through. + /// + /// For ELF, Mach-O and PE that view is the file (or, for a chained-fixup + /// Mach-O, a rebased copy with identical layout), so the file ranges carry + /// over unchanged. For wasm it is the reconstructed linear-memory image, + /// whose offsets are linear-memory addresses unrelated to file positions — + /// and which is *entirely* initialized data, so there is nothing to + /// exclude. + pub fn search_regions(&self) -> Vec<(usize, usize)> { + if self.format == BinaryFormat::Wasm { + let len = self.structure_search_data().len(); + return if len > 0 { vec![(0, len)] } else { Vec::new() }; + } + self.data_regions() + } + /// The raw binary data this context was built from. pub fn data(&self) -> &'a [u8] { self.data @@ -544,7 +679,18 @@ pub fn detect_format(data: &[u8]) -> BinaryFormat { } /// Classify a section by name and record it into the appropriate GoSections field. +/// +/// ELF links that use RELRO (`-buildmode=pie` / `c-shared` / `c-archive`) +/// rename every read-only-relocatable Go section with a `.data.rel.ro` prefix +/// (`cmd/link/internal/ld/data.go`, `genrelrosecname`), so `.typelink` becomes +/// `.data.rel.ro.typelink` and `.go.type` becomes `.data.rel.ro.go.type`. The +/// prefix is stripped before matching — otherwise a PIE binary looks like it +/// has no typelink section at all, which the moduledata layout arbitration +/// reads as a Go 1.27 signal. fn classify_section(name: &str, range: Option, result: &mut GoSections) { + // `.data.rel.ro` itself (the empty-suffix RELRO section) strips to "" and + // matches nothing, which is what we want. + let name = name.strip_prefix(".data.rel.ro").unwrap_or(name); match name { ".gopclntab" | "__gopclntab" => { result.has_gopclntab = true; @@ -566,9 +712,24 @@ fn classify_section(name: &str, range: Option, result: &mut GoSect ".itablink" | "__itablink" => { result.itablink = range; } + ".go.type" | "__go_type" => { + result.go_type = range; + } + ".go.func" | "__go_func" => { + result.go_func = range; + } ".go.fipsinfo" | "__go_fipsinfo" => { result.fipsinfo = range; } + ".text" | "__text" => { + result.text_section = range; + } + ".noptrdata" | "__noptrdata" => { + result.noptrdata = range; + } + ".data" | "__data" => { + result.data_section = range; + } _ => {} } } diff --git a/src/lib.rs b/src/lib.rs index e96cbb7..c674c6a 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -91,7 +91,8 @@ use crate::{ Arch, PclntabVersion, buildid, buildinfo, embed, gcprog, goslice::{GoSlice, GoStr}, inittask, inline, itab, - moduledata::{ModuleHash, Moduledata, ModuledataVersion, PtabEntry, TextSect}, + locate::ModuledataLocator, + moduledata::{LayoutHints, ModuleHash, Moduledata, ModuledataVersion, PtabEntry, TextSect}, name::decode_name, pclntab::{self, FuncData, ParsedPclntab, PclntabMeta}, strings as gostrings, types, @@ -179,6 +180,20 @@ impl<'a> GoBinary<'a> { report.push(ConfidenceSignal::BuildidNotePresent); report.raise_to(Confidence::High); } + // Type-metadata sections. `.typelink` / `.itablink` are the Go ≤1.26 + // spelling; Go 1.27 replaced both with `.go.type` / `.go.func`. All + // four names are Go-linker-specific, so each is structural proof. + for (present, section) in [ + (sections.typelink.is_some(), ".typelink"), + (sections.itablink.is_some(), ".itablink"), + (sections.go_type.is_some(), ".go.type"), + (sections.go_func.is_some(), ".go.func"), + ] { + if present { + report.push(ConfidenceSignal::TypeSectionPresent { section }); + report.raise_to(Confidence::High); + } + } let build_id = buildid::extract(&ctx); if build_id.is_some() { @@ -400,9 +415,8 @@ impl<'a> GoBinary<'a> { fn parse_moduledata_at(&self, va: u64) -> Option { let meta = self.pclntab_meta?; let bytes = self.ctx.slice_at_va(va)?; - let has_typelink = self.ctx.sections().typelink.is_some(); - let go_minor = self.go_version.and_then(parse_go_minor_version); - Moduledata::parse(bytes, meta.ptr_size, meta.version, has_typelink, go_minor) + let hints = layout_hints(&self.ctx, meta.version, self.go_version); + Moduledata::parse(bytes, meta.ptr_size, hints) } /// Virtual address of `runtime.text` — the first byte of Go-emitted code. @@ -411,16 +425,47 @@ impl<'a> GoBinary<'a> { /// In most cases callers should reach for [`Self::entry_va`] / /// [`Self::entry_rva`] instead, which fold both the `text_va` lookup and /// the (PE-only) image-base translation into one accessor. + /// + /// # Fallbacks + /// + /// Legacy (Go 1.2-1.15) binaries are parsed without a moduledata, and a + /// heavily-stripped modern binary may also lack one, so three sources are + /// tried in decreasing order of authority: + /// + /// 1. `moduledata.text`. + /// 2. The pclntab's own record of `runtime.text` — the pcHeader + /// `textStart` field on Go 1.18-1.25, or the lowest absolute function + /// PC on Go 1.2-1.17 (both surface as `header_text_start` / + /// `text_start`). + /// 3. The base of the executable text section. Go 1.26 stopped writing + /// `textStart` into the pcHeader, which leaves a moduledata-less 1.26+ + /// binary with no recorded value at all; the Go linker places + /// `runtime.text` at the start of `.text` / `__text` on every format, + /// so the section header supplies it. + /// + /// Returns `None` rather than `0` when nothing is available — a bogus + /// `Some(0)` would silently turn every [`Self::entry_va`] into a raw + /// `entry_off`. pub fn text_va(&self) -> Option { if let Some(m) = self.moduledata.as_ref() { return Some(m.text); } - // Legacy (Go 1.2-1.15) binaries are parsed without a moduledata, and - // stripped modern binaries may also lack one. The pclntab records - // `runtime.text` directly: for Go 1.2-1.15 it is the first function's - // absolute PC, and for Go 1.18+ it is the pcHeader `textStart` field - // (both surface as `header_text_start`). - self.pclntab().and_then(|p| p.header_text_start) + if let Some(pcl) = self.pclntab() { + if let Some(va) = pcl.header_text_start { + return Some(va); + } + // Go 1.16-1.17 store absolute PCs in the functab; the parser keeps + // the lowest one here to rebase `entry_off`, which makes it + // `runtime.text`. + if pcl.text_start != 0 { + return Some(pcl.text_start); + } + } + self.ctx + .sections() + .text_section + .map(|s| s.va) + .filter(|&va| va != 0) } /// Binary-level virtual address of a function's entry point. @@ -854,18 +899,51 @@ impl<'a> GoBinary<'a> { return Vec::new(); } - // One pass over the function table to map entry-offset -> name, so each - // init PC resolves without an O(nfunc) rescan per function. + // Resolve the init PCs to names with one pass over the function table. + // + // A binary has thousands of functions and a couple of dozen init + // tasks, so the pass collects only the entry offsets the tasks + // actually reference — indexing every function into a map costs more + // memory than the whole rest of this call and throws almost all of it + // away. It also walks `func_entries` rather than `functions()`, which + // would additionally decode a source file, line range and frame size + // for every function in the binary. let text_va = self.text_va(); - let mut names: std::collections::HashMap = std::collections::HashMap::new(); - for f in self.functions() { - names.entry(f.entry_offset).or_insert(f.name); + let entry_off_of = + |pc_va: u64| -> Option { u32::try_from(pc_va.checked_sub(text_va?)?).ok() }; + + let mut wanted: Vec = raw + .iter() + .flatten() + .filter_map(|pc| entry_off_of(*pc)) + .collect(); + wanted.sort_unstable(); + wanted.dedup(); + + let mut names: Vec<(u32, &str)> = Vec::with_capacity(wanted.len()); + if !wanted.is_empty() + && let Some(pcl) = self.pclntab() + { + for (entry_off, func_off) in pcl.func_entries() { + if wanted.binary_search(&entry_off).is_err() { + continue; + } + if let Some(func) = pcl.parse_func(func_off) + && let Some(name) = pcl.func_name(func.name_off as u32) + { + names.push((entry_off, name)); + } + } + names.sort_unstable_by_key(|(off, _)| *off); } let resolve = |pc_va: u64| -> Option<&str> { - let off = pc_va.checked_sub(text_va?)?; - let off = u32::try_from(off).ok()?; - names.get(&off).copied() + let off = entry_off_of(pc_va)?; + names + .binary_search_by_key(&off, |(o, _)| *o) + .ok() + .and_then(|i| names.get(i)) + .map(|(_, n)| *n) }; raw.into_iter() @@ -946,18 +1024,10 @@ impl<'a> GoBinary<'a> { /// Collect with `bin.types().collect::>()` if you need an owned /// container. pub fn types(&self) -> types::TypeIter<'_> { - let meta = match self.pclntab_meta { - Some(m) => m, - None => return types::extract_types_iter(&self.ctx, 0, None, None, None), + let (Some(meta), Some(md)) = (self.pclntab_meta, self.moduledata.as_ref()) else { + return types::TypeIter::empty(&self.ctx); }; - let go_version_minor = self.go_version().and_then(parse_go_minor_version); - types::extract_types_iter( - &self.ctx, - meta.ptr_size, - Some(meta.version), - Some(meta.offset), - go_version_minor, - ) + types::extract_types_iter(&self.ctx, md, self.type_abi(meta)) } /// Parse the type descriptor at a specific virtual address into a @@ -970,13 +1040,7 @@ impl<'a> GoBinary<'a> { pub fn type_at(&self, va: u64) -> Option> { let meta = self.pclntab_meta?; let types_base = self.moduledata.as_ref()?.types; - types::type_at_va( - &self.ctx, - va, - types_base, - meta.ptr_size, - self.legacy_names(), - ) + types::type_at_va(&self.ctx, va, types_base, self.type_abi(meta)) } /// Enumerate **every** reachable type descriptor, not just the @@ -1000,14 +1064,33 @@ impl<'a> GoBinary<'a> { }; types::extract_all_types( &self.ctx, - meta.ptr_size, self.types().collect(), types_base, etypes, - self.legacy_names(), + self.type_abi(meta), ) } + /// The per-binary constants every type-descriptor read depends on. + /// + /// The `abi.MapType` layout is the one that needs deriving: the map + /// descriptor changed shape six times between Go 1.7 and 1.27, and its size + /// feeds the position of every map type's `UncommonType`, so it has to be + /// settled before any descriptor is read. See [`types::MapLayout::infer`]. + fn type_abi(&self, meta: PclntabMeta) -> types::TypeAbi { + let md_version = self.moduledata.as_ref().map(|m| m.version); + types::TypeAbi { + ps: meta.ptr_size, + legacy_names: self.legacy_names(), + map_layout: types::MapLayout::infer( + self.go_version().and_then(parse_go_minor_version), + meta.version == PclntabVersion::Go120, + md_version == Some(ModuledataVersion::V5), + md_version == Some(ModuledataVersion::V4), + ), + } + } + /// Streaming iterator over Go string literals discovered by scanning the /// binary for `(ptr, len)` headers that resolve to in-binary UTF-8 bytes. /// @@ -1198,196 +1281,32 @@ fn parse_go_minor_version(version: &str) -> Option { /// Locate and parse the moduledata for accessor-only use (text/etext/types /// region addresses). /// -/// Prefer the dedicated `.go.module` section (Go 1.26+); otherwise scan for -/// moduledata via its pcHeader pointer (PE, and ELF / Mach-O before Go 1.26). -/// Returns `None` if the binary lacks VA mappings or moduledata can't be -/// located — callers degrade gracefully (the affected accessors return `None`). +/// Delegates to [`ModuledataLocator`], which owns every discovery strategy — +/// the `.go.module` section on Go 1.26+ ELF / Mach-O, and the `pcHeader`- +/// pointer scan everywhere else. Returns `None` if the binary lacks VA +/// mappings or the moduledata cannot be located; callers degrade gracefully +/// (the affected accessors return `None`). fn find_moduledata( ctx: &BinaryContext<'_>, pclntab: &ParsedPclntab<'_>, go_version: Option<&str>, ) -> Option { - if !ctx.has_va_mapping() { - return None; - } - - // Read through the address-space view so chained-fixup pointers (Mach-O - // plugins / CGO) are already rebased to real VAs. - let data = ctx.structure_search_data(); - let sections = ctx.sections(); - let go_minor = go_version.and_then(parse_go_minor_version); - let has_typelink = sections.typelink.is_some(); - - if let Some(ref range) = sections.go_module { - let end = range.offset.checked_add(range.size)?; - let md_data = data.get(range.offset..end)?; - return Moduledata::parse( - md_data, - pclntab.ptr_size, - pclntab.version, - has_typelink, - go_minor, - ); - } - - if ctx.format() == BinaryFormat::Wasm { - return find_moduledata_wasm(ctx, pclntab, has_typelink, go_minor); - } - - // No dedicated `.go.module` section. That section was added in Go 1.26, so - // older ELF / Mach-O binaries (and every PE) keep moduledata in - // `.noptrdata` with no name — locate it by scanning for its pcHeader - // pointer. - find_moduledata_by_scan(ctx, pclntab, has_typelink, go_minor) -} - -/// Wasm moduledata discovery: scan the linear-memory image for a -/// pointer-aligned `u64` equal to the pcHeader's linear-memory address, then -/// validate by parsing. -/// -/// For wasm, [`ParsedPclntab::offset`] is already a linear-memory address -/// (the parser ran on the reconstructed linear-memory image, not on file -/// bytes), so no `file_to_va` translation is needed. -fn find_moduledata_wasm( - ctx: &BinaryContext<'_>, - pclntab: &ParsedPclntab<'_>, - has_typelink: bool, - go_minor: Option, -) -> Option { - let lm = ctx.structure_search_data(); - let ps = pclntab.ptr_size as usize; - if ps == 0 { - return None; - } - let pclntab_va = pclntab.offset as u64; - let target_bytes: Vec = match pclntab.ptr_size { - 4 => (pclntab_va as u32).to_le_bytes().to_vec(), - 8 => pclntab_va.to_le_bytes().to_vec(), - _ => return None, - }; - - let mut offset: usize = 0; - while let Some(end) = offset.checked_add(ps) { - if end > lm.len() { - break; - } - let rem = offset.checked_rem(ps).unwrap_or(0); - if rem != 0 { - let bump = ps.saturating_sub(rem); - offset = match offset.checked_add(bump) { - Some(o) => o, - None => break, - }; - continue; - } - let window = match lm.get(offset..end) { - Some(w) => w, - None => break, - }; - if window == target_bytes.as_slice() { - let remaining = match lm.get(offset..) { - Some(r) => r, - None => break, - }; - if let Some(md) = Moduledata::parse( - remaining, - pclntab.ptr_size, - pclntab.version, - has_typelink, - go_minor, - ) && md.minpc < md.maxpc - && md.types != 0 - { - return Some(md); - } - } - offset = match offset.checked_add(ps) { - Some(o) => o, - None => break, - }; - } - None -} - -/// Validate a scanned moduledata candidate, accounting for the legacy layout. -/// -/// The modern check requires a non-zero `types` base and a `funcnametab` -/// pointer that maps into the file. The legacy (Go 1.5-1.15) layout has neither -/// `funcnametab` nor — before Go 1.7 — a `types` base, so it is validated -/// through its always-present `text` boundary instead. `minpc < maxpc` guards -/// both. -fn moduledata_scan_valid(ctx: &BinaryContext<'_>, md: &Moduledata) -> bool { - if md.minpc >= md.maxpc { - return false; - } - match md.version { - ModuledataVersion::V1 => md.text != 0 && ctx.va_to_file(md.text).is_some(), - _ => md.types != 0 && ctx.va_to_file(md.funcnametab.ptr).is_some(), - } + let hints = layout_hints(ctx, pclntab.version, go_version); + ModuledataLocator::new(ctx, pclntab.ptr_size, pclntab.offset, hints).locate() } -/// Section-less moduledata discovery (PE, and ELF / Mach-O before Go 1.26): -/// scan file bytes for a pointer-aligned value matching the pclntab VA — the -/// moduledata's first field is `pcHeader *pcHeader` — then validate by parsing. -fn find_moduledata_by_scan( +/// Collect the out-of-band signals [`Moduledata::parse`] uses to pick between +/// the V5 (Go 1.27+) and pre-V5 moduledata layouts. +fn layout_hints( ctx: &BinaryContext<'_>, - pclntab: &ParsedPclntab<'_>, - has_typelink: bool, - go_minor: Option, -) -> Option { - // The scan looks for the pcHeader pointer; on chained-fixup Mach-O it must - // run over the rebased view so the stored pointer matches the pclntab VA. - let data = ctx.structure_search_data(); - let pclntab_va = ctx.file_to_va(pclntab.offset)?; - let ps = pclntab.ptr_size as usize; - if ps == 0 { - return None; - } - - let target_bytes: Vec = match ps { - 4 => (pclntab_va as u32).to_le_bytes().to_vec(), - 8 => pclntab_va.to_le_bytes().to_vec(), - _ => return None, - }; - - let mut offset = 0usize; - while let Some(end) = offset.checked_add(ps) { - if end > data.len() { - break; - } - let rem = offset.checked_rem(ps).unwrap_or(0); - if rem != 0 { - let bump = ps.saturating_sub(rem); - offset = match offset.checked_add(bump) { - Some(o) => o, - None => break, - }; - continue; - } - let window = match data.get(offset..end) { - Some(w) => w, - None => break, - }; - if window == target_bytes.as_slice() { - let remaining = match data.get(offset..) { - Some(r) => r, - None => break, - }; - if let Some(md) = Moduledata::parse( - remaining, - pclntab.ptr_size, - pclntab.version, - has_typelink, - go_minor, - ) && moduledata_scan_valid(ctx, &md) - { - return Some(md); - } - } - offset = match offset.checked_add(ps) { - Some(o) => o, - None => break, - }; + pclntab_version: PclntabVersion, + go_version: Option<&str>, +) -> LayoutHints { + let sections = ctx.sections(); + LayoutHints { + pclntab_version, + go_minor: go_version.and_then(parse_go_minor_version), + has_typelink_section: sections.typelink.is_some(), + has_go_type_section: sections.go_type.is_some(), } - None } diff --git a/src/metadata.rs b/src/metadata.rs index a204c47..a4ce520 100644 --- a/src/metadata.rs +++ b/src/metadata.rs @@ -37,12 +37,23 @@ use crate::{ structures::pclntab::{FuncData, FuncEntryIter, ParsedPclntab}, }; -/// Whether `pkg` is a Go runtime package (`runtime` or `runtime/`). +/// Whether `pkg` is a Go runtime package. +/// +/// Covers all three spellings the runtime has used: `runtime` itself, +/// `runtime/` (e.g. `runtime/cgo`, and the pre-1.24 `runtime/internal/*` +/// tree), and `internal/runtime/` — the home the runtime's internal +/// packages (`internal/runtime/atomic`, `internal/runtime/maps`, +/// `internal/runtime/sys`, …) moved to in Go 1.24 and where they still live in +/// 1.27. Without the third form, half the runtime of a modern binary +/// classifies as ordinary internal library code. /// /// Shared with [`crate::structures::types::GoType::is_runtime`] so type-side /// and function-side classifications agree on one canonical rule. pub fn is_runtime_path(pkg: &str) -> bool { - pkg == "runtime" || pkg.starts_with("runtime/") + pkg == "runtime" + || pkg.starts_with("runtime/") + || pkg == "internal/runtime" + || pkg.starts_with("internal/runtime/") } /// Whether `pkg` is a Go-internal package (runtime, `internal/*`, `vendor/*`, diff --git a/src/structures/abitype.rs b/src/structures/abitype.rs index f83cc18..dee9499 100644 --- a/src/structures/abitype.rs +++ b/src/structures/abitype.rs @@ -16,11 +16,14 @@ //! - `FieldAlign_` (u8) //! - `Kind_` (u8) //! - `Equal` (uintptr, equality function pointer) -//! - `GCData` (uintptr, GC bitmap pointer) +//! - `GCData` (uintptr, GC pointer-mask bitmap — or a pointer to one; see +//! [`TFLAG_GC_MASK_ON_DEMAND`]) //! - `Str` (NameOff / i32) //! - `PtrToThis` (TypeOff / i32) //! -//! Source: `src/internal/abi/type.go:21-46` +//! Source: `src/internal/abi/type.go`. The struct itself has been stable since +//! Go 1.21; what changed in Go 1.27 is how `GCData` is populated — see +//! [`TFLAG_GC_MASK_ON_DEMAND`]. use crate::structures::util::{read_i32, read_u32, read_uintptr}; @@ -33,6 +36,27 @@ pub const TFLAG_EXTRA_STAR: u8 = 0x02; /// `TFlag` bit: the type has a user-defined name (not a composite literal type). pub const TFLAG_NAMED: u8 = 0x04; +/// `TFlag` bit: the type's `Equal` and hash functions may treat values as plain +/// memory (`TFlagRegularMemory`). +pub const TFLAG_REGULAR_MEMORY: u8 = 0x08; + +/// `TFlag` bit: `GCData` is **not** a pointer mask but a `**byte` — a slot the +/// runtime fills in with a lazily-built mask on first use +/// (`runtime.getGCMaskOnDemand`). +/// +/// The flag exists from Go 1.22, but its reach changed sharply in Go 1.27: +/// that release deleted type-level GC *programs* (the `type:.gcprog.*` symbols +/// became `type:.gcmask.*` under a new `runtime.gcmask.*` BSS carrier) and +/// dropped `abi.MaxPtrmaskBytes` from 2048 to 16. Below 1.27 almost no type +/// took the on-demand path; from 1.27 most non-trivial ones do. Check this bit +/// before treating [`AbiType::gcdata`] as the address of a bitmap. +pub const TFLAG_GC_MASK_ON_DEMAND: u8 = 0x10; + +/// `TFlag` bit: a value of this type is stored directly in an interface word +/// rather than behind a pointer (`TFlagDirectIface`). Mirrors the legacy +/// `KindDirectIface` bit in `Kind_`, which `AbiType::kind` masks off. +pub const TFLAG_DIRECT_IFACE: u8 = 0x20; + /// Parsed `abi.Type` -- the base type descriptor found at the start of every /// Go runtime type. #[derive(Debug, Clone, Copy, Default)] @@ -53,7 +77,13 @@ pub struct AbiType { pub kind_: u8, /// VA of the type's equality function (`Equal`), or `0` if none. pub equal: u64, - /// VA of the type's GC bitmap (`GCData`), or `0` if none. + /// VA of the type's GC pointer-mask bitmap (`GCData`), or `0` if none. + /// + /// When [`TFLAG_GC_MASK_ON_DEMAND`] is set this is instead the address of + /// a `*byte` slot the runtime populates at run time, so statically it + /// points at zeroed BSS rather than at a mask. Use + /// [`AbiType::gc_mask_va`] to get the bitmap address only when one is + /// actually present. pub gcdata: u64, /// Offset into the names table for this type's string representation. pub str_off: i32, @@ -140,6 +170,40 @@ impl AbiType { pub fn is_named(&self) -> bool { self.tflag & TFLAG_NAMED != 0 } + + /// Whether `Equal`/hash may treat values of this type as plain memory. + pub fn is_regular_memory(&self) -> bool { + self.tflag & TFLAG_REGULAR_MEMORY != 0 + } + + /// Whether [`Self::gcdata`] addresses a runtime-populated `*byte` slot + /// instead of a statically-emitted pointer mask. See + /// [`TFLAG_GC_MASK_ON_DEMAND`]. + pub fn gc_mask_on_demand(&self) -> bool { + self.tflag & TFLAG_GC_MASK_ON_DEMAND != 0 + } + + /// Whether a value of this type is stored directly in an interface word. + /// + /// Go carried this as `KindDirectIface` (bit 5 of `Kind_`) before moving it + /// into `TFlag`, and still sets both, so either bit is accepted. + pub fn is_direct_iface(&self) -> bool { + self.tflag & TFLAG_DIRECT_IFACE != 0 || self.kind_ & 0x20 != 0 + } + + /// VA of this type's statically-emitted GC pointer mask, or `None` when + /// the type has none or defers mask construction to run time. + /// + /// Prefer this over reading [`Self::gcdata`] directly: from Go 1.27 the + /// on-demand path is the common case, and the raw field then addresses an + /// empty BSS slot rather than a bitmap. + pub fn gc_mask_va(&self) -> Option { + if self.gcdata == 0 || self.gc_mask_on_demand() { + None + } else { + Some(self.gcdata) + } + } } #[cfg(test)] diff --git a/src/structures/buildinfo.rs b/src/structures/buildinfo.rs index bbf3aa9..ab2545e 100644 --- a/src/structures/buildinfo.rs +++ b/src/structures/buildinfo.rs @@ -102,22 +102,7 @@ const FLAG_VERSION_INL: u8 = 0x02; /// For the pointer format (Go < 1.18), only a version string scan is attempted. pub fn extract<'a>(ctx: &BinaryContext<'a>) -> Option> { let data = ctx.data(); - let sections = ctx.sections(); - - let search_data = if let Some(ref range) = sections.go_buildinfo { - let raw_end = range.offset.checked_add(range.size)?; - let end = raw_end.min(data.len()); - data.get(range.offset..end)? - } else { - data - }; - - let magic_pos = find_aligned_magic(search_data)?; - let header_start = if let Some(ref range) = sections.go_buildinfo { - range.offset.checked_add(magic_pos)? - } else { - magic_pos - }; + let header_start = find_magic(ctx, data)?; let header_end = header_start.checked_add(BUILDINFO_HEADER_SIZE)?; let header = data.get(header_start..header_end)?; @@ -151,26 +136,81 @@ pub fn extract<'a>(ctx: &BinaryContext<'a>) -> Option> { Some(info) } -/// Find the magic header at 16-byte alignment within `data`. +/// Locate the build-info header's offset within `data`. +/// +/// `go:buildinfo` is a writable-data symbol (`sym.SBUILDINFO`), so the search +/// narrows to the regions that can hold it before falling back to the whole +/// image: +/// +/// 1. the dedicated `.go.buildinfo` / `__go_buildinfo` section — exact, and +/// the usual case for ELF and Mach-O; +/// 2. the `.data` / `.noptrdata` sections — PE merges every Go data symbol +/// into `.data`, which is a small fraction of a Go binary (tens of KB +/// against megabytes of `.text` and `.rdata`); +/// 3. the data regions of the image +/// ([`BinaryContext::data_regions`]), so a stripped or unusual section +/// table still works. +fn find_magic(ctx: &BinaryContext<'_>, data: &[u8]) -> Option { + let sections = ctx.sections(); + let candidates = [ + sections.go_buildinfo.as_ref(), + sections.data_section.as_ref(), + sections.noptrdata.as_ref(), + ]; + for range in candidates.into_iter().flatten() { + let end = range.offset.checked_add(range.size)?.min(data.len()); + if let Some(region) = data.get(range.offset..end) + && let Some(pos) = find_aligned_magic(region) + { + return range.offset.checked_add(pos); + } + } + // Fall back to a sweep, but only over the regions that can hold data: the + // blob is a `sym.SBUILDINFO` symbol, so the executable section and the + // pclntab are excluded. That matters most for wasm, which names no + // sections at all and would otherwise sweep the whole module — twice, once + // per candidate list above — to prove a blob it never carries is absent. + for (from, to) in ctx.data_regions() { + if let Some(region) = data.get(from..to) + && let Some(pos) = find_aligned_magic(region) + { + return from.checked_add(pos); + } + } + None +} + +/// Find the build-info magic within `data`, preferring a 16-byte-aligned hit. /// -/// The Go linker aligns the build info to 16 bytes (macOS requirement). -/// We first scan with alignment checking, then fall back to an unaligned scan -/// in case the section offset shifted the alignment. +/// The Go linker aligns the symbol to [`BUILDINFO_ALIGN`] (a macOS +/// requirement), so an aligned occurrence is the real one; an unaligned hit is +/// still returned as a fallback in case a section offset shifted the +/// alignment. Both are answered in a **single** sweep — the previous +/// aligned-then-unaligned pair walked the buffer twice, which on PE and wasm +/// (neither of which has a `.go.buildinfo` section to narrow the search) meant +/// scanning the whole image twice over. fn find_aligned_magic(data: &[u8]) -> Option { - let mut pos: usize = 0; - while let Some(end) = pos.checked_add(BUILDINFO_HEADER_SIZE) { - if end > data.len() { + let mut first_unaligned: Option = None; + let mut from: usize = 0; + while let Some(rel) = data + .get(from..) + .and_then(|d| find_bytes(d, BUILDINFO_MAGIC)) + { + let at = from.checked_add(rel)?; + // A hit with no room for the full header cannot be the build info. + if at + .checked_add(BUILDINFO_HEADER_SIZE) + .is_none_or(|e| e > data.len()) + { break; } - let window = data.get(pos..)?; - if window.starts_with(BUILDINFO_MAGIC) - && (pos.checked_rem(BUILDINFO_ALIGN) == Some(0) || pos == 0) - { - return Some(pos); + if at.checked_rem(BUILDINFO_ALIGN) == Some(0) { + return Some(at); } - pos = pos.checked_add(1)?; + first_unaligned.get_or_insert(at); + from = at.checked_add(1)?; } - find_bytes(data, BUILDINFO_MAGIC) + first_unaligned } /// Read a varint-length-prefixed UTF-8 string, returning the string (borrowed) diff --git a/src/structures/descriptor.rs b/src/structures/descriptor.rs index 101ccaa..056e2e9 100644 --- a/src/structures/descriptor.rs +++ b/src/structures/descriptor.rs @@ -15,7 +15,7 @@ use crate::structures::{ functype::FuncTypeExtra, interfacetype::InterfaceTypeExtra, kind, - maptype::MapTypeExtra, + maptype::{MapLayout, MapTypeExtra}, method::{GoImethod, GoMethod}, structtype::{GoStructField, StructTypeExtra}, uncommon::UncommonType, @@ -23,10 +23,19 @@ use crate::structures::{ }; /// Compute the total descriptor size for a type, equivalent to Go's -/// `abi.Type.DescriptorSize()` from `src/internal/abi/type.go:750-799`. +/// `abi.Type.DescriptorSize()` from `src/internal/abi/type.go`. /// /// The total is: concrete_type_size + uncommon_type_size + variable_data + methods. -pub fn descriptor_size(type_data: &[u8], abi_type: &AbiType, ps: u8) -> Option { +/// +/// `map_layout` selects the [`MapTypeExtra`] shape, which changed in both Go +/// 1.24 and Go 1.27; getting it wrong mis-sizes every map descriptor and, with +/// it, the position of the trailing `UncommonType`. +pub fn descriptor_size( + type_data: &[u8], + abi_type: &AbiType, + ps: u8, + map_layout: MapLayout, +) -> Option { let p = ps as usize; let base_sz = AbiType::size(ps); @@ -55,7 +64,15 @@ pub fn descriptor_size(type_data: &[u8], abi_type: &AbiType, ps: u8) -> Option (base_sz.checked_add(MapTypeExtra::size(ps))?, 0), + kind::MAP => { + // `Probe` has to be resolved against this descriptor's own bytes + // before it can be sized. + let resolved = map_layout.resolve_for(type_data.get(base_sz..)?, ps); + ( + base_sz.checked_add(MapTypeExtra::size(ps, resolved))?, + 0usize, + ) + } kind::POINTER => (base_sz.checked_add(ElemTypeExtra::size(ps))?, 0), kind::SLICE => (base_sz.checked_add(ElemTypeExtra::size(ps))?, 0), kind::STRUCT => { @@ -123,7 +140,7 @@ mod tests { fn scalar_type_size() { let buf = make_abitype_buf(8, kind::INT, 0); let abi = AbiType::parse(&buf, 8).unwrap(); - let sz = descriptor_size(&buf, &abi, 8).unwrap(); + let sz = descriptor_size(&buf, &abi, 8, MapLayout::SwissSplitGroup).unwrap(); assert_eq!(sz, AbiType::size(8)); } @@ -132,7 +149,7 @@ mod tests { let mut buf = make_abitype_buf(8, kind::POINTER, 0); buf.extend(vec![0u8; ElemTypeExtra::size(8)]); let abi = AbiType::parse(&buf, 8).unwrap(); - let sz = descriptor_size(&buf, &abi, 8).unwrap(); + let sz = descriptor_size(&buf, &abi, 8, MapLayout::SwissSplitGroup).unwrap(); assert_eq!(sz, AbiType::size(8) + ElemTypeExtra::size(8)); } @@ -140,6 +157,6 @@ mod tests { fn unknown_kind_returns_none() { let buf = make_abitype_buf(8, 0xFF, 0); let abi = AbiType::parse(&buf, 8).unwrap(); - assert!(descriptor_size(&buf, &abi, 8).is_none()); + assert!(descriptor_size(&buf, &abi, 8, MapLayout::SwissSplitGroup).is_none()); } } diff --git a/src/structures/embed.rs b/src/structures/embed.rs index 9d4892b..26bab9f 100644 --- a/src/structures/embed.rs +++ b/src/structures/embed.rs @@ -69,50 +69,81 @@ pub fn extract<'a>(ctx: &'a BinaryContext<'a>, ptr_size: u8) -> Vec s, None => return Vec::new(), }; - let header_size = match p.checked_mul(3) { - Some(s) => s, - None => return Vec::new(), - }; let mut out = Vec::new(); let mut seen_arrays: Vec = Vec::new(); - let mut off = 0usize; - while let Some(end) = off.checked_add(header_size) { - if end > data.len() { - break; - } - // Candidate slice header (ptr, len, cap). - if let Some(arr_va) = read_slice_header(data, off, ptr_size, entry_size) - && !seen_arrays.contains(&arr_va.0) - && let Some(mut assets) = - parse_file_array(ctx, arr_va.0, arr_va.1, ptr_size, entry_size) + for (from, to) in ctx.search_regions() { + scan_region( + ctx, + data, + from, + to, + ptr_size, + entry_size, + &mut out, + &mut seen_arrays, + ); + } + out +} + +/// Scan `[start, end)` for `[]file` slice headers, appending every asset of +/// every array that validates. +/// +/// The three words of a `(ptr, len, cap)` header are consecutive, so the walk +/// keeps a rolling window and reads each word once rather than once per +/// candidate position, and tests `len`/`cap` — by far the more selective +/// fields — before looking at the pointer. +#[allow(clippy::too_many_arguments)] +fn scan_region<'a>( + ctx: &'a BinaryContext<'a>, + data: &'a [u8], + start: usize, + end: usize, + ptr_size: u8, + entry_size: usize, + out: &mut Vec>, + seen_arrays: &mut Vec, +) { + let p = ptr_size as usize; + let Some(start) = start.checked_next_multiple_of(p) else { + return; + }; + let end = end.min(data.len()); + // Need three words to form a header. + let Some(last) = end.checked_sub(p.saturating_mul(3)) else { + return; + }; + + let read = |off: usize| read_uintptr(data, off, ptr_size); + let (Some(mut ptr), Some(mut len)) = (read(start), read(start.saturating_add(p))) else { + return; + }; + + let mut off = start; + while off <= last { + let Some(cap) = read(off.saturating_add(p.saturating_mul(2))) else { + return; + }; + if len != 0 + && len == cap + && len <= MAX_ENTRIES_PER_FS + && ptr != 0 + && (len as usize).checked_mul(entry_size).is_some() + && !seen_arrays.contains(&ptr) + && let Some(mut assets) = parse_file_array(ctx, ptr, len, ptr_size, entry_size) { - seen_arrays.push(arr_va.0); + seen_arrays.push(ptr); out.append(&mut assets); } + ptr = len; + len = cap; off = match off.checked_add(p) { Some(o) => o, - None => break, + None => return, }; } - out -} - -/// Validate a `(ptr, len, cap)` slice-header candidate at `off`. Returns -/// `(array_va, len)` when it could plausibly be a `[]file` header. -fn read_slice_header(data: &[u8], off: usize, ps: u8, entry_size: usize) -> Option<(u64, u64)> { - let p = ps as usize; - let ptr = read_uintptr(data, off, ps)?; - let len = read_uintptr(data, off.checked_add(p)?, ps)?; - let cap = read_uintptr(data, off.checked_add(p.checked_mul(2)?)?, ps)?; - if ptr == 0 || len == 0 || len != cap || len > MAX_ENTRIES_PER_FS { - return None; - } - // The whole array must fit within addressable bytes; cheap pre-check using - // entry_size * len not overflowing. - (len as usize).checked_mul(entry_size)?; - Some((ptr, len)) } /// Parse exactly `count` `file` entries at array VA `arr_va`. Returns `None` @@ -126,11 +157,14 @@ fn parse_file_array<'a>( entry_size: usize, ) -> Option>> { let p = ps as usize; - let mut assets = Vec::with_capacity(count.min(64) as usize); + // Deliberately un-reserved: nearly every candidate the scan offers is + // rejected on its first entry, and reserving up front would allocate for + // each of those only to drop it unused. + let mut assets = Vec::new(); // The embed `files` list is sorted by (dir, base); enforcing strictly // increasing order is the decisive filter that rejects the many unrelated // `[]string`-shaped tables a blind scan would otherwise match. - let mut prev_key: Option<(String, String)> = None; + let mut prev_key: Option<(&str, &str)> = None; for i in 0..count { let entry_off = (i as usize).checked_mul(entry_size)?; let entry_va = arr_va.checked_add(entry_off as u64)?; @@ -167,9 +201,7 @@ fn parse_file_array<'a>( // Enforce the canonical embed ordering. let key = embed_sort_key(path); - if let Some(prev) = &prev_key - && *prev >= key - { + if prev_key.is_some_and(|prev| prev >= key) { return None; } prev_key = Some(key); @@ -189,15 +221,18 @@ fn parse_file_array<'a>( /// The `(dir, base)` sort key `embed` orders its file list by. /// /// Mirrors `embed.split`: strip a trailing `/`, then split at the last -/// remaining `/` (a missing dir becomes `"."`). -fn embed_sort_key(name: &str) -> (String, String) { +/// remaining `/` (a missing dir becomes `"."`). Borrowed from `name` rather +/// than owned — the key exists only to be compared against its predecessor, +/// and the blind scan evaluates it for every candidate entry it examines, so +/// owning it allocated twice per rejected entry. +fn embed_sort_key(name: &str) -> (&str, &str) { let n = name.strip_suffix('/').unwrap_or(name); match n.rfind('/') { Some(i) => ( - n.get(..i).unwrap_or(".").to_string(), - n.get(i.saturating_add(1)..).unwrap_or("").to_string(), + n.get(..i).unwrap_or("."), + n.get(i.saturating_add(1)..).unwrap_or(""), ), - None => (".".to_string(), n.to_string()), + None => (".", n), } } diff --git a/src/structures/itab.rs b/src/structures/itab.rs index 6917e99..2434e2a 100644 --- a/src/structures/itab.rs +++ b/src/structures/itab.rs @@ -219,8 +219,11 @@ fn itab_stride(ctx: &BinaryContext<'_>, itab_va: u64, ps: usize, ps_u8: u8) -> O return Some(base); } let inter_va = read_uintptr(buf, 0, ps_u8)?; - let nmethods = interface_method_count(ctx, inter_va, ps_u8).unwrap_or(1); - let extra = nmethods.saturating_sub(1).checked_mul(ps)?; + // No method count means no stride. Guessing one (the old code assumed a + // single method) silently misaligns the rest of the walk and turns every + // following record into fabricated itab pairs; the caller stops instead. + let nmethods = interface_method_count(ctx, inter_va, ps_u8)?; + let extra = nmethods.checked_sub(1)?.checked_mul(ps)?; base.checked_add(extra) } diff --git a/src/structures/locate.rs b/src/structures/locate.rs new file mode 100644 index 0000000..e406e62 --- /dev/null +++ b/src/structures/locate.rs @@ -0,0 +1,190 @@ +//! Finding the `moduledata` in a binary. +//! +//! Every Go binary has exactly one `runtime.firstmoduledata`, but where it is +//! and how to reach it depends on the format and the toolchain version: +//! +//! | Case | Strategy | +//! |-------------------------------|----------------------------------------------| +//! | ELF / Mach-O, Go 1.26+ | the dedicated `.go.module` / `__go_module` section | +//! | ELF / Mach-O, Go ≤ 1.25 | scan for the `pcHeader` pointer | +//! | PE (every version) | scan — the Go PE linker emits no named Go sections | +//! | wasm | scan the reconstructed linear-memory image | +//! +//! The scan works because `moduledata`'s first field is `pcHeader *pcHeader`, +//! which points at the pclntab: a pointer-aligned word equal to the pclntab's +//! address, whose surroundings then parse as a plausible moduledata, is the +//! moduledata. Candidates are validated rather than trusted — a pointer to the +//! pclntab can legitimately appear elsewhere in the image. +//! +//! [`ModuledataLocator`] owns all of that so the type reader and the top-level +//! [`crate::GoBinary`] share one implementation instead of each carrying a +//! copy of the scan. + +use crate::{ + formats::{BinaryContext, BinaryFormat}, + structures::moduledata::{LayoutHints, Moduledata, ModuledataVersion}, +}; + +/// Locates and parses a binary's `moduledata`. +/// +/// Construct once per binary and call [`Self::locate`]; the result is worth +/// caching, since the scan path is proportional to the size of the searched +/// region. +pub struct ModuledataLocator<'a> { + /// The binary being searched. + ctx: &'a BinaryContext<'a>, + /// Pointer size in bytes (4 or 8). + ptr_size: u8, + /// Offset of the pclntab within [`BinaryContext::structure_search_data`]. + pclntab_offset: usize, + /// Signals that decide the moduledata layout once bytes are found. + hints: LayoutHints, +} + +impl<'a> ModuledataLocator<'a> { + /// Build a locator for `ctx`. + /// + /// `pclntab_offset` is the pclntab's offset into the address-space view + /// ([`BinaryContext::structure_search_data`]), which is what both the + /// scan target and the wasm linear-memory addressing are derived from. + pub fn new( + ctx: &'a BinaryContext<'a>, + ptr_size: u8, + pclntab_offset: usize, + hints: LayoutHints, + ) -> Self { + Self { + ctx, + ptr_size, + pclntab_offset, + hints, + } + } + + /// Find and parse the moduledata, or return `None` if neither strategy + /// yields a structurally valid one. + pub fn locate(&self) -> Option { + if !self.ctx.has_va_mapping() || self.ptr_size == 0 { + return None; + } + self.in_section().or_else(|| self.by_scan()) + } + + /// Parse the moduledata out of the dedicated `.go.module` / `__go_module` + /// section (Go 1.26+ ELF and Mach-O). No search needed — the section *is* + /// the structure. + fn in_section(&self) -> Option { + let range = self.ctx.sections().go_module.as_ref()?; + let data = self.ctx.structure_search_data(); + let end = range.offset.checked_add(range.size)?; + Moduledata::parse(data.get(range.offset..end)?, self.ptr_size, self.hints) + } + + /// Scan the address-space view for the `pcHeader` pointer and parse the + /// first candidate whose fields validate. + /// + /// Writable data regions are searched first when the section table names + /// them: the moduledata is emitted into `.noptrdata` (ELF / Mach-O) or the + /// merged `.data` (PE), which on a real binary is a small fraction of the + /// image, and searching it first turns a whole-file sweep into a few tens + /// of kilobytes. The full image remains the fallback, so a stripped or + /// unusual section table costs correctness nothing. + fn by_scan(&self) -> Option { + let data = self.ctx.structure_search_data(); + let target = self.scan_target()?; + + // The section ranges are file offsets. They index the same bytes the + // scan walks for ELF, Mach-O and PE, but for wasm the searched view is + // the reconstructed linear-memory image, where a file offset means + // nothing — so wasm scans the image whole. It is the smaller of the + // two anyway, being only the module's initialized data. + if self.ctx.format() != BinaryFormat::Wasm { + let sections = self.ctx.sections(); + for range in [sections.noptrdata.as_ref(), sections.data_section.as_ref()] + .into_iter() + .flatten() + { + if let Some(found) = self.scan_region(data, range.offset, range.size, target) { + return Some(found); + } + } + } + self.scan_region(data, 0, data.len(), target) + } + + /// The pointer value the scan looks for: the pclntab's address in whatever + /// address space `structure_search_data` presents. + /// + /// For wasm that view *is* linear memory, so the pclntab's offset into it + /// is already the address the runtime stored. Every other format needs the + /// file-offset-to-VA translation. On chained-fixup Mach-O the view is the + /// rebased copy, so the stored pointer is a real VA by the time we see it. + fn scan_target(&self) -> Option { + if self.ctx.format() == BinaryFormat::Wasm { + Some(self.pclntab_offset as u64) + } else { + self.ctx.file_to_va(self.pclntab_offset) + } + } + + /// Scan `[start, start + len)` of `data` for `target` at pointer-aligned + /// positions, parsing and validating each hit. + /// + /// The comparison is done on whole words rather than byte slices: the + /// region is walked with `chunks_exact`, which the compiler turns into a + /// straight-line integer scan, and only an equal word costs a parse. + fn scan_region( + &self, + data: &[u8], + start: usize, + len: usize, + target: u64, + ) -> Option { + let p = self.ptr_size as usize; + // Align the start up to a pointer boundary; the linker never places + // moduledata unaligned, and a misaligned scan would double the work. + let start = start.checked_next_multiple_of(p)?; + let end = start.checked_add(len)?.min(data.len()); + let region = data.get(start..end)?; + + for (i, word) in region.chunks_exact(p).enumerate() { + let value = match p { + 4 => u32::from_le_bytes(word.try_into().ok()?) as u64, + 8 => u64::from_le_bytes(word.try_into().ok()?), + _ => return None, + }; + if value != target { + continue; + } + let Some(at) = i.checked_mul(p).and_then(|o| o.checked_add(start)) else { + continue; + }; + if let Some(md) = data + .get(at..) + .and_then(|rest| Moduledata::parse(rest, self.ptr_size, self.hints)) + && self.accept(&md) + { + return Some(md); + } + } + None + } + + /// Whether a scanned candidate is a real moduledata. + /// + /// A pointer to the pclntab can appear outside the moduledata, so a hit is + /// only accepted when the fields around it also hold: the PC range must be + /// non-empty, and the structure must be anchored by a field that maps back + /// into the image. The legacy (Go 1.5-1.15) layout has no `funcnametab` + /// and — before Go 1.7 — no `types` base, so it is anchored through its + /// always-present `text` boundary instead. + fn accept(&self, md: &Moduledata) -> bool { + if md.minpc >= md.maxpc { + return false; + } + match md.version { + ModuledataVersion::V1 => md.text != 0 && self.ctx.va_to_file(md.text).is_some(), + _ => md.types != 0 && self.ctx.va_to_file(md.funcnametab.ptr).is_some(), + } + } +} diff --git a/src/structures/maptype.rs b/src/structures/maptype.rs index a1d0c96..85774a5 100644 --- a/src/structures/maptype.rs +++ b/src/structures/maptype.rs @@ -1,119 +1,489 @@ -//! Go map type extra fields (`abi.SwissMapType`). +//! Go map type extra fields (`abi.MapType` / `abi.SwissMapType` / +//! `abi.OldMapType` / `runtime.maptype`). //! -//! The `MapType` follows the embedded `abi.Type` in the binary layout -//! for types of kind `Map`. This is the largest concrete type extra, -//! carrying ten pointer-sized fields and a `u32` flags word. +//! The map extra follows the embedded `abi.Type` in the binary layout for types +//! of kind `Map`. It is by a wide margin the least stable concrete-type extra: +//! its shape has changed **five** times across the releases gobin parses (six +//! layouts in all), and +//! the meaning of its flag bits changed once more on top of that. Reading one +//! layout with another's field list silently shifts every field past the +//! pointer block and — because the extra's size feeds +//! [`crate::structures::descriptor::descriptor_size`] — mislocates the trailing +//! `UncommonType`, which then yields a garbage method count. //! -//! Binary layout: -//! - 10 * pointer_size + 4 bytes of flags -//! - Plus 4 bytes of alignment padding when ps == 8 -//! - Total: 10*ps + 4 + (4 if ps==8) = 44 (ps=4) or 88 (ps=8) +//! ## Layouts //! -//! Fields (all uintptr unless noted): -//! - `Key` -- pointer to the key type descriptor -//! - `Elem` -- pointer to the element type descriptor -//! - `Group` -- pointer to the group type descriptor -//! - `Hasher` -- pointer to the hash function -//! - `GroupSize` -- size of a map group in bytes -//! - `KeysOff` -- offset of keys within a group -//! - `KeyStride` -- stride between consecutive keys -//! - `ElemsOff` -- offset of elements within a group -//! - `ElemStride` -- stride between consecutive elements -//! - `ElemOff` -- offset from key to its corresponding element -//! - `Flags` (u32) -- map type flags +//! Every layout is a run of pointer-sized fields followed by an 8-byte tail +//! (padded to pointer alignment on 64-bit), so the size is +//! `pointers * ptrSize + 8` throughout. //! -//! Source: `src/internal/abi/type.go:379-420` +//! | Go | [`MapLayout`] | Pointer fields | Tail | ps=8 | ps=4 | +//! |-----------|-----------------------|--------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------|------| +//! | ≤ 1.10 | `HmapWithHmapType` | `Key`, `Elem`, `Bucket`, `Hmap` | `keysize u8`, `indirectkey bool`, `valuesize u8`, `indirectvalue bool`, `bucketsize u16`, `reflexivekey bool`, `needkeyupdate bool` | 40 | 24 | +//! | 1.11 | `HmapBools` | `Key`, `Elem`, `Bucket` | same bool tail | 32 | 20 | +//! | 1.12-1.13 | `HmapFlags` | `Key`, `Elem`, `Bucket` | `keysize u8`, `elemsize u8`, `bucketsize u16`, `flags u32` | 32 | 20 | +//! | 1.14-1.23 | `HmapHasher` | `Key`, `Elem`, `Bucket`, `Hasher` | same as above | 40 | 24 | +//! | 1.24-1.26 | `Swiss` | + `GroupSize`, `SlotSize`, `ElemOff` (7 total) | `flags u32` | 64 | 32 | +//! | 1.27+ | `SwissSplitGroup` | + `GroupSize`, `KeysOff`, `KeyStride`, `ElemsOff`, `ElemStride`, `ElemOff` (10) | `flags u32` | 88 | 44 | +//! +//! What each change did: Go 1.11 dropped the `hmap` type pointer; 1.12 folded +//! the four tail booleans into a `flags` word; 1.14 added the `hasher` function +//! pointer; 1.24 replaced bucket-based `hmap` with Swiss tables; 1.27 split the +//! single `SlotSize`/`ElemOff` pair into explicit key/elem offsets and strides +//! so one descriptor can describe both the interleaved (`KVKVKV…`) and the +//! split (`KKKVVV…`, `GOEXPERIMENT=mapsplitgroup`) group layouts. +//! +//! ## Flag bits +//! +//! The `flags` word is **not** comparable across the hmap/Swiss boundary — Go +//! renumbered the bits when it introduced Swiss maps: +//! +//! | Property | hmap (1.12-1.23) | Swiss (1.24+) | +//! |-----------------|------------------|---------------| +//! | indirect key | `1 << 0` | `1 << 2` | +//! | indirect elem | `1 << 1` | `1 << 3` | +//! | reflexive key | `1 << 2` | — (dropped) | +//! | need key update | `1 << 3` | `1 << 0` | +//! | hash might panic| `1 << 4` | `1 << 1` | +//! +//! Read them through [`MapTypeExtra::flags`], a [`MapFlags`] that normalizes +//! all three encodings — including the pre-1.12 booleans — into `Option` +//! per property. [`MapTypeExtra::raw_flags`] keeps the undecoded word for +//! callers that want it. +//! +//! Sources: `src/runtime/type.go` (Go 1.7-1.20), `src/internal/abi/type.go` +//! (Go 1.21-1.23), `src/internal/abi/map_noswiss.go` + `map_swiss.go` (Go +//! 1.24-1.25), `src/internal/abi/map.go` (Go 1.26, 1.27). Sizes cross-checked +//! against `abi.RTypeSize` in `src/internal/abi/compiletype.go` (Go 1.27), +//! which gives `Map => CommonSize + 10*ptrSize + 4 (+4 padding when +//! ptrSize == 8)`. + +use crate::structures::util::{read_u16, read_u32, read_uintptr}; + +/// Which `abi` map-descriptor layout a binary uses. +/// +/// Determined from the Go version where one is available, and otherwise from +/// the structural evidence the rest of the binary carries — see +/// [`MapLayout::infer`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum MapLayout { + /// Go ≤ 1.10: `Key`, `Elem`, `Bucket`, `Hmap` plus the bool-flag tail. + HmapWithHmapType, + /// Go 1.11: the `Hmap` type pointer is gone; bool-flag tail retained. + HmapBools, + /// Go 1.12-1.13: the tail booleans became a `flags uint32`. + HmapFlags, + /// Go 1.14-1.23: `Hasher` reinstated as the fourth pointer field. + HmapHasher, + /// Go 1.24-1.26: Swiss tables with a single `SlotSize` / `ElemOff` pair. + Swiss, + /// Go 1.27+: Swiss tables carrying explicit key/elem offsets and strides. + SwissSplitGroup, + /// Go 1.20-1.25 with no recoverable version string — the moduledata layout + /// narrows the window but cannot separate `HmapHasher` (1.20-1.23) from + /// `Swiss` (1.24-1.25). Each descriptor is then classified from its own + /// bytes by [`MapLayout::resolve_for`]. + Probe, +} + +impl MapLayout { + /// Pick the layout from the Go minor version. Exact. + pub fn for_go_minor(minor: u32) -> Self { + match minor { + 0..=10 => Self::HmapWithHmapType, + 11 => Self::HmapBools, + 12 | 13 => Self::HmapFlags, + 14..=23 => Self::HmapHasher, + 24..=26 => Self::Swiss, + _ => Self::SwissSplitGroup, + } + } + + /// Pick the layout from whatever evidence the binary offers. + /// + /// The Go version string settles it outright. Without one, the moduledata + /// layout still dates the binary exactly at two boundaries — `V5` is Go + /// 1.27+ and `V4` is Go 1.26 — and the pclntab magic gives a floor: Swiss + /// maps arrived in Go 1.24 and therefore imply the Go 1.20 magic, while + /// anything below that magic is at most Go 1.19 and so `HmapHasher` + /// (the layout in force from 1.14; older magics narrow it no further, and + /// [`Self::resolve_for`] is not consulted because the ambiguity there is + /// between same-size layouts that no local evidence separates). + /// + /// The one window that stays open is a `V3` moduledata with the Go 1.20 + /// magic (Go 1.20-1.25), which straddles the 1.24 Swiss switch; that + /// resolves per descriptor via [`Self::Probe`]. + /// + /// `md_is_v5` / `md_is_v4` come from + /// [`crate::structures::moduledata::ModuledataVersion`]; `go120_magic` is + /// true for [`crate::structures::PclntabVersion::Go120`]. + pub fn infer(go_minor: Option, go120_magic: bool, md_is_v5: bool, md_is_v4: bool) -> Self { + if let Some(m) = go_minor { + return Self::for_go_minor(m); + } + if md_is_v5 { + return Self::SwissSplitGroup; + } + if md_is_v4 { + return Self::Swiss; + } + if !go120_magic { + return Self::HmapHasher; + } + Self::Probe + } + + /// Resolve [`Self::Probe`] against one descriptor's extra bytes; every + /// other variant returns itself. + /// + /// The two candidates in the ambiguous window — `HmapHasher` (Go + /// 1.20-1.23) and `Swiss` (Go 1.24-1.25) — both start with four + /// pointer-sized fields, so they diverge at `extra + 4*ptrSize`, and the + /// two readings of that word are disjoint in range: + /// + /// - **Swiss** reads `GroupSize uintptr` = `8 + 8*SlotSize`, and `SlotSize` + /// is at most 256 bytes because Go stores keys and elements larger than + /// 128 bytes indirectly (as pointers). So `GroupSize <= 2056` and every + /// bit above 15 is zero. + /// - **HmapHasher** packs `KeySize u8`, `ValueSize u8`, `BucketSize u16`, + /// `Flags u32` into the same word, and `BucketSize` — which lands in bits + /// 16..31 — is `8 + 8*(KeySize+ValueSize) + ptrSize`, never zero. + /// + /// A non-zero value above bit 15 therefore means `HmapHasher`; anything + /// else is `Swiss`. On 32-bit the same reasoning applies to the `u32` at + /// that offset, whose bits 16..31 hold `BucketSize` in the hmap reading. + pub fn resolve_for(self, extra: &[u8], ps: u8) -> Self { + if self != Self::Probe { + return self; + } + let Some(off) = (ps as usize).checked_mul(4) else { + return Self::Swiss; + }; + let Some(word) = read_uintptr(extra, off, ps) else { + return Self::Swiss; + }; + if word >> 16 != 0 { + Self::HmapHasher + } else { + Self::Swiss + } + } + + /// Whether the layout is one of the bucket-based `hmap` shapes (Go ≤ 1.23). + pub fn is_hmap(self) -> bool { + matches!( + self, + Self::HmapWithHmapType | Self::HmapBools | Self::HmapFlags | Self::HmapHasher + ) + } + + /// Number of pointer-sized fields the layout carries before its 8-byte + /// tail. + fn pointer_field_count(self) -> usize { + match self { + // Key, Elem, Bucket, Hmap. + Self::HmapWithHmapType => 4, + // Key, Elem, Bucket. + Self::HmapBools | Self::HmapFlags => 3, + // Key, Elem, Bucket, Hasher. + Self::HmapHasher => 4, + // …plus GroupSize, SlotSize, ElemOff. + Self::Swiss => 7, + // …plus KeysOff, KeyStride, ElemsOff, ElemStride. + Self::SwissSplitGroup => 10, + // Never reached: `Probe` is resolved before sizing. + Self::Probe => 7, + } + } +} -use crate::structures::util::{read_u32, read_uintptr}; +/// Semantic map properties, decoded from whichever flag encoding the +/// descriptor's layout uses. +/// +/// Each is `None` when the layout does not record that property (Swiss maps +/// dropped `reflexive_key`; the pre-1.12 bool tail has no `hash_might_panic`). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct MapFlags { + /// Keys are stored indirectly, as pointers. + pub indirect_key: Option, + /// Elements are stored indirectly, as pointers. + pub indirect_elem: Option, + /// `k == k` holds for every key (no NaN-like keys). + pub reflexive_key: Option, + /// An overwrite must update the stored key, not just the element. + pub need_key_update: Option, + /// The hash function can panic (interface-keyed maps). + pub hash_might_panic: Option, +} /// Parsed extra fields for a map type descriptor. -#[derive(Debug, Clone, Copy, Default)] +/// +/// Fields absent from the descriptor's [`MapLayout`] are `None` so callers can +/// tell "this Go version does not record it" from a genuine zero. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct MapTypeExtra { + /// Which layout the descriptor was parsed with. + pub layout: MapLayout, /// Virtual address of the key type descriptor. pub key: u64, /// Virtual address of the element type descriptor. pub elem: u64, - /// Virtual address of the group type descriptor. + /// Virtual address of the bucket (hmap) or slot-group (Swiss) type + /// descriptor. pub group: u64, - /// Virtual address of the hash function. - pub hasher: u64, - /// Size of a map group in bytes. - pub group_size: u64, - /// Offset of keys within a group. - pub keys_off: u64, - /// Stride between consecutive keys. - pub key_stride: u64, - /// Offset of elements within a group. - pub elems_off: u64, - /// Stride between consecutive elements. - pub elem_stride: u64, - /// Offset from key to its corresponding element. - pub elem_off: u64, - /// Map type flags. - pub flags: u32, + /// Virtual address of the `hmap` type descriptor + /// ([`MapLayout::HmapWithHmapType`] only — Go 1.11 removed the field). + pub hmap: Option, + /// Virtual address of the key-hashing function. `None` for Go 1.11-1.13, + /// which had no `hasher` field. + pub hasher: Option, + /// Size of a slot group in bytes (`GroupSize`). Swiss layouts only. + pub group_size: Option, + /// Size of one key/elem slot (`SlotSize`). [`MapLayout::Swiss`] only — + /// Go 1.27 replaced it with the explicit stride fields below. + pub slot_size: Option, + /// Offset of the keys array within a group (`KeysOff`). + /// [`MapLayout::SwissSplitGroup`] only. + pub keys_off: Option, + /// Stride between consecutive keys (`KeyStride`). + /// [`MapLayout::SwissSplitGroup`] only. + pub key_stride: Option, + /// Offset of the elements array within a group (`ElemsOff`). + /// [`MapLayout::SwissSplitGroup`] only. + pub elems_off: Option, + /// Stride between consecutive elements (`ElemStride`). + /// [`MapLayout::SwissSplitGroup`] only. + pub elem_stride: Option, + /// Offset from a key to its element within a slot (`ElemOff`). Present in + /// both Swiss layouts. + pub elem_off: Option, + /// Size of a key slot in bytes (`KeySize`). hmap layouts only. + pub key_size: Option, + /// Size of a value slot in bytes (`ValueSize` / `elemsize`). hmap layouts + /// only. + pub value_size: Option, + /// Size of a bucket in bytes (`BucketSize`). hmap layouts only. + pub bucket_size: Option, + /// Raw `flags` word. `None` for the pre-1.12 layouts, which encoded the + /// same properties as separate booleans. **Bit meanings differ between the + /// hmap and Swiss eras** — prefer [`Self::flags`], which normalizes them. + pub raw_flags: Option, + /// Semantic flags, normalized across all three encodings. + pub flags: MapFlags, } impl MapTypeExtra { - /// Binary size for the given pointer size. + /// Binary size of the extra for the given pointer size and layout. /// - /// Layout: 10*ps + 4 + padding. - /// - ps=4: 10*4 + 4 = 44 bytes (no padding needed) - /// - ps=8: 10*8 + 4 + 4 = 88 bytes (4 bytes padding for alignment) - pub fn size(ps: u8) -> usize { - let base = (ps as usize).saturating_mul(10).saturating_add(4); + /// Every layout is `pointer_fields * ptrSize` followed by an 8-byte tail — + /// either four small integers plus four booleans, or `u8 + u8 + u16 + u32`, + /// or (Swiss) a `u32` padded out to pointer alignment. That comes to + /// `pointers * ps + 8` on 64-bit and `pointers * ps + 8` on 32-bit for the + /// hmap layouts, and `pointers * ps + 4 (+4)` for the Swiss ones. + pub fn size(ps: u8, layout: MapLayout) -> usize { + let p = ps as usize; + let pointers = p.saturating_mul(layout.pointer_field_count()); + if layout.is_hmap() { + // u8 + u8 + u16 + (u32 | 4 bools) = 8 bytes, pointer-aligned on + // both 32- and 64-bit. + return pointers.saturating_add(8); + } + let base = pointers.saturating_add(4); if ps == 8 { - base.saturating_add(4) // alignment padding + base.saturating_add(4) // alignment padding for the trailing u32 } else { base } } - /// Parse from `data`. Data must start at the map type extra fields. + /// Parse from `data`, which must start at the map type's extra fields + /// (i.e. just past the embedded `abi.Type`). + /// + /// A [`MapLayout::Probe`] layout is resolved against these bytes first, so + /// the returned [`Self::layout`] is always a concrete variant. /// - /// Returns `None` if the buffer is too small. - pub fn parse(data: &[u8], ps: u8) -> Option { + /// Returns `None` if the buffer is too small for the resolved layout. + pub fn parse(data: &[u8], ps: u8, layout: MapLayout) -> Option { + let layout = layout.resolve_for(data, ps); let p = ps as usize; - if data.len() < Self::size(ps) { + if data.len() < Self::size(ps, layout) { return None; } let mut off: usize = 0; + // Read the next pointer-sized field and advance. + let next = |off: &mut usize| -> Option { + let v = read_uintptr(data, *off, ps)?; + *off = off.checked_add(p)?; + Some(v) + }; - let key = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let elem = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let group = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let hasher = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let group_size = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let keys_off = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let key_stride = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let elems_off = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let elem_stride = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let elem_off = read_uintptr(data, off, ps)?; - off = off.checked_add(p)?; - let flags = read_u32(data, off)?; - - Some(Self { + let key = next(&mut off)?; + let elem = next(&mut off)?; + let group = next(&mut off)?; + + let mut out = Self { + layout, key, elem, group, - hasher, - group_size, - keys_off, - key_stride, - elems_off, - elem_stride, - elem_off, - flags, - }) + hmap: None, + hasher: None, + group_size: None, + slot_size: None, + keys_off: None, + key_stride: None, + elems_off: None, + elem_stride: None, + elem_off: None, + key_size: None, + value_size: None, + bucket_size: None, + raw_flags: None, + flags: MapFlags::default(), + }; + + match layout { + MapLayout::HmapWithHmapType => { + out.hmap = Some(next(&mut off)?); + out.read_hmap_sizes(data, off)?; + out.read_bool_tail(data, off)?; + } + MapLayout::HmapBools => { + out.read_hmap_sizes(data, off)?; + out.read_bool_tail(data, off)?; + } + MapLayout::HmapFlags => { + out.read_hmap_sizes(data, off)?; + out.read_hmap_flags(data, off.checked_add(4)?)?; + } + MapLayout::HmapHasher => { + out.hasher = Some(next(&mut off)?); + out.read_hmap_sizes(data, off)?; + out.read_hmap_flags(data, off.checked_add(4)?)?; + } + MapLayout::Swiss => { + out.hasher = Some(next(&mut off)?); + out.group_size = Some(next(&mut off)?); + out.slot_size = Some(next(&mut off)?); + out.elem_off = Some(next(&mut off)?); + out.read_swiss_flags(data, off)?; + } + // `Probe` cannot survive `resolve_for`; treat it as the Go 1.27 + // layout so the match stays total without an unreachable arm. + MapLayout::SwissSplitGroup | MapLayout::Probe => { + out.hasher = Some(next(&mut off)?); + out.group_size = Some(next(&mut off)?); + out.keys_off = Some(next(&mut off)?); + out.key_stride = Some(next(&mut off)?); + out.elems_off = Some(next(&mut off)?); + out.elem_stride = Some(next(&mut off)?); + out.elem_off = Some(next(&mut off)?); + out.read_swiss_flags(data, off)?; + } + } + + Some(out) + } + + /// Read the `keysize u8`, `valuesize u8`, `bucketsize u16` block shared by + /// every hmap layout. `off` is the start of the 8-byte tail. + /// + /// The pre-1.12 layouts interleave booleans between the sizes + /// (`keysize, indirectkey, valuesize, indirectvalue, bucketsize`), so the + /// sizes sit at `+0` / `+2` there and at `+0` / `+1` from Go 1.12; the + /// bucket size is at `+4` and `+2` respectively. + fn read_hmap_sizes(&mut self, data: &[u8], off: usize) -> Option<()> { + let bools = matches!( + self.layout, + MapLayout::HmapWithHmapType | MapLayout::HmapBools + ); + let (value_at, bucket_at) = if bools { + (2usize, 4usize) + } else { + (1usize, 2usize) + }; + self.key_size = data.get(off).copied(); + self.value_size = data.get(off.checked_add(value_at)?).copied(); + self.bucket_size = read_u16(data, off.checked_add(bucket_at)?); + Some(()) + } + + /// Decode the Go ≤ 1.11 boolean tail into [`MapFlags`]. `off` is the start + /// of the 8-byte tail: `keysize, indirectkey, valuesize, indirectvalue, + /// bucketsize(2), reflexivekey, needkeyupdate`. + fn read_bool_tail(&mut self, data: &[u8], off: usize) -> Option<()> { + let at = |i: usize| -> Option { + off.checked_add(i) + .and_then(|o| data.get(o)) + .map(|&b| b != 0) + }; + self.flags = MapFlags { + indirect_key: at(1), + indirect_elem: at(3), + reflexive_key: at(6), + need_key_update: at(7), + // Introduced with the flags word in Go 1.12. + hash_might_panic: None, + }; + Some(()) + } + + /// Decode the Go 1.12-1.23 `flags uint32` (`runtime/map.go`: + /// `indirectkey 1`, `indirectvalue 2`, `reflexivekey 4`, `needkeyupdate 8`, + /// `hashMightPanic 16`). + fn read_hmap_flags(&mut self, data: &[u8], off: usize) -> Option<()> { + let f = read_u32(data, off)?; + self.raw_flags = Some(f); + self.flags = MapFlags { + indirect_key: Some(f & 1 != 0), + indirect_elem: Some(f & 2 != 0), + reflexive_key: Some(f & 4 != 0), + need_key_update: Some(f & 8 != 0), + hash_might_panic: Some(f & 16 != 0), + }; + Some(()) + } + + /// Decode the Go 1.24+ `Flags uint32` (`internal/abi/map.go`: + /// `MapNeedKeyUpdate 1`, `MapHashMightPanic 2`, `MapIndirectKey 4`, + /// `MapIndirectElem 8`). Swiss maps dropped `reflexivekey`. + fn read_swiss_flags(&mut self, data: &[u8], off: usize) -> Option<()> { + let f = read_u32(data, off)?; + self.raw_flags = Some(f); + self.flags = MapFlags { + indirect_key: Some(f & 4 != 0), + indirect_elem: Some(f & 8 != 0), + reflexive_key: None, + need_key_update: Some(f & 1 != 0), + hash_might_panic: Some(f & 2 != 0), + }; + Some(()) + } + + /// Byte stride between consecutive keys in a bucket / group, however the + /// descriptor's layout happens to record it. + /// + /// `SwissSplitGroup` states it directly as `KeyStride`; `Swiss` uses the + /// key/elem `SlotSize` as the stride for both; the hmap layouts pack keys + /// of `KeySize` contiguously in the bucket. `None` when the descriptor did + /// not carry the value. + pub fn key_stride(&self) -> Option { + match self.layout { + l if l.is_hmap() => self.key_size.map(u64::from), + MapLayout::Swiss => self.slot_size, + _ => self.key_stride, + } + } + + /// Byte stride between consecutive elements, however the descriptor's + /// layout records it. See [`Self::key_stride`]. + pub fn elem_stride(&self) -> Option { + match self.layout { + l if l.is_hmap() => self.value_size.map(u64::from), + MapLayout::Swiss => self.slot_size, + _ => self.elem_stride, + } } } @@ -122,68 +492,270 @@ mod tests { use super::*; #[test] - fn parse_64bit() { - let mut buf = vec![0u8; 88]; - let ps: u8 = 8; - let p = ps as usize; + fn sizes_match_upstream() { + // 64-bit: CommonSize is added by the caller; these are the extras only. + assert_eq!(MapTypeExtra::size(8, MapLayout::HmapWithHmapType), 40); + assert_eq!(MapTypeExtra::size(8, MapLayout::HmapBools), 32); + assert_eq!(MapTypeExtra::size(8, MapLayout::HmapFlags), 32); + assert_eq!(MapTypeExtra::size(8, MapLayout::HmapHasher), 40); + assert_eq!(MapTypeExtra::size(8, MapLayout::Swiss), 64); + assert_eq!(MapTypeExtra::size(8, MapLayout::SwissSplitGroup), 88); + // 32-bit: the trailing u32 needs no padding. + assert_eq!(MapTypeExtra::size(4, MapLayout::HmapWithHmapType), 24); + assert_eq!(MapTypeExtra::size(4, MapLayout::HmapBools), 20); + assert_eq!(MapTypeExtra::size(4, MapLayout::HmapFlags), 20); + assert_eq!(MapTypeExtra::size(4, MapLayout::HmapHasher), 24); + assert_eq!(MapTypeExtra::size(4, MapLayout::Swiss), 32); + assert_eq!(MapTypeExtra::size(4, MapLayout::SwissSplitGroup), 44); + } - // Fill each uintptr field with a recognizable value - for i in 0..10u64 { - let val = (i + 1) * 0x1000; - let off = i as usize * p; - buf[off..off + p].copy_from_slice(&val.to_le_bytes()); - } - // flags at offset 80 - buf[80..84].copy_from_slice(&0xABCDu32.to_le_bytes()); - - let m = MapTypeExtra::parse(&buf, 8).unwrap(); - assert_eq!(m.key, 0x1000); - assert_eq!(m.elem, 0x2000); - assert_eq!(m.group, 0x3000); - assert_eq!(m.hasher, 0x4000); - assert_eq!(m.group_size, 0x5000); - assert_eq!(m.keys_off, 0x6000); - assert_eq!(m.key_stride, 0x7000); - assert_eq!(m.elems_off, 0x8000); - assert_eq!(m.elem_stride, 0x9000); - assert_eq!(m.elem_off, 0xA000); - assert_eq!(m.flags, 0xABCD); + #[test] + fn for_go_minor_covers_every_boundary() { + use MapLayout::*; + let cases = [ + (7, HmapWithHmapType), + (10, HmapWithHmapType), + (11, HmapBools), + (12, HmapFlags), + (13, HmapFlags), + (14, HmapHasher), + (23, HmapHasher), + (24, Swiss), + (26, Swiss), + (27, SwissSplitGroup), + (28, SwissSplitGroup), + ]; + for (minor, want) in cases { + assert_eq!(MapLayout::for_go_minor(minor), want, "go1.{minor}"); + } } #[test] - fn size_ps4() { - assert_eq!(MapTypeExtra::size(4), 44); + fn infer_without_a_version_uses_structural_evidence() { + // V5 and V4 date the binary exactly. + assert_eq!( + MapLayout::infer(None, true, true, false), + MapLayout::SwissSplitGroup + ); + assert_eq!(MapLayout::infer(None, true, false, true), MapLayout::Swiss); + // Pre-Go120 magic rules out Swiss maps entirely. + assert_eq!( + MapLayout::infer(None, false, false, false), + MapLayout::HmapHasher + ); + // Go 1.20-1.25 stays ambiguous and is resolved per descriptor. + assert_eq!(MapLayout::infer(None, true, false, false), MapLayout::Probe); + // A version string always wins. + assert_eq!( + MapLayout::infer(Some(11), true, true, true), + MapLayout::HmapBools + ); + } + + /// `map[string]float64` as Go 1.26 emits it, taken byte-for-byte from + /// `tests/samples/types_go126_linux_amd64` at VA `0x4b06a0 + 0x30`. + fn go126_map_string_float64() -> Vec { + let mut d = Vec::new(); + for w in [ + 0x4a9ae0u64, // Key -> string + 0x4a9ea0, // Elem -> float64 + 0x4b2320, // Group + 0x4cf168, // Hasher + 0xc8, // GroupSize = 200 + 0x18, // SlotSize = 24 (16-byte string + 8-byte float64) + 0x10, // ElemOff = 16 + 0x1, // Flags = MapNeedKeyUpdate, then 4 bytes of padding + ] { + d.extend_from_slice(&w.to_le_bytes()); + } + d } #[test] - fn size_ps8() { - assert_eq!(MapTypeExtra::size(8), 88); + fn swiss_layout_reads_go126_fields() { + let d = go126_map_string_float64(); + let m = MapTypeExtra::parse(&d, 8, MapLayout::Swiss).unwrap(); + assert_eq!(m.layout, MapLayout::Swiss); + assert_eq!(m.key, 0x4a9ae0); + assert_eq!(m.elem, 0x4a9ea0); + assert_eq!(m.group, 0x4b2320); + assert_eq!(m.hasher, Some(0x4cf168)); + assert_eq!(m.group_size, Some(0xc8)); + assert_eq!(m.slot_size, Some(0x18)); + assert_eq!(m.elem_off, Some(0x10)); + assert_eq!(m.raw_flags, Some(1)); + assert_eq!(m.flags.need_key_update, Some(true)); + assert_eq!(m.flags.indirect_key, Some(false)); + assert_eq!(m.flags.reflexive_key, None, "Swiss maps dropped it"); + // Go 1.27-only fields must read as absent, not as zero. + assert!(m.keys_off.is_none()); + assert!(m.key_stride.is_none()); + assert!(m.elems_off.is_none()); + assert!(m.elem_stride.is_none()); } #[test] - fn too_short_returns_none() { - let buf = vec![0u8; 80]; - assert!(MapTypeExtra::parse(&buf, 8).is_none()); + fn go126_descriptor_is_not_read_with_the_go127_layout() { + // Regression guard for the bug this layout gating fixes: the Go 1.27 + // extra is 88 bytes, so applied to a 64-byte Go 1.26 descriptor it + // over-reads into whatever follows. + let d = go126_map_string_float64(); + assert_eq!(d.len(), 64); + assert!( + MapTypeExtra::parse(&d, 8, MapLayout::SwissSplitGroup).is_none(), + "the 1.27 layout must not fit a 1.26 descriptor" + ); } #[test] - fn parse_32bit() { - let mut buf = vec![0u8; 44]; - let ps: u8 = 4; - let p = ps as usize; + fn probe_picks_swiss_for_a_go126_descriptor() { + let d = go126_map_string_float64(); + assert_eq!(MapLayout::Probe.resolve_for(&d, 8), MapLayout::Swiss); + let m = MapTypeExtra::parse(&d, 8, MapLayout::Probe).unwrap(); + assert_eq!(m.layout, MapLayout::Swiss); + assert_eq!(m.group_size, Some(0xc8)); + } - for i in 0..10u32 { - let val = (i + 1) * 0x100; - let off = i as usize * p; - buf[off..off + p].copy_from_slice(&val.to_le_bytes()); + /// Go 1.14-1.23 `map[string]float64`: KeySize 16, ValueSize 8, + /// BucketSize `8 + 8*24 + 8 = 208`, Flags 4 (`reflexivekey`). + fn hmap_hasher_map_string_float64() -> Vec { + let mut d = Vec::new(); + for w in [0x4a9ae0u64, 0x4a9ea0, 0x4b2320, 0x4cf168] { + d.extend_from_slice(&w.to_le_bytes()); } - // flags at offset 40 - buf[40..44].copy_from_slice(&0x0001u32.to_le_bytes()); + d.push(16); // KeySize + d.push(8); // ValueSize + d.extend_from_slice(&208u16.to_le_bytes()); // BucketSize + d.extend_from_slice(&4u32.to_le_bytes()); // Flags = reflexivekey + d + } - let m = MapTypeExtra::parse(&buf, 4).unwrap(); - assert_eq!(m.key, 0x100); - assert_eq!(m.elem, 0x200); - assert_eq!(m.elem_off, 0xA00); - assert_eq!(m.flags, 1); + #[test] + fn probe_picks_hmap_for_a_bucket_descriptor() { + // BucketSize occupies bits 16..31 of the probe word, so it exceeds + // 0xFFFF and reads as hmap rather than as a Swiss GroupSize. + let d = hmap_hasher_map_string_float64(); + assert_eq!(MapLayout::Probe.resolve_for(&d, 8), MapLayout::HmapHasher); + let m = MapTypeExtra::parse(&d, 8, MapLayout::Probe).unwrap(); + assert_eq!(m.layout, MapLayout::HmapHasher); + assert_eq!(m.hasher, Some(0x4cf168)); + assert_eq!(m.key_size, Some(16)); + assert_eq!(m.value_size, Some(8)); + assert_eq!(m.bucket_size, Some(208)); + assert_eq!(m.raw_flags, Some(4)); + assert_eq!(m.flags.reflexive_key, Some(true)); + assert_eq!(m.flags.indirect_key, Some(false)); + assert_eq!(m.key_stride(), Some(16)); + assert_eq!(m.elem_stride(), Some(8)); + // Swiss-only fields must read as absent. + assert!(m.group_size.is_none()); + assert!(m.slot_size.is_none()); + } + + #[test] + fn pre_112_bool_tail_decodes_without_a_flags_word() { + // Go ≤1.10 `map[string]float64`: Key, Elem, Bucket, Hmap, then + // keysize 16, indirectkey 0, valuesize 8, indirectvalue 0, + // bucketsize 208, reflexivekey 1, needkeyupdate 0. + let mut d = Vec::new(); + for w in [0x4a9ae0u64, 0x4a9ea0, 0x4b2320, 0x4b3000] { + d.extend_from_slice(&w.to_le_bytes()); + } + d.extend_from_slice(&[16, 0, 8, 0]); + d.extend_from_slice(&208u16.to_le_bytes()); + d.extend_from_slice(&[1, 0]); + + let m = MapTypeExtra::parse(&d, 8, MapLayout::HmapWithHmapType).unwrap(); + assert_eq!(m.hmap, Some(0x4b3000)); + assert_eq!(m.hasher, None, "no hasher field before Go 1.14"); + assert_eq!(m.key_size, Some(16)); + assert_eq!(m.value_size, Some(8)); + assert_eq!(m.bucket_size, Some(208)); + assert_eq!( + m.raw_flags, None, + "the bool tail has no flags word to report" + ); + assert_eq!(m.flags.indirect_key, Some(false)); + assert_eq!(m.flags.indirect_elem, Some(false)); + assert_eq!(m.flags.reflexive_key, Some(true)); + assert_eq!(m.flags.need_key_update, Some(false)); + assert_eq!(m.flags.hash_might_panic, None); + } + + #[test] + fn go111_drops_the_hmap_pointer() { + let mut d = Vec::new(); + for w in [0x4a9ae0u64, 0x4a9ea0, 0x4b2320] { + d.extend_from_slice(&w.to_le_bytes()); + } + d.extend_from_slice(&[16, 1, 8, 0]); + d.extend_from_slice(&208u16.to_le_bytes()); + d.extend_from_slice(&[0, 1]); + + let m = MapTypeExtra::parse(&d, 8, MapLayout::HmapBools).unwrap(); + assert_eq!(m.group, 0x4b2320); + assert_eq!(m.hmap, None); + assert_eq!(m.key_size, Some(16)); + assert_eq!(m.value_size, Some(8)); + assert_eq!(m.bucket_size, Some(208)); + assert_eq!(m.flags.indirect_key, Some(true)); + assert_eq!(m.flags.need_key_update, Some(true)); + } + + #[test] + fn go112_flags_word_replaces_the_bools() { + let mut d = Vec::new(); + for w in [0x4a9ae0u64, 0x4a9ea0, 0x4b2320] { + d.extend_from_slice(&w.to_le_bytes()); + } + d.push(16); + d.push(8); + d.extend_from_slice(&208u16.to_le_bytes()); + d.extend_from_slice(&(1u32 | 16).to_le_bytes()); // indirectkey|hashMightPanic + + let m = MapTypeExtra::parse(&d, 8, MapLayout::HmapFlags).unwrap(); + assert_eq!(m.hasher, None, "hasher arrives in Go 1.14"); + assert_eq!(m.key_size, Some(16)); + assert_eq!(m.bucket_size, Some(208)); + assert_eq!(m.flags.indirect_key, Some(true)); + assert_eq!(m.flags.hash_might_panic, Some(true)); + assert_eq!(m.flags.reflexive_key, Some(false)); + } + + #[test] + fn split_group_layout_reads_every_stride() { + let mut d = Vec::new(); + for w in [ + 0x551000u64, // Key + 0x551100, // Elem + 0x551200, // Group + 0x551300, // Hasher + 0xc8, // GroupSize + 0x8, // KeysOff + 0x18, // KeyStride + 0x18, // ElemsOff + 0x18, // ElemStride + 0x10, // ElemOff + 0xc, // Flags = MapIndirectKey|MapIndirectElem + ] { + d.extend_from_slice(&w.to_le_bytes()); + } + let m = MapTypeExtra::parse(&d, 8, MapLayout::SwissSplitGroup).unwrap(); + assert_eq!(m.keys_off, Some(0x8)); + assert_eq!(m.key_stride(), Some(0x18)); + assert_eq!(m.elem_stride(), Some(0x18)); + assert_eq!(m.elem_off, Some(0x10)); + assert_eq!(m.flags.indirect_key, Some(true)); + assert_eq!(m.flags.indirect_elem, Some(true)); + assert_eq!(m.flags.need_key_update, Some(false)); + assert!(m.slot_size.is_none()); + } + + #[test] + fn too_short_returns_none() { + let d = vec![0u8; 16]; + assert!(MapTypeExtra::parse(&d, 8, MapLayout::Swiss).is_none()); + assert!(MapTypeExtra::parse(&d, 8, MapLayout::HmapHasher).is_none()); + assert!(MapTypeExtra::parse(&d, 8, MapLayout::HmapWithHmapType).is_none()); } } diff --git a/src/structures/mod.rs b/src/structures/mod.rs index c48e742..a073f1e 100644 --- a/src/structures/mod.rs +++ b/src/structures/mod.rs @@ -8,6 +8,10 @@ //! - [`buildinfo`] -- Version, module path, dependencies, and build settings //! - [`pclntab`] -- The PC/line table: function names, source files, line numbers //! +//! [`moduledata`] parses the linker-generated master record that ties the rest +//! together, and [`locate`] finds it — by section where the toolchain emits +//! one, and by scanning for its `pcHeader` pointer everywhere else. +//! //! ## Why These Structures Exist //! //! The Go runtime is more self-aware than a typical C runtime. It needs metadata for: @@ -40,6 +44,7 @@ pub mod inline; pub mod interfacetype; pub mod itab; pub mod kind; +pub mod locate; pub mod maptype; pub mod method; pub mod moduledata; diff --git a/src/structures/moduledata.rs b/src/structures/moduledata.rs index d974663..c14e0ae 100644 --- a/src/structures/moduledata.rs +++ b/src/structures/moduledata.rs @@ -14,7 +14,16 @@ //! | V4 | 1.26 | +epclntab | //! | V5 | 1.27+ | -typelinks, -itablinks, +typedesclen, +itaboffset | //! -//! Source: `src/runtime/symtab.go` (field offsets verified per release tag). +//! Source: `src/runtime/symtab.go` (field offsets verified per release tag, +//! most recently against `go1.27.1`). +//! +//! ## Picking a layout +//! +//! V5 moved `types`' neighbours, so a wrong guess shifts every field from +//! `etypes` onward while still passing a head-only validity check. The +//! out-of-band signals in [`LayoutHints`] are not sufficient on their own — PE +//! carries no `.typelink` section at *any* Go version — so [`Moduledata::parse`] +//! parses both candidates and keeps the one that is internally consistent. use crate::structures::{ PclntabVersion, @@ -263,20 +272,159 @@ pub enum ModuledataVersion { V5, } +/// Out-of-band signals used to pick the moduledata layout. +/// +/// Everything here is derived from the *containing binary* rather than from +/// the moduledata bytes themselves, so it has to be threaded in by the caller. +/// Grouping the signals keeps [`Moduledata::parse`] from growing a row of +/// positional booleans whose meaning is invisible at the call site. +#[derive(Debug, Clone, Copy)] +pub struct LayoutHints { + /// pclntab magic of the binary, which fixes the Go-version floor: `Go118` + /// implies 1.18+ (`rodata`/`gofunc`), `Go120` implies 1.20+ (`covctrs`). + pub pclntab_version: PclntabVersion, + /// Go minor version, when a version string was recovered. `None` for + /// binaries whose version was stripped or obfuscated away. + pub go_minor: Option, + /// Whether a `.typelink` / `__typelink` section was found. Its *absence* + /// is weak evidence of Go 1.27+ — PE, wasm, and RELRO ELF links spelled + /// `.data.rel.ro.typelink` have no such section at any version — so it is + /// never used alone. See [`Moduledata::parse`]. + pub has_typelink_section: bool, + /// Whether a `.go.type` / `__go_type` section was found. This one *is* a + /// positive Go 1.27+ signal: the section did not exist before. + pub has_go_type_section: bool, +} + +impl LayoutHints { + /// Whether the pclntab magic proves Go 1.18+, i.e. `rodata` and `gofunc` + /// are present in the moduledata. + fn has_rodata_gofunc(&self) -> bool { + matches!( + self.pclntab_version, + PclntabVersion::Go118 | PclntabVersion::Go120 + ) + } + + /// Whether the pclntab magic proves Go 1.20+, i.e. the `covctrs` / + /// `ecovctrs` pair is present (it sits *before* `types`, so its presence + /// shifts every later field). + fn has_covctrs(&self) -> bool { + matches!(self.pclntab_version, PclntabVersion::Go120) + } + + /// Whether the V5 (Go 1.27+) layout should be tried before the pre-V5 one. + /// + /// V5 needs the Go 1.20+ magic as a floor. Beyond that, a `.go.type` + /// section settles it outright; otherwise we fall back to the weak + /// "no typelink section" signal, which only counts when the recovered + /// version agrees (or there is no version to disagree). + fn prefer_v5(&self) -> bool { + self.has_covctrs() + && (self.has_go_type_section + || (!self.has_typelink_section && self.go_minor.is_none_or(|m| m >= 27))) + } +} + impl Moduledata { /// Parse a moduledata from raw bytes with version detection. /// /// Field presence is determined per-field: /// 1. pclntab magic Go120 -> has `covctrs`/`rodata`/`gofunc` (Go 1.20+) /// 2. Go minor version -> `inittasks` (1.21+), `epclntab` (1.26+) - /// 3. absent `.typelink` section (+ minor >= 27 when known) -> V5 layout - pub fn parse( - data: &[u8], - ps: u8, - pclntab_version: PclntabVersion, - has_typelink_section: bool, - go_version_minor: Option, - ) -> Option { + /// 3. V5 (Go 1.27+) -> see below + /// + /// # Choosing between the V5 and pre-V5 layouts + /// + /// V5 moved `types`' neighbours around (`+typedesclen`, `+itaboffset`, + /// `+itabsize`, `-typelinks`, `-itablinks`), so guessing wrong shifts every + /// field from `etypes` onward and yields a moduledata that still passes a + /// head-only validity check — `minpc`/`maxpc`/`funcnametab` are ahead of + /// the divergence — while silently reporting an empty types region, no + /// itabs, no init tasks and `has_main == false`. + /// + /// The hints alone cannot settle it: PE never emits a `.typelink` section + /// at any Go version, so on a PE binary whose version string was scrubbed + /// the only remaining signal points the wrong way. Instead of trusting the + /// hints, this parses *both* candidate layouts (hint-preferred first) and + /// returns the first one that is internally self-consistent — see + /// [`Moduledata::layout_self_consistent`]. Only if neither validates does + /// the hint-preferred layout win, so callers still get the head fields. + pub fn parse(data: &[u8], ps: u8, hints: LayoutHints) -> Option { + // The Go 1.5-1.15 moduledata has an entirely different head and is + // parsed separately; it has no V5/pre-V5 ambiguity. + if hints.pclntab_version == PclntabVersion::Go12 { + return Self::parse_go12_legacy(data, ps, hints.go_minor); + } + + let prefer_v5 = hints.prefer_v5(); + for v5 in [prefer_v5, !prefer_v5] { + // V5 requires the Go 1.20+ magic; never consider it below that. + if v5 && !hints.has_covctrs() { + continue; + } + if let Some(md) = Self::parse_modern(data, ps, hints, v5) + && md.layout_self_consistent() + { + return Some(md); + } + } + Self::parse_modern(data, ps, hints, prefer_v5) + } + + /// Whether the parsed layout is internally consistent — the arbiter used by + /// [`Self::parse`] to decide whether it guessed V5 correctly. + /// + /// Checks only relationships that hold in *every* real Go image, so a + /// correctly-guessed layout always passes and a mis-guessed one reliably + /// fails: + /// + /// - `etypes >= types`. Reading a V5 moduledata with the pre-V5 layout puts + /// `typedesclen` (a small length) where `etypes` belongs, so `etypes` + /// lands far below `types`. + /// - Every V5 sub-region fits inside `[types, etypes)` and the descriptor + /// region precedes the itab region. Reading a pre-V5 moduledata as V5 + /// picks up unrelated pointers as `typedesclen`/`itaboffset`, which then + /// dwarf the (collapsed) types span. + /// - `rodata`, `gofunc` and `epclntab` are image addresses at or above + /// `text`, never the small integers a misaligned read produces. + pub fn layout_self_consistent(&self) -> bool { + let Some(span) = self.etypes.checked_sub(self.types) else { + return false; + }; + if self.version == ModuledataVersion::V5 { + let (Some(typedesclen), Some(itaboffset), Some(itabsize)) = + (self.typedesclen, self.itaboffset, self.itabsize) + else { + return false; + }; + let Some(itab_end) = itaboffset.checked_add(itabsize) else { + return false; + }; + // Layout of `.go.type` (cmd/link/internal/ld/data.go, dodataSect + // STYPE case): ptrSize skip, `type:*`, typelink descriptors up to + // `typedesclen`, non-typelink descriptors, then itabs. + if typedesclen > span || itaboffset > span || itab_end > span { + return false; + } + if typedesclen > itaboffset { + return false; + } + } + // A misaligned read lands on lengths, flags and slice fields, which + // read as small integers rather than addresses inside the image. + [self.rodata, self.gofunc, self.epclntab] + .into_iter() + .flatten() + .all(|va| va >= self.text) + } + + /// Parse a Go 1.16+ moduledata with the layout fixed by `hints` and the + /// explicit `v5` choice. Shared by both arms of [`Self::parse`]'s + /// arbitration, which is why the V5 decision is a parameter rather than + /// something re-derived here. + fn parse_modern(data: &[u8], ps: u8, hints: LayoutHints, v5: bool) -> Option { + let go_version_minor = hints.go_minor; let p = ps as usize; let slice_sz = GoSlice::size(ps); // A Go `string` header is (ptr, len) = 2 pointers. @@ -306,9 +454,6 @@ impl Moduledata { let maxpc = read_uintptr(data, off, ps)?; off = advance(off, p)?; - // The Go 1.5-1.15 moduledata has an entirely different head (no - // `pcHeader` pointer; `pclntable []byte` first) and is parsed - // separately. // The middle/tail layout is determined per-field from the pclntab // magic and the Go minor version, verified against runtime/symtab.go // across releases: @@ -317,11 +462,7 @@ impl Moduledata { // - inittasks : Go 1.21+ // - epclntab : Go 1.26+ (absent in 1.24 / 1.25!) // - V5 (typedesclen + itaboffset/itabsize, no typelinks/itablinks): - // Go 1.27+ - if pclntab_version == PclntabVersion::Go12 { - return Self::parse_go12_legacy(data, ps, go_version_minor); - } - + // Go 1.27+, chosen by the caller let text = read_uintptr(data, off, ps)?; off = advance(off, p)?; let etext = read_uintptr(data, off, ps)?; @@ -343,14 +484,8 @@ impl Moduledata { // `covctrs` sits *before* `types`, so its presence shifts every later // field; `rodata`/`gofunc` sit *after* `etypes`. They were added in // different releases (1.20 vs 1.18), so they must be gated separately. - let has_covctrs = matches!(pclntab_version, PclntabVersion::Go120); - let has_rodata_gofunc = matches!( - pclntab_version, - PclntabVersion::Go118 | PclntabVersion::Go120 - ); - // V5 dropped the `.typelink` section, so its absence is a reliable - // structural signal; the version string disambiguates when present. - let v5 = has_covctrs && !has_typelink_section && go_version_minor.is_none_or(|m| m >= 27); + let has_covctrs = hints.has_covctrs(); + let has_rodata_gofunc = hints.has_rodata_gofunc(); let has_inittasks = v5 || go_version_minor.is_none_or(|m| m >= 21); // covctrs, ecovctrs (Go 1.20+) @@ -817,13 +952,93 @@ fn looks_like_slice_header(data: &[u8], off: usize, ps: u8) -> bool { mod tests { use super::*; + /// Hints for a binary with neither a `.typelink` nor a `.go.type` section + /// — the shape a PE, a wasm module, or a stripped RELRO ELF presents. + fn hints(pclntab_version: PclntabVersion, go_minor: Option) -> LayoutHints { + LayoutHints { + pclntab_version, + go_minor, + has_typelink_section: false, + has_go_type_section: false, + } + } + + /// Write a little-endian `u64` at `off`. + fn put(d: &mut [u8], off: usize, v: u64) { + d[off..off + 8].copy_from_slice(&v.to_le_bytes()); + } + + /// Field offsets of the 64-bit V5 (Go 1.27+) moduledata, counted from the + /// `pcHeader` pointer at 0. Derived by walking `parse_modern`'s reads. + mod v5_off { + pub const TEXT: usize = 176; + pub const ETEXT: usize = 184; + pub const TYPES: usize = 296; + pub const TYPEDESCLEN: usize = 304; + pub const ETYPES: usize = 312; + pub const ITABOFFSET: usize = 320; + pub const ITABSIZE: usize = 328; + pub const RODATA: usize = 336; + pub const GOFUNC: usize = 344; + pub const EPCLNTAB: usize = 352; + } + + /// Field offsets of the 64-bit V3 (Go 1.20-1.25) moduledata. Identical to + /// V5 up to `types`, then diverges: no `typedesclen`, no itab fields. + mod v3_off { + pub const TEXT: usize = 176; + pub const ETEXT: usize = 184; + pub const TYPES: usize = 296; + pub const ETYPES: usize = 304; + pub const RODATA: usize = 312; + pub const GOFUNC: usize = 320; + pub const TEXTSECTMAP_PTR: usize = 328; + pub const TEXTSECTMAP_LEN: usize = 336; + pub const TEXTSECTMAP_CAP: usize = 344; + } + + /// A structurally plausible 64-bit V5 moduledata: every sub-region of + /// `[types, etypes)` is in range and the segment pointers are real + /// addresses, so [`Moduledata::layout_self_consistent`] accepts it. + fn synthetic_v5() -> Vec { + let mut d = vec![0u8; 700]; + put(&mut d, v5_off::TEXT, 0x401000); + put(&mut d, v5_off::ETEXT, 0x4a0000); + put(&mut d, v5_off::TYPES, 0x500000); + put(&mut d, v5_off::TYPEDESCLEN, 0x1000); + put(&mut d, v5_off::ETYPES, 0x520000); + put(&mut d, v5_off::ITABOFFSET, 0x1f000); + put(&mut d, v5_off::ITABSIZE, 0x400); + put(&mut d, v5_off::RODATA, 0x490000); + put(&mut d, v5_off::GOFUNC, 0x4f0000); + put(&mut d, v5_off::EPCLNTAB, 0x4ffff0); + d + } + + /// A structurally plausible 64-bit V3 moduledata. The `textsectmap` slice + /// header at the position `epclntab` would occupy keeps the Go 1.26 probe + /// on the V3 side when the minor version is unknown. + fn synthetic_v3() -> Vec { + let mut d = vec![0u8; 700]; + put(&mut d, v3_off::TEXT, 0x401000); + put(&mut d, v3_off::ETEXT, 0x4a0000); + put(&mut d, v3_off::TYPES, 0x4a2000); + put(&mut d, v3_off::ETYPES, 0x4d9000); + put(&mut d, v3_off::RODATA, 0x4a2000); + put(&mut d, v3_off::GOFUNC, 0x55c000); + put(&mut d, v3_off::TEXTSECTMAP_PTR, 0x400000); + put(&mut d, v3_off::TEXTSECTMAP_LEN, 1); + put(&mut d, v3_off::TEXTSECTMAP_CAP, 1); + d + } + #[test] fn version_detection_go116() { // V2 moduledata needs enough space for the full prefix + version-specific section // Prefix: 1 ptr + 6 slices + 4 ptrs + 8 skipped ptrs = 1*8 + 6*24 + 4*8 + 8*8 = 248 // V2 tail: 3 ptrs + 2 ptrs + 1 slice + 2 slices = 3*8 + 2*8 + 24 + 2*24 = 112 let data = vec![0u8; 400]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go116, false, None); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go116, None)); assert!(md.is_some()); assert_eq!(md.unwrap().version, ModuledataVersion::V2); } @@ -833,7 +1048,7 @@ mod tests { // Go 1.18-1.19 (Go118 magic): rodata/gofunc present (V3), but no // covctrs. Distinct from the V2 (Go116) layout. let data = vec![0u8; 500]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go118, false, Some(19)) + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go118, Some(19))) .expect("Go118 moduledata should parse"); assert_eq!(md.version, ModuledataVersion::V3); assert!(md.rodata.is_some(), "Go 1.18+ has rodata"); @@ -843,7 +1058,7 @@ mod tests { #[test] fn version_detection_go116_is_v2_no_rodata() { let data = vec![0u8; 500]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go116, false, Some(16)) + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go116, Some(16))) .expect("Go116 moduledata should parse"); assert_eq!(md.version, ModuledataVersion::V2); assert!(md.rodata.is_none(), "Go 1.16-1.17 has no rodata"); @@ -852,7 +1067,7 @@ mod tests { #[test] fn version_detection_go120_minor_22() { let data = vec![0u8; 500]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go120, false, Some(22)); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, Some(22))); assert!(md.is_some()); assert_eq!(md.unwrap().version, ModuledataVersion::V3); } @@ -861,7 +1076,7 @@ mod tests { fn version_detection_go120_minor_25_is_v3_no_epclntab() { // Go 1.24 / 1.25 have NO epclntab field -> V3 layout, not V4. let data = vec![0u8; 500]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go120, false, Some(25)); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, Some(25))); assert!(md.is_some()); assert_eq!(md.unwrap().version, ModuledataVersion::V3); } @@ -870,7 +1085,7 @@ mod tests { fn version_detection_go120_minor_26_is_v4() { // epclntab was added in Go 1.26 -> V4. let data = vec![0u8; 500]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go120, false, Some(26)); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, Some(26))); assert!(md.is_some()); assert_eq!(md.unwrap().version, ModuledataVersion::V4); } @@ -878,43 +1093,131 @@ mod tests { #[test] fn version_detection_go120_minor_27_v5() { // minor > 26 and no typelink section -> V5 (Go 1.27+). - let mut data = vec![0u8; 600]; - // itaboffset lands at byte 320 in the 64-bit V5 walk (see the V5 - // branch comment for the field sequence). Plant a recognizable - // value to prove the new fields are actually read. - data[320..328].copy_from_slice(&0xdead_beefu64.to_le_bytes()); - data[328..336].copy_from_slice(&0x40u64.to_le_bytes()); - let md = Moduledata::parse(&data, 8, PclntabVersion::Go120, false, Some(27)) + let data = synthetic_v5(); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, Some(27))) .expect("V5 moduledata should parse"); assert_eq!(md.version, ModuledataVersion::V5); - assert_eq!(md.itaboffset, Some(0xdead_beef)); - assert_eq!(md.itabsize, Some(0x40)); + assert_eq!(md.typedesclen, Some(0x1000)); + assert_eq!(md.itaboffset, Some(0x1f000)); + assert_eq!(md.itabsize, Some(0x400)); assert!(md.typelinks.is_none()); assert!(md.itablinks.is_none()); // rodata/gofunc are still present in V5 (regression guard against // the old speculative branch that hardcoded them to None). - assert!(md.rodata.is_some()); - assert!(md.gofunc.is_some()); + assert_eq!(md.rodata, Some(0x490000)); + assert_eq!(md.gofunc, Some(0x4f0000)); + assert_eq!(md.epclntab, Some(0x4ffff0)); } #[test] fn v4_falls_back_to_typelinks_not_v5() { // minor 27 but a typelink section is present -> stay on V4 layout. let data = vec![0u8; 600]; - let md = Moduledata::parse(&data, 8, PclntabVersion::Go120, true, Some(27)) - .expect("should parse"); + let md = Moduledata::parse( + &data, + 8, + LayoutHints { + pclntab_version: PclntabVersion::Go120, + go_minor: Some(27), + has_typelink_section: true, + has_go_type_section: false, + }, + ) + .expect("should parse"); assert_eq!(md.version, ModuledataVersion::V4); } + #[test] + fn pre_v5_layout_survives_a_missing_version_string() { + // Regression guard: PE binaries never carry a `.typelink` section, so + // for a version-scrubbed PE the only hint points at V5. Reading a + // pre-V5 moduledata with the V5 layout collapses the types span while + // leaving a huge `typedesclen`, which the consistency check rejects — + // so the parser must fall back to V3 rather than return the garbage. + let data = synthetic_v3(); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, None)) + .expect("V3 moduledata should parse"); + assert_eq!(md.version, ModuledataVersion::V3); + assert_eq!(md.types, 0x4a2000); + assert_eq!(md.etypes, 0x4d9000); + assert_eq!(md.rodata, Some(0x4a2000)); + assert_eq!(md.gofunc, Some(0x55c000)); + assert!(md.typedesclen.is_none()); + assert!(md.itaboffset.is_none()); + } + + #[test] + fn v5_layout_wins_without_a_version_string() { + // The mirror case: a genuine V5 moduledata read as V3/V4 puts + // `typedesclen` where `etypes` belongs, leaving `etypes < types`. The + // consistency check rejects that, so V5 is chosen even though no + // version string is available. + let data = synthetic_v5(); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, None)) + .expect("V5 moduledata should parse"); + assert_eq!(md.version, ModuledataVersion::V5); + assert_eq!(md.itaboffset, Some(0x1f000)); + } + + #[test] + fn go_type_section_selects_v5_despite_a_stale_typelink_section() { + // `.go.type` only exists from Go 1.27, so it outranks the weak + // "a typelink section is present" signal. + let data = synthetic_v5(); + let md = Moduledata::parse( + &data, + 8, + LayoutHints { + pclntab_version: PclntabVersion::Go120, + go_minor: None, + has_typelink_section: true, + has_go_type_section: true, + }, + ) + .expect("V5 moduledata should parse"); + assert_eq!(md.version, ModuledataVersion::V5); + } + + #[test] + fn v5_is_never_chosen_below_the_go120_magic() { + // V5 requires covctrs, which the Go118 magic rules out — so even with + // every V5 hint set the layout must stay pre-V5. + let data = synthetic_v5(); + let md = Moduledata::parse( + &data, + 8, + LayoutHints { + pclntab_version: PclntabVersion::Go118, + go_minor: None, + has_typelink_section: false, + has_go_type_section: true, + }, + ) + .expect("should parse"); + assert_ne!(md.version, ModuledataVersion::V5); + } + + #[test] + fn inconsistent_layouts_fall_back_to_the_preferred_hint() { + // When neither candidate validates (truncated / garbage tail), the + // hint-preferred layout is returned so callers still get the head + // fields rather than nothing at all. + let mut data = synthetic_v5(); + // Break both layouts: types above etypes with a nonsensical span. + put(&mut data, v5_off::TYPES, 0x900000); + put(&mut data, v5_off::ETYPES, 0x100000); + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, Some(27))) + .expect("should still parse"); + assert_eq!(md.version, ModuledataVersion::V5); + assert_eq!(md.minpc, 0); + } + #[test] fn go12_legacy_parses_v1() { // Minimal legacy (Go 1.5-1.15) moduledata head: pclntable slice, then // ftab/filetab/findfunctab/minpc/maxpc, text/etext, four data ranges, // end/gcdata/gcbss, types/etypes. The tail is read best-effort. let mut data = vec![0u8; 512]; - let put = |d: &mut [u8], off: usize, v: u64| { - d[off..off + 8].copy_from_slice(&v.to_le_bytes()); - }; // pclntable slice @0 (ptr, len, cap). put(&mut data, 0, 0x5000); put(&mut data, 8, 0x100); @@ -926,7 +1229,7 @@ mod tests { put(&mut data, 200, 0x9000); // types put(&mut data, 208, 0xa000); // etypes - let md = Moduledata::parse(&data, 8, PclntabVersion::Go12, false, Some(10)) + let md = Moduledata::parse(&data, 8, hints(PclntabVersion::Go12, Some(10))) .expect("legacy moduledata should parse"); assert_eq!(md.version, ModuledataVersion::V1); assert_eq!(md.pclntable.ptr, 0x5000); @@ -945,6 +1248,6 @@ mod tests { #[test] fn too_short_returns_none() { let data = vec![0u8; 10]; - assert!(Moduledata::parse(&data, 8, PclntabVersion::Go120, false, Some(25)).is_none()); + assert!(Moduledata::parse(&data, 8, hints(PclntabVersion::Go120, Some(25))).is_none()); } } diff --git a/src/structures/pclntab.rs b/src/structures/pclntab.rs index df00eb0..9d42191 100644 --- a/src/structures/pclntab.rs +++ b/src/structures/pclntab.rs @@ -42,7 +42,8 @@ //! 7 1 ptrSize Pointer size (4/8) //! 8 ptrSize nfunc Number of functions //! 8+1*ps ptrSize nfiles Number of source files -//! 8+2*ps ptrSize (unused) Formerly textStart (Go 1.18+) +//! 8+2*ps ptrSize textStart `runtime.text` VA; Go 1.18-1.25 only +//! (Go 1.26+ leaves the slot zeroed) //! 8+3*ps ptrSize funcnameOffset Offset to funcnametab //! 8+4*ps ptrSize cuOffset Offset to cutab //! 8+5*ps ptrSize filetabOffset Offset to filetab @@ -179,8 +180,15 @@ pub struct ParsedPclntab<'a> { /// it so `FuncData::entry_off` is a relative offset for every version. pub text_start: u64, /// The pcHeader's `textStart` field (the `runtime.text` VA), added to the - /// header in Go 1.18. `None` for Go 1.16-1.17, which lack the field. - /// Mirrors `moduledata.text`. + /// header in Go 1.18 and **retired in Go 1.26**, which left the slot in + /// place but stopped writing it because storing it required a relocation + /// (`runtime/symtab.go`: *"The next field used to be textStart. This is no + /// longer stored… Code should use the moduledata text field instead."*). + /// + /// `None` for Go 1.16-1.17, which lack the field, and for Go 1.26+, where + /// the slot reads as zero — a zero is reported as absent rather than as + /// address `0`, so callers do not silently rebase every function entry + /// against the bottom of the address space. Mirrors `moduledata.text`. pub header_text_start: Option, } @@ -1582,10 +1590,13 @@ fn parse_header( }; // The pcHeader gained a `textStart` field at index 2 in Go 1.18; Go - // 1.16-1.17 (off_base == 2) do not have it. + // 1.16-1.17 (off_base == 2) do not have it, and Go 1.26+ zeroed it out + // (see `ParsedPclntab::header_text_start`). `runtime.text` is never 0 in a + // real image — even wasm, whose PCs start at 0, carries the value in the + // moduledata rather than here — so a zero means "not recorded". let header_text_start = if off_base > 2 { let off = advance_n(8, 2, ps)?; - read_uintptr(data, off, ptr_size) + read_uintptr(data, off, ptr_size).filter(|&va| va != 0) } else { None }; @@ -1940,7 +1951,12 @@ mod tests { go_module: None, typelink: None, itablink: None, + go_type: None, + go_func: None, fipsinfo: None, + noptrdata: None, + data_section: None, + text_section: None, }; let parsed = scan_relaxed(&data, §ions).unwrap(); diff --git a/src/structures/types.rs b/src/structures/types.rs index aa6b3cc..4fa938d 100644 --- a/src/structures/types.rs +++ b/src/structures/types.rs @@ -9,9 +9,15 @@ //! 1. **Typelink path** (ELF `.typelink`, Mach-O `__typelink`): An array of `int32` //! offsets from `moduledata.types`. Each offset points to an `abi.Type`. //! -//! 2. **Descriptor-walking path** (PE, or future Go without typelinks): Walk from -//! `moduledata.types + PtrSize` to `moduledata.etypes`, advancing by each type's -//! `DescriptorSize`. Same algorithm as the Go runtime's `moduleTypelinks()`. +//! 2. **Descriptor-walking path** (Go 1.27+, which has no typelink table): Walk +//! from `moduledata.types + PtrSize` to `moduledata.types + typedesclen`, +//! advancing by each type's `DescriptorSize` with pointer alignment — the +//! same algorithm as the Go runtime's `moduleTypelinks()`. Nothing past +//! `typedesclen` is walkable: the linker groups every `type:`-prefixed +//! read-only symbol under one carrier and sorts the non-typelink remainder +//! by size, so that region interleaves descriptors with +//! `type:.namedata.*` blobs and ends in the inline itab array. Those +//! descriptors are reachable only through [`extract_all_types`]. //! //! 3. **PE moduledata discovery**: PE binaries lack Go-specific section names. //! We find moduledata by scanning `.data` for a pointer matching the pclntab VA @@ -20,16 +26,17 @@ //! ## Source References //! //! - Type descriptors: `src/internal/abi/type.go` -//! - Type walking: `src/runtime/type.go:522-545` (`moduleTypelinks`) -//! - Moduledata: `src/runtime/symtab.go:402-450` +//! - Type walking: `src/runtime/type.go` (`moduleTypelinks`) +//! - Type-section layout: `src/cmd/link/internal/ld/data.go` (`dodataSect`, +//! `sym.STYPE` case) — which also records `typedesclen` and `itaboffset` +//! - Moduledata: `src/runtime/symtab.go` use std::collections::{HashSet, VecDeque}; use crate::{ - formats::{BinaryContext, BinaryFormat}, + formats::BinaryContext, metadata::{is_internal_path, is_runtime_path, is_stdlib_path}, structures::{ - PclntabVersion, abitype::AbiType, arraytype::ArrayTypeExtra, chantype::ChanTypeExtra, @@ -37,10 +44,9 @@ use crate::{ elemtype::ElemTypeExtra, functype::FuncTypeExtra, interfacetype::InterfaceTypeExtra, - maptype::MapTypeExtra, method::GoImethod, method::GoMethod, - moduledata::{Moduledata, ModuledataVersion}, + moduledata::Moduledata, name::{ NAME_FLAG_EMBEDDED, NAME_FLAG_EXPORTED, decode_name, decode_name_and_tag, decode_name_with_flags, @@ -51,6 +57,26 @@ use crate::{ }, }; +/// Re-exported so callers reading [`TypeDetail::Map`] do not have to reach +/// into `structures::maptype` for the layout tag and raw descriptor fields. +pub use crate::structures::maptype::{MapFlags, MapLayout, MapTypeExtra}; + +/// The per-binary constants every type-descriptor read depends on. +/// +/// All three are fixed for a whole binary but vary between binaries, and all +/// three change how the *same* bytes decode, so they travel together rather +/// than as a row of positional parameters. +#[derive(Debug, Clone, Copy)] +pub struct TypeAbi { + /// Pointer size in bytes (4 or 8). + pub ps: u8, + /// Go ≤ 1.16 encodes type-name lengths as a 2-byte big-endian `uint16`; + /// 1.17+ uses a varint. See [`crate::structures::name::decode_name`]. + pub legacy_names: bool, + /// Which `abi.MapType` shape this binary's map descriptors use. + pub map_layout: MapLayout, +} + /// A type extracted deterministically from Go type descriptors. /// /// All string fields borrow from the underlying binary data via the lifetime @@ -254,17 +280,20 @@ pub enum TypeDetail<'a> { key_va: u64, /// Virtual address of the element (value) type descriptor. elem_va: u64, - /// Virtual address of the internal bucket/group type descriptor. + /// Virtual address of the internal bucket (Go ≤ 1.23) or slot-group + /// (Go 1.24+) type descriptor. group_va: u64, - /// Virtual address of the key-hashing function. - hasher_va: u64, - /// Byte stride between keys in a bucket/group. - key_stride: u64, - /// Byte stride between elements in a bucket/group. - elem_stride: u64, - /// Map implementation flags (`abi.MapType.Flags` / - /// `maptype.flags` — indirect-key/elem, reflexive-key, etc.). - flags: u32, + /// Every remaining descriptor field, tagged with the [`MapLayout`] it + /// was read under: the hasher, the per-version size/stride fields, and + /// the normalized [`MapFlags`]. `abi.MapType` has had six shapes, so + /// only the three addresses above are common to all of them; reach for + /// [`MapTypeExtra::key_stride`] / [`MapTypeExtra::elem_stride`] for + /// layout-independent strides. + /// + /// Boxed because the full descriptor is far larger than any other + /// variant's payload, and map types are a small minority of any + /// binary's type set. + extra: Box, }, /// Pointer type: `*T`. Pointer { @@ -549,29 +578,43 @@ pub struct TypeIter<'a> { ctx: &'a BinaryContext<'a>, data: &'a [u8], types_base_va: u64, - ps: u8, - /// Pre-1.17 (`Go116`) name length encoding (2-byte big-endian vs varint). - legacy: bool, + abi: TypeAbi, strategy: TypeIterStrategy<'a>, } enum TypeIterStrategy<'a> { /// Iterate `int32` offsets from a typelink array. Typelinks { tl_data: &'a [u8], pos: usize }, - /// Walk `[types_base_va + ps .. etypes_va]` advancing by `DescriptorSize`. - Walk { td: u64, etypes_va: u64 }, + /// Walk a contiguous descriptor region `[td, end_va)`, advancing by + /// `DescriptorSize` with pointer alignment — the same stepper + /// `runtime.moduleTypelinks` uses on Go 1.27+. + Walk { + /// VA of the next descriptor to parse. + td: u64, + /// One past the last byte of the region. + end_va: u64, + /// Remaining tolerance for unparseable records before the walk gives + /// up. Without a budget one bad byte would either truncate the whole + /// enumeration or spin over a large region one word at a time. + skips_left: u32, + }, /// No types reachable. Empty, } impl<'a> TypeIter<'a> { - fn empty(ctx: &'a BinaryContext<'a>) -> Self { + /// An iterator that yields nothing — the result when a binary has no + /// moduledata, no VA mapping, or no types region. + pub fn empty(ctx: &'a BinaryContext<'a>) -> Self { Self { ctx, data: ctx.structure_search_data(), types_base_va: 0, - ps: 0, - legacy: false, + abi: TypeAbi { + ps: 0, + legacy_names: false, + map_layout: MapLayout::SwissSplitGroup, + }, strategy: TypeIterStrategy::Empty, } } @@ -599,8 +642,7 @@ impl<'a> Iterator for TypeIter<'a> { file_off, type_va, self.types_base_va, - self.ps, - self.legacy, + self.abi, self.ctx, ) { @@ -610,38 +652,64 @@ impl<'a> Iterator for TypeIter<'a> { } None } - TypeIterStrategy::Walk { td, etypes_va } => { - let p = self.ps as u64; + TypeIterStrategy::Walk { + td, + end_va, + skips_left, + } => { + let p = self.abi.ps as u64; if p == 0 { return None; } - while *td < *etypes_va { + while *td < *end_va { *td = align_up_u64(*td, p)?; - if *td >= *etypes_va { + if *td >= *end_va { return None; } - let file_off = self.ctx.va_to_file(*td)?; + let here = *td; + // Step over an unparseable record rather than ending the + // walk. On Go 1.27+ this is the only type-enumeration + // strategy there is, so aborting on the first bad byte + // would silently truncate a whole binary's type list. + let mut skip = || -> Option<()> { + *skips_left = skips_left.checked_sub(1)?; + *td = here.checked_add(p)?; + Some(()) + }; + let Some(file_off) = self.ctx.va_to_file(here) else { + skip()?; + continue; + }; let remaining = match self.data.get(file_off..) { - Some(d) if d.len() >= AbiType::size(self.ps) => d, + Some(d) if d.len() >= AbiType::size(self.abi.ps) => d, + // Past the end of the mapped image: nothing follows. _ => return None, }; - let abi_type = AbiType::parse(remaining, self.ps)?; - let desc_size = match descriptor::descriptor_size(remaining, &abi_type, self.ps) - { + let (Some(abi_type), _) = (AbiType::parse(remaining, self.abi.ps), ()) else { + skip()?; + continue; + }; + let desc_size = match descriptor::descriptor_size( + remaining, + &abi_type, + self.abi.ps, + self.abi.map_layout, + ) { Some(s) if s > 0 => s, - _ => return None, + _ => { + skip()?; + continue; + } }; let go_type = build_go_type( &abi_type, remaining, self.data, self.types_base_va, - self.ps, - self.legacy, + self.abi, self.ctx, ); - let here = *td; - *td = td.checked_add(desc_size as u64)?; + *td = here.checked_add(desc_size as u64)?; if let Some(mut t) = go_type { t.descriptor_va = here; return Some(t); @@ -654,67 +722,52 @@ impl<'a> Iterator for TypeIter<'a> { } } -/// Construct a streaming type iterator. The constructor performs moduledata -/// discovery up front (cheap on ELF / Mach-O, scan-based on PE) so each -/// [`Iterator::next`] call does only the per-type work. +/// Construct a streaming iterator over the binary's **reflection-visible** +/// type descriptors — the set the `typelink` table used to name. +/// +/// The constructor performs moduledata discovery up front (cheap on ELF / +/// Mach-O, scan-based on PE) so each [`Iterator::next`] call does only the +/// per-type work. /// /// Strategy selection: -/// 1. Dedicated `.typelink` / `__typelink` section if present. -/// 2. `moduledata.typelinks` slice (PE / older Go). -/// 3. Descriptor walk over `[types .. etypes]`. +/// 1. Dedicated `.typelink` / `__typelink` section if present (Go ≤ 1.26). +/// 2. `moduledata.typelinks` slice (PE / wasm / RELRO ELF, Go ≤ 1.26). +/// 3. Descriptor walk. Go 1.27 removed both tables and instead sorts the +/// typelink descriptors to the front of the types region, recording their +/// total length in `moduledata.typedesclen`; walking +/// `[types + ptrSize, types + typedesclen)` reproduces the old table +/// exactly, and is what `runtime.moduleTypelinks` itself does. Pre-1.27 +/// binaries that reach this branch have no such bound and fall back to +/// `etypes`. /// 4. Empty iterator if none of the above is available. +/// +/// The descriptors Go 1.27 keeps *after* `typedesclen` are not separately +/// enumerable: the linker groups every `type:`-prefixed read-only symbol under +/// one carrier and sorts the non-typelink remainder by size, so that region +/// interleaves descriptors with `type:.namedata.*` blobs and no boundary +/// between them is recorded. Those types are reached through +/// [`extract_all_types`] instead. pub fn extract_types_iter<'a>( ctx: &'a BinaryContext<'a>, - ptr_size: u8, - pclntab_version: Option, - pclntab_offset: Option, - go_version_minor: Option, + md: &Moduledata, + abi: TypeAbi, ) -> TypeIter<'a> { - if !ctx.has_va_mapping() { + if !ctx.has_va_mapping() || abi.ps == 0 || md.types == 0 { return TypeIter::empty(ctx); } - - // Read all runtime structures through the address-space view: file - // bytes for ELF/Mach-O/PE, the reconstructed linear-memory image for - // wasm. Wasm pclntab/moduledata/types live in linear memory and span - // multiple disjoint data segments, so accessing them via file offsets - // alone would split structures at segment boundaries. + // Read runtime structures through the address-space view: file bytes for + // ELF/Mach-O/PE, the reconstructed linear-memory image for wasm. Wasm type + // descriptors span multiple disjoint data segments, so addressing them by + // file offset alone would split structures at segment boundaries. let data = ctx.structure_search_data(); let sections = ctx.sections(); - let pv = pclntab_version.unwrap_or(PclntabVersion::Go120); - let has_typelink = sections.typelink.is_some(); - // Go ≤1.16 encodes type-name lengths as a 2-byte big-endian uint16; 1.17+ - // uses a varint. See `name::decode_name`. - let legacy = matches!(pv, PclntabVersion::Go116 | PclntabVersion::Go12); - - // Find moduledata: the dedicated `.go.module` section (Go 1.26+), else a - // pointer-scan for its pcHeader pointer. The section is absent on PE, on - // wasm, and on ELF / Mach-O before Go 1.26, so the scan is the fallback for - // every non-section case. - let moduledata = if let Some(ref range) = sections.go_module { - let end = match range.offset.checked_add(range.size) { - Some(e) => e, - None => return TypeIter::empty(ctx), - }; - let md_data = match data.get(range.offset..end) { - Some(s) => s, - None => return TypeIter::empty(ctx), - }; - Moduledata::parse(md_data, ptr_size, pv, has_typelink, go_version_minor) - } else { - discover_moduledata_via_pcheader( - ctx, - ptr_size, - pv, - has_typelink, - go_version_minor, - pclntab_offset, - ) - }; - let md = match moduledata { - Some(m) if m.types != 0 => m, - _ => return TypeIter::empty(ctx), + let iter = |strategy| TypeIter { + ctx, + data, + types_base_va: md.types, + abi, + strategy, }; // Strategy 1: dedicated typelink section. @@ -722,14 +775,7 @@ pub fn extract_types_iter<'a>( && let Some(end) = range.offset.checked_add(range.size) && let Some(tl_data) = data.get(range.offset..end) { - return TypeIter { - ctx, - data, - types_base_va: md.types, - ps: ptr_size, - legacy, - strategy: TypeIterStrategy::Typelinks { tl_data, pos: 0 }, - }; + return iter(TypeIterStrategy::Typelinks { tl_data, pos: 0 }); } // Strategy 1b: typelinks slice from moduledata (Go 1.16-1.26 PE). @@ -739,108 +785,55 @@ pub fn extract_types_iter<'a>( && let Some(tl_end) = tl_file_off.checked_add(tl_byte_len) && let Some(tl_data) = data.get(tl_file_off..tl_end) { - return TypeIter { - ctx, - data, - types_base_va: md.types, - ps: ptr_size, - legacy, - strategy: TypeIterStrategy::Typelinks { tl_data, pos: 0 }, - }; + return iter(TypeIterStrategy::Typelinks { tl_data, pos: 0 }); } // Strategy 2: walk the type descriptor region. - if md.etypes > md.types { - let td = md.types.saturating_add(ptr_size as u64); // skip ptrSize header - return TypeIter { - ctx, - data, - types_base_va: md.types, - ps: ptr_size, - legacy, - strategy: TypeIterStrategy::Walk { - td, - etypes_va: md.etypes, - }, - }; + match typelink_walk_range(md, abi.ps) { + Some((td, end_va)) => iter(TypeIterStrategy::Walk { + td, + end_va, + skips_left: WALK_SKIP_BUDGET, + }), + None => TypeIter::empty(ctx), } - - TypeIter::empty(ctx) } -/// Find moduledata by scanning the address-space view for a pointer-aligned -/// value equal to the pclntab's VA. -/// -/// The first field of moduledata is `pcHeader *pcHeader`, which points to the -/// pclntab. For PE we scan `data` (file bytes) for that pointer at pointer- -/// aligned positions; for wasm we scan the reconstructed linear-memory image -/// the same way (offsets in the image are linear-memory addresses, exactly -/// what the runtime stores). -fn discover_moduledata_via_pcheader( - ctx: &BinaryContext<'_>, - ps: u8, - pv: PclntabVersion, - has_typelink: bool, - go_version_minor: Option, - pclntab_offset: Option, -) -> Option { - let data = ctx.structure_search_data(); - let pclntab_off = pclntab_offset?; - let pclntab_va = if ctx.format() == BinaryFormat::Wasm { - pclntab_off as u64 - } else { - ctx.file_to_va(pclntab_off)? - }; +/// How many unparseable records a descriptor walk tolerates before giving up. +/// Generous enough to step over inter-region padding and the odd descriptor +/// kind we do not model, small enough that a walk over non-type bytes stops +/// quickly instead of grinding through a whole segment one word at a time. +const WALK_SKIP_BUDGET: u32 = 64; - let p = ps as usize; - - if p == 0 { - return None; - } - let search_start = data.len().checked_div(4).unwrap_or(0); // skip code region - let target_bytes = match ps { - 4 => (pclntab_va as u32).to_le_bytes().to_vec(), - 8 => pclntab_va.to_le_bytes().to_vec(), - _ => return None, +/// `[start, end)` VAs of the typelink (reflection-visible) descriptor region. +/// +/// On V5 the region is bounded by `moduledata.typedesclen`, exactly as +/// `runtime.moduleTypelinks` reads it; the descriptors past that bound are +/// non-typelink types and then itabs, neither of which belongs in the +/// typelink enumeration. Pre-V5 binaries have no such bound — they only reach +/// this walk when both typelink tables are unavailable — so the whole types +/// region is used. +/// +/// The `ptrSize` skip at the head is the slot the linker reserves so that no +/// type reference has offset zero; `type:*` sits there. +/// +/// Nothing past `typedesclen` can be walked the same way. The linker groups +/// *every* `type:`-prefixed read-only symbol under the same carrier and sorts +/// the non-typelink remainder by size, so `[typedesclen, itaboffset)` is a mix +/// of non-typelink descriptors and `type:.namedata.*` blobs with no recorded +/// boundary between them. Those descriptors are reachable only by following +/// references — see [`extract_all_types`]. +fn typelink_walk_range(md: &Moduledata, ptr_size: u8) -> Option<(u64, u64)> { + let start = md.types.checked_add(u64::from(ptr_size))?; + let end = match md.typedesclen { + Some(len) => md.types.checked_add(len)?, + None => md.etypes, }; - - let mut offset = search_start; - while let Some(end) = offset.checked_add(p) { - if end > data.len() { - break; - } - - let rem = offset.checked_rem(p).unwrap_or(0); - if rem != 0 { - let bump = p.saturating_sub(rem); - offset = match offset.checked_add(bump) { - Some(o) => o, - None => break, - }; - continue; - } - - if data.get(offset..end) == Some(target_bytes.as_slice()) { - let remaining = match data.get(offset..) { - Some(r) => r, - None => break, - }; - if let Some(md) = Moduledata::parse(remaining, ps, pv, has_typelink, go_version_minor) { - let anchored = match md.version { - ModuledataVersion::V1 => md.text != 0 && ctx.va_to_file(md.text).is_some(), - _ => md.types != 0 && ctx.va_to_file(md.funcnametab.ptr).is_some(), - }; - if md.minpc < md.maxpc && anchored { - return Some(md); - } - } - } - offset = match offset.checked_add(p) { - Some(o) => o, - None => break, - }; + if end > start { + Some((start, end)) + } else { + None } - None } /// Parse a single `abi.Type` at the given file offset and build a `GoType`. @@ -849,13 +842,12 @@ fn parse_type_at<'a>( file_off: usize, type_va: u64, types_base_va: u64, - ps: u8, - legacy: bool, + abi: TypeAbi, ctx: &BinaryContext<'a>, ) -> Option> { let remaining = data.get(file_off..)?; - let abi_type = AbiType::parse(remaining, ps)?; - let mut t = build_go_type(&abi_type, remaining, data, types_base_va, ps, legacy, ctx)?; + let abi_type = AbiType::parse(remaining, abi.ps)?; + let mut t = build_go_type(&abi_type, remaining, data, types_base_va, abi, ctx)?; t.descriptor_va = type_va; Some(t) } @@ -865,12 +857,11 @@ pub fn type_at_va<'a>( ctx: &'a BinaryContext<'a>, va: u64, types_base_va: u64, - ps: u8, - legacy: bool, + abi: TypeAbi, ) -> Option> { let data = ctx.structure_search_data(); let file_off = ctx.va_to_file(va)?; - let t = parse_type_at(data, file_off, va, types_base_va, ps, legacy, ctx)?; + let t = parse_type_at(data, file_off, va, types_base_va, abi, ctx)?; // Validate this is a real descriptor, not a mid-descriptor / non-type // address a stray reference pointed at: the kind must be known and the // name must be clean (Go type names never contain control characters). @@ -891,11 +882,10 @@ pub fn type_at_va<'a>( /// (typically `typelink`), e.g. a struct used only as a pointer's element. pub fn extract_all_types<'a>( ctx: &'a BinaryContext<'a>, - ptr_size: u8, seeds: Vec>, types_base: u64, etypes: u64, - legacy: bool, + abi: TypeAbi, ) -> Vec> { const CAP: usize = 2_000_000; // Every Go type descriptor lives in [types, etypes). Bounding the traversal @@ -917,7 +907,7 @@ pub fn extract_all_types<'a>( if !in_range(va) || !visited.insert(va) { continue; } - let t = match type_at_va(ctx, va, types_base, ptr_size, legacy) { + let t = match type_at_va(ctx, va, types_base, abi) { Some(t) => t, None => continue, }; @@ -1030,10 +1020,14 @@ fn build_go_type<'a>( type_data: &'a [u8], full_data: &'a [u8], types_base_va: u64, - ps: u8, - legacy: bool, + abi: TypeAbi, ctx: &BinaryContext<'a>, ) -> Option> { + let TypeAbi { + ps, + legacy_names: legacy, + map_layout, + } = abi; let kind = TypeKind::from_raw(abi_type.kind()); // Resolve name via Str (NameOff from types base) @@ -1119,15 +1113,12 @@ fn build_go_type<'a>( .unwrap_or(TypeDetail::None), TypeKind::Map => type_data .get(base_sz..) - .and_then(|d| MapTypeExtra::parse(d, ps)) + .and_then(|d| MapTypeExtra::parse(d, ps, map_layout)) .map(|m| TypeDetail::Map { key_va: m.key, elem_va: m.elem, group_va: m.group, - hasher_va: m.hasher, - key_stride: m.key_stride, - elem_stride: m.elem_stride, - flags: m.flags, + extra: Box::new(m), }) .unwrap_or(TypeDetail::None), TypeKind::Pointer => type_data @@ -1167,7 +1158,9 @@ fn build_go_type<'a>( TypeKind::Func => align_up(base_sz.saturating_add(FuncTypeExtra::SIZE), ps as usize)? .saturating_sub(base_sz), TypeKind::Interface => InterfaceTypeExtra::size(ps), - TypeKind::Map => MapTypeExtra::size(ps), + TypeKind::Map => { + MapTypeExtra::size(ps, map_layout.resolve_for(type_data.get(base_sz..)?, ps)) + } TypeKind::Pointer | TypeKind::Slice => ElemTypeExtra::size(ps), TypeKind::Struct => StructTypeExtra::size(ps), _ => 0, diff --git a/tests/integration.rs b/tests/integration.rs index 9c791ff..cc06eb9 100644 --- a/tests/integration.rs +++ b/tests/integration.rs @@ -52,6 +52,10 @@ const BASIC_EMBED: &str = "tests/samples/embed_go126_darwin_arm64"; const BASIC_EMBED_STRIPPED: &str = "tests/samples/embed_go126_darwin_arm64_stripped"; const BASIC_EMBED_LINUX: &str = "tests/samples/embed_go126_linux_amd64"; const COVER_LINUX: &str = "tests/samples/basic_go126_linux_amd64_cover"; +const BASIC_GO127_LINUX: &str = "tests/samples/basic_go127_linux_amd64"; +const BASIC_GO127_PIE: &str = "tests/samples/basic_go127_linux_amd64_pie"; +const BASIC_GO124_WINDOWS: &str = "tests/samples/basic_go124_windows_amd64.exe"; +const BASIC_GO124_WINDOWS_NOVERSION: &str = "tests/samples/basic_go124_windows_amd64_noversion.exe"; fn load(path: &str) -> Vec { std::fs::read(path).unwrap_or_else(|e| panic!("Failed to read {path}: {e}")) @@ -777,6 +781,101 @@ mod buildinfo { mod moduledata { use super::*; + /// A PE binary never carries a `.typelink` section at any Go version, so + /// with its version string scrubbed the only layout hint points at Go 1.27. + /// Picking V5 there shifts every field past `types` and silently reports an + /// empty types region, no itabs, no init tasks and `has_main == false`. + /// The layout must instead be arbitrated structurally, giving byte-for-byte + /// the same extraction as the intact binary. + #[test] + fn layout_survives_a_scrubbed_version_string() { + use gobin::structures::moduledata::ModuledataVersion; + + let intact_data = load(BASIC_GO124_WINDOWS); + let intact = GoBinary::parse(&intact_data).unwrap(); + let scrubbed_data = load(BASIC_GO124_WINDOWS_NOVERSION); + let scrubbed = GoBinary::parse(&scrubbed_data).unwrap(); + + // The premise: the version really is unrecoverable from the scrubbed + // copy, so the parser cannot fall back on it. + assert!(intact.go_version().unwrap().starts_with("go1.24")); + assert!( + !scrubbed + .go_version() + .is_some_and(|v| v.starts_with("go1.24")), + "fixture should have no usable version string" + ); + + let a = intact.moduledata().unwrap(); + let b = scrubbed.moduledata().unwrap(); + assert_eq!(b.version, ModuledataVersion::V3); + assert_eq!(a.version, b.version); + assert_eq!((a.types, a.etypes), (b.types, b.etypes)); + assert_eq!( + (a.rodata, a.gofunc, a.epclntab), + (b.rodata, b.gofunc, b.epclntab) + ); + assert_eq!(a.has_main, b.has_main); + assert!(a.has_main); + + assert_eq!(intact.types().count(), scrubbed.types().count()); + assert_eq!(intact.itab_pairs().count(), scrubbed.itab_pairs().count()); + assert_eq!(intact.init_order().len(), scrubbed.init_order().len()); + assert!( + scrubbed.types().count() > 100, + "types must not collapse to 0" + ); + assert!(scrubbed.itab_pairs().count() > 0); + assert!(!scrubbed.init_order().is_empty()); + } + + /// `-buildmode=pie` renames every read-only-relocatable Go section with a + /// `.data.rel.ro` prefix. If the classifier does not strip it, a PIE binary + /// looks like it has no type sections at all — which for Go ≤1.26 is read + /// as a Go 1.27 signal, and for 1.27 loses the positive `.go.type` signal. + #[test] + fn pie_relro_section_names_are_recognized() { + use gobin::structures::moduledata::ModuledataVersion; + + let plain_data = load(BASIC_GO127_LINUX); + let plain = GoBinary::parse(&plain_data).unwrap(); + let pie_data = load(BASIC_GO127_PIE); + let pie = GoBinary::parse(&pie_data).unwrap(); + + assert!( + pie.context().sections().go_type.is_some(), + ".data.rel.ro.go.type must classify as .go.type" + ); + assert!(pie.context().sections().go_func.is_some()); + assert_eq!(pie.moduledata().unwrap().version, ModuledataVersion::V5); + // Same program, same toolchain: the extraction must not differ. + assert_eq!(plain.types().count(), pie.types().count()); + assert_eq!(plain.itab_pairs().count(), pie.itab_pairs().count()); + assert_eq!(plain.functions().count(), pie.functions().count()); + } + + /// Go 1.26 stopped writing `textStart` into the pcHeader, leaving the slot + /// zeroed. A moduledata-less 1.26+ binary must not report `runtime.text` as + /// address 0 — that turns every entry VA into a raw `entry_off` without any + /// error surfacing. + #[test] + fn text_va_is_never_zero() { + for f in super::matrix::discover() { + let data = load(&f.path); + let Some(bin) = GoBinary::parse(&data) else { + continue; + }; + if let Some(va) = bin.text_va() { + // wasm addresses code in a separate PC space whose text base + // legitimately is 0; every other format maps text above 0. + if bin.context().format() == BinaryFormat::Wasm { + continue; + } + assert_ne!(va, 0, "{}: text_va() reported address 0", f.path); + } + } + } + #[test] fn moduledata_segments_and_identity() { let data = load(BASIC_LINUX); @@ -1067,6 +1166,34 @@ mod moduledata { mod embed_and_fips { use super::*; + /// `embed.FS` recovery reads through the address-space view, which for + /// wasm is the reconstructed linear-memory image rather than the file. + /// Narrowing that search with file-offset section ranges finds nothing — + /// silently, since a binary with no embeds legitimately returns empty. + #[test] + fn embedded_assets_recovered_from_wasm() { + let data = load("tests/samples/embed_go127_wasip1_wasm"); + let bin = GoBinary::parse(&data).unwrap(); + let assets = bin.embedded_assets(); + assert!( + !assets.is_empty(), + "wasm embed fixture must yield its embedded files" + ); + assert!( + assets.iter().any(|a| a.path.ends_with(".txt")), + "expected the embedded text assets, got {:?}", + assets.iter().map(|a| a.path).collect::>() + ); + // Directories carry no bytes; files do. + for a in &assets { + if a.is_dir { + assert!(a.data.is_empty(), "{}: dir must have no bytes", a.path); + } else { + assert!(!a.data.is_empty(), "{}: file must have bytes", a.path); + } + } + } + fn assert_embed_assets(path: &str) { let data = load(path); let bin = GoBinary::parse(&data).unwrap(); @@ -1278,6 +1405,229 @@ mod wasm { mod types { use super::*; + /// `abi.MapType` has shipped three different shapes — bucket-based `hmap` + /// (≤1.23), Swiss tables (1.24-1.26), and Swiss with explicit key/elem + /// strides (1.27+) — and each descriptor's size feeds the position of its + /// trailing `UncommonType`. Sweep the corpus and assert every map + /// descriptor is read with the layout its Go version actually emitted, with + /// the version-specific fields populated and the others genuinely absent. + #[test] + fn map_descriptors_use_the_layout_of_their_go_version() { + use gobin::structures::types::{MapLayout, TypeDetail}; + + let mut checked = 0usize; + let mut seen: BTreeSet = BTreeSet::new(); + for f in super::matrix::discover() { + let minor = super::matrix::tag_minor(&f.gotag); + // Type descriptors arrived in Go 1.7. + if minor < 7 { + continue; + } + let expected = MapLayout::for_go_minor(minor); + let data = load(&f.path); + let Some(bin) = GoBinary::parse(&data) else { + continue; + }; + let mut saw_map = false; + for t in bin.types() { + let TypeDetail::Map { extra, .. } = &t.detail else { + continue; + }; + saw_map = true; + assert_eq!( + extra.layout, expected, + "{}: {} parsed with {:?}, expected {:?}", + f.path, t.name, extra.layout, expected + ); + match expected { + MapLayout::HmapWithHmapType | MapLayout::HmapBools => { + assert!(extra.bucket_size.is_some_and(|b| b > 0), "{}", f.path); + assert!(extra.hasher.is_none(), "{}: hasher arrives in 1.14", f.path); + assert!( + extra.raw_flags.is_none(), + "{}: the bool tail has no flags word", + f.path + ); + assert!(extra.group_size.is_none(), "{}", f.path); + assert_eq!( + extra.hmap.is_some(), + expected == MapLayout::HmapWithHmapType, + "{}: the hmap pointer exists only up to Go 1.10", + f.path + ); + } + MapLayout::HmapFlags | MapLayout::HmapHasher => { + assert!(extra.bucket_size.is_some_and(|b| b > 0), "{}", f.path); + assert!(extra.raw_flags.is_some(), "{}", f.path); + assert!(extra.hmap.is_none(), "{}", f.path); + assert_eq!( + extra.hasher.is_some(), + expected == MapLayout::HmapHasher, + "{}: the hasher pointer arrives in Go 1.14", + f.path + ); + assert!(extra.group_size.is_none(), "{}", f.path); + assert!(extra.slot_size.is_none(), "{}", f.path); + assert!(extra.key_stride.is_none(), "{}", f.path); + } + MapLayout::Swiss => { + assert!(extra.group_size.is_some_and(|g| g > 0), "{}", f.path); + assert!(extra.slot_size.is_some(), "{}", f.path); + // The 1.27-only stride fields must read as absent, + // not as a zero scavenged from the next descriptor. + assert!(extra.keys_off.is_none(), "{}", f.path); + assert!(extra.key_stride.is_none(), "{}", f.path); + assert!(extra.bucket_size.is_none(), "{}", f.path); + } + MapLayout::SwissSplitGroup => { + assert!(extra.group_size.is_some_and(|g| g > 0), "{}", f.path); + assert!(extra.keys_off.is_some(), "{}", f.path); + assert!(extra.key_stride.is_some(), "{}", f.path); + assert!(extra.elem_stride.is_some(), "{}", f.path); + assert!(extra.slot_size.is_none(), "{}", f.path); + assert!(extra.bucket_size.is_none(), "{}", f.path); + } + MapLayout::Probe => unreachable!("resolved before parsing"), + } + // The flags word is a 5-bit mask in every encoding; a value + // outside that range means we read past the descriptor's end. + if let Some(raw) = extra.raw_flags { + assert!( + raw < 64, + "{}: {} has implausible map flags {raw:#x}", + f.path, + t.name + ); + } + // `reflexive_key` exists in every hmap encoding and in none of + // the Swiss ones, which is a second, independent check that the + // right decoder ran. + assert_eq!( + extra.flags.reflexive_key.is_some(), + expected.is_hmap(), + "{}: {} reflexive-key presence disagrees with {:?}", + f.path, + t.name, + expected + ); + seen.insert(format!("{expected:?}")); + checked += 1; + } + // Every fixture that reaches `fmt`/`reflect` carries map types. + if f.prog == "types" { + assert!( + saw_map, + "{}: types harness should surface map types", + f.path + ); + } + } + assert!(checked > 50, "expected a broad sweep, checked {checked}"); + // Every layout era must be represented by a real fixture, so the + // corpus cannot silently lose coverage of one of the six boundaries. + let want: BTreeSet = [ + "HmapWithHmapType", + "HmapBools", + "HmapFlags", + "HmapHasher", + "Swiss", + "SwissSplitGroup", + ] + .into_iter() + .map(str::to_string) + .collect(); + assert_eq!( + seen, want, + "map-layout coverage gap: the corpus exercises {seen:?}" + ); + } + + /// Go 1.27 replaced the typelink table with a walk bounded by + /// `moduledata.typedesclen`. Walking to `etypes` instead runs off the end + /// of the typelink descriptors into non-typelink types, `type:.namedata.*` + /// blobs, and finally the inline itab array — surfacing unnamed junk types. + #[test] + fn v5_type_walk_stops_at_typedesclen() { + use gobin::structures::moduledata::ModuledataVersion; + + for f in super::matrix::discover() { + let data = load(&f.path); + let Some(bin) = GoBinary::parse(&data) else { + continue; + }; + let Some(md) = bin.moduledata() else { continue }; + if md.version != ModuledataVersion::V5 { + continue; + } + let typedesclen = md.typedesclen.expect("V5 records typedesclen"); + let itaboffset = md.itaboffset.expect("V5 records itaboffset"); + assert!(typedesclen > 0 && typedesclen <= itaboffset, "{}", f.path); + let bound = md.types + typedesclen; + + let types: Vec<_> = bin.types().collect(); + assert!(!types.is_empty(), "{}: V5 must still yield types", f.path); + for t in &types { + assert!( + t.descriptor_va >= md.types && t.descriptor_va < bound, + "{}: type {:?} at {:#x} lies outside [types, types+typedesclen) = [{:#x}, {:#x})", + f.path, + t.name, + t.descriptor_va, + md.types, + bound + ); + assert!( + !t.name.is_empty(), + "{}: unnamed type at {:#x} — the walk over-ran its bound", + f.path, + t.descriptor_va + ); + } + } + } + + /// Go 1.27 lowered `abi.MaxPtrmaskBytes` from 2048 to 16 and deleted + /// type-level GC programs, so most non-trivial types now defer their + /// pointer mask to run time. `gc_mask_va` must report those as absent + /// rather than handing back the address of an empty BSS slot. + #[test] + fn gc_mask_on_demand_is_modelled() { + use gobin::structures::abitype::AbiType; + + let data = load(BASIC_GO127_LINUX); + let bin = GoBinary::parse(&data).unwrap(); + let mut on_demand = 0usize; + let mut inline_mask = 0usize; + for t in bin.types() { + let Some(off) = bin.context().va_to_file(t.descriptor_va) else { + continue; + }; + let Some(abi) = bin + .context() + .structure_search_data() + .get(off..) + .and_then(|d| AbiType::parse(d, 8)) + else { + continue; + }; + if abi.gc_mask_on_demand() { + on_demand += 1; + assert!( + abi.gc_mask_va().is_none(), + "on-demand types have no static mask" + ); + } else if abi.gcdata != 0 { + inline_mask += 1; + assert_eq!(abi.gc_mask_va(), Some(abi.gcdata)); + } + } + assert!(inline_mask > 0, "small types still carry a static mask"); + // Whether any type in a given program crosses the (Go 1.27) 16-byte + // threshold depends on the program, so `on_demand` is only required to + // be consistent, not non-zero. + let _ = on_demand; + } + #[test] fn type_details_present() { use gobin::structures::types::TypeDetail; @@ -1652,6 +2002,16 @@ mod types { // runtime.* / runtime/internal/* are runtime AND internal, never stdlib assert!(is_runtime_path("runtime")); assert!(is_runtime_path("runtime/internal/atomic")); + // Go 1.24 moved the runtime's internal packages under `internal/`; + // they are still runtime code, not ordinary internal library code. + assert!(is_runtime_path("internal/runtime/atomic")); + assert!(is_runtime_path("internal/runtime/maps")); + assert!(is_runtime_path("internal/runtime/sys")); + assert!(is_internal_path("internal/runtime/maps")); + assert!(!is_stdlib_path("internal/runtime/maps")); + // …but a non-runtime `internal/` package is not runtime. + assert!(!is_runtime_path("internal/abi")); + assert!(!is_runtime_path("internal/runtimefoo")); assert!(is_internal_path("runtime")); assert!(!is_stdlib_path("runtime")); @@ -1772,7 +2132,7 @@ mod types { #[test] fn enum_display_strings_are_stable() { - use gobin::structures::types::TypeDetail; + use gobin::structures::types::{MapLayout, MapTypeExtra, TypeDetail}; // Confidence assert_eq!(format!("{}", Confidence::None), "none"); @@ -1817,10 +2177,9 @@ mod types { key_va: 0x1, elem_va: 0x2, group_va: 0, - hasher_va: 0, - key_stride: 0, - elem_stride: 0, - flags: 0, + extra: Box::new( + MapTypeExtra::parse(&[0u8; 88], 8, MapLayout::SwissSplitGroup).unwrap(), + ), } .kind_str(), "map" @@ -1930,17 +2289,17 @@ mod matrix { structures::{Arch, PclntabVersion, moduledata::ModuledataVersion}, }; - struct Fixture { - path: String, - prog: String, - gotag: String, - goos: String, - goarch: String, + pub struct Fixture { + pub path: String, + pub prog: String, + pub gotag: String, + pub goos: String, + pub goarch: String, } /// Discover every fixture binary, parsing its /// `_go__[_variant]` filename. - fn discover() -> Vec { + pub fn discover() -> Vec { let dir = "tests/samples"; let mut out = Vec::new(); for entry in std::fs::read_dir(dir).unwrap() { @@ -1954,9 +2313,16 @@ mod matrix { continue; } let name = fname.strip_suffix(".exe").unwrap_or(&fname); - // Peel an optional build variant (stripped / cover / fips) to expose - // the underlying _go__ name. - let core = ["_stripped", "_cover", "_fips"] + // `_noversion` fixtures have their Go version string deliberately + // scrubbed, so the version-derived assertions below do not apply. + // They are exercised by `moduledata::layout_survives_a_scrubbed_ + // version_string` instead. + if name.ends_with("_noversion") { + continue; + } + // Peel an optional build variant (stripped / cover / fips / pie) to + // expose the underlying _go__ name. + let core = ["_stripped", "_cover", "_fips", "_pie"] .iter() .find_map(|v| name.strip_suffix(v)) .unwrap_or(name); @@ -1979,7 +2345,7 @@ mod matrix { /// Minor version from a `go1` tag (`go17` → 7, `go126` → 26). Every /// Go release is 1.x, so stripping `go1` is unambiguous. - fn tag_minor(gotag: &str) -> u32 { + pub fn tag_minor(gotag: &str) -> u32 { gotag .strip_prefix("go1") .and_then(|s| s.parse().ok()) diff --git a/tests/samples/README.md b/tests/samples/README.md index 6588af8..802ec66 100644 --- a/tests/samples/README.md +++ b/tests/samples/README.md @@ -16,9 +16,11 @@ matrix in `build.sh` automatically extends test coverage. ``` `variant` ∈ `stripped` (`-ldflags="-s -w"`), `cover` (`-cover`), `fips` -(`GOFIPS140`). `.exe` = PE/Windows. Every backbone fixture is `linux/amd64`; -the `go116/go120/go124/go126/go127` anchors additionally get cross-compiled -`darwin/arm64` (Mach-O), `windows/amd64` (PE), and `wasip1/wasm`. +(`GOFIPS140`), `pie` (`-buildmode=pie`), `noversion` (a post-processed copy with +the Go version string overwritten). `.exe` = PE/Windows. Every backbone fixture +is `linux/amd64`; the `go116/go120/go124/go126/go127` anchors additionally get +cross-compiled `darwin/arm64` (Mach-O), `windows/amd64` (PE), and +`wasip1/wasm`. ## Version → layout @@ -31,12 +33,31 @@ detect (one `basic_go_linux_amd64` fixture per row, plus the variants): | `go15` | Go12 | V1 | V1 head with no `types`/`itablinks` (`[]*_type`) | | `go17`, `go18` | Go12 | V1 | +types/typelinks/typemap (1.7); +plugin/textsect (1.8) | | `go19` | Go12 | V1 | V1 tail without `hasmain`/`bad` | -| `go110`,`go112`,`go115` | Go12 | V1 | full V1 (+hasmain/bad 1.10; +funcID 1.12) | +| `go110`,`go111`,`go112`,`go113`,`go115` | Go12 | V1 | full V1 (+hasmain/bad 1.10; +funcID 1.12) | | `go116`, `go117` | Go116 | V2 | new pcHeader / functab / `_func` | | `go118` | Go118 | V3 | u32 functab offsets, `_func` +flag; +rodata/gofunc | | `go120`,`go121`,`go123`,`go124`,`go125` | Go120 | V3 | `_func` +startLine; +covctrs (1.20), +inittasks (1.21) | | `go126` | Go120 | V4 | +epclntab, `.go.module` section | -| `go127` | Go120 | V5 | inline itabs, no typelinks | +| `go127` | Go120 | V5 | inline itabs, no typelinks, `.go.type`/`.go.func` sections | + +The corpus separately pins the `abi.MapType` layout, which has changed **six** +times and is not tied to the moduledata version — so each era needs its own +fixture. The integration test `types::map_descriptors_use_the_layout_of_their_go_version` +sweeps the whole corpus and additionally asserts that all six eras are present, +so this coverage cannot be lost silently: + +| Tags | Map layout | What changed at the boundary | +|-----------------------------|--------------------|--------------------------------------------| +| `go17`, `go110` | `HmapWithHmapType` | `Key, Elem, Bucket, Hmap` + a bool tail | +| `go111` | `HmapBools` | Go 1.11 drops the `Hmap` type pointer | +| `go112`, `go113` | `HmapFlags` | Go 1.12 folds the bools into `flags uint32`| +| `go115`…`go123` | `HmapHasher` | Go 1.14 adds the `Hasher` pointer | +| `go124`, `go125`, `go126` | `Swiss` | Go 1.24 replaces buckets with Swiss tables | +| `go127` | `SwissSplitGroup` | Go 1.27 adds explicit key/elem strides | + +The flag *bits* were renumbered at the Swiss boundary too, so `MapTypeExtra` +normalizes them; `go120` (`hashMightPanic` on an interface-keyed map) and +`go126` exercise both encodings. ## Programs (sources under `src/`) @@ -62,6 +83,29 @@ detect (one `basic_go_linux_amd64` fixture per row, plus the variants): (CGO) it is a Mach-O dylib using **chained fixups** (`plugin_go126_darwin_arm64.so`); see below. +## Adversarial variants + +Two fixtures exist to cover inputs a triage pipeline actually sees, where the +parser cannot fall back on the usual markers: + +- **`basic_go127_linux_amd64_pie`** — `-buildmode=pie` prefixes every + read-only-relocatable Go section with `.data.rel.ro` + (`.data.rel.ro.go.type`, and pre-1.27 `.data.rel.ro.typelink`). Without the + prefix stripping in `formats::classify_section`, a PIE binary looks like it + has no type sections at all. +- **`embed_go127_wasip1_wasm`** — the only fixture where the address-space + view is not the file. `embed.FS` recovery searches the reconstructed + linear-memory image, so any attempt to narrow that search with file-offset + section ranges finds nothing — silently, because a binary with no embeds + legitimately returns an empty list. +- **`basic_go124_windows_amd64_noversion.exe`** — a byte-identical copy of + `basic_go124_windows_amd64.exe` with every `go1.24` literal zeroed, the way an + obfuscator leaves one. PE never carries a `.typelink` section at any Go + version, so with no version string the only layout hint points at the Go 1.27 + moduledata; the fixture pins the structural arbitration that stops the parser + acting on it. Generated by `build.sh` from the intact binary, not by a + toolchain. + ## Rebuilding `build.sh` is the single, self-contained builder. It runs on a **linux/amd64 @@ -77,12 +121,8 @@ cleanly from linux. ./build.sh --list # print the planned fixture set and exit ``` -Two fixtures need a host the script cannot assume: +One fixture needs a host the script cannot assume: -- **`go127` (V5)** is the unreleased 1.27 dev tree, built via `gotip`. Bootstrap - it once with `go install golang.org/dl/gotip@latest && gotip download` - (into `$HOME/sdk/gotip`); `build.sh` then includes it automatically and skips - it when absent. - **`plugin_go126_darwin_arm64.so`** needs CGO + a darwin C toolchain for the Mach-O chained-fixup test, so it is built on a mac: ```sh diff --git a/tests/samples/basic_go111_linux_amd64 b/tests/samples/basic_go111_linux_amd64 new file mode 100755 index 0000000..8c018b0 Binary files /dev/null and b/tests/samples/basic_go111_linux_amd64 differ diff --git a/tests/samples/basic_go112_linux_amd64 b/tests/samples/basic_go112_linux_amd64 index 114056e..ff99e14 100755 Binary files a/tests/samples/basic_go112_linux_amd64 and b/tests/samples/basic_go112_linux_amd64 differ diff --git a/tests/samples/basic_go113_linux_amd64 b/tests/samples/basic_go113_linux_amd64 new file mode 100755 index 0000000..1cd8fb2 Binary files /dev/null and b/tests/samples/basic_go113_linux_amd64 differ diff --git a/tests/samples/basic_go123_linux_amd64 b/tests/samples/basic_go123_linux_amd64 index e8fc7de..01fd6ad 100755 Binary files a/tests/samples/basic_go123_linux_amd64 and b/tests/samples/basic_go123_linux_amd64 differ diff --git a/tests/samples/basic_go124_darwin_arm64 b/tests/samples/basic_go124_darwin_arm64 index 1538d19..c3c7513 100755 Binary files a/tests/samples/basic_go124_darwin_arm64 and b/tests/samples/basic_go124_darwin_arm64 differ diff --git a/tests/samples/basic_go124_linux_amd64 b/tests/samples/basic_go124_linux_amd64 index f0bf6f9..52aff4f 100755 Binary files a/tests/samples/basic_go124_linux_amd64 and b/tests/samples/basic_go124_linux_amd64 differ diff --git a/tests/samples/basic_go124_linux_amd64_fips b/tests/samples/basic_go124_linux_amd64_fips index 21423b5..419d441 100755 Binary files a/tests/samples/basic_go124_linux_amd64_fips and b/tests/samples/basic_go124_linux_amd64_fips differ diff --git a/tests/samples/basic_go124_wasip1_wasm b/tests/samples/basic_go124_wasip1_wasm index 1269588..3da1b03 100755 Binary files a/tests/samples/basic_go124_wasip1_wasm and b/tests/samples/basic_go124_wasip1_wasm differ diff --git a/tests/samples/basic_go124_windows_amd64.exe b/tests/samples/basic_go124_windows_amd64.exe index 9fb1b32..9ef2a89 100755 Binary files a/tests/samples/basic_go124_windows_amd64.exe and b/tests/samples/basic_go124_windows_amd64.exe differ diff --git a/tests/samples/basic_go124_windows_amd64_noversion.exe b/tests/samples/basic_go124_windows_amd64_noversion.exe new file mode 100644 index 0000000..7eee35e Binary files /dev/null and b/tests/samples/basic_go124_windows_amd64_noversion.exe differ diff --git a/tests/samples/basic_go127_darwin_arm64 b/tests/samples/basic_go127_darwin_arm64 index afc4dd0..13d685f 100755 Binary files a/tests/samples/basic_go127_darwin_arm64 and b/tests/samples/basic_go127_darwin_arm64 differ diff --git a/tests/samples/basic_go127_darwin_arm64_stripped b/tests/samples/basic_go127_darwin_arm64_stripped index 896c751..8cb6a84 100755 Binary files a/tests/samples/basic_go127_darwin_arm64_stripped and b/tests/samples/basic_go127_darwin_arm64_stripped differ diff --git a/tests/samples/basic_go127_linux_amd64 b/tests/samples/basic_go127_linux_amd64 index 2c1f8fb..103d6fe 100755 Binary files a/tests/samples/basic_go127_linux_amd64 and b/tests/samples/basic_go127_linux_amd64 differ diff --git a/tests/samples/basic_go127_linux_amd64_cover b/tests/samples/basic_go127_linux_amd64_cover new file mode 100755 index 0000000..46e3fe4 Binary files /dev/null and b/tests/samples/basic_go127_linux_amd64_cover differ diff --git a/tests/samples/basic_go127_linux_amd64_fips b/tests/samples/basic_go127_linux_amd64_fips new file mode 100755 index 0000000..803290c Binary files /dev/null and b/tests/samples/basic_go127_linux_amd64_fips differ diff --git a/tests/samples/basic_go127_linux_amd64_pie b/tests/samples/basic_go127_linux_amd64_pie new file mode 100755 index 0000000..bf5840d Binary files /dev/null and b/tests/samples/basic_go127_linux_amd64_pie differ diff --git a/tests/samples/basic_go127_linux_amd64_stripped b/tests/samples/basic_go127_linux_amd64_stripped new file mode 100755 index 0000000..f7ea896 Binary files /dev/null and b/tests/samples/basic_go127_linux_amd64_stripped differ diff --git a/tests/samples/basic_go127_wasip1_wasm b/tests/samples/basic_go127_wasip1_wasm index 6e28690..12134a3 100755 Binary files a/tests/samples/basic_go127_wasip1_wasm and b/tests/samples/basic_go127_wasip1_wasm differ diff --git a/tests/samples/basic_go127_windows_amd64.exe b/tests/samples/basic_go127_windows_amd64.exe index 93e81f5..3a3a974 100755 Binary files a/tests/samples/basic_go127_windows_amd64.exe and b/tests/samples/basic_go127_windows_amd64.exe differ diff --git a/tests/samples/basic_go127_windows_amd64_stripped.exe b/tests/samples/basic_go127_windows_amd64_stripped.exe index 68aaa5e..92500f3 100755 Binary files a/tests/samples/basic_go127_windows_amd64_stripped.exe and b/tests/samples/basic_go127_windows_amd64_stripped.exe differ diff --git a/tests/samples/build.sh b/tests/samples/build.sh index 62c10b3..bf192d4 100755 --- a/tests/samples/build.sh +++ b/tests/samples/build.sh @@ -2,11 +2,10 @@ # # Rebuild every test fixture under tests/samples/ from the sources in src/. # -# ONE script for the whole corpus. It runs on a linux/amd64 host and is largely +# ONE script for the whole corpus. It runs on a linux/amd64 host and is fully # self-contained: it downloads each released Go toolchain from go.dev on demand -# — no system Go and no container engine. (The single exception is the -# unreleased go127/V5 fixture, built via a gotip tree the caller bootstraps; see -# the gotip note below.) A linux host is used because the pre-1.16 toolchains +# — no system Go, no container engine, no gotip bootstrap. A linux host is used +# because the pre-1.16 toolchains # have no darwin/arm64 build (and crash under qemu user-emulation), while every # modern format (Mach-O / PE / Wasm) cross-compiles cleanly from linux. The cgo # harness needs a host C compiler (gcc), which a linux/amd64 box has. @@ -56,7 +55,9 @@ mkdir -p "$OUT" "$CACHE" "$WORK" # go18 1.8.7 +textsectmap, ptab, pluginpath, pkghashes # go19 1.9.7 V1 tail without hasmain/bad # go110 1.10.8 +hasmain, +bad -# go112 1.12.17 _func gains funcID +# go111 1.11.13 maptype drops the `hmap` type pointer +# go112 1.12.17 _func gains funcID; maptype bools collapse into `flags` +# go113 1.13.15 last maptype without `hasher` (added in 1.14) # go115 1.15.15 last Go12-pclntab release # go116 1.16.15 new pcHeader (magic Go116); moduledata V2 # go117 1.17.13 last Go116-magic @@ -66,16 +67,19 @@ mkdir -p "$OUT" "$CACHE" "$WORK" # go123 1.23.12 interior V3 sample # go124 1.24.13 GOFIPS140 available # go125 1.25.11 interior V3 sample -# go126 1.26.4 moduledata V4 (+epclntab); newest stable anchor +# go126 1.26.4 moduledata V4 (+epclntab) +# go127 1.27.1 moduledata V5 (-typelinks/-itablinks, +typedesclen, +# +itaboffset/itabsize); `.go.type`/`.go.func` sections; +# abi.MapType gains the split-group stride fields; newest +# stable anchor # --------------------------------------------------------------------------- -# go127 (moduledata V5) is the unreleased 1.27 dev tree, built via gotip; it is -# skipped automatically when $HOME/sdk/gotip is absent. VERSIONS=( "go12:1.2.2" "go14:1.4.3" "go15:1.5.4" "go17:1.7.6" "go18:1.8.7" - "go19:1.9.7" "go110:1.10.8" "go112:1.12.17" "go115:1.15.15" + "go19:1.9.7" "go110:1.10.8" "go111:1.11.13" "go112:1.12.17" "go113:1.13.15" + "go115:1.15.15" "go116:1.16.15" "go117:1.17.13" "go118:1.18.10" "go120:1.20.14" "go121:1.21.13" "go123:1.23.12" "go124:1.24.13" "go125:1.25.11" "go126:1.26.4" - "go127:tip" + "go127:1.27.1" ) # Anchor versions that get cross-compiled format variants. Mach-O/PE build from @@ -83,17 +87,19 @@ VERSIONS=( FORMAT_ANCHORS=("go116" "go120" "go124" "go126" "go127") WASIP1_ANCHORS=("go121" "go124" "go126" "go127") # Versions that get a `types` harness (the type system arrived in Go 1.7). -TYPES_VERSIONS=("go17" "go110" "go116" "go120" "go126") +# `go123` and `go126` bracket the two `abi.MapType` layout changes (buckets -> +# Swiss in 1.24, Swiss -> split-group strides in 1.27), which the map-descriptor +# stride assertions in `mod matrix` depend on. +TYPES_VERSIONS=( + "go17" "go110" "go111" "go112" "go113" "go116" "go120" "go123" "go126" "go127" +) # Versions that get a `generics` harness (type parameters arrived in Go 1.18). -GENERICS_VERSIONS=("go118" "go120" "go124" "go126") +GENERICS_VERSIONS=("go118" "go120" "go124" "go126" "go127") # Versions that get a `cgo` harness (linux/amd64, CGO_ENABLED=1 + host gcc). -CGO_VERSIONS=("go116" "go126") +CGO_VERSIONS=("go116" "go126" "go127") # Minor version of a tag/full-version string ("1.9.7" -> 9, "go116" -> 16). -# The unreleased tip toolchain (Go 1.27 dev) maps to a large sentinel so all the -# ">= N" feature gates treat it as newest. minor_of() { - [[ "$1" == "tip" || "$1" == "go127" ]] && { echo 99; return; } local v="${1#go1}"; v="${v#1.}"; echo "${v%%.*}" } in_list() { local x=$1; shift; local e; for e in "$@"; do [[ "$e" == "$x" ]] && return 0; done; return 1; } @@ -110,8 +116,10 @@ selected() { # Download + extract a toolchain into the cache (idempotent); echo its GOROOT. fetch_toolchain() { # fetch_toolchain local v="$1" - # Go 1.27 is unreleased: use a gotip tree the caller bootstrapped into - # $HOME/sdk/gotip (`go install golang.org/dl/gotip@latest && gotip download`). + # An unreleased dev toolchain (for the *next* Go, when one is being tracked + # ahead of its release) can be bootstrapped into $HOME/sdk/gotip with + # `go install golang.org/dl/gotip@latest && gotip download`; the matrix uses + # released versions only, so this branch is normally unused. if [[ "$v" == "tip" ]]; then [[ -x "$HOME/sdk/gotip/bin/go" ]] && echo "$HOME/sdk/gotip" || return 1 return 0 @@ -216,11 +224,66 @@ for entry in "${VERSIONS[@]}"; do build "$tag" "$ver" basic darwin arm64 "_fips" GOFIPS140=v1.0.0 ;; go127) build "$tag" "$ver" basic darwin arm64 "_stripped" -ldflags "-s -w" build "$tag" "$ver" basic windows amd64 "_stripped" -ldflags "-s -w" - build "$tag" "$ver" types linux amd64 "" - build "$tag" "$ver" generics linux amd64 "" ;; + build "$tag" "$ver" basic linux amd64 "_stripped" -ldflags "-s -w" + build "$tag" "$ver" basic linux amd64 "_cover" -cover + build "$tag" "$ver" basic linux amd64 "_fips" GOFIPS140=v1.0.0 + # -buildmode=pie moves every read-only-relocatable Go section under + # a `.data.rel.ro` prefix (`.data.rel.ro.go.type`, and pre-1.27 + # `.data.rel.ro.typelink`). Without a PIE fixture the section + # classifier's prefix stripping is untested, and an unprefixed + # lookup makes a PIE binary look like it has no typelink section at + # all — which the moduledata arbitration reads as a Go 1.27 signal. + build "$tag" "$ver" basic linux amd64 "_pie" -buildmode=pie + build "$tag" "$ver" minimal linux amd64 "" + build "$tag" "$ver" embed linux amd64 "" + build "$tag" "$ver" embed linux amd64 "_stripped" -ldflags "-s -w" + # A wasm embed fixture pins the one case where the address-space + # view is not the file: `embed.FS` recovery searches the + # reconstructed linear-memory image, so any attempt to narrow that + # search with file-offset section ranges silently finds nothing. + build "$tag" "$ver" embed wasip1 wasm "" ;; esac done +# --------------------------------------------------------------------------- +# Derived fixtures: produced by post-processing a built binary rather than by +# invoking a toolchain. +# --------------------------------------------------------------------------- + +# A Go binary with every occurrence of its version string overwritten, the way +# an obfuscator (garble) or a repacker leaves one. With no version to key on, +# the moduledata parser has to pick its layout from structural evidence alone — +# and on PE, which never carries a `.typelink` section, the only hint points at +# the Go 1.27 layout. Guessing wrong there silently empties the types, itabs, +# init tasks and inline tree, so this fixture pins the arbitration. +scrub_version() { # scrub_version + local src="$OUT/$1" dst="$OUT/$2" lit="$3" + if [[ $list_only -eq 1 ]]; then echo " $2"; return 0; fi + if [[ ! -s "$src" ]]; then echo " !! $2: missing source $1"; fail+=("$2"); return 0; fi + echo " $2" + if python3 - "$src" "$dst" "$lit" <<'PYEOF' +import sys + +src, dst, lit = sys.argv[1], sys.argv[2], sys.argv[3].encode() +data = bytearray(open(src, "rb").read()) +n = 0 +i = data.find(lit) +while i >= 0: + data[i : i + len(lit)] = b"\x00" * len(lit) + n += 1 + i = data.find(lit, i + len(lit)) +if n == 0: + sys.exit(f"no occurrence of {lit!r} in {src}") +open(dst, "wb").write(bytes(data)) +print(f" scrubbed {n} occurrence(s)") +PYEOF + then ok=$((ok + 1)); else echo " !! scrub failed"; rm -f "$dst"; fail+=("$2"); fi +} + +if [[ ${#SELECT[@]} -eq 0 ]] || selected go124; then + scrub_version basic_go124_windows_amd64.exe basic_go124_windows_amd64_noversion.exe go1.24 +fi + if [[ $list_only -eq 1 ]]; then exit 0; fi echo "Done. Built $ok fixtures into $OUT." if [[ ${#fail[@]} -gt 0 ]]; then printf 'Failed:\n'; printf ' %s\n' "${fail[@]}"; fi diff --git a/tests/samples/cgo_go127_linux_amd64 b/tests/samples/cgo_go127_linux_amd64 new file mode 100755 index 0000000..401fa9e Binary files /dev/null and b/tests/samples/cgo_go127_linux_amd64 differ diff --git a/tests/samples/embed_go127_linux_amd64 b/tests/samples/embed_go127_linux_amd64 new file mode 100755 index 0000000..3c1f961 Binary files /dev/null and b/tests/samples/embed_go127_linux_amd64 differ diff --git a/tests/samples/embed_go127_linux_amd64_stripped b/tests/samples/embed_go127_linux_amd64_stripped new file mode 100755 index 0000000..94db40f Binary files /dev/null and b/tests/samples/embed_go127_linux_amd64_stripped differ diff --git a/tests/samples/embed_go127_wasip1_wasm b/tests/samples/embed_go127_wasip1_wasm new file mode 100755 index 0000000..2d42107 Binary files /dev/null and b/tests/samples/embed_go127_wasip1_wasm differ diff --git a/tests/samples/generics_go127_linux_amd64 b/tests/samples/generics_go127_linux_amd64 index 96a623d..236207a 100755 Binary files a/tests/samples/generics_go127_linux_amd64 and b/tests/samples/generics_go127_linux_amd64 differ diff --git a/tests/samples/minimal_go127_linux_amd64 b/tests/samples/minimal_go127_linux_amd64 new file mode 100755 index 0000000..dde5e71 Binary files /dev/null and b/tests/samples/minimal_go127_linux_amd64 differ diff --git a/tests/samples/types_go111_linux_amd64 b/tests/samples/types_go111_linux_amd64 new file mode 100755 index 0000000..634340a Binary files /dev/null and b/tests/samples/types_go111_linux_amd64 differ diff --git a/tests/samples/types_go112_linux_amd64 b/tests/samples/types_go112_linux_amd64 new file mode 100755 index 0000000..9ded885 Binary files /dev/null and b/tests/samples/types_go112_linux_amd64 differ diff --git a/tests/samples/types_go113_linux_amd64 b/tests/samples/types_go113_linux_amd64 new file mode 100755 index 0000000..688a285 Binary files /dev/null and b/tests/samples/types_go113_linux_amd64 differ diff --git a/tests/samples/types_go123_linux_amd64 b/tests/samples/types_go123_linux_amd64 new file mode 100755 index 0000000..590ef09 Binary files /dev/null and b/tests/samples/types_go123_linux_amd64 differ diff --git a/tests/samples/types_go127_linux_amd64 b/tests/samples/types_go127_linux_amd64 index 8cfa0cd..065907b 100755 Binary files a/tests/samples/types_go127_linux_amd64 and b/tests/samples/types_go127_linux_amd64 differ