Rust symbol demangling

[zokier]

This came up in HN discussion that Bloaty does not have currently demangling for Rust symbols, and such feature would be useful.

There are at least two different libraries that implement Rust demangling:

1. GNU libiberty: https://github.com/gcc-mirror/gcc/blob/master/libiberty/rust-demangle.c

2. rustc-demangle: https://github.com/alexcrichton/rustc-demangle

The latter has the downside of being implemented in Rust, so a small wrapper is needed to make it work with C++ code. Might need bit of work to make it play with CMake nicely.

I have made a quick proof of concept version based on rustc-demangle here: https://github.com/zokier/bloaty/tree/rust_demangle

[zokier]

Shifting the discussion to this issue..

I think I'd prefer to just make this part of shortsymbols/fullsymbols instead of making a separate "rustsymbols". I assume that Rust symbols won't successfully demangle as C++ (and vice-versa), so we can just try both demanglers and use whatever works. That seems like it will be more graceful for mixed C++/Rust binaries.

That would be indeed nicer, but I'm not sure if it is easily achievable, because Rust and C++ manglings are so similar. I'm not expert on this, so do not take my word on this, but testing shortsymbols/fullsymbols on a Rust executable results partial demangling:

[zokier@zarch bloaty]$ ./bloaty -d shortsymbols /tmp/tinyrocket/target/release/tinyrocket
	 VM SIZE                                                                                        FILE SIZE
 --------------                                                                                  --------------
  65.7%   640Ki [2303 Others]                                                                      875Ki  42.8%
   0.0%       0 [section .debug_str]                                                               259Ki  12.7%
   0.0%       0 [section .debug_info]                                                              227Ki  11.2%
   0.0%       0 [section .debug_ranges]                                                            107Ki   5.3%
   0.0%       0 [section .debug_pubnames]                                                         94.6Ki   4.6%
   9.2%  89.5Ki idna::uts46::find_char::hf4b73b1817809953                                         89.5Ki   4.4%
   0.0%       0 [section .debug_line]                                                             81.1Ki   4.0%
   7.2%  69.9Ki unicode_normalization::normalize::d::h2e71db9e521747a3                            70.0Ki   3.4%
   0.0%       0 [section .debug_pubtypes]                                                         48.4Ki   2.4%
   3.3%  32.1Ki rocket::config::RocketConfig::override_from_env::h2f1179166e400e61                32.2Ki   1.6%
   3.3%  31.9Ki hyper::server::listener::spawn_with::_$u7b$$u7b$closure$u7d$$u7d$::h3691293af625  32.0Ki   1.6%
   1.7%  16.6Ki rocket::ignite::hee6613aa2a5fcf8a                                                 16.6Ki   0.8%
   1.7%  16.2Ki rocket::config::config::Config::set_raw::hb13b072d56297a7e                        16.3Ki   0.8%
   1.6%  15.4Ki unicode_bidi::char_data::bidi_class::he12c0681b23bfdb2                            15.5Ki   0.8%
   0.0%       0 [section .debug_macro]                                                            13.2Ki   0.6%
   1.3%  13.2Ki tinyrocket::main::h8716512f8746ab53                                               13.2Ki   0.6%
   1.3%  12.4Ki idna::uts46::decode_slice::hd86b0f78245a9e42                                      12.5Ki   0.6%
   1.2%  12.1Ki std::sys_common::backtrace::output::hc58f0399059446b5                             12.2Ki   0.6%
   1.1%  10.7Ki unicode_normalization::normalize::compose::h51bcd650cea02388                      10.8Ki   0.5%
   0.8%  7.83Ki rocket::request::request::Request::format::h73a5bf51e7bffce6                      7.92Ki   0.4%
   0.7%  6.45Ki rocket::rocket::Rocket::dispatch::hf52688098d427981                               6.53Ki   0.3%
 100.0%   974Ki TOTAL                                                                             2.00Mi 100.0%

[zokier@zarch bloaty]$ ./bloaty -d fullsymbols /tmp/tinyrocket/target/release/tinyrocket
	 VM SIZE                                                                                        FILE SIZE
 --------------                                                                                  --------------
  65.7%   640Ki [2316 Others]                                                                      875Ki  42.8%
   0.0%       0 [section .debug_str]                                                               259Ki  12.7%
   0.0%       0 [section .debug_info]                                                              227Ki  11.2%
   0.0%       0 [section .debug_ranges]                                                            107Ki   5.3%
   0.0%       0 [section .debug_pubnames]                                                         94.6Ki   4.6%
   9.2%  89.5Ki idna::uts46::find_char::hf4b73b1817809953                                         89.5Ki   4.4%
   0.0%       0 [section .debug_line]                                                             81.1Ki   4.0%
   7.2%  69.9Ki unicode_normalization::normalize::d::h2e71db9e521747a3                            70.0Ki   3.4%
   0.0%       0 [section .debug_pubtypes]                                                         48.4Ki   2.4%
   3.3%  32.1Ki rocket::config::RocketConfig::override_from_env::h2f1179166e400e61                32.2Ki   1.6%
   3.3%  31.9Ki hyper::server::listener::spawn_with::_$u7b$$u7b$closure$u7d$$u7d$::h3691293af625  32.0Ki   1.6%
   1.7%  16.6Ki rocket::ignite::hee6613aa2a5fcf8a                                                 16.6Ki   0.8%
   1.7%  16.2Ki rocket::config::config::Config::set_raw::hb13b072d56297a7e                        16.3Ki   0.8%
   1.6%  15.4Ki unicode_bidi::char_data::bidi_class::he12c0681b23bfdb2                            15.5Ki   0.8%
   0.0%       0 [section .debug_macro]                                                            13.2Ki   0.6%
   1.3%  13.2Ki tinyrocket::main::h8716512f8746ab53                                               13.2Ki   0.6%
   1.3%  12.4Ki idna::uts46::decode_slice::hd86b0f78245a9e42                                      12.5Ki   0.6%
   1.2%  12.1Ki std::sys_common::backtrace::output::hc58f0399059446b5                             12.2Ki   0.6%
   1.1%  10.7Ki unicode_normalization::normalize::compose::h51bcd650cea02388                      10.8Ki   0.5%
   0.8%  7.83Ki rocket::request::request::Request::format::h73a5bf51e7bffce6                      7.92Ki   0.4%
   0.7%  6.45Ki rocket::rocket::Rocket::dispatch::hf52688098d427981                               6.53Ki   0.3%
 100.0%   974Ki TOTAL                                                                             2.00Mi 100.0%

[zokier@zarch bloaty]$ ./bloaty -d rustsymbols /tmp/tinyrocket/target/release/tinyrocket
	 VM SIZE                                                        FILE SIZE
 --------------                                                  --------------
  63.2%   616Ki [1279 Others]                                      812Ki  39.8%
   0.0%       0 [section .debug_str]                               259Ki  12.7%
   0.0%       0 [section .debug_info]                              227Ki  11.2%
   0.0%       0 [section .debug_ranges]                            107Ki   5.3%
   0.0%       0 [section .debug_pubnames]                         94.6Ki   4.6%
   9.2%  89.5Ki idna::uts46::find_char                            89.5Ki   4.4%
   0.0%       0 [section .debug_line]                             81.1Ki   4.0%
   7.2%  69.9Ki unicode_normalization::normalize::d               70.0Ki   3.4%
   0.0%       0 [section .debug_pubtypes]                         48.4Ki   2.4%
   2.2%  21.3Ki core::ptr::drop_in_place                          43.8Ki   2.1%
   3.3%  32.1Ki rocket::config::RocketConfig::override_from_env   32.2Ki   1.6%
   3.3%  31.9Ki hyper::server::listener::spawn_with::{{closure}}  32.0Ki   1.6%
   2.0%  19.6Ki <yansi::Paint<T> as core::fmt::Display>::fmt      21.6Ki   1.1%
   1.4%  13.2Ki <&'a T as core::fmt::Display>::fmt                18.2Ki   0.9%
   1.7%  16.6Ki rocket::ignite                                    16.6Ki   0.8%
   1.7%  16.2Ki rocket::config::config::Config::set_raw           16.3Ki   0.8%
   1.6%  15.4Ki unicode_bidi::char_data::bidi_class               15.5Ki   0.8%
   0.9%  8.76Ki <&'a T as core::fmt::Debug>::fmt                  14.5Ki   0.7%
   1.1%  10.7Ki <alloc::raw_vec::RawVec<T, A>>::double            14.2Ki   0.7%
   0.0%       0 [section .debug_macro]                            13.2Ki   0.6%
   1.3%  13.2Ki tinyrocket::main                                  13.2Ki   0.6%
 100.0%   974Ki TOTAL                                             2.00Mi 100.0%
[zokier@zarch bloaty]$ 

[luser]

An annoying user-visible piece of this is that Rust symbols have a hash suffix on them (to allow linking multiple major versions of the same crate into the same binary, if desired), you can see the ::hNNNNN suffixes in the output above. This means that trying to use bloaty to do a symbol-wise diff between two versions of a binary that includes Rust code doesn't work, because the same symbols in the binaries can wind up with different hashes. :-|

n.b. the current in-use Rust mangling scheme is backwards-compatible with Itanium C++ name mangling for compatibility reasons. However, there's a Rust Symbol Mangling (v0) RFC which has not yet stabilized that specifies a Rust-specific mangling that is unambiguous.

[kupiakos]

Are there any further updates on this?

[Enselic]

Workaround

As a workaround until this is solved you can use --demangle=none and filter the output through rustfilt which demangles symbols from stdin using the above mentioned rustc-demangle library:

$ cargo install rustfilt
$ bloaty target/release/hello-world -d symbols --demangle=none | rustfilt
    FILE SIZE        VM SIZE
 --------------  --------------
  69.9%   286Ki  67.8%   226Ki    [541 Others]
   4.0%  16.3Ki   4.8%  16.2Ki    std::backtrace_rs::symbolize::gimli::resolve
   3.4%  13.8Ki   4.1%  13.7Ki    std::backtrace_rs::symbolize::gimli::Context::new
   2.5%  10.1Ki   3.0%  10.1Ki    gimli::read::dwarf::Unit<R>::new
   2.5%  10.1Ki   3.0%  10.0Ki    miniz_oxide::inflate::core::decompress
   1.9%  7.68Ki   2.3%  7.55Ki    addr2line::ResUnit<R>::find_function_or_location::{{closure}}
   1.7%  6.92Ki   2.0%  6.86Ki    addr2line::Lines::parse

Improved Symbols

I recommend building with RUSTFLAGS="-Csymbol-mangling-version=v0" because that adds more detail to symbols. In particular it includes information about specific generic arguments. Note that in order to get that kind of symbols for the standard library you need to rebuild the standard library yourself with RUSTFLAGS="-Csymbol-mangling-version=v0". It's easy with -Zbuild-std but it requires a nightly toolchain:

$ RUSTFLAGS="-Csymbol-mangling-version=v0" cargo +nightly build -Zbuild-std=std --release
$ bloaty target/release/hello-world -d symbols --demangle=none | rustfilt
    FILE SIZE        VM SIZE
 --------------  --------------
  77.9%   415Ki  76.6%   300Ki    [821 Others]
   2.0%  10.6Ki   2.7%  10.5Ki    miniz_oxide::inflate::core::decompress
   1.4%  7.54Ki   1.9%  7.45Ki    std::backtrace_rs::symbolize::gimli::resolve
   1.1%  5.86Ki   1.5%  5.70Ki    gimli::read::unit::parse_attribute::<gimli::read::endian_slice::EndianSlice<gimli::endianity::LittleEndian>>
   1.0%  5.58Ki   1.4%  5.39Ki    <addr2line::Context<gimli::read::endian_slice::EndianSlice<gimli::endianity::LittleEndian>>>::parse_units
   1.0%  5.41Ki   1.3%  5.21Ki    <addr2line::function::Function<gimli::read::endian_slice::EndianSlice<gimli::endianity::LittleEndian>>>::parse_children

Common Source of Rust Binary Bloat

I'd like to serve one important insight when it comes to the binary size of a Rust program. In particular a small "hello world" program.

Around 83% of the binary size of a Rust hello world program comes from code to support the nice default Rust backtraces upon panics (both for panic=unwind and panic=abort). In particular there is (assembly) code for parsing binary file sections and debug info. If you build the standard library without that so called "backtrace" feature which is enabled by default, Rust hello world goes from 334 kB to 55 kB:

$ RUSTFLAGS="-Csymbol-mangling-version=v0" cargo +nightly build --release --config profile.release.codegen-units=1 --config profile.release.lto=true --config profile.release.strip=true -Zbuild-std -Zbuild-std-features=
$ ls -l target/release/hello-world
-rwxr-xr-x 2 martin martin 55552 Jan  4 21:31 target/release/hello-world

In the above command we enable no standard library features by using the "empty" features directive -Zbuild-std-features=. The default set of enabled standard library features is "panic-unwind", "backtrace" and "default". It's the "backtrace" feature that contributes significant binary "bloat" for Rust hello world programs. So by not building the standard library with that future we can reduce binary size significantly.

You can reduce binary size even further by using more of the tricks from https://github.com/johnthagen/min-sized-rust than I used in the above example command.

Note that with debug symbols and e.g. gdb (preferably the rust-gdb wrapper) you can of course still obtain backtraces of your programs even if you built without the "backtrace" feature. The "backtrace" feature is basically just embedding a 300-ish kB backtracer in each Rust program.

[cerisier]

Hello everyone, I was wondering what was the state of this.

#112 has been there for a few years now.
It uses the C interface of the native Rust rust_demangle and was done at a time where v0 mangling scheme that differentiates Rust symbols (_R) from C++ symbols (_Z) was not a thing.

Meanwhile, things have evolved and 2 pure C as well a 1 pure C++ implementations are available:

1. https://github.com/rust-lang/rustc-demangle/tree/main/crates/native-c from the official rustc_demangle crate.
2. https://github.com/gcc-mirror/gcc/blob/master/libiberty/rust-demangle.c from GNU libiberty (thx @eddyb).
3. https://github.com/llvm/llvm-project/blob/main/llvm/lib/Demangle/RustDemangle.cpp from llvm.

I would like to suggest 2 decisions to get this moving forward:

1. Decide on which implementation to use (Rust, C++ or one of the 2 C implementation)
2. Decide on demangling strategy (Do we try to demangle legacy rust symbols as well or only v0).

Rust symbol demangling

The v0 Rust mangling scheme is still not the default but is used by many and already supported in a wide variety of tools.
The legacy Rust mangling scheme which intersect with the Itanium C++ ABI it to my knowledge, still not safely differentiable from real C++ symbols.

I would like to suggest 2 approaches to handle this:

1. Demangle v0 symbols by looking for the _R prefix and otherwise fallback to current itanium demangling.
2. Try to demangle v0 and legacy Rust symbols and otherwise fallback to current itanium demangling.

In 2. this means always running rust demangling.
In both case, I suggest that demangling is automatic from the perspective of the user. (no special option for rust).

I'm willing to work on this and continue #112 or create a new PR if we can reach an agreement here.

cc @haberman @eddyb @zokier

[cerisier]

Anyone has an opinion on the above ? I'm more than happy to work on an PR to bring the support.

[the80srobot]

Being a Google project, it's pretty likely the authors have been laid off or moved on. I think @cerisier you might need to fork the project. (I've been having the same problem with multiple Google-led projects.)