fix(install): copy dylib dependencies alongside binaries (#17055)

[sximelon]

What does this PR try to resolve?

Fixes #17055.

When a crate depends on a dylib or cdylib, rustc implicitly enables -C prefer-dynamic, which causes libstd to be linked as a shared library. cargo install only copies the binary to ~/.cargo/bin/, leaving the required .so files behind, so the binary fails at runtime.

This PR addresses the issue with two changes:

1. Copy shared libraries alongside the binary — a new install_dylibs() function collects .so/.dylib files from three sources: root-unit cdylib/dylib targets, a directory scan of root_output and deps_output (for dependency crates), and the sysroot target libdir (for libstd).

2. Fix rpath after install — fix_rpath_for_install() adds $ORIGIN (Linux, via patchelf) or @loader_path (macOS, via install_name_tool) to the installed binary's rpath, so it can find the shared libraries in its own directory. If the tool is not available, a warning is emitted.

How to test and review this PR?

cargo build --bin cargo
./target/debug/cargo install --force cargo_install_dylib_bug

libstd and project dylibs should be installed alongside the binary
ls ~/.cargo/bin/*.so

If patchelf is installed, the binary should work:
cargo_install_dylib_bug

if patchelf is not installed:
LD_LIBRARY_PATH=~/.cargo/bin cargo_install_dylib_bug

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: @ehuss, @epage, @weihanglo
• @ehuss, @epage, @weihanglo expanded to ehuss, epage, weihanglo
• Random selection from ehuss, epage, weihanglo

[epage]

Our contrib ask for issues to be the focus for problem and solution exploration with PRs for implementation discussions. Since the Issue is not S-accepted, I'm moving this to a draft.

[sximelon]

I've investigated this and put together a fix at #17078. Here's what I found:

Root cause: cargo install only copies the binary to ~/.cargo/bin/, ignoring any .so/.dylib files the binary depends on. The Compilation struct already tracks cdylibs (and I added the missing dylibs field), but cargo_install.rs never uses them.

The fix has two parts:

1. Copy shared libraries: a new install_dylibs() function collects .so/.dylib from three sources — root-unit cdylib/dylib targets, root_output + deps_output directory scan (for dependency crates), and the sysroot target libdir (for libstd). These are copied to ~/.cargo/bin/ alongside the binary.

2. Fix rpath: fix_rpath_for_install() runs patchelf --add-rpath $ORIGIN (Linux) or install_name_tool -add_rpath @loader_path (macOS) on the installed binary so it looks for shared libraries in its own directory. If the tool is unavailable, a warning is emitted.

What I tried and why it didn't work:

• Injecting -Clink-arg=-Wl,-rpath,$ORIGIN at compile time via target_rustc_args fails for packages with multiple targets (lib + bin).
• Setting CARGO_ENCODED_RUSTFLAGS doesn't work because cargo reads env vars from a startup snapshot.
• Both approaches also change compilation fingerprints, breaking existing tests.

Would appreciate guidance on whether this approach looks reasonable.

[weihanglo]

@sximelon please discuss in the issue, and I wanted to say this is not a trivial features that we should also look like prior arts from other systems, and then decide how Cargo handles dylibs installation.