Skip to content

ci(nightly): drop the musl linker override so rustc keeps the link (#683) - #717

Open
logbie wants to merge 1 commit into
mainfrom
warden/drop-musl-linker-override
Open

ci(nightly): drop the musl linker override so rustc keeps the link (#683)#717
logbie wants to merge 1 commit into
mainfrom
warden/drop-musl-linker-override

Conversation

@logbie

@logbie logbie commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Fixes #683.

nightly.yml's build-linux job sets both CC_x86_64_unknown_linux_musl and CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER to musl-gcc. The second is unnecessary and actively harmful: musl-gcc is a spec-file wrapper around the host gcc, and its specs select musl's dynamic loader. On an image where that holds, the job silently produces a binary with a PT_INTERP segment while still going green — the exact condition #616 was closed to prevent — and -C target-feature=+crt-static cannot win the link back once musl-gcc is driving it.

Rust's x86_64-unknown-linux-musl target ships its own self-contained musl libc and static-links by default, so removing the override moves the build toward rustc's default rather than away from it.

What changed

One deleted line, plus a comment recording why the pairing must not come back.

       TARGET: x86_64-unknown-linux-musl
       CC_x86_64_unknown_linux_musl: musl-gcc
-      CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER: musl-gcc

CC_ is kept deliberately — cc-rs needs it to compile aws-lc-sys' C and assembly for the musl target, and that is the one dependency in the graph with a musl story to get wrong.

Deliberately not included

The issue also floats adding CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_RUSTFLAGS: "-C target-feature=+crt-static". I left it out. It is redundant with the target default, and it introduces a second rustflags source that has to union correctly with the cfg(target_os = "linux") block in .cargo/config.toml carrying the 8 MB stack-size link-arg the parser depends on. The issue author verified that union empirically, but it turns a pure deletion into a change with a non-obvious interaction, and the parser's stack budget is not worth risking for redundancy. The comment states the intent instead. Happy to add it if you'd rather have it explicit.

Risk and verification

Risk class R0 — CI mechanics, no behavioural change to the language. No src/ change, no TestPrograms/ exposure, no version touched. Per testing.md this needs no manufactured failing test: the existing check is the test.

Verified locally:

  • actionlint 1.7.7 on nightly.yml: zero findings, identical to main before the change.
  • YAML parses; build-linux.env now resolves to exactly {TARGET, CC_x86_64_unknown_linux_musl, CARGO_PROFILE_RELEASE_DEBUG}.
  • No MUSL_LINKER reference remains anywhere in the tree outside the explanatory comment.

What I could not verify, stated plainly: build-linux runs only in nightly.yml, and there is no PR-triggered musl lane — .github/workflows/verify-linux-musl.yml is registered but absent from the tree. So PR CI cannot exercise this job, and the authoritative proof is the Assert the binaries are statically linked step (PT_INTERP absence on both wfl and wfl-lsp) on the first full nightly after merge. That step is unchanged and is the correct detector.

I did not try to prove it by dispatching nightly.yml from this branch, because the release job is guarded only by should_build == 'true' with no github.ref condition — it holds contents: write and read-modify-writes the nightly release plus the Spaces rolling pointers, SHA256SUMS and status.json. A branch dispatch would have published unmerged artifacts over the real nightly. That gap is written up in #683 and deserves its own fix.

Since the next nightly is the verification, it is worth a glance at tomorrow's build-linux log after this merges. If PT_INTERP ever does appear, the assertion fails loudly and this is trivially revertible.

Opened by the WFL repo warden (automated maintenance pass). Not merged by me — CI and a human decide.


Open in Devin Review

Summary by CodeRabbit

  • Bug Fixes
    • Improved Linux musl build configuration to support more reliable compilation.
    • Preserved Rust’s default musl linker behavior while configuring the C compiler.

CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=musl-gcc hands the musl
link to musl-gcc, a spec-file wrapper around the host gcc whose specs
select musl's dynamic loader. Under it the build can silently emit a
binary with a PT_INTERP segment while still going green, and
-C target-feature=+crt-static cannot win the link back. That is exactly
the condition #616 was closed to prevent.

Rust's x86_64-unknown-linux-musl target ships its own self-contained
musl libc and static-links by default, so removing the override moves
the build toward rustc's default rather than away from it. CC_ stays:
cc-rs genuinely needs it to compile aws-lc-sys for the musl target.

The behaviour is currently benign on blacksmith-8vcpu-ubuntu-2404, so
this is a latent dependency on an unpinned property of the runner image
rather than a live breakage. The "Assert the binaries are statically
linked" step (PT_INTERP absence, both binaries) remains the detector and
is unchanged.

Closes #683
Copilot AI lite review requested due to automatic review settings August 17, 2026 09:07
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aa305038-0dca-4bff-92f3-9a5dd626cc20

📥 Commits

Reviewing files that changed from the base of the PR and between 36de4fa and 13abffc.

📒 Files selected for processing (1)
  • .github/workflows/nightly.yml

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The nightly Linux musl build continues to use musl-gcc for C compilation. It no longer configures musl-gcc as Rust’s linker, so Rust uses its default musl linker configuration.

Changes

Linux musl build

Layer / File(s) Summary
Musl linker configuration
.github/workflows/nightly.yml
The workflow keeps CC_x86_64_unknown_linux_musl=musl-gcc and removes CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=musl-gcc.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 13abf

This is a localized nightly CI configuration change with no application-code impact; no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: removing the musl linker override from the nightly build.
Linked Issues check ✅ Passed The PR removes the unnecessary linker override, retains CC_ for aws-lc-sys, and preserves the static-linkage assertion required by issue #683.
Out of Scope Changes check ✅ Passed The changes are limited to the nightly workflow and its explanatory comment, with no unrelated source, test, or version changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch warden/drop-musl-linker-override

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Open in Devin Review

Comment on lines +468 to 480
# cc-rs needs this so aws-lc-sys' C and assembly compile for the musl
# target. It is deliberately NOT paired with a matching
# CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER. That variable would hand
# the link to musl-gcc - a spec-file wrapper around the host gcc whose
# specs select musl's DYNAMIC loader - so the build silently emits a
# binary carrying a PT_INTERP segment, and `-C target-feature=+crt-static`
# cannot win the link back once musl-gcc is driving it. Rust's
# x86_64-unknown-linux-musl target ships its own self-contained musl and
# static-links by default, so the correct move is to leave the link to
# rustc rather than take it away. Dropping that override is wfl#683; the
# regression it prevents is wfl#616 reappearing. The `Assert the binaries
# are statically linked` step below is what proves this held.
CC_x86_64_unknown_linux_musl: musl-gcc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Removing the linker override changes which compiler drives the link, and the stack-size link-arg must still pass through

With CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER gone, rustc falls back to its default linker driver (cc, i.e. the host gcc on the Blacksmith image) combined with the self-contained musl objects it ships. That path must still accept the -C link-arg=-Wl,-z,stack-size=8388608 rustflag from the cfg(target_os = "linux") block in .cargo/config.toml:10-11, which the musl target matches — the parser depends on that 8 MB stack. Host cc passes -Wl,... through unchanged, so the argument survives, but nothing in CI asserts the resulting stack-size program header, so a silent regression here would only show up as deep-recursion parser failures at runtime rather than as a red build. Worth considering an assertion on the PT_GNU_STACK/stack-size value alongside the existing PT_INTERP check.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +468 to +479
# cc-rs needs this so aws-lc-sys' C and assembly compile for the musl
# target. It is deliberately NOT paired with a matching
# CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER. That variable would hand
# the link to musl-gcc - a spec-file wrapper around the host gcc whose
# specs select musl's DYNAMIC loader - so the build silently emits a
# binary carrying a PT_INTERP segment, and `-C target-feature=+crt-static`
# cannot win the link back once musl-gcc is driving it. Rust's
# x86_64-unknown-linux-musl target ships its own self-contained musl and
# static-links by default, so the correct move is to leave the link to
# rustc rather than take it away. Dropping that override is wfl#683; the
# regression it prevents is wfl#616 reappearing. The `Assert the binaries
# are statically linked` step below is what proves this held.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 No PR-triggered lane exercises this job, so the change is unverified until the next nightly

build-linux exists only in nightly.yml and is gated on check-for-changes.outputs.should_build == 'true', so no pull-request event can run it. The Assert the binaries are statically linked step is therefore the sole detector and it first executes after merge. If the default-linker path fails outright (e.g. host cc cannot find the self-contained musl objects), the nightly build breaks rather than degrading, which is the safer failure mode, but the merge is effectively unvalidated at review time.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the Linux musl nightly build workflow to rely on rustc’s default musl linking behavior (static by default) by removing a harmful linker override that can silently produce binaries with a PT_INTERP segment.

Changes:

  • Removed CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER: musl-gcc from the build-linux job environment.
  • Added an explanatory comment documenting why CC_x86_64_unknown_linux_musl is required (for cc-rs/aws-lc-sys) while the linker override must not be reintroduced.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13abffc9a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +477 to +479
# rustc rather than take it away. Dropping that override is wfl#683; the
# regression it prevents is wfl#616 reappearing. The `Assert the binaries
# are statically linked` step below is what proves this held.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exercise the changed linker path before merge

The build-linux job was checked and is only exercised by this scheduled/manually dispatched nightly workflow, so the unchanged static-link and Debian smoke checks do not validate this new linker selection until after it reaches main. If rustc's inferred self-contained setup cannot link the musl-gcc-built native archives on the runner, the first full nightly fails after merge; moreover, the commit's stated R0 classification is incorrect because deleting the linker override changes executable build behavior. Add a non-publishing PR or disposable staging lane that builds both binaries through this path and runs the existing PT_INTERP and Debian checks before relying on the default linker.

AGENTS.md reference: AGENTS.md:L129-L137

Useful? React with 👍 / 👎.

@logbie

logbie commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

CI is complete and fully green: 18 substantive checks pass, 2 skipped (Bump Version, which only runs on main, and claude). MERGEABLE / CLEAN.

Notably green: Repository Hygiene on both lanes and config-lint — which closes the one residual risk I flagged when opening this, since scripts/check_repo_hygiene.py needs Python 3.11+ (tomllib) and could not be run locally. CI has now proven it.

The verification limit stated in the description is unchanged and worth repeating before merge: none of these 18 checks exercise build-linux. That job exists only in nightly.yml, so the musl linkage property this PR is about is not covered by any of the green above. The authoritative check is the Assert the binaries are statically linked step (absence of PT_INTERP on both wfl and wfl-lsp) on the first full nightly after this merges.

One scheduling note: the last three nightlies were designed no-change skips, because main has not moved since 36de4fa7 on 08-14. Merging this will itself change main, so the next nightly should run a full build and exercise the job — no manual dispatch needed, and a manual dispatch from a branch is the thing to avoid here anyway (see the unguarded release job noted in #683).

Posted by the WFL repo warden (automated maintenance pass).

@logbie

logbie commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Stale-PR check, 2026-08-19 warden pass. This PR is now ~48h idle with no blocker on it — 18 substantive checks pass, 2 skipped (Bump Version, main-only; claude, opt-in), MERGEABLE/CLEAN, zero review requests outstanding. Nothing here is waiting on the author.

The reason for the nudge rather than silence is that the situation around it has changed. main has not moved since 2026-08-14T18:43Z — 110 hours — and this morning's nightly was the fifth consecutive no-change skip (should_build=false, run 32218549080). The last real full build was 2026-08-15 (run 31866229680), and there has been no published nightly artifact since nightly-2026-08-15. So the repo's green board is currently green partly because nothing is exercising it.

That matters specifically for this PR. The defect it fixes — CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=musl-gcc letting rustc hand the link to a wrapper that can silently emit a dynamically linked binary (#683) — is latent, not active: on the 2026-08-15 nightly the "Assert the binaries are statically linked" step passed. That is precisely why it is worth merging now rather than after it bites. The assertion passing today is a property of the runner image, not of the configuration; the configuration is what this PR makes correct. Merging it also silently un-reintroduces the #616 glibc-floor regression risk that the musl lane exists to prevent.

Next action: maintainer merge. There is no author-side work left. Once merged, main moves, the next scheduled nightly does a real build, and the static-linkage assertion gets exercised against the corrected config — which is the verification that has been carried forward unresolved for four passes now.

Posted by the WFL repo warden (automated triage pass).

@logbie

logbie commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

Still open and still green — 24h since the last note, and this is the fourth pass flagging it. Rather than repeat yesterday's status, here is the one thing I could check today that nobody had checked yet: whether the three open warden PRs can be merged as a batch. They can.

Test-merged locally against main @ 36de4fa7:

order result
#717#718 clean (Auto-merging .github/workflows/nightly.yml)
#718#717 clean
#716 + #717 + #718 together clean, exit 0

#717 and #718 are the two that share .github/workflows/nightly.yml, so that was the real question. #717's hunk is the env: block of the musl job (~L465); #718's is the packaging step (~L550). They are far enough apart that git resolves them without help, in either direction. There is no merge order to get right, and no rebase needed on whichever lands second.

And the combined tree is lint-clean. actionlint 1.7.7 reports 5 findings on main today — all of them actions/setup-python@v4 node16-runner deprecation (ci.yml:49,223,394,697, versioning.yml:28) — and 0 findings on the all-three merge. That is exactly the set #716 removes; nothing new has appeared underneath it.

Current state of all three: each is MERGEABLE, 18 checks passing, 2 skipped (Bump Version, claude), no conflicts, no drift — main has not moved since 2026-08-14T18:43Z, so the check runs are still evaluated against the same tree they were run against.

Why this matters beyond tidiness: the scheduled nightly has now taken five consecutive designed no-change skips (08-16 → 08-20) because should_build=false on a main that has been frozen 134 hours. The last full nightly build and published artifact is nightly-2026-08-15. These three PRs are the only things in the queue that would move main and put a real build back on the board — which is also the only way to re-verify the static-linkage assertion this PR exists to protect, and to confirm #718's reproducible-tarball fix in the environment that actually publishes.

No action needed from the author; this is a maintainer merge decision. Deliberately posted once, here, rather than duplicated onto #716 and #718 — the finding is about the queue, not about any single PR.

Posted by the WFL repo warden (automated triage pass).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly.yml: CARGO_TARGET_..._MUSL_LINKER=musl-gcc can silently produce a dynamically-linked binary (re-opens #616 conditions)

2 participants