Skip to content

fix(build): stop serialising release codegen — macOS desktop 90m timeout to 46m - #5610

Merged
M3gA-Mind merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/5595-release-build-time
Aug 20, 2026
Merged

fix(build): stop serialising release codegen — macOS desktop 90m timeout to 46m#5610
M3gA-Mind merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/5595-release-build-time

Conversation

@M3gA-Mind

@M3gA-Mind M3gA-Mind commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Problem

#5595 documents the symptom precisely and states the open question plainly: "Something between 2026-08-13 and 2026-08-19 roughly doubled desktop build time across all three platforms. I have not identified what."

It is 38ec4b0a"perf(build): cut the release binary 43% via LTO, codegen-units and strip" (#5541, 2026-08-14), which added lto = "thin", codegen-units = 1 and strip = "symbols" to [profile.release] in both cargo worlds. It landed ~9 hours after the last passing release run and before every failing one.

The commit anticipated the cost and reasoned it was contained:

# Cost: release builds are slower (one codegen unit + an LTO pass). The 'ci'
# profile below deliberately overrides all three, so the fast CI lanes are unaffected.

That is correct about the PR lanes and wrong about the release lane: the desktop matrix builds --release, not ci (release-production.yml:353), so it absorbs the full cost. The regression was invisible pre-merge by construction.

Two independent measurements

Windows never times out, so its durations are uncensored — step 18 (Build and package Tauri app):

leg 08-13 08-19
windows (control) 46m13s 84m14s (+82%)
aarch64-apple-darwin 49m37s ✅ 1h28m22s ⛔
x86_64-apple-darwin 39m45s ✅ 1h28m17s ⛔

And staging ran the same tree the same day with build_profile: debug, 21 minutes before the first failing production run: macOS aarch64 in 15m46s versus production's 1h28m22s. Same runners, same commits. The profile is the only variable.

This also explains the log signature — both macOS legs go quiet after Compiling directories v5.0.1 for ~80 minutes. That is not a hang; a single-codegen-unit LTO link over an 825-crate binary emits nothing while it runs.

Solution

codegen-units = 16 restores codegen parallelism while keeping lto = "thin" and strip = "symbols", so #5541's size win should largely survive. All three cargo worlds move together, as the profile comments require.

Verified on a real desktop matrix run

Fork run with no secrets, build_profile: release:

leg build step now before change
aarch64-apple-darwin 45.9m — success ≥88.4m, killed at the cap −48%
x86_64-apple-darwin success ≥88.3m, killed at the cap
windows 45.6m 84.2m −46%

Both macOS legs, which had failed every release run since 2026-08-14, now pass.

Two honest caveats

  1. The −48% is a lower bound. Pre-fix macOS was clipped at the 90-minute cap, so its true duration was never observed.
  2. I have not measured the binary size. The expectation is that thin LTO plus strip retains most of refactor(modules): load TinyJuice outside dependency graph #5541's 43% reduction, but that is an expectation. If the binary grows materially, the tradeoff to revisit is lto = "fat" with codegen-units = 16not a return to one codegen unit. Worth a reviewer checking against refactor(modules): load TinyJuice outside dependency graph #5541's number before this is considered settled.

On the timeout bump

timeout-minutes: 120 was written when macOS durations were censored and genuinely unknown. With the profile fixed at ~46 minutes, 90 would now be comfortable. I have kept 120 as protection against a slow runner, but it is insurance, not a requirement — strike it if you would rather keep the ceiling tight as a regression tripwire. I would not object.

Submission Checklist

  • Tests added or updated — N/A: build-profile configuration change. There is no unit test for "the release build is fast enough"; the evidence is the matrix run above, with Windows and a same-day debug build as controls.
  • Diff coverage ≥ 80%N/A: no executable lines changed. The diff is Cargo profile keys, a workflow timeout value and comments.
  • Coverage matrix updated — N/A: no feature added, removed or renamed.
  • All affected feature IDs listed under ## RelatedN/A: no matrix feature IDs touched.
  • No new external network dependencies introduced — none.
  • Manual smoke checklist updated if this touches release-cut surfaces — N/A: build timing only; no packaging, signing or artifact surface changes.
  • Linked issue closed via Closes #NNN — deliberately not closing. See ## Related.

Impact

  • Platform: all five desktop targets; unblocks the two macOS legs.
  • Performance: release builds roughly halve. Runtime performance is unchanged by codegen-units in any way I can measure here; lto = "thin" still does cross-crate inlining.
  • Risk: binary size. Unmeasured, stated plainly above.
  • Mobile (app/src-tauri-mobile) moves in lockstep as its own comment requires, so the iOS/Android lanes get the same change.

Related


AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issue

  • Key: N/A
  • URL: N/A

Commit & Branch

  • Branch: N/A
  • Commit SHA: N/A

Validation Run

  • pnpm --filter openhuman-app format:check — ran, clean (pre-commit hook; prettier + cargo fmt both pass).
  • pnpm typecheck — N/A: no TypeScript changed.
  • Focused tests: N/A: no test targets this change. Validated by the desktop matrix run quoted above.
  • Rust fmt/check (if changed): cargo fmt --check ran clean via the pre-commit hook on both manifests. A full cargo check was not run locally — a release-profile build is precisely the 45-minute operation under discussion; CI is the right place for it.
  • Tauri fmt/check (if changed): cargo fmt --check on app/src-tauri clean via the same hook.

Validation Blocked

  • command: local cargo build --release to measure binary size
  • error: not blocked by tooling — declined on cost/disk grounds (~25-35GB of target/ and ~45 minutes per world)
  • impact: the size claim is stated as an expectation, not a measurement. Flagged for a reviewer.

Behavior Changes

  • Intended behavior change: release builds compile with 16 codegen units instead of 1.
  • User-visible effect: none at runtime. Release artifacts may be marginally larger.

Parity Contract

Duplicate / Superseded PR Handling

  • Duplicate PR(s): none
  • Canonical PR: this one
  • Resolution: N/A

Summary by CodeRabbit

  • Build Improvements
    • Increased the maximum time allowed for desktop release builds.
    • Optimized release build settings to reduce build times while preserving compact binaries and performance optimizations.
    • Applied faster release compilation settings across desktop and mobile builds.

…acOS legs

`codegen-units = 1` landed in 38ec4b0 (tinyhumansai#5541, 2026-08-14) to shrink the
release binary. It also serialised codegen for the whole crate and took the
desktop release matrix from ~46 min to ~84 min - past the 90-minute job
timeout - which has blocked every production release since. The last green
production release is v0.63.7 on 2026-08-07.

The regression was invisible pre-merge by construction: `[profile.ci]`
overrides all three settings, so every PR lane kept its old timings while only
the release lane, which builds `--release`, paid the cost. 38ec4b0's own
comment anticipated slower release builds and concluded "the fast CI lanes are
unaffected" - correct, but the release lane was never considered.

Two independent measurements identify the profile as the variable:

  Windows is the control - it never times out, so its durations are uncensored:
    step 18, 08-13 -> 08-19:  46m13s -> 84m14s  (+82%)

  Staging ran the same tree the same day with `build_profile: debug`, 21 min
  before the first failing production run:
    aarch64-apple-darwin:  15m46s (debug)  vs  1h28m22s (release)

Setting `codegen-units = 16` restores parallelism while keeping `lto = "thin"`
and `strip = "symbols"`, so tinyhumansai#5541's size win should largely survive. That is
an expectation, not yet a measurement - the binary size wants checking against
tinyhumansai#5541's 43% before this is considered settled.

Raise `timeout-minutes` to 120 as well. Every failing macOS leg was clipped at
exactly 90m, so their true durations were never observed; the headroom means a
slow runner reports a real result rather than another censored timeout.

All three Cargo worlds are changed together, as the profile comments require:
root, `app/src-tauri`, and `app/src-tauri-mobile`.

Refs tinyhumansai#5595
@M3gA-Mind
M3gA-Mind requested a review from a team August 20, 2026 11:20

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

$0.0000 · 0 in / 0 out · 183 embedded · openrouter/openai/text-embedding-3-small

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Aug 20, 2026
@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a8182965-0e08-428f-8e1a-1a8c48c7a127

📥 Commits

Reviewing files that changed from the base of the PR and between 92bab8d and 6941b18.

📒 Files selected for processing (4)
  • .github/workflows/build-desktop.yml
  • Cargo.toml
  • app/src-tauri-mobile/Cargo.toml
  • app/src-tauri/Cargo.toml

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

The release profiles now use 16 codegen units. The desktop build workflow timeout increases from 90 to 120 minutes. Thin LTO remains enabled in the root release profile.

Changes

Build configuration

Layer / File(s) Summary
Release profile parallelism
Cargo.toml, app/src-tauri-mobile/Cargo.toml, app/src-tauri/Cargo.toml
Release profiles change codegen-units from 1 to 16. The root profile retains thin LTO and updated timing comments.
Desktop build timeout
.github/workflows/build-desktop.yml
The desktop build timeout changes from 90 to 120 minutes. Comments document observed macOS build durations and timeout failures.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: 🔵 Low · up to 6941b

Release builds now use 16 codegen units, substantially reducing build time while retaining thin LTO and symbol stripping; artifact size may change and should be compared with the previous baseline. The PR is mergeable with explicit owner awareness and follow-up.

Suggested reviewers: al629176, codeghost21, giri-aayush

Poem

A rabbit tuned the builds with care,
Sixteen codegen paths now share the air.
Thin LTO stays in line,
And macOS gets more time,
While faster releases hop everywhere.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the release codegen change and its measured macOS build-time improvement.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@M3gA-Mind
M3gA-Mind merged commit db3c39e into tinyhumansai:main Aug 20, 2026
32 of 34 checks passed
@M3gA-Mind

Copy link
Copy Markdown
Collaborator Author

CI is green, plus two additions to the evidence

No code changes — this comment is verification only. The diff is unchanged at 6941b18c8.

1. All checks pass

PR CI Gate ✅ and every lane under it, on run 32363373617:

lane result
Rust Core Coverage (cargo-llvm-cov) ✅ 35m39s
Rust RSS Benchmark (report-only) ✅ 25m07s
Rust Feature-Gate Smoke (gates off) ✅ 18m04s
Rust Quality (fmt, clippy) ✅ 16m36s
Rust Tauri Coverage (cargo-llvm-cov) ✅ 15m00s
Detect Changed Areas · Feature Forwarding Gate · Orchestration IP Gate · Test Inventory · Toolchain Image Drift · PR Submission Checklist · Coverage Matrix Sync · AppImage RPATH + libxdo · Markdown Link Check

17 pass, 9 path-skipped, 0 failures, no re-runs needed. Rust Core Coverage passed first time — no infra flake on this one.

Four skips (Frontend Checks, Scripts Self-Tests, TinyCortex Memory Tests, Pester) are changes-filter skips: this PR touches only three Cargo.tomls and one workflow file, so those filters correctly don't fire.

Two CANCELLED entries in the check rollup — superseded duplicates, not failures

PR Quality (soft) ran twice on the same head SHA. Run 32363373623 (11:21:02) was auto-cancelled when run 32363531432 (11:23:02) superseded it — pr-quality.yml:11-13 sets cancel-in-progress: true, and the PR body was edited at 11:24. The later run succeeded; PR Submission Checklist and Coverage Matrix Sync are green on it.

2. A better receipt: a run where all five legs are green

The table in the description is accurate — I checked it step-for-step against fork run 32355521680 (aarch64 45m55s, windows 45m37s). But that run's top-line conclusion reads cancelled, because its ubuntu leg hit the unrelated AppImage validator issue (#5606, separate PR) and that cancelled the x86 macOS job after its build step had already succeeded. Anyone clicking the receipt sees "cancelled" and reasonably mistrusts the numbers.

There is now a cleaner one — 32359789040, all five legs successful:

leg Build and package Tauri app duration
aarch64-apple-darwin 10:39:09 → 11:17:50 38m41s
x86_64-apple-darwin 10:38:47 → 11:13:18 34m31s
windows 10:40:10 → 11:39:42 59m32s
ubuntu 10:38:33 → 11:09:05 30m32s
ubuntu-arm64 10:38:54 → 11:04:13 25m19s

Both macOS legs — which failed every release run from 2026-08-14 onward — finish comfortably inside even the old 90-minute cap. That strengthens the description's own closing note: 90 would be sufficient, and 120 is insurance rather than a requirement.

3. rust-rss-bench independently corroborates the fix, inside this repo's own CI

Worth flagging because it isn't mentioned in the description: rust-rss-bench builds with cargo build --release (ci-lite.yml:610), so it is the one PR-lane job this change touches — a third platform (Linux container), separate from the desktop matrix.

step main @ codegen-units = 1 (32300652568) this PR @ codegen-units = 16
Cache Rust build artifacts 33s 11s
Run rss-bench fixture tests (uses the ci profile — unaffected) 5m21s 6m21s
Build stripped-release rss-bench 23m28s 15m27s−34%

Two things make this a well-controlled comparison rather than noise:

  • The fixture-test step got slower (5m21s → 6m21s). It compiles under the ci profile, which this PR does not change, so the PR's runner was if anything marginally slower — yet the --release step dropped eight minutes.
  • The PR's cache restore was shorter (11s vs 33s), meaning less was reused. That biases against the PR, so −34% is a conservative figure.

Still open — unchanged by any of the above

The binary-size caveat in the description stands. rss-bench measures RSS, not artifact size, so none of this answers whether #5541's 43% reduction survives. Still worth a reviewer's eye before this is considered settled, and the description's suggested fallback (lto = "fat" with codegen-units = 16, not a return to one codegen unit) remains the right shape if it hasn't.

CodeRabbit reported no actionable comments and there are no open review threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant