Skip to content

Build aarch64 packages on native ARM64 runners - #171

Open
scottjones wants to merge 3 commits into
omacom:masterfrom
scottjones:aarch64-ci
Open

Build aarch64 packages on native ARM64 runners#171
scottjones wants to merge 3 commits into
omacom:masterfrom
scottjones:aarch64-ci

Conversation

@scottjones

Copy link
Copy Markdown
Contributor

Build aarch64 packages on native ARM64 runners

Omarchy runs on Apple Silicon today through @malik-na's
omarchy-mac, and the one thing that makes an
install slow is that every AUR package is compiled on the laptop. pkgs.omarchy.org serves
no aarch64 tree, so there is nothing to download.

The interesting part is that this repo is already set up for it. bin/build, bin/sign,
bin/update-repo and bin/sync-repo all take --arch; build/Dockerfile bootstraps an
Arch Linux ARM rootfs with the ALARM keyring and [alarm]/[aur] repos; sync-repo writes
to $MIRROR/$ARCH; and should_build_for_arch() already skips packages whose arch=()
excludes aarch64. omarchy-keyring is even published for aarch64 so the ARM builder image
can bootstrap. What is missing is something that runs it.

What this adds

A Build aarch64 Packages workflow on ubuntu-24.04-arm — GitHub's ARM64 runners, free
for public repositories. The README currently tells you to emulate ARM64 on an x86_64 host
with QEMU and warns it is slower; a native runner removes the emulation.

It is workflow_dispatch only, so it costs nothing until someone triggers it, and takes an
optional package list so a full build can be run in batches.

It deliberately does not publish. Signing and syncing need the repository key and the
pkgs.omarchy.org credentials, so the workflow uploads packages as an artifact and leaves
the maintainer steps alone:

bin/repo sign --arch aarch64
bin/repo promote --arch aarch64
bin/repo sync --arch aarch64

PKGBUILD changes

tensaku and tzupdate build cleanly on aarch64 — both are plain Rust builds with nothing
architecture-specific — but carry arch=('x86_64'). Both are AUR-synced, so the change is a
.omarchy/patches/ entry alongside the PKGBUILD edit, matching how umu-launcher and others
handle local modifications, so the next sync-aur does not revert it.

These two are not a guess: both are built and running on an M2 Max at the same versions in
this PR (tensaku 0.26.7, tzupdate 3.1.0).

Coverage

Of the 115 packages in pkgbuilds/, 76 need no change at all — 52 already declare
aarch64 and 24 are arch=any. Most of the remaining 39 are legitimately x86-only
(nvidia-580xx-utils, intel-ipu7-camera, asusctl, supergfxctl,
macbook12-spi-driver-dkms) or proprietary binaries with no ARM build.

One build-system fix came out of this

The first CI run failed on the very first package, and the cause is worth flagging beyond
this PR. bin/build bind-mounts build-output/ and pkgs.omarchy.org/ into the builder
container, which works as the image's builder user — uid 1000, from the useradd in
build/Dockerfile. A GitHub runner is uid 1001, and make_dir_writable() chowns those
directories to the host user, so the container loses write access. repo-add then fails
to create the incremental omarchy-build.db — silently, because its output goes to
/dev/nullpacman -Sy cannot open it, and every package dies with "Could not resolve
all dependencies".

This is invisible on a workstation, where the developer is usually uid 1000 as well and the
two users coincide. It will affect anything that automates this repo on any runner whose uid
differs, on either architecture. The workflow opens the modes before bin/build runs, since
chown preserves them; a deeper fix in make_dir_writable() felt out of scope for this PR,
as it would change behavior for the existing x86_64 flow.

Verification

Run on a native ARM64 runner, not described in theory:

  • The runner-architecture guard passed (uname -m = aarch64), so no emulation was involved.
  • bin/build --arch aarch64 --package tzupdate reported Built: 1 Failed: 0 in roughly two
    minutes.
  • The artifact contains tzupdate-3.1.0-1-aarch64.pkg.tar.xz, whose .PKGINFO reads
    arch = aarch64 and whose /usr/bin/tzupdate is an ELF 64-bit LSB pie executable, ARM aarch64.

CI run for the verification above: https://github.com/scottjones/omarchy-pkgs/actions/runs/32306145100

The build system already supports aarch64 end to end -- bin/build,
bin/sign, bin/update-repo and bin/sync-repo all take --arch, the
Dockerfile bootstraps an Arch Linux ARM rootfs, and omarchy-keyring is
published for aarch64 so that image can bootstrap -- but nothing runs
it, so pkgs.omarchy.org serves no aarch64 tree.

GitHub's ARM64 runners are free for public repositories, so the build
needs no QEMU. The workflow is dispatch-only and takes an optional
package list, and it stops at uploading artifacts: signing and syncing
need credentials only a maintainer has.
The builder container works as its own uid 1000 user, while a GitHub
runner is uid 1001, and make_dir_writable() chowns the mounted output
directories to the host user. The container then cannot write its
incremental omarchy-build database, pacman -Sy fails to open it, and no
makedepends resolve -- the build dies on the first package.

This is invisible on a workstation, where the developer is uid 1000 too.
@scottjones

Copy link
Copy Markdown
Contributor Author

There are now eight open aarch64 PRs on this repo and we're starting to collide. Posting a map here since this PR is the one several of the others already reference.

How they fit

Layer PR Author
Native ARM64 runner #171 (this one) @scottjones
Reusable repo artifact / omarchy.db publication #223 @birkskyum
Recipes + builder toolchain #240 @riverscn
arch=() flips #195, #197, #244 @oceanapplications, @andrewpatrickflynn
Platform bring-up #221 (Snapdragon fw), #222 (Limine) @birkskyum

These are complementary rather than competing. #223 says so directly — it carries this branch's commits so it can be exercised today, and plans to rebase to only its follow-up commits once this merges. #240 is the mirror image: its docs/aarch64-follow-up.md explicitly defers "native AArch64 production scheduling", states it "does not add an emulated GitHub Actions job", and asks that release acceptance be performed on a native AArch64 builder. That is exactly what this PR adds, and it isn't emulated — ubuntu-24.04-arm, free for public repos.

Where we actually conflict

Until a few minutes ago three PRs flipped arch=() on the same two packages, under three different patch filenames for an identical change:

Package #171 #195 #240
tensaku build-for-aarch64.patch add-aarch64-arch.patch aarch64.patch
tzupdate build-for-aarch64.patch add-aarch64-arch.patch aarch64.patch

I've removed those from this PR (see below), so what remains is #195 and #240, which also overlap on asdcontrol, github-copilot-cli, hyprland-preview-share-picker, omasnap, qmk-hid, symfony-cli and v4l2-relayd.

The diffs agree — arch=('x86_64')arch=('x86_64' 'aarch64') in every case. It's a naming and ownership problem, not a design disagreement.

Proposed sequencing

  1. Done — I've dropped the tensaku/tzupdate changes from this PR. It's now two commits and two files (.github/workflows/build-aarch64.yml and a README note), +154/-1, with no file in common with Add aarch64 to arch=() for packages that build natively on ARM64 #195 or Add native AArch64 package build support #240. The remaining overlap with Publish a reusable aarch64 repository artifact #223 is the workflow itself, which is the dependency rather than a conflict.
  2. Merge this, so Publish a reusable aarch64 repository artifact #223 can rebase down to its own commits.
  3. Let Add native AArch64 package build support #240 own the arch=() flips and settle on its aarch64.patch convention, with Add aarch64 to arch=() for packages that build natively on ARM64 #195/1password: build for aarch64 #197/grok-bot: aarch64 + 0.30.0 Linux deb names #244 converging on it.

Happy to reorder if maintainers would rather take #240 first — the point is mainly that these want an order rather than parallel resolution of the same files.

@omarchybot

Copy link
Copy Markdown
Collaborator

Re-reviewed at 60d8452 (previous pass was 913d235). Reviewed by Claude Opus 5 and by Codex at xhigh reasoning as an independent second reviewer; where the two agree below I say so, and Codex's independence is not currently guaranteed, so agreement is worth less than the two findings it contributed on its own.

What changed since the last pass. The branch was rebased rather than extended: 913d235 is no longer an ancestor of 60d8452, the "Build tensaku and tzupdate for aarch64" commit was dropped, and the two remaining commits are byte-identical in content to the ones already reviewed. The whole pkgbuilds/ tree at this head is byte-identical to the merge base — no orphaned .omarchy/patches entry, no PKGBUILD referencing a patch that is no longer on disk, and arch=('x86_64') restored in both tensaku and tzupdate. Codex checked every .patch mention in every PKGBUILD against the files on disk and found no dangling reference. So there is no new code in this delta, and two earlier findings are now moot: the collision with #9 and #195 over the same two arch=() flips is gone, and so is the concern about those patches relying on patch default fuzz to survive an AUR bump. The map in your comment checks out with one correction — this PR and #240 still share README.md.

Still open, and new.

  1. .github/workflows/build-aarch64.yml:14-16 (medium) — the maintainer publication recipe in the header comment does not publish a usable repository. bin/promote-build:172-174 deliberately skips omarchy-build.db* when moving packages into production, and bin/update-repo:34-39 is the only thing that runs build/update-repo.sh, whose repo-add at line 66 is what produces omarchy.db. Following signpromotesync as written uploads package files and no database, so pacman clients pointed at the aarch64 tree cannot see any of it. The recipe also omits --mirror, and helpers/paths.sh:11-19 defaults MIRROR=edge, so promoting a stable artifact would read the wrong paths. Adding bin/repo update --arch aarch64 --mirror <mirror> before sync, and threading --mirror through all four, is the fix — but it is your publish flow and we have not run it, so we have left the file alone rather than push a recipe we could not exercise.

  2. .github/workflows/build-aarch64.yml:41-43 (low) — the suggested fallback of driving the build "in batches with the packages input" is not dependency-complete. build/build.sh:440-442 selects only the packages named, build/build.sh:511 counts a dependency only when it is also in the selected set, and build/build.sh:60-61 configures the production [omarchy] repo only when a database already exists — which on a clean runner it does not. pkgbuilds/omarchy/PKGBUILD:32 hard-pins omarchy-settings=${pkgver}, so a batch containing omarchy without omarchy-settings fails at makepkg -s. The unscoped default build is unaffected; it is the batching advice that would not work as described. Codex found both of these; neither was in our first pass.

  3. .github/workflows/build-aarch64.yml:129-136 (low, carried over) — path: logs/ can never match. helpers/paths.sh:11 puts LOG_DIR at logs/, and bin/repo:26,39 is the only thing that creates or writes it; this workflow calls bin/build directly. With if-no-files-found: ignore the step is silent, so a failed build produces no diagnostic artifact at all despite the step suggesting otherwise.

  4. .github/workflows/build-aarch64.yml:69-87 (low, carried over) — the chmod -R 777. The diagnosis is right, and we traced it: helpers/docker-helpers.sh:62-69 runs sudo chown -R $(id -u):$(id -g), which succeeds on a runner with passwordless sudo, leaving the bind-mounted directories owned by uid 1001 while build/Dockerfile ends USER builder at uid 1000; chown does not touch the mode, so opening it first is what lets both write. On the pinned ubuntu-24.04-arm label each job gets a fresh single-tenant VM that is destroyed afterwards, so 0777 on two workspace directories is not a privilege or integrity boundary here — it would be a different matter on a persistent or self-hosted runner, which this workflow does not use. A named POSIX ACL (setfacl -R -m u:1000:rwx -m d:u:1000:rwx) would survive that recursive chown and leave "other" non-writable, if you would rather not have the 777 in the file at all; Codex confirmed the ARM64 runner image ships acl. Worth knowing that all three open aarch64 PRs landed on the same answer independently: Add native AArch64 package build support #240 rewrites make_dir_writable() to chmod -R a+rwX and asserts mode 777 in its own test, and Publish a reusable aarch64 repository artifact #223's copy of this workflow does chmod -R a+rwX build-output pkgs.omarchy.org src. This is not out of line with where the repo is going.

Re-verified clean at this head. No fork can reach this workflow: on: carries only workflow_dispatch — no pull_request, pull_request_target, workflow_call or workflow_run anywhere in the workflow set — dispatch needs repository write access, and a fork running its own copy gets its own secrets, not this repository's. No signing key, publishing credential or write-scoped token is reachable: the job is permissions: contents: read, persist-credentials: false, it never invokes bin/sign, bin/repo or bin/sync-repo, and BASECAMP_CHATBOT_URL is scoped to the failure-notification step alone, matching the pattern already in sync-aur.yml and sync-upstream.yml. No shared-database race: helpers/paths.sh:17-18 makes both output paths architecture- and mirror-specific, nothing here writes the published tree, concurrent dispatches get separate VMs, and the artifact name carries github.run_id. The quoting holds — --package "$PACKAGES" as a single argument is reconstructed identically by the arg loop at bin/build:33-40, and the compgen -G non-match is safely inside an if under set -e. The artifact path matches BUILD_OUTPUT_DIR. Codex agreed on all of this, having been asked the same questions independently.

What was not tested. Nothing was executed. This repository has no test suite, and the only thing that can exercise a GitHub Actions workflow on ubuntu-24.04-arm is GitHub's own runner — so every claim above comes from reading the tree, not from a green run. We parsed the YAML to confirm it is well-formed and that on: resolves to workflow_dispatch alone, and traced the uid and path reasoning through bin/build, helpers/docker-helpers.sh, helpers/paths.sh, build/build.sh and build/Dockerfile. Whether a full unscoped aarch64 build actually completes inside the 360-minute cap is unknown and can only be found out by dispatching it.

Where this sits. Waiting on you for item 1 — a few lines in the header comment. Nothing was pushed to your branch. For the maintainer there is a sequencing question this comment's map understates: #223 does not carry this branch's workflow, it replaces .github/workflows/build-aarch64.yml with its own 145-line rewrite that adds a workflow_call trigger taking pkgs_repository and pkgs_ref inputs. On this one file the two are competing implementations rather than stacked, and #223 as it stands also reintroduces the build-for-aarch64.patch files this PR just removed.

@birkskyum

Copy link
Copy Markdown

@scottjones if you can land this one, i can rebase the prs i have that build on top.

The maintainer recipe in the header comment published packages without a
database. bin/promote-build skips omarchy-build.db* when it moves packages
into the published tree, and bin/repo update is the only thing that runs the
repo-add that produces omarchy.db, so sign -> promote -> sync uploaded package
files that no pacman client could resolve. Add the update step, and thread
--mirror through all four: helpers/paths.sh defaults MIRROR to edge, so the
recipe as written would have read the wrong tree for a stable artifact.

The batching suggestion on timeout-minutes was not dependency-complete either.
build/build.sh builds only the named packages and counts a dependency only
when it is also in the selected set, and it configures the production
[omarchy] repo only when a database already exists, which is never true on a
clean runner. A batch containing omarchy without omarchy-settings fails at
makepkg -s on its pinned omarchy-settings=${pkgver}. Say what the input is
actually for rather than offering it as a way to split a full build.

The log upload could never match. LOG_DIR is logs/, written only by bin/repo,
and this workflow calls bin/build directly; build/build.sh writes no log files
at all. With if-no-files-found: ignore the step was silent about it, implying
a diagnostic artifact that never existed. The job log is the diagnostic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@scottjones

Copy link
Copy Markdown
Contributor Author

Thanks — items 1, 2 and 3 are fixed in 7d50d88. All three were comment-only, so the workflow's behaviour is unchanged; what changed is that the comments no longer describe things that do not work.

1. Publish recipe (medium). Confirmed, and the diagnosis was exactly right. bin/promote-build:86,123,173 skips omarchy-build.db* on every path that moves a file into the published tree, bin/repo update is the only caller of build/update-repo.sh, and its repo-add at line 66 is what writes omarchy.db. bin/sync-repo:235-237 then uploads --include "omarchy.*" out of $REPO_DIR — so with no update step there is simply no database file there to upload, and the packages land unresolvable. The recipe now reads:

bin/repo sign    --arch aarch64 --mirror <mirror>
bin/repo promote --arch aarch64 --mirror <mirror>
bin/repo update  --arch aarch64 --mirror <mirror>
bin/repo sync    --arch aarch64 --mirror <mirror>

with --mirror threaded through all four for the MIRROR=edge default you flagged, a note on where to unpack the artifact, and a short paragraph explaining why update is not optional so the next person does not trim it back out.

2. Batching advice (low). Confirmed. build/build.sh:511 only counts a dependency when it is also in PACKAGES_TO_BUILD, and build/build.sh:60-61 gates the [omarchy] repo on a database that does not exist on a clean runner, so a batch has to be dependency-closed. pkgbuilds/omarchy/PKGBUILD:32 pins omarchy-settings=${pkgver} and is the obvious way to trip it. The comment no longer offers the input as a way to split a full build; it says what the input is for and what a batch has to satisfy.

3. Log upload (low). Confirmed, and slightly worse than described — not only does this workflow call bin/build directly while logs/ is written only by bin/repo:26,39, but build/build.sh writes no log file anywhere; it is all stdout. There was nothing for the step to collect under any arrangement, so I removed it rather than repointing it. The job log is the diagnostic.

4. chmod -R 777 (low). Left as is, deliberately. Your own analysis is the reason: ubuntu-24.04-arm gives each job a fresh single-tenant VM that is destroyed afterwards, so this is not a privilege boundary, and the mode has to be opened before make_dir_writable()'s recursive chown for the uid 1000/1001 split to work at all. setfacl would be tidier in the abstract, but #240 rewrites make_dir_writable() to chmod -R a+rwX and #223 does chmod -R a+rwX in its own copy of this workflow — so switching to ACLs here would make this the only one of the three doing something different, for no gain on an ephemeral runner. Happy to revisit if make_dir_writable() gets fixed properly, which would let this step disappear entirely.

On the sequencing note at the end: agreed, and thanks for catching it — I had #223 as stacked on this branch, and it is a competing rewrite of the same file rather than a follow-up. That is worth settling before either merges. cc @birkskyum, since it changes what "rebase on top" means for #223 specifically — #221 and #222 are unaffected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants