Skip to content

feat(test): automated headless VM e2e harness + coverage matrix - #14

Merged
AdamJHall merged 1 commit into
mainfrom
feat/vm-e2e-harness
Jun 23, 2026
Merged

feat(test): automated headless VM e2e harness + coverage matrix#14
AdamJHall merged 1 commit into
mainfrom
feat/vm-e2e-harness

Conversation

@AdamJHall

Copy link
Copy Markdown
Owner

What

Adds the automated, headless VM end-to-end harness that docs/vm-validation.md called for
— the last-mile proof that a real archinstall does the right thing end-to-end on a booted
system
, not just that our render parses.

test/e2e/vm/e2e.py (pure stdlib — no pip) drives QEMU headless over a serial socket:
boot the live ISO → archwright install --yes → inject a harness-only Phase-B autorun
(NOPASSWD sudoers + serial-getty autologin + bootstrap/validate scripts, none of which leaks
into real configs) → reboot → archwright bootstrap → assert the installed system. Runs are
fully isolated per descriptor and run concurrently via --jobs (per-descriptor flock guards
against collisions).

task vm-e2e -- lvm-multi      # one descriptor
task vm-e2e -- --jobs 5       # whole matrix, 5 VMs at once
task vm-e2e-list

Coverage (18 descriptors)

  • Disk layouts: lvm (single / multi / multi-volume), btrfs (+compress / snapper), plain
    (every swap type), systemd-boot, LUKS (lvm_on_luks + luks).
  • Stage/feature coverage (validated by lib/features.sh): reflector, custom kernel,
    plymouth, hooks, setup, services, dotfiles (injected local chezmoi repo), flatpak, KDE.

Results from a full matrix run (archinstall 4.3)

14 green. The bootloader / swap / encryption / snapper reverse-engineered shapes are
confirmed. Three shapes surfaced real bugs, written up in docs/bugs/ (not fixed here):

Bug Effect
btrfs-subvolume-not-used-as-root.md btrfs installs to the top-level subvol; configured @ unused
lvm-multivolume-pv-fstype-empty.md multi-volume LVM renders an empty PV fs_type → archinstall aborts
plymouth-bootctl-update-fails-systemd-boot.md Phase-B bootctl update fails on systemd-boot (via the always-on plymouth stage)
flatpak-system-remote-add-polkit-hang.md flatpak stage hangs on a polkit prompt (system-wide remote-add as the user)

Also

  • docs/vm-validation.md reworked into a results/closeout doc with links to the bugs above.
  • New CONTRIBUTING.md: add → test (manual vm.sh + automated vm-e2e) → commit → PR.
  • vm / vm-fresh / vm-disk reframed as the interactive companion to vm-e2e;
    stale e2e-disks.yml comment fixed.

Testing

go build, go vet, go test ./..., gofmt -l . all clean. The harness itself is exercised
locally (needs /dev/kvm); see test/e2e/vm/README.md#requirements for host deps. Encryption
descriptors assert the on-disk LUKS shape on the ISO (encrypted Phase B is unimplemented); the
harness validates boot → bootstrap → assertions, not graphical desktop rendering (use
task vm-disk for that).

🤖 Generated with Claude Code

The last-mile validation docs/vm-validation.md called for: a pure-stdlib Python
orchestrator (test/e2e/vm/e2e.py) that drives QEMU headless over a serial socket
to run the full flow unattended — boot the live ISO, `install --yes`, inject a
harness-only Phase-B autorun (NOPASSWD sudoers + serial-getty autologin +
bootstrap/validate scripts, none of which touches real configs), reboot, run
`bootstrap`, and assert the installed system. Runs are isolated per descriptor
(own rundir/disks/serial socket/NVRAM) and can run concurrently via `--jobs` with
a per-descriptor flock guarding against collisions. `task vm-e2e` / `vm-e2e-list`.

Coverage (18 descriptors): disk layouts — lvm (single/multi/multi-volume), btrfs
(+compress/snapper), plain (every swap type), systemd-boot, LUKS (lvm_on_luks +
luks, asserted on the ISO since encrypted Phase B is unimplemented); and stage
features validated by lib/features.sh — reflector, custom kernel, plymouth, hooks,
setup, services, dotfiles (injected local chezmoi repo), flatpak, KDE desktop.

A full matrix run (archinstall 4.3) confirmed the bootloader/swap/encryption/
snapper shapes and surfaced 3 real bugs, written up in docs/bugs/: btrfs installs
to the top-level subvolume (configured @ unused); multi-volume LVM renders the PV
partition with an empty fs_type (archinstall aborts); and the Phase-B `bootctl
update` refresh fails on systemd-boot (reached via the always-on plymouth stage).
A fourth — the flatpak stage hangs on a polkit prompt — is documented too.

docs/vm-validation.md is updated to a results/closeout doc; CONTRIBUTING.md
documents add → test (manual vm.sh + automated vm-e2e) → commit → PR; the
vm/vm-fresh/vm-disk tasks are reframed as the interactive companion to vm-e2e.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AdamJHall
AdamJHall merged commit 18335fc into main Jun 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant