Skip to content

Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion - #6063

Open
eriknordmark wants to merge 16 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-resize
Open

Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion#6063
eriknordmark wants to merge 16 commits into
lf-edge:masterfrom
eriknordmark:kvm-to-k-resize

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion

Long-lived devices installed with EVE's old small boot-disk layout (IMGA/IMGB
as small as 300 MB, a small ESP) cannot hold the EVE-k (Kubernetes) rootfs and
its container/k3s tooling. Upgrading replaces rootfs contents but never
repartitions, so the small geometry persists. This branch repartitions the boot
disk in place to the larger EVE-k geometry at the moment the controller pushes a
cross-flavor base-OS update — growing ESP/IMGA/IMGB, and first shrinking the
/persist filesystem when that's the only way to free the space — so an
in-field device can be converted to EVE-k without re-imaging and without losing
/persist data or device identity.

A read-only pre-flight classifies the disk into proceed / grow / shrink /
insufficient. grow happens online; shrink backs up the device identity to
the CONFIG partition and reboots into an offline shrink+grow under storage-init;
insufficient declines the conversion cleanly and the device stays on its
current flavor. The design is idempotent and fail-safe: no failure mode bricks
the device, and even a power loss mid-shrink recovers to a bootable, remotely
manageable state.

Interrupting the shrink also does not corrupt data on the filesystem being
resized: resize2fs flushes each relocated extent before repointing any inode at
the new location, and the smaller size commits only at the very end, so a reset
resumes from the old geometry with every file addressing either its untouched
original or a complete copy. Fault injection bears this out — 36 conversions with
193 watchdog resets landing inside the shrink, no corrupted app volume and no
structural fsck finding. It holds as far as the storage stack honors the flush;
buffering in the kernel and in the storage controller means a flush may be
acknowledged before the bytes reach the medium, which QEMU-based testing cannot
exercise, so the identity files are additionally verified by digest.

See docs/STORAGE-RESIZING.md for the full design.

What's here

  • pkg/storage-resizer — standalone Go binary doing all the disk work
    (check/backup/shrink/grow/restore/cleanup) via diskfs/partitionresizer, kept
    out of the pillar module.
  • storage-init — runs the offline shrink/grow and post-mount
    restore/cleanup, re-syncing /config so PCR14 stays stable.
  • pkg/pillar/diskconvert — pillar orchestration that execs the resizer and
    selects the next step; unit-tested behind a fake Runner.
  • baseosmgr / nodeagent / zedagent — drive the conversion at the cross-flavor
    seam, reboot into the offline shrink when needed, and report
    ZDEVICE_STATE_CONVERTING with a sub-state.
  • volumemgr / upgradeconverter — recognize blobs already on /persist across
    the kvm→k namespace change so pre-staged content isn't re-downloaded; gate the
    cross-flavor block on the actual volume set.

In-flight items before merge

  • The repartition and refused legs of the test plan now pass end to end — seven
    topologies (ext4 shrink/grow, two-disk ext4, two-disk ZFS, ZFS grow, and the two
    refusal cases), 7/7. The power-loss recovery cases are now covered by watchdog fault injection
    on an integration branch carrying this stack. Still outstanding: bare-metal power
    loss (see above), arm64, and a run on this branch alone rather than the
    integration branch.
  • We should probably convert these eden tests to evetest but that can be done in a separate PR.

How to test and validate this PR

  • Unit: pkg/pillar/diskconvert orchestration tests (fake Runner) and the
    pkg/storage-resizer check/shrink/grow logic.
  • E2E under eden + swtpm on a device installed with the old small boot-disk
    geometry: push a cross-flavor (EVE-kvm→EVE-k) base-OS update and confirm
    each pre-flight class behaves correctly — grow (online), shrink
    (offline shrink+grow under storage-init, identity backed up to CONFIG and
    restored), and insufficient (conversion declined cleanly, device stays on
    its current flavor). Confirm /persist data and device identity survive,
    and that a power loss mid-shrink recovers to a bootable, remotely
    manageable state.

Changelog notes

Adds in-field boot-disk repartitioning so a device installed with EVE's old
small boot-disk layout can be converted between EVE-kvm and EVE-k on a
cross-flavor base-OS update, without re-imaging and without losing /persist
data or device identity. Requires that there are no app instances or volume instances
on the device.

PR Backports

  • 17.0-stable: No.
  • 16.0-stable: No.
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@eriknordmark eriknordmark changed the title Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion [WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion Jun 19, 2026
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 2 times, most recently from 6858d0a to 90fa14a Compare June 20, 2026 18:37
@github-actions
github-actions Bot requested a review from jsfakian June 20, 2026 21:59
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 2 times, most recently from 053e426 to 36977ab Compare June 22, 2026 12:59
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 7 times, most recently from 42fd8e5 to 58e45a6 Compare July 9, 2026 22:04
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 9, 2026
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 27.03488% with 502 lines in your changes missing coverage. Please review.
✅ Project coverage is 25.23%. Comparing base (aadcfa9) to head (b25f0c5).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
pkg/pillar/cmd/baseosmgr/convert.go 22.59% 137 Missing ⚠️
pkg/pillar/vault/handler_zfs.go 1.50% 131 Missing ⚠️
pkg/pillar/cmd/baseosmgr/baseosmgr.go 0.00% 58 Missing and 1 partial ⚠️
pkg/pillar/diskconvert/convert.go 46.66% 48 Missing ⚠️
...illar/cmd/upgradeconverter/containerd_namespace.go 71.68% 20 Missing and 12 partials ⚠️
pkg/pillar/cmd/baseosmgr/handlebaseos.go 0.00% 18 Missing ⚠️
pkg/pillar/cmd/nodeagent/handlebaseos.go 0.00% 18 Missing ⚠️
pkg/pillar/cmd/nodeagent/nodeagent.go 0.00% 18 Missing ⚠️
pkg/pillar/cmd/zedagent/reportinfo.go 0.00% 13 Missing ⚠️
pkg/pillar/zfs/zfs.go 0.00% 13 Missing ⚠️
... and 3 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6063      +/-   ##
==========================================
+ Coverage   24.87%   25.23%   +0.35%     
==========================================
  Files         506      519      +13     
  Lines       93177    95575    +2398     
==========================================
+ Hits        23181    24114     +933     
- Misses      68178    69444    +1266     
- Partials     1818     2017     +199     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 10, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 11, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 11, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Jul 14, 2026
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Aug 3, 2026
Record lf-edge#6259, note that lf-edge#6063 is now 17 commits, refresh lf-edge#6240's head, and mark
the CDI upload-pod teardown logging as a pre-merge copy of an upstream commit.
Spell out why lf-edge#6242 is not replayed here and that its NAD-CRD race is still
open in the Go ApplyMultusCNI. Add the lf-edge#6240/lf-edge#6259 test interaction and the
rebase-due note: master is 63 commits ahead of the shared merge-base.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 2 times, most recently from c131d6d to 3a5f104 Compare August 4, 2026 14:12
eriknordmark added a commit to eriknordmark/eve that referenced this pull request Aug 4, 2026
eriknordmark and others added 2 commits August 28, 2026 13:31
After zedagent restarts pubVolumeConfig, volumemgr has reconciled its
on-disk recovery (populateExistingVolumesFormat*) with the controller-
intended set into VolumeStatus. Signal restart on pubVolumeStatus at the
tail of handleVolumeRestart so subscribers can use Restarted() as a gate
on the published set being in a known state, rather than Synchronized()
which only reflects the socket-level handshake.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-flavor BaseOs upgrade between EVE-k and non-EVE-k was
unconditionally rejected. The rationale is that /persist/vault/volumes
layout differs across flavors, so existing volumes can break across the
switch. On a device with no volumes there is nothing to disturb, so the
block is overly strict.

Subscribe to VolumeConfig (from zedagent) and VolumeStatus (from
volumemgr) in baseosmgr and only error out on the flavor mismatch when
either set is non-empty. Wait at startup for both publications to signal
restart so the first BaseOsConfig after boot evaluates against the true
volume set; Synchronized() is not sufficient because it only confirms
the socket handshake.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@eriknordmark
eriknordmark marked this pull request as ready for review August 28, 2026 12:55
@eriknordmark eriknordmark changed the title [WIP] Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion Aug 28, 2026
@eriknordmark
eriknordmark requested a review from rene August 28, 2026 13:02
@eriknordmark
eriknordmark force-pushed the kvm-to-k-resize branch 2 times, most recently from 72b7bb8 to c008be4 Compare August 28, 2026 14:12
eriknordmark and others added 10 commits August 28, 2026 16:27
Pillar's containerd client uses namespace eve-user-apps on EVE-kvm and
k8s.io on EVE-k (pkg/pillar/containerd/containerd.go init()). The user
containerd's content store at /persist/vault/containerd survives the
kvm→k upgrade, but the bolt metadata DB records blob ownership per
namespace, so pillar on EVE-k sees the store as empty under k8s.io
and re-downloads every blob that EVE-kvm had previously fetched.
Each blob's on-disk file gets overwritten with bit-identical content
on a fresh content.WriteBlob; pre-staged ContentTrees burn full
registry bandwidth on every upgrade.

Add a post-vault converter that runs once on the first EVE-k boot to
copy blob metadata records from eve-user-apps to k8s.io directly in
the bolt DB at /persist/vault/containerd/io.containerd.metadata.v1.bolt.
Metadata-only, no blob copies. The post-vault phase fits cleanly: the
vault is decrypted but the user containerd has not yet been started by
the kube container, so the bolt DB is idle and safe to mutate via
bbolt (already vendored). A sentinel file at /persist/vault/
containerd/.eve-namespace-port-done short-circuits subsequent boots.
No-op on EVE-kvm and on fresh-install EVE-k that has no prior
eve-user-apps records.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
populateInitBlobStatus() reconstructs BlobStatus from the user
containerd's content store on volumemgr startup so that blobs already
on /persist (e.g. those pulled before an EVE-kvm → EVE-k upgrade) are
recognized and don't trigger a re-download. On EVE-k it filters by the
EVEDownloadedLabel because the user containerd is shared with k3s and
we must not publish BlobStatus for k3s/longhorn/etc. pod images.

Pillar's CAS writes EVEDownloadedLabel only on manifest-shaped blobs
(and on the named image entry), not on the individual layer blobs the
manifest references. The manifest carries containerd.io/gc.ref.content.*
labels that point at every layer — the same set containerd's GC uses
to keep layers alive while the manifest is reachable. Filtering by the
label per-blob therefore rejects every layer with "Ignoring the blob
... not downloaded by eve", forcing volumemgr to re-fetch each layer
even though they're physically present on /persist.

Walk those labels transitively from each directly-labeled blob to
recover the full set pillar considers its own. Foreign manifests
(k3s pod images) don't carry the label so they don't seed the walk —
no false positives for shared blobs.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
A device whose /persist is ZFS and that is converted in-field from
EVE-kvm to EVE-k carries over a vault stored as a filesystem dataset,
but EVE-k expects the vault as a zvol formatted ext4 alongside a
separate etcd-storage zvol. The carried-over filesystem vault therefore
never reached a mounted state on the first EVE-k boot, so attestation
escrow could not complete and the device reboot-looped back to kvm.

On the first EVE-k boot the unlock path now detects that the vault is
still a filesystem dataset, mounts it so the device boots, and migrates
it in place to the zvol layout: it stages a new zvol, copies the vault
contents into it (containerd content store and metadata, downloader,
verifier, and configs all live inside the vault, so blobs and content
trees come across), then destroys the old dataset and renames the new
zvol into place. An empty etcd-storage zvol is created since etcd starts
fresh on EVE-k. The staging zvol survives until the rename and remount
succeed, so an interrupted attempt is reconstructed on the next boot.

The migration helper is parameterized on encryption: the TPM path
creates the new zvols encrypted with the staged unlock key, the no-TPM
SetupDefaultVault path creates them unencrypted (matching how no-TPM ZFS
vaults are created), so both paths end up with the same zvol layout.

This covers a ZFS device carrying content trees and blobs but no app
volumes; app-volume conversion is handled separately.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pick up the ZDEVICE_STATE_CONVERTING device state and the ZDeviceSubState
enum, which the boot-disk conversion reports to the controller while a
device is being repartitioned between the EVE-kvm and EVE-k layouts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Mirror the eve-api ZDeviceState CONVERTING value and the ZDeviceSubState
enum as types.DeviceState and types.DeviceSubState, with four sub-states:
unspecified, rebooting-to-resize, installing, and rebooting-to-target.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Standalone Go binary doing the disk work (check/backup/shrink/grow/
restore/cleanup) via diskfs/partitionresizer, plus a run-watchdog
subcommand that holds/pets /dev/watchdog across a long offline op. The
--no-pet/--attempt escalating fault-injection mode is implemented but
only used by the separate stress-test PR; the product pets. The grow
also creates the reserved second EFI System partition (ESP-B) so a
converted device reaches the same layout as a fresh EVE-k install.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Execs the storage-resizer and selects the next conversion step; unit-tested behind a fake Runner.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
On a cross-flavor (kvm<->k) base-OS update, run the read-only pre-flight, proceed/grow/shrink/decline accordingly, reboot into the offline shrink when needed, and report ZDEVICE_STATE_CONVERTING with a sub-state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Handle the baseosmgr-requested reboot into the early-boot offline shrink/grow during a kvm<->k conversion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Report ZDEVICE_STATE_CONVERTING and the conversion sub-state to the controller.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
eriknordmark and others added 4 commits August 28, 2026 16:43
Run the offline shrink/grow and post-mount restore/cleanup under storage-init (re-syncing /config so PCR14 stays stable), feed (pet) the hardware watchdog across it, and raise the resize reboot budget. Wire the eve-storage-resizer image into pillar's and storage-init's Dockerfiles.

The statically linked resizer adds about 1.5MB to the non-k rootfs, which puts the squashfs image a little over the 290MB early-warning check, so that check moves to 291MB. The compatibility constraint it guards is the 300MB rootfs partition on devices installed before 10.2.0, which this still clears by 9MB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
When a watchdog reset interrupts the offline resize2fs shrink of /persist,
the next boot's e2fsck reconnects the orphaned subtrees it repairs into
/persist/lost+found. That rescued data is no longer owned by any agent
(newlogd, the image CAS GC, ...), so nothing ever reclaims it, yet it keeps
consuming P3 space and counts against the low-disk threshold.

Report the directory's usage so the space is attributable to the controller
and a nonzero value flags that a filesystem repair occurred, and add it to
the boot-time low-disk cleanup list. It is placed last, since it may hold the
only rescued copy of the data, so an emergency low-disk boot reclaims it only
after the known-disposable directories did not free enough.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
The /config backup protects the identity and connectivity files against a
shrink that wipes /persist, but nothing detected a shrink that damaged them
in place: a file left at its original size and structurally valid while
holding a partially-copied mix of old and new bytes passes the
present/non-empty/PEM/JSON checks.

Whether an interrupted relocation can actually produce such a file is not
settled. resize2fs flushes each relocated extent before it repoints any inode
at the new location, so on a storage stack that honors the flush a reset
leaves every file addressing either the untouched original or a complete copy.
On real hardware that ordering is only as good as the layers beneath it:
buffering in the kernel and in the storage controller means a flush may be
acknowledged before the bytes reach the medium, so a power cut may tear a copy
after all. Verifying costs one hash per critical file and removes the need to
know.

The backup step now records the SHA-256 of each copy while /persist is still
intact, and the restore step re-hashes the live copies before restoring
anything and reports the comparison on the console. It reports rather than
repairs: the device keeps running until the reboot into the offline resize,
so the saved config and the DPC list can legitimately differ by then, and
overwriting one with the stale backup would lose the ssh keys and network
config it carries. Which files to restore stays with the per-type validators.
The digest manifest lives in the backup directory, so the existing cleanup
paths remove it with the copies and it cannot perturb the measure-config PCR.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@eriknordmark

eriknordmark commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

The yetus -1 on this PR is a structural consequence of adding a new Go module (pkg/storage-resizer), not a problem in the diff.

yetus — the -1 on golangcilint is a false positive. Yetus scores that subsystem by diffing the line count of golangci-lint's per-module output. master produces 503 lines, this branch 504, and the one extra line is:

pkg/storage-resizer/0 issues.

That is the new module reporting zero problems, counted as "1 new issue". In the yetus-scan artifact, results-codespell.txt, results-markdownlint.txt and results-revive.txt are empty, and results-golangcilint.txt contains only the line above. The four real findings from the earlier run (codespell unparseable, two MD040 fenced blocks, revive's builtin-shadowing clear plus the missing const-block comment, and four unchecked fmt.Fprint*) are all fixed. Since the extra line is inherent to the module existing, this vote is expected to recur.

scan-pr / osv-scan — resolved, and worth recording why. osv-scanner reads the go directive as the stdlib version, and a brand-new go.mod has no baseline to diff against, so at go 1.25.7 every stdlib CVE fixed after 1.25.7 was reported as new and failed the job. Declaring go 1.25.12 — matching pkg/kube/kube-init and the Go shipped in lfedge/eve-alpine — aligns the new module with a version already present in master, and the job now reports Total 0 packages affected by 0 known vulnerabilities. Its annotation still lists seven stdlib advisories fixed in 1.25.13 (GO-2026-5026, -5972, -6088, -6089, -6090, -6091, -6218), but those are shared with the modules already on 1.25.12 and do not fail the check. Going to 1.25.13 is not possible here regardless: the build image ships Go 1.25.12 and build.yml sets network: no, so a higher directive cannot download a toolchain. The vendored klauspost/compress finding from the earlier run is also fixed (1.18.6 → 1.18.7).

One project-wide change worth a reviewer's attention: the non-k ROOTFS_MAXSIZE_MB check moves from 290 to 291 MB. The statically linked resizer adds ~1.5 MB to the non-k squashfs rootfs, which put it ~164 KiB over 290; master itself sits at ~288.7 MB, so there was only ~1.3 MB of headroom. The compatibility constraint being guarded is the 300 MB pre-10.2.0 rootfs partition, which 291 still clears by 9 MB. Rationale is in the storage-init,pillar: run storage-resizer during the offline resize commit message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

main-quest The fate of the project rests on this PR. Prioritise review to advance the storyline!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant