Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion - #6063
Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion#6063eriknordmark wants to merge 16 commits into
Conversation
6858d0a to
90fa14a
Compare
053e426 to
36977ab
Compare
643a8f4 to
5dbc563
Compare
42fd8e5 to
58e45a6
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6063 +/- ##
==========================================
+ Coverage 24.87% 25.23% +0.35%
==========================================
Files 506 519 +13
Lines 93177 95575 +2398
==========================================
+ Hits 23181 24114 +933
- Misses 68178 69444 +1266
- Partials 1818 2017 +199 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
55c10f6 to
5f6ec99
Compare
Record lf-edge#6259, note that lf-edge#6063 is now 17 commits, refresh lf-edge#6240's head, and mark the CDI upload-pod teardown logging as a pre-merge copy of an upstream commit. Spell out why lf-edge#6242 is not replayed here and that its NAD-CRD race is still open in the Go ApplyMultusCNI. Add the lf-edge#6240/lf-edge#6259 test interaction and the rebase-due note: master is 63 commits ahead of the shared merge-base. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c131d6d to
3a5f104
Compare
3a5f104 to
0f44597
Compare
After zedagent restarts pubVolumeConfig, volumemgr has reconciled its on-disk recovery (populateExistingVolumesFormat*) with the controller- intended set into VolumeStatus. Signal restart on pubVolumeStatus at the tail of handleVolumeRestart so subscribers can use Restarted() as a gate on the published set being in a known state, rather than Synchronized() which only reflects the socket-level handshake. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-flavor BaseOs upgrade between EVE-k and non-EVE-k was unconditionally rejected. The rationale is that /persist/vault/volumes layout differs across flavors, so existing volumes can break across the switch. On a device with no volumes there is nothing to disturb, so the block is overly strict. Subscribe to VolumeConfig (from zedagent) and VolumeStatus (from volumemgr) in baseosmgr and only error out on the flavor mismatch when either set is non-empty. Wait at startup for both publications to signal restart so the first BaseOsConfig after boot evaluates against the true volume set; Synchronized() is not sufficient because it only confirms the socket handshake. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
0f44597 to
40d09e0
Compare
72b7bb8 to
c008be4
Compare
Pillar's containerd client uses namespace eve-user-apps on EVE-kvm and k8s.io on EVE-k (pkg/pillar/containerd/containerd.go init()). The user containerd's content store at /persist/vault/containerd survives the kvm→k upgrade, but the bolt metadata DB records blob ownership per namespace, so pillar on EVE-k sees the store as empty under k8s.io and re-downloads every blob that EVE-kvm had previously fetched. Each blob's on-disk file gets overwritten with bit-identical content on a fresh content.WriteBlob; pre-staged ContentTrees burn full registry bandwidth on every upgrade. Add a post-vault converter that runs once on the first EVE-k boot to copy blob metadata records from eve-user-apps to k8s.io directly in the bolt DB at /persist/vault/containerd/io.containerd.metadata.v1.bolt. Metadata-only, no blob copies. The post-vault phase fits cleanly: the vault is decrypted but the user containerd has not yet been started by the kube container, so the bolt DB is idle and safe to mutate via bbolt (already vendored). A sentinel file at /persist/vault/ containerd/.eve-namespace-port-done short-circuits subsequent boots. No-op on EVE-kvm and on fresh-install EVE-k that has no prior eve-user-apps records. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
populateInitBlobStatus() reconstructs BlobStatus from the user containerd's content store on volumemgr startup so that blobs already on /persist (e.g. those pulled before an EVE-kvm → EVE-k upgrade) are recognized and don't trigger a re-download. On EVE-k it filters by the EVEDownloadedLabel because the user containerd is shared with k3s and we must not publish BlobStatus for k3s/longhorn/etc. pod images. Pillar's CAS writes EVEDownloadedLabel only on manifest-shaped blobs (and on the named image entry), not on the individual layer blobs the manifest references. The manifest carries containerd.io/gc.ref.content.* labels that point at every layer — the same set containerd's GC uses to keep layers alive while the manifest is reachable. Filtering by the label per-blob therefore rejects every layer with "Ignoring the blob ... not downloaded by eve", forcing volumemgr to re-fetch each layer even though they're physically present on /persist. Walk those labels transitively from each directly-labeled blob to recover the full set pillar considers its own. Foreign manifests (k3s pod images) don't carry the label so they don't seed the walk — no false positives for shared blobs. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A device whose /persist is ZFS and that is converted in-field from EVE-kvm to EVE-k carries over a vault stored as a filesystem dataset, but EVE-k expects the vault as a zvol formatted ext4 alongside a separate etcd-storage zvol. The carried-over filesystem vault therefore never reached a mounted state on the first EVE-k boot, so attestation escrow could not complete and the device reboot-looped back to kvm. On the first EVE-k boot the unlock path now detects that the vault is still a filesystem dataset, mounts it so the device boots, and migrates it in place to the zvol layout: it stages a new zvol, copies the vault contents into it (containerd content store and metadata, downloader, verifier, and configs all live inside the vault, so blobs and content trees come across), then destroys the old dataset and renames the new zvol into place. An empty etcd-storage zvol is created since etcd starts fresh on EVE-k. The staging zvol survives until the rename and remount succeed, so an interrupted attempt is reconstructed on the next boot. The migration helper is parameterized on encryption: the TPM path creates the new zvols encrypted with the staged unlock key, the no-TPM SetupDefaultVault path creates them unencrypted (matching how no-TPM ZFS vaults are created), so both paths end up with the same zvol layout. This covers a ZFS device carrying content trees and blobs but no app volumes; app-volume conversion is handled separately. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pick up the ZDEVICE_STATE_CONVERTING device state and the ZDeviceSubState enum, which the boot-disk conversion reports to the controller while a device is being repartitioned between the EVE-kvm and EVE-k layouts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Mirror the eve-api ZDeviceState CONVERTING value and the ZDeviceSubState enum as types.DeviceState and types.DeviceSubState, with four sub-states: unspecified, rebooting-to-resize, installing, and rebooting-to-target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Standalone Go binary doing the disk work (check/backup/shrink/grow/ restore/cleanup) via diskfs/partitionresizer, plus a run-watchdog subcommand that holds/pets /dev/watchdog across a long offline op. The --no-pet/--attempt escalating fault-injection mode is implemented but only used by the separate stress-test PR; the product pets. The grow also creates the reserved second EFI System partition (ESP-B) so a converted device reaches the same layout as a fresh EVE-k install. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Execs the storage-resizer and selects the next conversion step; unit-tested behind a fake Runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
On a cross-flavor (kvm<->k) base-OS update, run the read-only pre-flight, proceed/grow/shrink/decline accordingly, reboot into the offline shrink when needed, and report ZDEVICE_STATE_CONVERTING with a sub-state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Handle the baseosmgr-requested reboot into the early-boot offline shrink/grow during a kvm<->k conversion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Report ZDEVICE_STATE_CONVERTING and the conversion sub-state to the controller. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
c008be4 to
4024af8
Compare
Run the offline shrink/grow and post-mount restore/cleanup under storage-init (re-syncing /config so PCR14 stays stable), feed (pet) the hardware watchdog across it, and raise the resize reboot budget. Wire the eve-storage-resizer image into pillar's and storage-init's Dockerfiles. The statically linked resizer adds about 1.5MB to the non-k rootfs, which puts the squashfs image a little over the 290MB early-warning check, so that check moves to 291MB. The compatibility constraint it guards is the 300MB rootfs partition on devices installed before 10.2.0, which this still clears by 9MB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
When a watchdog reset interrupts the offline resize2fs shrink of /persist, the next boot's e2fsck reconnects the orphaned subtrees it repairs into /persist/lost+found. That rescued data is no longer owned by any agent (newlogd, the image CAS GC, ...), so nothing ever reclaims it, yet it keeps consuming P3 space and counts against the low-disk threshold. Report the directory's usage so the space is attributable to the controller and a nonzero value flags that a filesystem repair occurred, and add it to the boot-time low-disk cleanup list. It is placed last, since it may hold the only rescued copy of the data, so an emergency low-disk boot reclaims it only after the known-disposable directories did not free enough. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The /config backup protects the identity and connectivity files against a shrink that wipes /persist, but nothing detected a shrink that damaged them in place: a file left at its original size and structurally valid while holding a partially-copied mix of old and new bytes passes the present/non-empty/PEM/JSON checks. Whether an interrupted relocation can actually produce such a file is not settled. resize2fs flushes each relocated extent before it repoints any inode at the new location, so on a storage stack that honors the flush a reset leaves every file addressing either the untouched original or a complete copy. On real hardware that ordering is only as good as the layers beneath it: buffering in the kernel and in the storage controller means a flush may be acknowledged before the bytes reach the medium, so a power cut may tear a copy after all. Verifying costs one hash per critical file and removes the need to know. The backup step now records the SHA-256 of each copy while /persist is still intact, and the restore step re-hashes the live copies before restoring anything and reports the comparison on the console. It reports rather than repairs: the device keeps running until the reboot into the offline resize, so the saved config and the DPC list can legitimately differ by then, and overwriting one with the stale backup would lose the ssh keys and network config it carries. Which files to restore stays with the per-type validators. The digest manifest lives in the backup directory, so the existing cleanup paths remove it with the copies and it cannot perturb the measure-config PCR. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4024af8 to
b25f0c5
Compare
|
The yetus yetus — the That is the new module reporting zero problems, counted as "1 new issue". In the scan-pr / osv-scan — resolved, and worth recording why. osv-scanner reads the One project-wide change worth a reviewer's attention: the non-k |
Boot-disk repartition for in-field EVE-kvm ↔ EVE-k conversion
Long-lived devices installed with EVE's old small boot-disk layout (IMGA/IMGB
as small as 300 MB, a small ESP) cannot hold the EVE-k (Kubernetes) rootfs and
its container/k3s tooling. Upgrading replaces rootfs contents but never
repartitions, so the small geometry persists. This branch repartitions the boot
disk in place to the larger EVE-k geometry at the moment the controller pushes a
cross-flavor base-OS update — growing ESP/IMGA/IMGB, and first shrinking the
/persistfilesystem when that's the only way to free the space — so anin-field device can be converted to EVE-k without re-imaging and without losing
/persistdata or device identity.A read-only pre-flight classifies the disk into
proceed/grow/shrink/insufficient.growhappens online;shrinkbacks up the device identity tothe CONFIG partition and reboots into an offline shrink+grow under storage-init;
insufficientdeclines the conversion cleanly and the device stays on itscurrent flavor. The design is idempotent and fail-safe: no failure mode bricks
the device, and even a power loss mid-shrink recovers to a bootable, remotely
manageable state.
Interrupting the shrink also does not corrupt data on the filesystem being
resized: resize2fs flushes each relocated extent before repointing any inode at
the new location, and the smaller size commits only at the very end, so a reset
resumes from the old geometry with every file addressing either its untouched
original or a complete copy. Fault injection bears this out — 36 conversions with
193 watchdog resets landing inside the shrink, no corrupted app volume and no
structural fsck finding. It holds as far as the storage stack honors the flush;
buffering in the kernel and in the storage controller means a flush may be
acknowledged before the bytes reach the medium, which QEMU-based testing cannot
exercise, so the identity files are additionally verified by digest.
See
docs/STORAGE-RESIZING.mdfor the full design.What's here
pkg/storage-resizer— standalone Go binary doing all the disk work(check/backup/shrink/grow/restore/cleanup) via
diskfs/partitionresizer, keptout of the pillar module.
storage-init— runs the offline shrink/grow and post-mountrestore/cleanup, re-syncing
/configso PCR14 stays stable.pkg/pillar/diskconvert— pillar orchestration that execs the resizer andselects the next step; unit-tested behind a fake Runner.
seam, reboot into the offline shrink when needed, and report
ZDEVICE_STATE_CONVERTINGwith a sub-state./persistacrossthe kvm→k namespace change so pre-staged content isn't re-downloaded; gate the
cross-flavor block on the actual volume set.
In-flight items before merge
topologies (ext4 shrink/grow, two-disk ext4, two-disk ZFS, ZFS grow, and the two
refusal cases), 7/7. The power-loss recovery cases are now covered by watchdog fault injection
on an integration branch carrying this stack. Still outstanding: bare-metal power
loss (see above), arm64, and a run on this branch alone rather than the
integration branch.
How to test and validate this PR
pkg/pillar/diskconvertorchestration tests (fake Runner) and thepkg/storage-resizercheck/shrink/grow logic.geometry: push a cross-flavor (EVE-kvm→EVE-k) base-OS update and confirm
each pre-flight class behaves correctly —
grow(online),shrink(offline shrink+grow under storage-init, identity backed up to CONFIG and
restored), and
insufficient(conversion declined cleanly, device stays onits current flavor). Confirm
/persistdata and device identity survive,and that a power loss mid-shrink recovers to a bootable, remotely
manageable state.
Changelog notes
Adds in-field boot-disk repartitioning so a device installed with EVE's old
small boot-disk layout can be converted between EVE-kvm and EVE-k on a
cross-flavor base-OS update, without re-imaging and without losing /persist
data or device identity. Requires that there are no app instances or volume instances
on the device.
PR Backports
Checklist
And the last but not least:
check them.
🤖 Generated with Claude Code