qvm-template-upgrade: add orchestration workflow and in-VM agent

[nihalxkumar]

This PR introduces the qvm-template-upgrade dom0 command-line utility, that performs an in-place N -> N+1 distribution upgrade of Debian and Fedora TemplateVM or StandaloneVM

fixes: QubesOS/qubes-issues#8605
GSoC 2026 project: Automate Template Version Upgrade

[ben-grande]

Yay, started. As I've done a review now, I will receive a notification every time you commit. Let me know when you need another look or have doubts by mentioning me.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 97.62931% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.31%. Comparing base (4973653) to head (c3f550e).

Files with missing lines	Patch %	Lines
vmupdate/agent/source/apt/apt_cli.py	88.88%	10 Missing ⚠️
vmupdate/agent/source/apt/apt_api.py	30.00%	7 Missing ⚠️
vmupdate/agent/source/common/package_manager.py	81.48%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #213      +/-   ##
==========================================
- Coverage   71.42%   69.31%   -2.12%     
==========================================
  Files          12       31      +19     
  Lines        1344     3021    +1677     
==========================================
+ Hits          960     2094    +1134     
- Misses        384      927     +543     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ben-grande]

PipelineRetryFailed

[nihalxkumar]

It's showing a successful run

[ben-grande]

Some tests are not enabled on this repo, such as mypy, black and pylint. See this as an example: https://github.com/QubesOS/qubes-core-admin/blob/main/.gitlab-ci.yml. Can you create a separate PR to enable those checks?

[nihalxkumar]

Sure, will do

[nihalxkumar]

We can squash here if it's looks fine. Will also have to rebase as this is 24 commits behind.

[ben-grande]

Looks fine to squash.

[marmarek]

Oh no, commit message in the other PR closed it ...

[marmarek]

I see a conflict here - CI will not run until it's resolved.

[ben-grande]

I have taken a look through the new code. Thanks for the progress, I like to see it evolving. Just minor review though

[ben-grande]

https://github.com/QubesOS/qubes-core-admin-linux/pull/213/changes/a0289a754c4936f215ac7f6920d678b4821ec213..a90eee15a3ce76eb66ca5d87eea4bf923789955e

Can you share more details of what fails on cleanup?

[nihalxkumar]

When I tried upgrading fedora 41 -> 42 there was a cleanup failure which happened after the successful version upgrade.

logs as shared privately on tuesday:

https://gist.github.com/nihalxkumar/01fb990deab3960cb28d680773bb1089#file-upgrade42-log-L48-L55

we can see Complete! in the above log.

https://gist.github.com/nihalxkumar/0e4edccb4f4409b1a4daf5c44b6f576f

After version_upgrade() returns success, the agent still runs cleanup. The old code used the cleanup exit code when deciding the final agent exit code.
For dnf this includes DNFCLI.clean() (dnf clean packages), and the transport also removes /run/qubes-update/.

a90eee1 (last commit) is based on that failure boundary. After the release transaction has succeeded, a later cleanup failure should not cause rollback of the upgraded clone.

[ben-grande]

But we can't ignore all cleanup failures. We don't know what is causing it, so this needs to be investigated.

[ben-grande]

a90eee1 wasn't ignoring all cleanup failures 😅

a90eee1

How not? It was ignoring all cleanup failures if pkg_mng.version_upgrade returned EXIT.OK. Can you explain in more details? I didn't check if this change remained in the latest commits.

[ben-grande]

f66d1e7

Unsure about this one also. If shutdown failed, the upgrade might have failed despite the package manager not returning an error during upgrade. Which exception were you receiving?

[nihalxkumar]

a90eee1 wasn't ignoring all cleanup failures 😅

a90eee1

How not? It was ignoring all cleanup failures if pkg_mng.version_upgrade returned EXIT.OK. Can you explain in more details? I didn't check if this change remained in the latest commits.

By "all" I thought we are including the regular-updates cleanup behavior as well, which were untouched. On the version-upgrade path, though, you're right: once the upgrade returned EXIT.OK, a90eee1 suppressed any non-zero clean() code. My reasoning was that a failing clean() after a completed release transaction is just a cache issue, but the dom0 tool rolls back on any agent exit != EXIT.OK (template_upgrade.py#L307) a fatal cleanup code would delete a good upgrade. So I tolerated it only on the success path and kept it fatal for normal updates.

I've since dropped that tolerance change. As in every run dnf clean actually returned 0, so it was never being triggered; and the rollbacks I was chasing were exit-26 (ERR_VM_UNHANDLED), whereas a cleanup failure can only ever be exit-25 (ERR_VM_CLEANUP) (dnf_cli.py#L145-L151), which is already in VM_HANDLED (exit_codes.py#L34-L40) and never escalates to 26. So cleanup wasn't the cause. The real source of exit-26 was the teardown, which bd9a723 addresses.

[nihalxkumar]

f66d1e7

Unsure about this one also. If shutdown failed, the upgrade might have failed despite the package manager not returning an error during upgrade. Which exception were you receiving?

The exception was an AttributeError:

2026-06-26 07:52:44,585 Cannot shutdown fedora-42-xfce, because of error: 'DeviceCollection' object has no attribute 'get_assigned_devices'

It's raised in the pre-existing _has_assigned_pci_devices helper (qube_connection.py#L109-L112, added in 12e9e32 3 months ago) while __exit__ shutsdown the clone.

To your concern that a failed shutdown could mean a failed upgrade: in this case it didn't. The agent finishes cleanly: distro-sync exit 0, dnf clean packages exit 0 and the failure only appears afterwards, at shutdown (logs shared earlier):

2026-06-26 05:40:02,642 [Agent] run command: dnf clean all
2026-06-26 05:40:04,387 [Agent] command exit code: 0
2026-06-26 05:40:04,391 [Agent] run command: dnf --releasever=42 distro-sync --best --allowerasing --assumeyes
2026-06-26 07:52:09,778 [Agent] command exit code: 0
2026-06-26 07:52:09,798 [Agent] version-upgrade out: Removed 24 files, 11 directories (total of 87 MiB). 0 errors occurred.
2026-06-26 07:52:09,808 [Agent] version-upgrade out: 
2026-06-26 07:52:09,817 [Agent] Notify dom0 about upgrades.
2026-06-26 07:52:38,827 [Agent] run command: dnf clean packages
2026-06-26 07:52:39,630 [Agent] command exit code: 0
2026-06-26 07:52:43,427 Remove /run/qubes-update/
2026-06-26 07:52:43,427 run command in fedora-42-xfce: rm -r /run/qubes-update/
2026-06-26 07:52:43,428 Wait for output
2026-06-26 07:52:44,585 Cannot shutdown fedora-42-xfce, because of error: 'DeviceCollection' object has no attribute 'get_assigned_devices'
2026-06-26 07:52:44,586 agent output: 
2026-06-26 07:52:44,586 agent output: 
2026-06-26 07:52:44,587 agent exit code: 0

My dom0 is on qubes-core-admin-client-4.2.18, whose DeviceCollection still exposes assignments()/attached() and not get_assigned_devices(). The latter is the R4.3/main device API that this branch targets, so on an actual R4.3 dom0 this specific AttributeError shouldn't fire.

bd9a723 logs the shutdown error instead of letting a teardown exception propagate out of _run_agent, get caught by update_qube's broad except, and surface as ERR_VM_UNHANDLED which then rolls back and deletes a clone whose upgrade already completed. It also appends the agent's out/err to the UpgradeError, so the real cause reaches the tool log next time instead of being discarded.

We could also harden the source instead, _has_assigned_pci_devices fall back across both device APIs (or catch AttributeError there) so the shutdown path itself is robust.

[ben-grande]

My dom0 is on qubes-core-admin-client-4.2.18, whose DeviceCollection still exposes assignments()/attached() and not get_assigned_devices(). The latter is the R4.3/main device API that this branch targets, so on an actual R4.3 dom0 this specific AttributeError shouldn't fire.

Can you upgrade to R4.3? It would make testing much easier, cause currently you are testing an EOL release.

bd9a723 logs the shutdown error instead of letting a teardown exception propagate out of _run_agent, get caught by update_qube's broad except, and surface as ERR_VM_UNHANDLED which then rolls back and deletes a clone whose upgrade already completed. It also appends the agent's out/err to the UpgradeError, so the real cause reaches the tool log next time instead of being discarded.

The exception should be raised so the command fails. It won't be raised when you have R4.3.

[nihalxkumar]

I upgraded my system to R4.3 and the upgrade ran successfully end to end upgrade with no errors. I have reset the branch to d3ab852 as of june15.

The shutdown exception propagates, and failure detail it was trying to surface is already in /var/log/qubes/update-<name>.log, so no replacement was needed.

Next thing is disk handling for temporary cache volume

[ben-grande]

Hi. Could you also take a look at:

• Debian integration
• API usage for better progress reporting

[ben-grande]

In the future, this tool will be tested by openqa and when it is good enough, it should serve this use case:

• https://github.com/QubesOS/openqa-tests-qubesos/blob/152c563a4aea5eaa5446354f410fb5a4fa64af76/tests/distupgrade_template.pm#L41

In other words, it should be able to do, what this command from the link, does.

---

After you add API support, check if upgrading simultaneously with CLI and API breaks things or it is safely handled.

[nihalxkumar]

@ben-grande Should debian integration be also included in this PR?

[ben-grande]

Yes please.

[nihalxkumar]

1271315 (fedora disk handling) automates the manual cache-volume procedure. I went with attaching a throwaway cache volume to the clone every time instead of just as a fallback when dnf complains about space because a whole-release distro-sync on a template's small root disk will basically always need it, so it's provisioned upfront instead of waiting for the first attempt to fail.

dom0 creates a sparse 5GB image on disk, formats it ext4, and attaches it to the clone as a loop device through qvm-block, same as the manual steps. The one deliberate difference from the doc is that the agent mounts it by filesystem UUID rather than a device path like /dev/xvdi which isn't guaranteed stable.

Everything is torn down afterwards regardless of outcome: the loop device is detached and the image deleted on success, failure, or rollback, and a lock file lets a later run detect and reclaim an image left behind by one that got killed mid-way.

Open to design decisions, after the reviews I will test it on my system and then we can take on Debian upgrade.

[parulin]

I have some suggestion for better integration with the rest of the docs. If you have any question, feel free to ask.

This file is in doc/tools instead of doc/manpages, why?

[parulin]

As I understand, this tool should also be promoted in qubes-doc, in the user/templates/*/*-upgrade.rst files for fedora and debian.

[nihalxkumar]

@parulin you are mistaking this repository as qubes-core-admin-client

[nihalxkumar]

Debian upgrade end-to-end successful

https://gist.github.com/nihalxkumar/3f64e3a056e9ce367e9787d5bd94b9df
https://gist.github.com/nihalxkumar/212e885811589b85fdd23dafdfddf887