FS#68808 - [qemu] consider qemu-headless removal when bumping to 5.2.0

Attached to Project: Arch Linux
Opened by tinywrkb (tinywrkb) - Tuesday, 01 December 2020, 12:05 GMT
Last edited by David Runge (dvzrv) - Monday, 02 May 2022, 15:56 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Anatol Pomozov (anatolik)
David Runge (dvzrv)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Resurrecting #63162, where after some optional features of QEMU moved into external runtime loadable modules and made it possible to move so depends in optdepends.
This bug report suggests to consider removing qemu-headless as the full build should be able to run correctly even when missing the libs required for GUI and audio related features, and this is as long as they are not enabled at runtime.
Libs that are needed for GUI and audio features should be moved into optdepends so users can still have a minimal installation.

In the 5.2.0 release, the following features have been moved into modules: spice, opengl.

On my system, comparing the output of "$ ldd qemu-system-x86_64" between headless 5.1.0-3 and git master full build, the only extra libs in the full build are:

libdrm.so.2
libepoxy.so.0
libexpat.so.1
libgbm.so.1
libvirglrenderer.so.1
libwayland-server.so.0
libX11.so.6
libXau.so.6
libxcb.so.1
libXdmcp.so.6

I believe these are all coming from enabling virtio-gpu/virgl as they also appear in my private headless build that includes virgl.
I wouldn't be surprised if these will disappear also from the ldd output in post 5.2.0 releases as some work has been done to move virtio-gpu into a module but we should ping Gerd Hoffmann to inquire about this.
Even in a headless system, one can have a GPU so virgl is very much wanted in the headless package, in other words I think this difference in the lib list is negligible and does not justify the existence of a split package.
In contrast to #63162, I didn't test QEMU's behavior when these modules' dependencies are missing so proper testing is needed.


Links:

* #63162: https://bugs.archlinux.org/task/63162
* opengl module commit: https://github.com/qemu/qemu/commit/c8263659f1268a0f3502568d7663f722b2461935
* spice module commit: https://github.com/qemu/qemu/commit/cbe5fa11789035c43fd2108ac6f45848954954b5
* virtio-gpu module commit: https://github.com/qemu/qemu/commit/7b0de5b7969e695d6235525654a8a28a0d82a0e0
This task depends upon

Closed by  David Runge (dvzrv)
Monday, 02 May 2022, 15:56 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed with qemu >= 7.0.0
Comment by Josef Mutzenbacher (Joe2020) - Saturday, 05 December 2020, 20:14 GMT
Sometimes it seems as if "headless" is used in a ambiguous way, as it could refer to the host, the guest or both as well.
Maybe there should be a definition what is actually meant or another postfix should be used, like "qemu-minimal".

While I see the advantage of heaving a unified qemu package, I don't think a term like "the only extra libs are..." meets the point.
Those extra libs are a pita in some usecases.

There are hosts dedicated to pure hosting of (sometimes hundreds) of guests. The more minimalistic the host can be the better the situation is.
In such scenarios you don't want to have any X11, wayland, drm or OpenGL stuff installed.

My dedicated "hosting-hosts" use openssh, libvirt, qemu-headless and iSCSI. Sufficient to get the job done.

To make a long story short: My suggestion is to not use the ongoing modularization of qemu-5.2 to unify [qemu] and [qemu-headless], but to further strip down [qemu-headless] - at least for now.
Comment by Toolybird (Toolybird) - Thursday, 03 February 2022, 22:25 GMT
So, latest QEMU is now even more modular than ever. There are many options depending on how far you want to go with pkg splits etc. This is what I've come up with so far:

- new pkg "qemu-core". This takes over from "qemu-headless". It's even slimmer than current headless which should keep the minimalists happy. There is no qxl, no spice (or related shite), no GUI, no OpenGL, no audio, basically just core deps as per Fedora. At the pkg relationship level, this "provides" and "conflicts" with qemu-headless to ensure smooth upgrades.

- qemu. This now depends on "qemu-core" and supplies the additional modular bits for GUI, audio, spice, OpenGL, etc.

The main advantage is we don't have 2 separate QEMU builds anymore thus simplifying the PKGBUILD a lot. The main problem could be the lack of spice in core (I don't know if headless fans depend on spice or not?).

I've made a prototype PKGBUILD [1] based on the above. To be honest, I haven't tested it thoroughly yet, but it appears sane to me. It's based on my personal setup so will need slight tweaking to be usable on Arch.

Any comments? If you think this is the right approach I'll give it some proper testing and submit a version compatible with Arch.

[1] https://gitlab.com/-/snippets/2248456
Comment by tinywrkb (tinywrkb) - Thursday, 03 February 2022, 22:48 GMT
This sounds like the suggested package split doesn't change much and saves a rebuild. Sounds good.
It doesn't change the fact that at this point qemu is packaged incorrectly, and optional runtime dependencies are in the depends() array instead of the optdepends() array.
I personally gave up, I don't care anymore, I switched to running qemu+libvirt in a Podman Pod.
Comment by David Runge (dvzrv) - Thursday, 03 February 2022, 23:01 GMT
@tinywrkb, @Toolybird: Already got that on my list (see https://bugs.archlinux.org/task/73041#comment204489). Should happen before end of February :)
Comment by Toolybird (Toolybird) - Thursday, 03 February 2022, 23:18 GMT
> optional runtime dependencies are in the depends() array instead of the optdepends() array

Yeah, that is certainly another valid solution, similar to how the brltty dep is handled. I was aiming for a more "balanced" approach between old and new setups. I might rethink that though..
Comment by Toolybird (Toolybird) - Friday, 04 February 2022, 23:24 GMT
After thinking about it some more, placing all the modular bits into optional deps should be the ultimate goal. But I can forsee this resulting in an explosion of support requests in the forum. The Wiki will need editing too.

Going with my current approach would be less disruptive from a "keeping most existing setups working" POV. Consider it transitional.

> Should happen before end of February :)

@dvzrv, no stress, you seem to have a lot on your plate already!
Comment by David Runge (dvzrv) - Friday, 25 February 2022, 11:37 GMT
I have worked on a complete revamp of the PKGBUILD over the past few days:

You can see the results here: https://gitlab.archlinux.org/dvzrv/pkgbuilds/-/blob/master/qemu/PKGBUILD

The upside would be:
* everything is split up and can be installed in a very granular fashion
* no extra compilation required
* a few meta packages replace the functionality of existing packages (or actual packages): qemu (qemu-desktop), qemu-headless (qemu-base), qemu-{,headless-}arch-extra (qemu-emulators-full)
* accelerator tests are split out (qemu-tests) and not installed by default
* docs are split out (qemu-docs) and not installed by default
* a new meta package allows for installing everything: qemu-full

The downside would be:
* **many** packages to sign
* changes to the wiki are required
* writing a website announcement as the default (install the qemu package) would no longer do what it did before

All in all I believe that the modularity wins over the status quo or only modularizing half-way though, as the reduction in attack surface by being able to pick and match components is the superior approach (although it will be a bit more annoying to package).
Comment by David Runge (dvzrv) - Wednesday, 16 March 2022, 16:25 GMT
I have now split out the meta packages replacing some of the current main packages:
https://gitlab.archlinux.org/dvzrv/pkgbuilds/-/blob/master/qemu-meta/PKGBUILD

The qemu pkgbase gained binfmt support in the qemu-user package and I have added stubs for qemu-user-static as well (this is currently blocked by pcre and glib2 not yet providing static libs).
https://gitlab.archlinux.org/dvzrv/pkgbuilds/-/blob/master/qemu/PKGBUILD

Please note that the PKGBUILDs are still kinda WIP (they should still get a few .install files that explain the difference between the old and the new setup).
I intend to get back to this early April, as I am currently on holidays.

It is not unlikely that qemu 7.0 is released before I'm back, @anatolik it would be great if you could either hold off on the upgrade then or implement the proposed changes.
I currently lack the successive time to spend on this for proper testing in staging/testing before releasing it (before I'm back).
Comment by Gerd v. Egidy (gvegidy) - Tuesday, 22 March 2022, 17:49 GMT
I was just about to open an issue for requesting a split out qemu-img package when I saw this.

My reason for wanting this is to allow qcow2 image creation and extraction on a live cd (https://www.system-rescue.org/),
but without installing all of qemu and it's disk space usage.

I would appreciate if the new PKGBUILD with the many split packages would become the official build.
Comment by David Runge (dvzrv) - Sunday, 10 April 2022, 10:35 GMT
I'll try to pick up work on this in the coming days again. Had to spend some time on catching up on packaging after vacation.
Comment by Toolybird (Toolybird) - Thursday, 21 April 2022, 22:51 GMT
@dvzrv, good work on the big re-org! I haven't reviewed in full but here's a few quick comments:

- the splitting seems a bit extreme, almost un-Arch like. I worry this will lead to requests for other pkgs to be more finely split, à la other big distros. "Well, it's fine for QEMU why not pkg XYZ?"

- some new configure switches seem to be unnecessary and just cause duplication:

--enable-lto : this simply enables "-flto=auto" which is already covered by our devtools

--extra-ldflags="$LDFLAGS"
--extra-cflags="$CFLAGS" : our *FLAGS are already well picked up. These switches have a specific purpose which is documented here [1]. Was there a reason you added these?

--audio-drv-list="pa,jack,alsa,oss" : we previously addressed this here [2]. You've probably based this on the summary displayed at end of configure which can be misleading in my experience.

Anyway, overall this is definitely an improvement, well done!

PS - sorry for late reply. Been busy on toolchain stuff...

[1] https://gitlab.com/qemu-project/qemu/-/commit/a2866660
[2] https://bugs.archlinux.org/task/73041
Comment by David Runge (dvzrv) - Friday, 22 April 2022, 07:03 GMT
@Toolybird: Thanks for the heads up! I will fix the configure flags (to be quite frank, a large "configure script" in front of meson is very counterproductive and it is easier to forget things that were there for testing purposes :S).

In regards to the splitting of the package:
My stance on this is, that although on Arch we do not split e.g. headers and libs of libraries, we do have split packages and we do use them to cut down on explicit installation size (e.g. when splitting out docs) and on implicit installation size (e.g. when splitting out functionality that pulls in other dependencies).
In qemu's case it is a bit of both. The project and some of its components are quite large, which certainly warrants a more finely grained access to them. Meta packages (also those done later on by users for their specific use-cases) are a great way to pick sets of functionality and are far more easy (and faster) to change than recompiling all of qemu only to change a dependency.
Although I have to admit that the split package amount is high (and makepkg is slow working with this), I do not believe there is such a thing as "un-Arch like" in this regard (as in: it does not conflict with Arch's principles). The current approach has been chosen quite pragmatically, as it provides the largest possible set of user choices (e.g. tooling such as qemu-img can now be installed separately) while allowing very targeted scenarios with a small footprint.j

Loading...