FS#64175 - [libvirt] fails to detect OVMF/UEFI firmware (regression)

Attached to Project: Community Packages
Opened by Jaume Delclòs Coll (cosarara) - Saturday, 19 October 2019, 09:40 GMT
Last edited by Anatol Pomozov (anatolik) - Saturday, 11 January 2020, 00:30 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture All
Severity Low
Priority High
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 28
Private No

Details

Description:
The behaviour is described in detail in this forum post: https://bbs.archlinux.org/viewtopic.php?id=250066
In short, even when ovmf is installed and /etc/libvirt/qemu.conf is properly configured, libvirt cannot find the ovmf images.

Additional info:
* The bug happens with libvirt-5.8.0-1
* Disappears when downgrading to libvirt-5.6.0-1

Steps to reproduce:
From https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_an_OVMF-based_guest_VM

* After installing qemu, libvirt, ovmf, and virt-manager, add the path to your OVMF firmware image and runtime variables template to your libvirt config so virt-install or virt-manager can find those later on.

/etc/libvirt/qemu.conf

nvram = [
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]

* You can now enable and start libvirtd.service and its logging component virtlogd.socket.

* Add your user to the libvirt user group to ensure authentication.

* Run virt-manager and create a new VM with the System kvm/qemu connection.

* When the VM creation wizard asks you to name your VM (final step before clicking "Finish"), check the "Customize before install" checkbox.

* In the "Overview" section, set your firmware to "UEFI". Except you can't because the option is grayed out (this is the bug).
This task depends upon

Closed by  Anatol Pomozov (anatolik)
Saturday, 11 January 2020, 00:30 GMT
Reason for closing:  Fixed
Additional comments about closing:  libvirt-5.10.0-2
Comment by Joshua Marsh (icub3d) - Saturday, 19 October 2019, 14:52 GMT
I noticed in the qemu.conf that the config for nvram was "obsolete". It looks like they are expecting JSON configs like in /usr/share/qemu/firmware/ to find firmware (see: https://git.qemu.org/gitweb.cgi?p=qemu.git;a=blob_plain;f=docs/interop/firmware.json;hb=HEAD).

If you install extra/qemu-arch-extra, it will populated some of the firmwares that are already setup in /usr/share/qemu/firmware/. Those will then show up in the virt-manager firmware drop-down list.

I'm wondering if we need such a config for ovmf?

Comment by Julian Phillips (chr0mag) - Saturday, 19 October 2019, 17:08 GMT
I just tried this by copying the '60-edk2-ovmf.json' example located here: https://src.fedoraproject.org/rpms/edk2/c/674b3c8a27a8 -- to '/etc/qemu/10-ovmf.json', changing the paths to match those of the Arch 'ovmf' package and launching virt-manager. It now finds the OVMF firmware. So yes, at least one such config should be included in the 'ovmf' package. The above Fedora link is a good starting point for example configs.

Also note that existing VMs that use the ovmf firmware run just fine, presumably because the nvram tag in the image XML definition points directly to the per-image firmware in /var/lib/libvirt/qemu/nvram/. I suspect it's also possible to create a new VM using 'virsh install' and passing it an XML config pointing directly to the base nvram firmware but I haven't actually tried this.
Comment by loqs (loqs) - Saturday, 19 October 2019, 17:10 GMT Comment by David J. Haines (dhaines) - Saturday, 19 October 2019, 21:17 GMT
This regression presents itself similarly via `virt-install`:

```
$ virt-install --name test --disk size=64 --network network=default --os-variant archlinux --boot uefi --memory 8192 -d
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (cli:208) Launched with command line: /usr/share/virt-manager/virt-install --name test --disk size=64 --network network=default --os-variant archlinux --boot uefi --memory 8192 -d
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (virt-install:207) Distilled --network options: ['network=default']
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (virt-install:139) Distilled --disk options: ['size=64']
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (cli:224) Requesting libvirt URI default
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (cli:227) Received libvirt URI qemu:///system
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (storage:140) Found default pool name=default target=/var/lib/libvirt/images
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (storage:208) refreshing pool=default
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (disk:225) Creating volume 'test.qcow2' on pool 'default'
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (disk:359) disk.set_vol_install: name=test.qcow2 poolxml=
<pool type='dir'>
<name>default</name>
<uuid>a853f421-5115-4559-b757-a85b11dd337e</uuid>
<capacity unit='bytes'>274810798080</capacity>
<allocation unit='bytes'>11137085440</allocation>
<available unit='bytes'>263673712640</available>
<source>
</source>
<target>
<path>/var/lib/libvirt/images</path>
<permissions>
<mode>0755</mode>
<owner>0</owner>
<group>0</group>
</permissions>
</target>
</pool>

[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (guest:463) Setting Guest osinfo name <_OsVariant name=generic>
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (installer:396) No media for distro detection.
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (installer:398) installer.detect_distro returned=None
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (guest:463) Setting Guest osinfo name <_OsVariant name=archlinux>
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (osdict:326) No recommended value found for key='ram', using minimum=536870912 * 2
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (cli:263) File "/usr/share/virt-manager/virt-install", line 1012, in <module>
fail(main_e)
File "/usr/share/virt-manager/virtinst/cli.py", line 263, in fail
log.debug("".join(traceback.format_stack()))

[Sat, 19 Oct 2019 17:15:56 virt-install 8171] ERROR (cli:264) Did not find any UEFI binary path for arch 'x86_64'
[Sat, 19 Oct 2019 17:15:56 virt-install 8171] DEBUG (cli:266)
Traceback (most recent call last):
File "/usr/share/virt-manager/virt-install", line 1005, in <module>
sys.exit(main())
File "/usr/share/virt-manager/virt-install", line 993, in main
guest, installer = build_guest_instance(conn, options)
File "/usr/share/virt-manager/virt-install", line 560, in build_guest_instance
installer.set_install_defaults(guest)
File "/usr/share/virt-manager/virtinst/install/installer.py", line 340, in set_install_defaults
guest.set_defaults(None)
File "/usr/share/virt-manager/virtinst/guest.py", line 729, in set_defaults
self._set_default_uefi()
File "/usr/share/virt-manager/virtinst/guest.py", line 787, in _set_default_uefi
path = self.get_uefi_path()
File "/usr/share/virt-manager/virtinst/guest.py", line 559, in get_uefi_path
"arch '%s'") % self.os.arch)
RuntimeError: Did not find any UEFI binary path for arch 'x86_64'
```
Comment by Julian Phillips (chr0mag) - Saturday, 19 October 2019, 23:00 GMT
Sorry, I conflated 'virt-install' with 'virsh create/define'. The latter 2 accept as arguments a domain XML file which you can point directly to the nvram firmware. The following allows you to create a new UEFI/OVMF VM in v5.8:
*virsh dumpxml <existing-uefi-vm-name> virsh-test.xml -- to get a template to work with
*edit virsh-test.xml -- change the name, delete the description, uuid, os/nvram tags and set the only hard disk to the arch iso.
*virsh define virsh-test.xml --validate
*open virt-manager -- the new UEFI/OVMF VM should appear and can be launched successfully

Hand editing XML is awful, but it does work.
Comment by Toolybird (Toolybird) - Sunday, 20 October 2019, 00:01 GMT
This smells like an accidental upstream change (ie: a bug). I doubt that upstream intentionally set out to break existing setups, especially when reading the comments in these commits:

https://www.redhat.com/archives/libvir-list/2019-May/msg00003.html
https://www.redhat.com/archives/libvir-list/2019-September/msg00485.html

The future is clearly the new FW descriptor JSON thing, but existing setups shouldn't just fall over. I might file an upstream bug after some more digging..

Comment by David J. Haines (dhaines) - Sunday, 20 October 2019, 01:50 GMT
The workaround mentioned above is almost, but not quite, correct. Try this instead (as root):

```
mkdir -p /etc/qemu/firmware
sed 's#qemu/edk2-x86_64-code.fd#ovmf/x64/OVMF_CODE.fd#;s#qemu/edk2-i386-vars.fd#ovmf/x64/OVMF_VARS.fd#' < /usr/share/qemu/firmware/60-edk2-x86_64.json > /etc/qemu/firmware/10-ovmf-workaround.json
systemctl restart libvirtd
```
Comment by Jeremy Audet (ichimonji10) - Monday, 21 October 2019, 02:02 GMT
I can confirm that the workaround provided by dhaines works. (Thanks, btw!)
Comment by Toolybird (Toolybird) - Monday, 21 October 2019, 20:21 GMT
Went ahead and filed an upstream bug[1] but no response yet.

We should really just get with the times and provide a JSON file via the ovmf package.

I've tested a patch and submitted it here[2]

It achieves exactly the same effect as the workaround / config edits mentioned above.

1. https://bugzilla.redhat.com/show_bug.cgi?id=1763477
2.  FS#64206 
Comment by Jeremy Audet (ichimonji10) - Tuesday, 22 October 2019, 17:27 GMT
I can also confirm that the patch provided by Toolybird works. See bug report for details. (Yes, I removed `/etc/qemu/firmware` as part of my testing procedure.)
Comment by Sim Sun (imi415) - Wednesday, 23 October 2019, 07:46 GMT
Added a similar file to /usr/share/qemu/firmware did solved this problem.
Here is my file for testing: /usr/share/qemu/firmware/60-ovmf-x86_64.json
Comment by Cyrus Frost (cyfrost) - Friday, 25 October 2019, 16:44 GMT
Can confirm that the workaround provided by dhaines works, thank you.
Comment by Toolybird (Toolybird) - Sunday, 10 November 2019, 20:13 GMT
Upstream has spoken and the verdict is in - libvirt is working as designed and Arch needs to adapt.

There are 2 problems:

1. qemu defaults to installing pre-built edk2 blobs and JSON files. The mere presence of a Firmware Desciptor JSON file on a system is enough for libvirt to ignore the nvram setting in /etc/libvirt/qemu.conf. This is by design.

2. Arch needs to modernize and ship an appropriate JSON file with the OVMF package.


1. can be fixed by following the Fedora lead in their qemu spec file:

# Provided by package edk2
rm -rf %{buildroot}%{_datadir}/%{name}/edk2*
rm -rf %{buildroot}%{_datadir}/%{name}/firmware/*edk2*.json

2. can be fixed by applying the patch provided in  FS#64206 
Comment by Dario (dario86) - Tuesday, 19 November 2019, 00:46 GMT
The patched package works, however virt-manager fails to boot a newly created UEFI machine with the following error message: unable to find any master var store for loader: /usr/share/ovmf/x64/OVMF_CODE.fd. To boot the machine I must add the following line in the <os> section of the XML file in Overview: <nvram template="/usr/share/ovmf/x64/OVMF_VARS.fd">/dev/mapper/nvram</nvram>

Edit: sorry, this is the line that works for me: <nvram template="/usr/share/ovmf/x64/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/PARABOLA_VARS.fd</nvram>
Comment by Diana (DianaNites) - Saturday, 14 December 2019, 23:25 GMT
Can confirm these packages are still kinda broken when you need UEFI, and apparently have been for months now, as I discovered today...
Comment by Anthony Rabbito (anthr76) - Sunday, 15 December 2019, 02:58 GMT
This was halfway working before I updated a small amount of packages on my system today. Before today I would be able to use existing VMs and run them. Now I cannot start any VMs that have OVMF on them. Previously I was able to create a BIOS VM and point it to the OVMF vars manually in XML edits.

This is frustrating. I'm surprised there's only a few amount of people being effected by this.
Comment by Martin Rys (C0rn3j) - Sunday, 15 December 2019, 10:53 GMT
> Previously I was able to create a BIOS VM and point it to the OVMF vars manually in XML edits.
Stop frankensteining things. Apply the JSON file like the patch does.

Most people aren't affected by this because they're happy running BIOS VMs only.

EDIT: After updating libvirt from 5.9 to 5.10 and restarting libvirtd I am running into the same issue, looks like the new version broke things even further.
Comment by Samuel Roberts (sproberts92) - Monday, 16 December 2019, 22:00 GMT
I can confirm that the workarounds above worked on libvirt 5.9 but not 5.10. The error is now:

libvirt.libvirtError: operation failed: unable to find any master var store for loader: /usr/share/ovmf/x64/OVMF_CODE.fd

So it looks like the nvram vars file isn't being created, where before it was created automatically.

I'm running with a user session, so for me it should be created the following path (based on existing VMs where it was created automatically):
~/.config/libvirt/qemu/nvram/<vm-name>_VARS.fd

But this isn't happening.
Comment by Martin Rys (C0rn3j) - Monday, 16 December 2019, 22:03 GMT
Someone elsewhere mentioned that deleting the contents of /usr/share/qemu/firmware made it work again along with the other workarounds.
Comment by Samuel Roberts (sproberts92) - Monday, 16 December 2019, 22:56 GMT
Great, it does indeed work when combined with an nvram entry added to the xml as mentioned above. And so far it seems to continue to work when I put everything back in /usr/share/qemu/firmware too.

Still a regression from the original behaviour, but definitely usable for now, thanks.
Comment by Stavros Papadopoulos (stafpapa) - Thursday, 26 December 2019, 12:37 GMT
I can confirm that for libvirt 5.10 adding this line <nvram template="/usr/share/ovmf/x64/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram> (taken from https://bugs.archlinux.org/task/64206) to the os section of the xml, fixes the issue and UEFI is recognized correctly. No need to erase /usr/share/qemu/firmware. Have tested with archlinux guest and it works fine.

The issue is however is windows 10, which although now boots up, it is not able to load the iso for installation from the cdrom. Even trying exit->boot manager and selecting the DVD rom, it is still not able to start the installation process (the screen just blinks and reloads).

Comment by Stuart Cardall (itoffshore) - Wednesday, 01 January 2020, 19:00 GMT
I found a fix here with the json that needs to be shipped with ovmf pkg:

https://forum.level1techs.com/t/solved-ovmf-linuxarch-mandjaro-ovmf-uefi-not-available-in-virt-manager/149555

create: /usr/share/qemu/firmware/60-ovmf-x86_64.json
----------------------------------------------------
{
"description": "UEFI OVMF firmware for x86_64",
"interface-types": [
"uefi"
],
"mapping": {
"device": "flash",
"executable": {
"filename": "/usr/share/ovmf/x64/OVMF_CODE.fd",
"format": "raw"
},
"nvram-template": {
"filename": "/usr/share/ovmf/x64/OVMF_VARS.fd",
"format": "raw"
}
},
"targets": [
{
"architecture": "x86_64",
"machines": [
"pc-i440fx-*",
"pc-q35-*"
]
}
],
"features": [
"acpi-s3",
"amd-sev",
"verbose-dynamic"
],
"tags": [
]
}

& finally = systemctl restart libvirtd

Comment by edac val (edacval) - Thursday, 02 January 2020, 20:32 GMT Comment by Eli Schwartz (eschwartz) - Wednesday, 08 January 2020, 04:11 GMT
  • Field changed: Attached to Project (Arch Linux → Community Packages)
  • Field changed: Summary (Libvirt fails to detect OVMF/UEFI firmware (regression) → [libvirt] fails to detect OVMF/UEFI firmware (regression))
Wrong project. The best way to submit a bug is through the "add new bug" link on the project details.
Comment by Anatol Pomozov (anatolik) - Wednesday, 08 January 2020, 04:44 GMT
 FS#64206  suggests that a config file should be added to OVMF. It is still not clear for me how libvirt release forces OVMF to be become broken.
Comment by Anatol Pomozov (anatolik) - Wednesday, 08 January 2020, 21:04 GMT
Please check ovmf in [testing] and let me know if it works for you.
Comment by Dario (dario86) - Thursday, 09 January 2020, 00:02 GMT
It works, but I still need to add the following line in the XML of a newly created virtual machine in the <os> section:
<nvram template="/usr/share/ovmf/x64/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/[whatever name of the virtual machine]_VARS.fd</nvram>
If I don't add this line I get the error message: «unable to find any master var store for loader: /usr/share/ovmf/x64/OVMF_CODE.fd».
Comment by loqs (loqs) - Thursday, 09 January 2020, 00:30 GMT Comment by Anatol Pomozov (anatolik) - Thursday, 09 January 2020, 00:40 GMT
Dario it seems a different bug. Also - here is upstream tracker for this issue https://bugzilla.redhat.com/show_bug.cgi?id=1776949

UPDATE: scratch this statement. I was confused this issue with  FS#64206 
Comment by Dario (dario86) - Thursday, 09 January 2020, 01:05 GMT
@anatolik, @loqs, patched libvirt now works as expected.
Comment by Anatol Pomozov (anatolik) - Thursday, 09 January 2020, 05:09 GMT
Ioqs thank you for the debugging.

libvirt-5.10.0-2 with the upstream patches been pushed to [testing]. Please install this and 'ovmf' package from [testing] and let us know the results.
Comment by Dario (dario86) - Thursday, 09 January 2020, 06:27 GMT
Packages in [testing] work as expected.

Loading...