FS#66528 - [edk2-ovmf] Existing VMs no longer boot with compatability symlinks

Attached to Project: Arch Linux
Opened by Connor (kryesh) - Sunday, 03 May 2020, 23:57 GMT
Last edited by David Runge (dvzrv) - Tuesday, 05 May 2020, 14:10 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Anatol Pomozov (anatolik)
David Runge (dvzrv)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

libvirt/virt-manager VMs using the following firmware path no longer boot:
"/usr/share/ovmf/x64/OVMF_CODE.fd"

After investigating the issue a little bit I noticed that
"/usr/share/ovmf/x64/OVMF_CODE.fd" was symlinked to "/usr/share/edk2-ovmf/x64/OVMF_CODE.secboot.fd"

moving "/usr/share/ovmf/x64/OVMF_CODE.fd" to "/usr/share/ovmf/x64/OVMF_CODE.secboot.fd" and creating a new symlink "/usr/share/ovmf/x64/OVMF_CODE.fd" -> "/usr/share/edk2-ovmf/x64/OVMF_CODE.fd" resolved the issue and all VMs using the firmware now boot correctly

Additional info:
* package version(s)
edk2-ovmf 202002-7
libvirt 5.10.0-3
virt-manager 2.2.1-2
linux-zen 5.6.8.zen1-1

Tested using ubuntu 18.04,19.04, and a windows 10 vm

Steps to reproduce:
After updating edk2-ovmf to version 202002-7, existing VMs using firmware located at "/usr/share/ovmf/x64/OVMF_CODE.fd" no longer boot
Updating symlink to point to the non-secureboot firmware allows the VMs to boot

libvirt os config from ubuntu 19.04 vm:
<os>
<type arch="x86_64" machine="pc-i440fx-4.1">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/ubuntu19.04_VARS.fd</nvram>
<boot dev="hd"/>
</os>
This task depends upon

Closed by  David Runge (dvzrv)
Tuesday, 05 May 2020, 14:10 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed with edk2-ovmf 202002-8
Comment by Anatol Pomozov (anatolik) - Monday, 04 May 2020, 01:21 GMT
What is the best way to reproduce this problem?
Comment by Connor (kryesh) - Monday, 04 May 2020, 01:39 GMT
I'm not certain about specific steps to reproduce, but when have have my system configured as shown in the attached screenshots the vm doesn't boot and shows a black screen.

Applying the symlink changes described in the initial report and restarting the vm allows it to boot normally

Is there any specific information you'd like about how my system is configured? I haven't taken any steps to enable secureboot on either the host or the vm
Comment by Anatol Pomozov (anatolik) - Monday, 04 May 2020, 02:49 GMT
I need a test case that shows this problem is fixed.

Additionally there were a few edk2 refactorings recently (and it introduced a few issues with the firmware files e.g. https://bugs.archlinux.org/task/66322) plus we are going to have more refactoring in QEMU (see https://bugs.archlinux.org/task/66460). I want to have some usecase (like yours) that shows that firmware location change does not introduce any new regressions.
Comment by Anatol Pomozov (anatolik) - Monday, 04 May 2020, 03:22 GMT
Here is proposed fix. Connor if you have chance could you please build the new package with provided patch and make sure that edk2-ovmf still works for you.
Comment by David Runge (dvzrv) - Monday, 04 May 2020, 05:56 GMT
@kryesh: Thanks for the report.

The symlinks are correct though: The ovmf package *only* provided "secure boot" firmware descriptors (but without `SMM_REQUIRE`) and no "non-secure boot" firmware descriptors.
There is a a configuration option to make SMM [1] enabled "secure boot" firmware descriptors work (see  FS#66322  [2]).

Please test this first, as otherwise you boot without secure boot!

[1] https://en.wikipedia.org/wiki/System_Management_Mode
[2] https://bugs.archlinux.org/task/66322#comment188602
Comment by David Runge (dvzrv) - Monday, 04 May 2020, 05:59 GMT
@kryesh: As a follow-up to explain what @nl6720 was referencing, please read this comment [1] in  FS#59465 .

[1] https://bugs.archlinux.org/task/59465#comment172528
Comment by Toolybird (Toolybird) - Monday, 04 May 2020, 08:03 GMT
> The symlinks are correct though: The ovmf package *only* provided "secure boot" firmware descriptors

@dvzrv, I think you might be mistaken here, which likely explains the bug.

Personally I've never bothered with "secure boot" and everything was working fine with the ovmf package.

In the new package, 2 of the symlinks look wrong as pointed out by @kryesh. Apologies for not spotting it earlier.
Comment by Connor (kryesh) - Monday, 04 May 2020, 09:33 GMT
@anatolik Thanks for the patch! the VMs booted without any issues using the patched package.

@dvzrv Thank you for the explanation! After spending some time tinkering after work it looks like with my current VM configuration I might have shot myself in the foot - reading through the pages you linked they mention needing to use q35 rather than i440fx. Creating a new vm with q35 and the secureboot firmware works perfectly.

Looks like I should probably move the disks from my current vms to new libvirt configs with the q35 chipset anyway as it looks like that option will have better support moving forwards.

Thanks both of you for the help!
Comment by David Runge (dvzrv) - Monday, 04 May 2020, 09:54 GMT
> @dvzrv, I think you might be mistaken here, which likely explains the bug.

@Toolybird: No, I'm not (see attached ovmf PKGBUILD from revision r382068).

> Personally I've never bothered with "secure boot" and everything was working fine with the ovmf package.

That doesn't really matter, as it's apparently what you have been using ;-)
As pointed out earlier, the SMM setting is now included (but previously was not), which is likely the cause of troubles moving forward.

@kryesh: Glad this was of help! If you have any suggestions or improvements, it's always great to have a more correct and up-to-date wiki about this! :)
   PKGBUILD (3.4 KiB)
Comment by nl6720 (nl6720) - Monday, 04 May 2020, 10:44 GMT Comment by Toolybird (Toolybird) - Monday, 04 May 2020, 12:37 GMT
> No, I'm not

Ugh, my point was that there is indeed a bug. The fact is - folk's VM's stopped booting[1]. The compatibility symlinks were not compatible. Thankfully Anatol's patch fixes that.

It's true that the previous ovmf pkg had the secure boot setting, but it was off by default so most folks wouldn't notice a problem. It's also true that the JSON descriptor file that I hacked together was not ideal due to the ambiguity surrounding secure boot. Anyway, thanks to your good work we now have a much saner setup going forward. Thanks.

[1]: https://www.reddit.com/r/VFIO/comments/gc4pql/vm_not_booting_after_latest_arch_upgrade/
Comment by David Runge (dvzrv) - Monday, 04 May 2020, 17:29 GMT
@Toolybird: My apologies! I looked at this under the assumption, that ovmf was in fact providing (as in requiring) "secure boot", in which case the symlinks would have been correct.
But as it stands ovmf only provided the *possibility* of "secure boot" enabled firmware descriptors and never enforced them (in the qemu .json descripter files)!

Now that all this confusion is cleared up I'm happy to change the symlinks!

Loading...