Arch Linux

@Haxxer64: Thanks for the report.

Note: Please make sure to attach the relevant file(s) to the ticket using the attachment functionality, as we otherwise have to rely upon non-free paste services such as pastebin.com, which are not very privacy friendly.

In regards to the issue: I do not use this type of setup so this will require some time to investigate.
There have been some changes [1] to the package since 202108-1:

* upgrade and CSM support [2]
* separating CSM images [3]
* add descriptor files for CSM, add TPM_ENABLE where possible [4]
* better naming scheme for descriptor files, S4 support, images for IA32 UEFI on x86_64 [5]

For good measure, are you able to test whether a new machine exhibits the same behavior?
I believe, that your issue is probably solved by an update to one of your configuration files (e.g. updated firmware location).

[1] https://github.com/archlinux/svntogit-packages/commits/packages/edk2/trunk
[2] https://github.com/archlinux/svntogit-packages/commit/c34ae40ea9e1dc243ebfc640a491681ff84af15d#diff-3e341d2d9c67be01819b25b25d5e53ea3cdf3a38d28846cda85a195eb9b7203a
[3] https://github.com/archlinux/svntogit-packages/commit/13a9fbbd85507bc3894277190b9de9f24b8bce47#diff-3e341d2d9c67be01819b25b25d5e53ea3cdf3a38d28846cda85a195eb9b7203a
[4] https://github.com/archlinux/svntogit-packages/commit/1b4a048efcdb771e431444ec6927c8a5d4982ded#diff-3e341d2d9c67be01819b25b25d5e53ea3cdf3a38d28846cda85a195eb9b7203a
[5] https://github.com/archlinux/svntogit-packages/commit/a123cfd55e608a31899b666a627028f968e89afd#diff-3e341d2d9c67be01819b25b25d5e53ea3cdf3a38d28846cda85a195eb9b7203a

@dvzrv, I have attached the libvirt XML.

I will try creating a new machine and see if this behavior repros, and get back to you.

Note that the firmware file path in my VM XML (/usr/share/edk2-ovmf/x64/OVMF_CODE.fd) is a valid file path. Is it possible that the nvram file (/var/lib/libvirt/qemu/nvram/win10-work_VARS.fd) is not compatible with the new firmware and needs to be recreated?

> Is it possible that the nvram file (/var/lib/libvirt/qemu/nvram/win10-work_VARS.fd) is not compatible with the new firmware and needs to be recreated?

That is possible. Do note, that changing the nvram file may have adverse effects on your guest OS. Please always make sure to have backups!

Just to note: edk2-ovmf 202111-4 appears to work for me another another person on the team with GPU passthrough (although we both have a dedicated GPU for just the VM).

also can you try providing ovmf debug output?
to do that change your libvirt.xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

and at the end of <domain> add

<qemu:commandline>
<qemu:arg value='-debugcon'/>
<qemu:arg value='stdio'/>
<qemu:arg value='-global'/>
<qemu:arg value='isa-debugcon.iobase=0x402'/>
</qemu:commandline>

then attach /var/log/libvirt/qemmu/vmname.log

Alexander, I added the extra qemu arguments like you suggested, and ran the VM using both old and new packages to see the difference between the logs. And they both look identical. But I'm able to repro the problem consistently. If you have any more tips to get any more logs I'm happy to collect them.

Note that I rebooted my machine after upgrading and downgrading this package to absolutely make sure they get picked up by any running services (I wasn't sure if any services use this package in the background).

@Svenstaro, when you run your VM with GPU passthrough, do you see the TianoCore logo or does it boot right into Windows? One of my VMs with GPU passthrough works with the new package but the TianoCore logo doesn't show up. The VM which doesn't boot up has Bitlocker enabled and is probably using some UEFI function to ask for the Bitlocker recovery key and I'm guessing that's what's causing it to fail.

Since I have an easy repro, I'm happy to try to build this package at various commits and kind of do a binary search to try and find the exact commit which causes this break. Can you please give me some tips on what this process would look like? I'm not sure how to build this package from source, and the range of commits I should be looking at.

I figured out how to build this package from source and was able to narrow down the break to this commit:
https://github.com/archlinux/svntogit-packages/commit/c34ae40ea9e1dc243ebfc640a491681ff84af15d

This is the commit which does the big upgrade of edk2 to 202111. I will continue digging into this and report back. If you have any tips that will be appreciated :)

> I figured out how to build this package from source and was able to narrow down the break to this commit:

yep that was expected. can you try building debug variant. and provide logs. maybe it will show something new.
to do that: change _build_type=RELEASE to _build_type=DEBUG in ovmf pkgbuild.

Ah, the debug build produced more log messages. I've attached it, hope it helps.

I'm also attaching the log output produced by the good package (also built with the debug config) so you can compare the 2 files to see what stands out

indeed according to -Graphics Console Started, Mode: 0
this doesn't happen on previous ovmf.

to understand if it's upstream regression or hour. you can update working pkg build to latest ovmf. I think it should build.
to do that just change pkgver.

Found the upstream commit which breaks my VM:
https://github.com/tianocore/edk2/commit/b8675deaa819631db2667df63f89799fe65fc906

All commits on master before this commit work successfully, and this one consistently fails.

I applied the reverse patch of the commit I mentioned above on top of tianocore/edk2 master branch and the VM boots up fine, which further proves that it's that particular commit which is bad. But it's not obvious why it's causing a failure.

I attached the log file when I ran the VM after applying the reverse patch on top of master

> But it's not obvious why it's causing a failure.

for me too. I recommend contacting upstream about it.
logs which you are gathered for us should be enough for them at list for start.

I think this ticket can be closed as A upstream issue.

I am talking to the upstream developers about this issue now and have filed a bug report:
https://bugzilla.tianocore.org/show_bug.cgi?id=3771

Can we please keep this ticket open until we release a patched version with a fix? This new version will break many people's workflows. This way it will they'll know what's going on and that a fix is pending?

Yeah sounds fair, let's keep it open. We usually close things as upstream if there's nothing we can do about it (like the nvidia packages) but it's fine to track the upstream progress in this case.

If there is a fix (e.g. revert) to apply that is ACK'ed by upstream, I'm happy to apply it.

This is the patch that the original author of the breaking change wants to submit to fix the bug:

https://github.com/stefanberger/edk2/commit/0207e3bb476b3efe12dee19688d7f035202cbc8d

They are still in the process of getting a code review done for it, so let's wait until this change gets approved.

I'll let you know if they do another patched release. If they do, then we can pick it up. Otherwise we can temporarily apply the patch ourselves until we pick up a newer release.

The fix has now been committed in the upstream repository.
The PR https://github.com/tianocore/edk2/pull/2325 has been checked in as https://github.com/tianocore/edk2/commit/ee1f8262b83dd88b30091e6e81221ff299796099

Can you please make a new release with this patch?
https://github.com/tianocore/edk2/commit/ee1f8262b83dd88b30091e6e81221ff299796099.patch

@Haxxer64: Please check whether 202111-5 in [testing] fixes your issue.

@dvzrv: Yes, I just tried out that package and it fixes the GPU passthrough issue. Thanks!

@Haxxer64 thanks for the confirmation and the upstream communication. Much appreciated!