FS#79857 - [grub] 2:2.12rc1-3 can fail in presence of XFS

Attached to Project: Arch Linux
Opened by Steven McFeely (smcfeely) - Thursday, 05 October 2023, 05:17 GMT
Last edited by Tobias Powalowski (tpowa) - Friday, 06 October 2023, 19:01 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Tobias Powalowski (tpowa)
Christian Hesse (eworm)
Architecture x86_64
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

Description:
After upgrading from grub 2:2.12rc1-1 to grub 2:2.12rc1-3, grub fails to load with the error message:

error: file '/grub/x86_64-efi/normal.mod' not found
Entering rescue mode...
grub rescue>

Additional info:
* I can confirm this happens with grub version 2:2.12rc1-3. I have not tried to use 2:2.12rc1-2. I can also confirm this issue occurs on both my laptop and desktop.

* Using an Arch Linux installation ISO I had on hand, I was able to remount everything and confirm that '/boot/grub/x86_64-efi/normal.mod' did exist on both systems. Therefore, the issue is not that the files do not exist.

* While unlikely, it is possible this is due to the following upstream bug: https://savannah.gnu.org/bugs/?64514
This bug was the only one I saw dated after the release of 2:2.12rc1-1 that mentioned normal.mod. That said, the error messages I received were about failing to find normal.mod, not a licensing issue.

* The attached file, 'grub', is the grub config file for the desktop while 'grub (system 2)' is for the laptop.

* For reference, on the desktop /boot is encrypted with luks1. The error message occurs after the decryption of /boot has completed. The laptop does not have an encrypted /boot.

Steps to reproduce:
1) Upgrade to grub 2:2.12rc1-3 using sudo pacman -Syu
2) As per the pacman output, reinstall grub and then regenerate the config files.
a) sudo grub-install --target=x86_64-efi --efi-directory=esp --bootloader-id=GRUB
b) sudo grub-mkconfig -o /boot/grub/grub.cfg
3) reboot

Note: For both my laptop and desktop esp is /efi
   grub (2.2 KiB)
   grub (system 2) (2.1 KiB)
This task depends upon

Closed by  Tobias Powalowski (tpowa)
Friday, 06 October 2023, 19:01 GMT
Reason for closing:  Fixed
Additional comments about closing:  2.12rc1-4
Comment by Toolybird (Toolybird) - Thursday, 05 October 2023, 07:39 GMT
Seems to work fine here i.e. cannot repro. The changes between the versions are minimal. There are no other reports (yet) so this suggests a config issue on your machine.

> sudo grub-install --target=x86_64-efi --efi-directory=esp --bootloader-id=GRUB

Did you really run it exactly like that? i.e. forgot to substitute "esp" with the correct mount point?
Comment by Tobias Powalowski (tpowa) - Thursday, 05 October 2023, 11:51 GMT
-3 only added new unifont and fixed ntfs module against security flaws.
Comment by Tomáš Golembiovský (nyoxi) - Thursday, 05 October 2023, 13:52 GMT
This is a problem also with grub-2:2.12rc1-3 at least on my machine. The source of the problem is that grub cannot properly read the XFS filesystem. When trying to list the content with 'ls /grub/x86_64-efi` from rescue shell it fails with `error: invalid XFS directory entry` and only some files are listed. Steven, can you confirm a similar thing is what happens also on your machine? What is the underlying filesystem of /boot?

This is apparently caused by more stricter check that were recently added to grub. It looks like the fix for it is on the way, but it's still going through development/reviews: https://lists.gnu.org/archive/html/grub-devel/2023-09/msg00110.html

I don't think the severity is Low in this case. The system becomes unbootable and there is no way to fix it from rescue shell. The only option is to boot from some Live image and reinstall older version of grub or move /boot onto different filesystem.
Comment by Tomáš Golembiovský (nyoxi) - Thursday, 05 October 2023, 14:05 GMT
Oh, I noticed just now that the bug was actually filed today for grub-2:2.12rc1-3.
Comment by loqs (loqs) - Thursday, 05 October 2023, 15:14 GMT
@nyoxi did the proposed fix you linked to fix your issue? If you have not yet tested it see attached diff.
   PKGBUILD.diff (8 KiB)
Comment by Tomáš Golembiovský (nyoxi) - Thursday, 05 October 2023, 15:17 GMT
I investigated a bit and did few more checks. This seems introduced in commit [1] which is new in 2.12-rc1. I didn't try rebuilding grub without it but I can confirm the issue exhibits also with grub-2:2.12rc1-1. What is more troubling that the issue does not exhibit every time which makes it harder to reproduce. Running grub-install sometimes breaks the boot and sometimes not, probably depending on how the files get laid out on the filesystem.

[1] https://git.savannah.gnu.org/cgit/grub.git/commit/?id=ef7850c757fb3dd2462a512cfa0ff19c89fcc0b1
Comment by Tomáš Golembiovský (nyoxi) - Thursday, 05 October 2023, 15:20 GMT
@loqs no I did not try the fix. I would be cautious with including it until it passes review. I can try rebuilding grub tomorrow to see if that helps or whether reverting the originating patch helps.
Comment by agapito fernandez (agapito) - Thursday, 05 October 2023, 16:37 GMT
I hit this bug a couple of months ago after a new installation on a XFS partition, chrooting my system and reinstalling grub fixed it. Last night I hit it again after upgrading grub, but i had no luck this time.

Fortunately I have another grub installed from another Arch's installation on a Btrfs partition and I could boot my main system thanks to the fact that I had already edited the grub.cfg previously to allow me to launch my main system kernel.

In short, and as others have commented before, this bug only seems to affect XFS users.
Comment by Steven McFeely (smcfeely) - Thursday, 05 October 2023, 17:47 GMT
Strictly speaking, I ran:

> sudo grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id=GRUB

I can also confirm that all of my partitions, except the EFI partition, use XFS.
Comment by Tobias Powalowski (tpowa) - Friday, 06 October 2023, 09:10 GMT
Please confirm working grub from here:
https://pkgbuild.com/~tpowa/grub-xfs-test/

It contains this fix:
https://lists.gnu.org/archive/html/grub-devel/2023-09/msg00113.html
Comment by agapito fernandez (agapito) - Friday, 06 October 2023, 15:13 GMT
@tpowa Not working, same error here.
Comment by Tomáš Golembiovský (nyoxi) - Friday, 06 October 2023, 15:55 GMT
I confirm that. Applying [1] does not help. However, reverting [2] or applying [3] fixes the issue.

[1] https://lists.gnu.org/archive/html/grub-devel/2023-09/msg00113.html
[2] https://git.savannah.gnu.org/cgit/grub.git/commit/?id=ef7850c757fb3dd2462a512cfa0ff19c89fcc0b1
[3] https://lists.gnu.org/archive/html/grub-devel/2023-09/msg00110.html
Comment by Tobias Powalowski (tpowa) - Friday, 06 October 2023, 16:36 GMT
I think we need both:
https://lists.gnu.org/archive/html/grub-devel/2023-09/msg00110.html
https://lists.gnu.org/archive/html/grub-devel/2023-09/msg00113.html

built a new one. Please try -5 package:
https://pkgbuild.com/~tpowa/grub-xfs-test/
Comment by agapito fernandez (agapito) - Friday, 06 October 2023, 17:41 GMT
-5 package works fine. Thanks.
Comment by Tobias Powalowski (tpowa) - Friday, 06 October 2023, 18:47 GMT
Thanks, building and uploading fixed -4 package to [testing} now.

Loading...