Arch Linux

Root FS is ext4. (I've seen the bug report on f2fs.)

Is there anything in the journal from a boot with the issue? Have you bisected [1] the kernel to find the causal commit?

[1] https://wiki.archlinux.org/title/Kernel#Debugging_regressions

Nothing in the logs, it doesn't reach the point where it can start logging. Is there a way to increase the log level for this phase of the boot process? It'd be helpful to know what exactly happens after loading the initramfs-linux.img and to compare that against what happens in linux 6.1.

Re bisecting: that would require a large number of kernel-recompiles since this is a major version number. I was hoping there'd be an easier way to debug this

earlycon=efifb earlyprintk=vga,keep ignore_loglevel see [1] for more kernel parameters. [2][3] are 6.1 from upstream built with Arch's config for 6.2. [4][5] are 6.2 from upstream built with Arch's config for 6.2. [6][7] are the fist bisection point.

[1] https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
[2] https://drive.google.com/file/d/122nGyut0LIlTVSTVltNHN2qlHnPfocaS/view?usp=share_link linux-6.1-1-x86_64.pkg.tar.zst
[3] https://drive.google.com/file/d/1mYofi49qOp-9k684Fn8aRHqmv_o3ctAR/view?usp=share_link linux-headers-6.1-1-x86_64.pkg.tar.zst
[4] https://drive.google.com/file/d/1iuZb9sKtfTmtyUDdP8_wbQjt43cSjToN/view?usp=share_link linux-6.2-1-x86_64.pkg.tar.zst
[5] https://drive.google.com/file/d/1dShJFAepKUlZiq6kZAaElfqnuNQHtCVS/view?usp=share_link linux-headers-6.2-1-x86_64.pkg.tar.zst
[6] https://drive.google.com/file/d/1o5aEkYDluU1jh4LihKeSq_LVZXwn9RLn/view?usp=share_link linux-6.1.r7411.g1ca06f1c1ace-1-x86_64.pkg.tar.zst
[7] https://drive.google.com/file/d/12NhoGt6d4cR2DJB8-_u9ZmcfhU-J-yYz/view?usp=share_link linux-headers-6.1.r7411.g1ca06f1c1ace-1-x86_64.pkg.tar.zst

Thanks! That is helpful. Here are my findings:
- kernel [2] boots fine, with the kernel parameters you provided I get nice early output printed to the console
- kernel [4] doesn't boot, no output printed after "Loading ../../initramfs-linux.img...ok"
- kernel [6] doesn't boot, same as [4]

$ git bisect bad
Bisecting: 3754 revisions left to test after this (roughly 12 steps)
[8715c6d3100fc7c6edddf29af4a399a1c12d028c] Merge tag 'for-6.2/dm-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm

https://drive.google.com/file/d/1zEXtYbbFXlKJe7hBUqvKg2R2eyNTtv1h/view?usp=share_link linux-6.1.r3656.g8715c6d3100f-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1vwTuSmkFqJuqtBlNMO71JTpAwgYh8BRV/view?usp=share_link linux-headers-6.1.r3656.g8715c6d3100f-1-x86_64.pkg.tar.zst

Thanks! This one boots fine

$ git bisect good
Bisecting: 2320 revisions left to test after this (roughly 11 steps)
[66efff515a6500d4b4976fbab3bee8b92a1137fb] Merge tag 'amd-drm-next-6.2-2022-12-07' of https://gitlab.freedesktop.org/agd5f/linux into drm-next

https://drive.google.com/file/d/1ECAtK6NnP_y2Fz07gkanRgfDnxk3Rutk/view?usp=share_link linux-6.1rc6.r1434.g66efff515a65-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1ab5AQwhcYlVQQvIYaNxayKmyLHKlvQpN/view?usp=share_link linux-headers-6.1rc6.r1434.g66efff515a65-1-x86_64.pkg.tar.zst

This one is good

I found an old 6-core machine to bisect the kernel now. The build asks a bunch of "... [Y/n/?] (NEW)" questions. Do you just hit enter to use the default until it repeats the same question again and then you choose the non-default answer?

(By the way, this old machine is still quite a powerful server that's much faster than the tiny thin laptop that I'm seeing the kernel issue one)

Keep hitting enter to select the default option. There should not be any repeated questions. Attached is the src tarball from this build.

$ git bisect good
Bisecting: 1144 revisions left to test after this (roughly 10 steps)
[c5589c436d4646e0dc23f64264db8e04cf67c88f] Merge tag 'regulator-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [c9c3395d5e3dcc6daee66c6908354d47bf98cb0c] Linux 6.2
git bisect bad c9c3395d5e3dcc6daee66c6908354d47bf98cb0c
# status: waiting for good commit(s), bad commit known
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
# bad: [1ca06f1c1acecbe02124f14a37cce347b8c1a90c] Merge tag 'xtensa-20221213' of https://github.com/jcmvbkbc/linux-xtensa
git bisect bad 1ca06f1c1acecbe02124f14a37cce347b8c1a90c
# good: [8715c6d3100fc7c6edddf29af4a399a1c12d028c] Merge tag 'for-6.2/dm-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
git bisect good 8715c6d3100fc7c6edddf29af4a399a1c12d028c
# good: [66efff515a6500d4b4976fbab3bee8b92a1137fb] Merge tag 'amd-drm-next-6.2-2022-12-07' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
git bisect good 66efff515a6500d4b4976fbab3bee8b92a1137fb

https://drive.google.com/file/d/1dpBhjUHI0xv34_OOqTEkx4Ak7Qz_sIgm/view?usp=share_link linux-6.1.r6266.gc5589c436d46-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1zGVTmyVXh_8dUmo6P0jEEAIynzRrO0LF/view?usp=share_link linux-headers-6.1.r6266.gc5589c436d46-1-x86_64.pkg.tar.zst

(By the way, this old machine is still quite a powerful server that's much faster than the tiny thin laptop that I'm seeing the kernel issue one)

Thanks again for your help! Can I ask you this one: How do you fix your git describe?
My git describe sais:
# git describe
fatal: No tags can describe '...'

The tags are in the mainline repo. If you are using https://github.com/archlinux/linux you can add mainline and fetch the tags using

$ git remote add mainline https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ git fetch --tags mainline

Thanks again for your help! Can I ask you this one: How do you fix your git describe?
My git describe sais:
# git describe
fatal: No tags can describe '...'

Cool. Thanks!

Bad luck. I thought to optimize this a bit and used this
# make LSMOD=/tmp/mylsmod localmodconfig

But then got too many "good" bisect results in a row that I started to suspect something and redid the latest "bad" you created for me: that one turned out good now. So not building some module that I don't use seems to fix the problem.

Now I have to redo the bisect without localmodconfig

The only update I have now is that your latest build (r6266.gc5589c436d46) is "good".

I'm building the next step. But could you please also create one for me? Thanks!

$ git bisect good
Bisecting: 579 revisions left to test after this (roughly 9 steps)
[86a0b4255e84563739d137ad374af6c7215bb3ff] Merge tag 'input-for-v6.2-rc0' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input

https://drive.google.com/file/d/1P0qJuebah3lApygaU7bSAfT3I_kdkEye/view?usp=share_link linux-6.1.r6831.g86a0b4255e84-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1K-yS8CPDwOdVwRbjxYL95IDX15bfeB8Q/view?usp=share_link linux-headers-6.1.r6831.g86a0b4255e84-1-x86_64.pkg.tar.zst

Bad luck. I thought to optimize this a bit and used this
# make LSMOD=/tmp/mylsmod localmodconfig

But then got too many "good" bisect results in a row that I started to suspect something and redid the latest "bad" you created for me: that one turned out good now. So not building some module that I don't use seems to fix the problem.

Now I have to redo the bisect without localmodconfig

The only update I have now is that your latest build (r6266.gc5589c436d46) is "good".

I'm building the next step. But could you please also create one for me? Thanks!

Assuming 86a0b4255e84563739d137ad374af6c7215bb3ff r6831.g86a0b4255e84 is good:
$ git bisect good
Bisecting: 376 revisions left to test after this (roughly 8 steps)
[0e2c9884cbbae00f956d881848669790d73be43d] Merge branches 'clk-mediatek', 'clk-trace', 'clk-qcom' and 'clk-microchip' into clk-next

https://drive.google.com/file/d/1YS-xom4bhfczASAI1esDq4hTR4dwugbe/view?usp=share_link linux-6.1.r204.g0e2c9884cbba-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/18W_IjJtia1PJrb8EBXLfuhuyhpJPYpmf/view?usp=share_link linux-headers-6.1.r204.g0e2c9884cbba-1-x86_64.pkg.tar.zst

# cat bisect.log
git bisect start
# status: waiting for both good and bad commits
# bad: [c9c3395d5e3dcc6daee66c6908354d47bf98cb0c] Linux 6.2
git bisect bad c9c3395d5e3dcc6daee66c6908354d47bf98cb0c
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
# bad: [1ca06f1c1acecbe02124f14a37cce347b8c1a90c] Merge tag 'xtensa-20221213' of https://github.com/jcmvbkbc/linux-xtensa
git bisect bad 1ca06f1c1acecbe02124f14a37cce347b8c1a90c
# good: [8715c6d3100fc7c6edddf29af4a399a1c12d028c] Merge tag 'for-6.2/dm-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
git bisect good 8715c6d3100fc7c6edddf29af4a399a1c12d028c
# good: [66efff515a6500d4b4976fbab3bee8b92a1137fb] Merge tag 'amd-drm-next-6.2-2022-12-07' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
git bisect good 66efff515a6500d4b4976fbab3bee8b92a1137fb
# good: [c5589c436d4646e0dc23f64264db8e04cf67c88f] Merge tag 'regulator-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator
git bisect good c5589c436d4646e0dc23f64264db8e04cf67c88f
# good: [86a0b4255e84563739d137ad374af6c7215bb3ff] Merge tag 'input-for-v6.2-rc0' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 86a0b4255e84563739d137ad374af6c7215bb3ff
# good: [0e2c9884cbbae00f956d881848669790d73be43d] Merge branches 'clk-mediatek', 'clk-trace', 'clk-qcom' and 'clk-microchip' into clk-next
git bisect good 0e2c9884cbbae00f956d881848669790d73be43d
# good: [71946a25f357a51dcce849367501d7fb04c0465b] Merge tag 'mmc-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
git bisect good 71946a25f357a51dcce849367501d7fb04c0465b
# good: [4eb77fa102ebc2a48d61941fd0293b0aeed00fee] Merge tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 4eb77fa102ebc2a48d61941fd0293b0aeed00fee
# good: [a70210f41566131f88d31583f96e36cb7f5d2ad0] Merge tag 'x86_microcode_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good a70210f41566131f88d31583f96e36cb7f5d2ad0
# good: [3ef3ace4e2ecf4aa4c8ddff1d35683671a09b05e] Merge tag 'x86_cpu_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 3ef3ace4e2ecf4aa4c8ddff1d35683671a09b05e
# good: [a70210f41566131f88d31583f96e36cb7f5d2ad0] Merge tag 'x86_microcode_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good a70210f41566131f88d31583f96e36cb7f5d2ad0
# bad: [4cb1fc6fffe4910845e183d1a2dfe9509ba1062c] Merge tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm
git bisect bad 4cb1fc6fffe4910845e183d1a2dfe9509ba1062c
# good: [74c344e6f153dd9ae97c99ad751723e4030d4af9] ARM: 9267/1: Define Armv8 registers in AArch32 state
git bisect good 74c344e6f153dd9ae97c99ad751723e4030d4af9
# good: [59e2cf8d21e05391c42628eb9fb5bb40f9d9698f] ARM: 9275/1: Drop '-mthumb' from AFLAGS_ISA
git bisect good 59e2cf8d21e05391c42628eb9fb5bb40f9d9698f
# good: [740afa4d39414516c36836ad88bed8294c72ba5f] Merge tag 'x86_sev_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 740afa4d39414516c36836ad88bed8294c72ba5f
# good: [ba290d4f1f7733732896567d2a1874f01e20fff4] ARM: 9277/1: Make the dumped instructions are consistent with the disassembled ones
git bisect good ba290d4f1f7733732896567d2a1874f01e20fff4
# good: [aaa4dd1b47f5ff5ef477fec5dcc6c397b457f1c2] ARM: 9279/1: support function error injection
git bisect good aaa4dd1b47f5ff5ef477fec5dcc6c397b457f1c2
# first bad commit: [4cb1fc6fffe4910845e183d1a2dfe9509ba1062c] Merge tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm

Strange, because commit 4cb1fc6 is related to linux-arm.

Now building v6.2-arch1 with 4cb1fc6 reverted...

It works!

Now, how do I report this upstream?

Merge commits are unlikely bug causes as all the commits they contained are added then the merge commit comes last.
How are you reverting 4cb1fc6?
$ git revert -n 4cb1fc6
error: commit 4cb1fc6fffe4910845e183d1a2dfe9509ba1062c is a merge but no -m option was given.
fatal: revert failed
$ git revert -n -m 1 4cb1fc6
Auto-merging arch/arm/Kconfig
Auto-merging arch/arm/Makefile
CONFLICT (content): Merge conflict in arch/arm/Makefile
Auto-merging arch/arm/boot/compressed/Makefile
error: could not revert 4cb1fc6fffe4... Merge tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'

Are both the linked kernels below good?

https://drive.google.com/file/d/1qKv6O0iIaGOVLNE3aOwgTkQDXzqnCuK6/view?usp=share_link linux-6.1.r7318.g4eb77fa102eb-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Pb049rRUlI4sj7RCk5bFJmq8wwf8g3HG/view?usp=share_link linux-headers-6.1.r7318.g4eb77fa102eb-1-x86_64.pkg.tar.zst

https://drive.google.com/file/d/11efcj8J0tZNoH-d0szf2h-Kv46XRF4Rd/view?usp=share_link linux-headers-6.1.r7370.ga70210f41566-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1avMhLcJ2iNoycqTXyZdn7DFNgqTOuEPY/view?usp=share_link linux-6.1.r7370.ga70210f41566-1-x86_64.pkg.tar.zst

# git revert -m 1 4cb1fc6
And then manually merged the conflict, which was just a style difference I think.

Let me try the two kernels you sent me

In fact, those two you just sent were both bad

Something has gone wrong with your bisect as you found both of those to be good. Please try

https://drive.google.com/file/d/1P0qJuebah3lApygaU7bSAfT3I_kdkEye/view?usp=share_link linux-6.1.r6831.g86a0b4255e84-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1K-yS8CPDwOdVwRbjxYL95IDX15bfeB8Q/view?usp=share_link linux-headers-6.1.r6831.g86a0b4255e84-1-x86_64.pkg.tar.zst

https://drive.google.com/file/d/1YS-xom4bhfczASAI1esDq4hTR4dwugbe/view?usp=share_link linux-6.1.r204.g0e2c9884cbba-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/18W_IjJtia1PJrb8EBXLfuhuyhpJPYpmf/view?usp=share_link linux-headers-6.1.r204.g0e2c9884cbba-1-x86_64.pkg.tar.zst

I am going to assume both of the above are good and build 71946a25f357a51dcce849367501d7fb04c0465b
Edit:
https://drive.google.com/file/d/1d3lTY1Q07ybkU_dNRkpruyyRO_xT34Au/view?usp=share_link linux-6.1.r7019.g71946a25f357-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1WX2It-BlgE6Z510TOkEdrqbL2dbLkQst/view?usp=share_link linux-headers-6.1.r7019.g71946a25f357-1-x86_64.pkg.tar.zst

Thank you, loqs! These are good, good, and good

$ git bisect bad
Bisecting: 50 revisions left to test after this (roughly 6 steps)
[e8dfdf3162eb549d064b8c10b1564f7e8ee82591] arm64: efi: Recover from synchronous exceptions occurring in firmware

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
# status: waiting for bad commit, 1 good commit known
# bad: [c9c3395d5e3dcc6daee66c6908354d47bf98cb0c] Linux 6.2
git bisect bad c9c3395d5e3dcc6daee66c6908354d47bf98cb0c
# bad: [1ca06f1c1acecbe02124f14a37cce347b8c1a90c] Merge tag 'xtensa-20221213' of https://github.com/jcmvbkbc/linux-xtensa
git bisect bad 1ca06f1c1acecbe02124f14a37cce347b8c1a90c
# good: [8715c6d3100fc7c6edddf29af4a399a1c12d028c] Merge tag 'for-6.2/dm-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
git bisect good 8715c6d3100fc7c6edddf29af4a399a1c12d028c
# good: [66efff515a6500d4b4976fbab3bee8b92a1137fb] Merge tag 'amd-drm-next-6.2-2022-12-07' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
git bisect good 66efff515a6500d4b4976fbab3bee8b92a1137fb
# good: [c5589c436d4646e0dc23f64264db8e04cf67c88f] Merge tag 'regulator-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator
git bisect good c5589c436d4646e0dc23f64264db8e04cf67c88f
# good: [86a0b4255e84563739d137ad374af6c7215bb3ff] Merge tag 'input-for-v6.2-rc0' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 86a0b4255e84563739d137ad374af6c7215bb3ff
# good: [0e2c9884cbbae00f956d881848669790d73be43d] Merge branches 'clk-mediatek', 'clk-trace', 'clk-qcom' and 'clk-microchip' into clk-next
git bisect good 0e2c9884cbbae00f956d881848669790d73be43d
# good: [71946a25f357a51dcce849367501d7fb04c0465b] Merge tag 'mmc-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
git bisect good 71946a25f357a51dcce849367501d7fb04c0465b
# bad: [4eb77fa102ebc2a48d61941fd0293b0aeed00fee] Merge tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 4eb77fa102ebc2a48d61941fd0293b0aeed00fee

https://drive.google.com/file/d/1nDZA_yEWdazHWXow75T7a6RB-jUYoaqX/view?usp=share_link linux-6.1rc8.r45.ge8dfdf3162eb-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/18ojO6ypfEnhEVkpXTO9ZWnGgHiAqyPlt/view?usp=share_link linux-headers-6.1rc8.r45.ge8dfdf3162eb-1-x86_64.pkg.tar.zst

Thanks! This one is good

$ git bisect good
Bisecting: 22 revisions left to test after this (roughly 5 steps)
[717e6eb49bdd98357d14c90d60a3409196b33cfc] Merge tag 'integrity-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity

https://drive.google.com/file/d/1s5PDaO2tA8R0L-leojWOFPLBYBra9KAx/view?usp=share_link linux-6.1.r7250.g717e6eb49bdd-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1rjVRHFvqYaEdgqppabAOxE_hJrIdi--T/view?usp=share_link linux-headers-6.1.r7250.g717e6eb49bdd-1-x86_64.pkg.tar.zst

Thanks! This one is good (in that it passes the step described above, but it gets stuck a little after and doesn't finish booting up)

Calling it good for now, may have to revisit it.

$ git bisect good
Bisecting: 11 revisions left to test after this (roughly 4 steps)
[c6355995ba471d7ad574174e593192ce805c7e1a] x86/boot/compressed: Move startup32_load_idt() into .text section

https://drive.google.com/file/d/1j5ByJ21Vz4FeM2nWfVQ86iIZf5TwgU6X/view?usp=share_link linux-6.1rc6.r11.gc6355995ba47-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1XFuwI_heuuC-d7QLVYNTQeTFZTyes0gw/view?usp=share_link linux-headers-6.1rc6.r11.gc6355995ba47-1-x86_64.pkg.tar.zst

Thanks. This one is good

$ git bisect good
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[cc3fdda2876e58a7e83e558ab51853cf106afb6a] x86/efi: Make the deprecated EFI handover protocol optional

https://drive.google.com/file/d/1VRNwYJRQnt4Q0fkFXqOxT0MbOtOiYrJf/view?usp=share_link linux-6.1rc6.r17.gcc3fdda2876e-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1qxekCMK4LuI2hqmag98ZDJR3nOxueYsM/view?usp=share_link linux-headers-6.1rc6.r17.gcc3fdda2876e-1-x86_64.pkg.tar.zst

Good!

$ git bisect good
Bisecting: 2 revisions left to test after this (roughly 2 steps)
[8b9ed79c2d587bec5f603d66801478a5af9af842] Merge tag 'x86_asm_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

https://drive.google.com/file/d/1qbEOco9ZcdWQFOUjqmMhH9_cuSrLo_UB/view?usp=share_link linux-6.1.r7298.g8b9ed79c2d58-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1okhLDGIP42Oe_rJ92iGkRrCSe0tPKpAH/view?usp=share_link linux-headers-6.1.r7298.g8b9ed79c2d58-1-x86_64.pkg.tar.zst

good

$ git bisect good
Bisecting: 0 revisions left to test after this (roughly 1 step)
[60253f100c5846029f1370e51be6ebaeb160dcec] x86/boot: Remove x86_32 PIC using %ebx workaround

https://drive.google.com/file/d/1J2vBbzLdXPRTmnLf9h47VTkOQzhTmwao/view?usp=share_link linux-6.1rc6.r19.g60253f100c58-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1N8MBVl4BIDKeIpGwTqPt1AT5ybWUiYyY/view?usp=share_link linux-headers-6.1rc6.r19.g60253f100c58-1-x86_64.pkg.tar.zst

This one is also good

$ git bisect good
4eb77fa102ebc2a48d61941fd0293b0aeed00fee is the first bad commit
commit 4eb77fa102ebc2a48d61941fd0293b0aeed00fee
Merge: 8b9ed79c2d58 60253f100c58
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Tue Dec 13 14:45:29 2022 -0800

Merge tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 boot updates from Borislav Petkov:
"A of early boot cleanups and fixes.

- Do some spring cleaning to the compressed boot code by moving the
EFI mixed-mode code to a separate compilation unit, the AMD memory
encryption early code where it belongs and fixing up build
dependencies. Make the deprecated EFI handover protocol optional
with the goal of removing it at some point (Ard Biesheuvel)

- Skip realmode init code on Xen PV guests as it is not needed there

- Remove an old 32-bit PIC code compiler workaround"

* tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/boot: Remove x86_32 PIC using %ebx workaround
x86/boot: Skip realmode init code when running as Xen PV guest
x86/efi: Make the deprecated EFI handover protocol optional
x86/boot/compressed: Only build mem_encrypt.S if AMD_MEM_ENCRYPT=y
x86/boot/compressed: Adhere to calling convention in get_sev_encryption_bit()
x86/boot/compressed: Move startup32_check_sev_cbit() out of head_64.S
x86/boot/compressed: Move startup32_check_sev_cbit() into .text
x86/boot/compressed: Move startup32_load_idt() out of head_64.S
x86/boot/compressed: Move startup32_load_idt() into .text section
x86/boot/compressed: Pull global variable reference into startup32_load_idt()
x86/boot/compressed: Avoid touching ECX in startup32_set_idt_entry()
x86/boot/compressed: Simplify IDT/GDT preserve/restore in the EFI thunk
x86/boot/compressed, efi: Merge multiple definitions of image_offset into one
x86/boot/compressed: Move efi32_pe_entry() out of head_64.S
x86/boot/compressed: Move efi32_entry out of head_64.S
x86/boot/compressed: Move efi32_pe_entry into .text section
x86/boot/compressed: Move bootargs parsing out of 32-bit startup code
x86/boot/compressed: Move 32-bit entrypoint code into .text section
x86/boot/compressed: Rename efi_thunk_64.S to efi-mixed.S

arch/x86/Kconfig | 17 ++
arch/x86/boot/compressed/Makefile | 8 +-
arch/x86/boot/compressed/efi_mixed.S | 345 ++++++++++++++++++++++++++++++++
arch/x86/boot/compressed/efi_thunk_64.S | 189 -----------------
arch/x86/boot/compressed/head_32.S | 4 -
arch/x86/boot/compressed/head_64.S | 303 ++--------------------------
arch/x86/boot/compressed/mem_encrypt.S | 152 ++++++++++++--
arch/x86/boot/cpuflags.c | 15 +-
arch/x86/boot/header.S | 2 +-
arch/x86/boot/tools/build.c | 2 +
arch/x86/include/asm/realmode.h | 1 +
arch/x86/include/asm/x86_init.h | 4 +
arch/x86/kernel/setup.c | 2 +-
arch/x86/kernel/x86_init.c | 3 +
arch/x86/realmode/init.c | 8 +-
arch/x86/xen/enlighten_pv.c | 2 +
drivers/firmware/efi/libstub/x86-stub.c | 2 +-
17 files changed, 547 insertions(+), 512 deletions(-)
create mode 100644 arch/x86/boot/compressed/efi_mixed.S
delete mode 100644 arch/x86/boot/compressed/efi_thunk_64.S

Which is a merge commit. Did something go wrong in the bisect?

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476
# bad: [c9c3395d5e3dcc6daee66c6908354d47bf98cb0c] Linux 6.2
git bisect bad c9c3395d5e3dcc6daee66c6908354d47bf98cb0c
# bad: [1ca06f1c1acecbe02124f14a37cce347b8c1a90c] Merge tag 'xtensa-20221213' of https://github.com/jcmvbkbc/linux-xtensa
git bisect bad 1ca06f1c1acecbe02124f14a37cce347b8c1a90c
# good: [8715c6d3100fc7c6edddf29af4a399a1c12d028c] Merge tag 'for-6.2/dm-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
git bisect good 8715c6d3100fc7c6edddf29af4a399a1c12d028c
# good: [66efff515a6500d4b4976fbab3bee8b92a1137fb] Merge tag 'amd-drm-next-6.2-2022-12-07' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
git bisect good 66efff515a6500d4b4976fbab3bee8b92a1137fb
# good: [c5589c436d4646e0dc23f64264db8e04cf67c88f] Merge tag 'regulator-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator
git bisect good c5589c436d4646e0dc23f64264db8e04cf67c88f
# good: [86a0b4255e84563739d137ad374af6c7215bb3ff] Merge tag 'input-for-v6.2-rc0' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good 86a0b4255e84563739d137ad374af6c7215bb3ff
# good: [0e2c9884cbbae00f956d881848669790d73be43d] Merge branches 'clk-mediatek', 'clk-trace', 'clk-qcom' and 'clk-microchip' into clk-next
git bisect good 0e2c9884cbbae00f956d881848669790d73be43d
# good: [71946a25f357a51dcce849367501d7fb04c0465b] Merge tag 'mmc-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
git bisect good 71946a25f357a51dcce849367501d7fb04c0465b
# bad: [4eb77fa102ebc2a48d61941fd0293b0aeed00fee] Merge tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 4eb77fa102ebc2a48d61941fd0293b0aeed00fee
# good: [e8dfdf3162eb549d064b8c10b1564f7e8ee82591] arm64: efi: Recover from synchronous exceptions occurring in firmware
git bisect good e8dfdf3162eb549d064b8c10b1564f7e8ee82591
# good: [717e6eb49bdd98357d14c90d60a3409196b33cfc] Merge tag 'integrity-v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity
git bisect good 717e6eb49bdd98357d14c90d60a3409196b33cfc
# good: [c6355995ba471d7ad574174e593192ce805c7e1a] x86/boot/compressed: Move startup32_load_idt() into .text section
git bisect good c6355995ba471d7ad574174e593192ce805c7e1a
# good: [cc3fdda2876e58a7e83e558ab51853cf106afb6a] x86/efi: Make the deprecated EFI handover protocol optional
git bisect good cc3fdda2876e58a7e83e558ab51853cf106afb6a
# good: [8b9ed79c2d587bec5f603d66801478a5af9af842] Merge tag 'x86_asm_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 8b9ed79c2d587bec5f603d66801478a5af9af842
# good: [60253f100c5846029f1370e51be6ebaeb160dcec] x86/boot: Remove x86_32 PIC using %ebx workaround
git bisect good 60253f100c5846029f1370e51be6ebaeb160dcec
# first bad commit: [4eb77fa102ebc2a48d61941fd0293b0aeed00fee] Merge tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

https://drive.google.com/file/d/1N54hIB512pwnbj_UtlT7JGQ6rrupfS7d/view?usp=share_link linux-6.2-6-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Jck5vs3vw_1KaoTkQBkiZcquBcKBfDUV/view?usp=share_link linux-headers-6.2-6-x86_64.pkg.tar.zst
This reverts 60253f100c5846029f1370e51be6ebaeb160dcec which was found to be good.
Edit:
https://drive.google.com/file/d/1ecip1xqDPfnNQ-o-W2a0x04Gk6UPsOW2/view?usp=share_link linux-6.2-7-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1odAA4UdzeixkphX5ca7ANX6KZ4MSltme/view?usp=share_link linux-headers-6.2-7-x86_64.pkg.tar.zst
This is git revert -n -m 1 4eb77fa102ebc2a48d61941fd0293b0aeed00fee on v6.2

Now, this 6.2-6 one is bad again, but wasn't that expected given that it reverts a comment that was found to be good?

The commit message of 4eb77fa seems to be quite related to the issue I'm seeing. I don't know enough about merge commits not being usual culprits. But could you do one more kernel for me, reverting 4eb77fa from 6.2 or from 6.2.1?

In my own bisect run, I had not run 'make clean' after each compile, having some trust in the makefile taking care of the dependency tree. I hope that later this week I'll find time to recompile some of the kernels you built for me and confirm the same behavior.

See my edit to my previous post, has the build you just requested.

This one is good!

Thanks so much with your help bisecting this!

Now what is the reason why a merge commit is unlikely?

And how can this best be reported upstream?

The merge commit comes after all the commits it contains. So if the merge is bad a previous commit should also be bad. In this case one or more of:

$ git log 4eb77fa102ebc2a48d61941fd0293b0aeed00fee^..4eb77fa102ebc2a48d61941fd0293b0aeed00fee --oneline
4eb77fa102eb Merge tag 'x86_boot_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
60253f100c58 x86/boot: Remove x86_32 PIC using %ebx workaround
f1e525009493 x86/boot: Skip realmode init code when running as Xen PV guest
cc3fdda2876e x86/efi: Make the deprecated EFI handover protocol optional
61de13df9590 x86/boot/compressed: Only build mem_encrypt.S if AMD_MEM_ENCRYPT=y
30c9ca16a527 x86/boot/compressed: Adhere to calling convention in get_sev_encryption_bit()
9d7eaae6a071 x86/boot/compressed: Move startup32_check_sev_cbit() out of head_64.S
b5d854cd4b6a x86/boot/compressed: Move startup32_check_sev_cbit() into .text
9ea813be3d34 x86/boot/compressed: Move startup32_load_idt() out of head_64.S
c6355995ba47 x86/boot/compressed: Move startup32_load_idt() into .text section
d73a257f7f86 x86/boot/compressed: Pull global variable reference into startup32_load_idt()
6aac80a8da46 x86/boot/compressed: Avoid touching ECX in startup32_set_idt_entry()
630f337f0c4f x86/boot/compressed: Simplify IDT/GDT preserve/restore in the EFI thunk
4b52016247ae x86/boot/compressed, efi: Merge multiple definitions of image_offset into one
7f22ca396778 x86/boot/compressed: Move efi32_pe_entry() out of head_64.S
73a6dec80e2a x86/boot/compressed: Move efi32_entry out of head_64.S
91592b5c0c2f x86/boot/compressed: Move efi32_pe_entry into .text section
5c3a85f35b58 x86/boot/compressed: Move bootargs parsing out of 32-bit startup code
e2ab9eab324c x86/boot/compressed: Move 32-bit entrypoint code into .text section
cb8bda8ad443 x86/boot/compressed: Rename efi_thunk_64.S to efi-mixed.S

Ideally you would identify which commit in the merge is triggering the issue. Not sure which mailing list covers x86_boot. You could also try the kernel bugzilla again not sure which category to file it in.

Enable EFI_HANDOVER_PROTOCOL
https://drive.google.com/file/d/1uuBgBRloLH6ISAbIgNWD81TAPwumYT8d/view?usp=share_link linux-6.2-8-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1NX-mg_4dSx43E6bcsB3IQ5zcFyDlsRzf/view?usp=share_link linux-headers-6.2-8-x86_64.pkg.tar.zst

git revert -n 60253f100c58
git revert -n f1e525009493
git revert -n cc3fdda2876e
git revert -n 61de13df9590
git revert -n 30c9ca16a527
git revert -n 9d7eaae6a071
git revert -n b5d854cd4b6a
git revert -n 9ea813be3d34
git revert -n c6355995ba47

Which is roughly half of the commits from 4eb77fa102eb
https://drive.google.com/file/d/1WYs-5mzVsR4BbN2eoui1KqRrwuUEC9Dc/view?usp=share_link linux-6.2-9-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1nZMrUGKZz5gWL4fcXAstPjPrWdQA4a6U/view?usp=share_link linux-headers-6.2-9-x86_64.pkg.tar.zst

These both boot fine. What was the 6.2-8 kernel exactly?

I get this error building the kernel using 'makepkg -ef' if I don't include the line "make -s kernelrelease > version" in the build() function. But you don't have that line there in your PKGBUILD. How does that still work for you? Do you not build using 'makepkg -ef'?
---
ZSTD /home/daan/build/linux/trunk/pkg/linux/usr/lib/modules/6.1.0-1/kernel/sound/x86/snd-hdmi-lpe-audio.ko.zst
ZSTD /home/daan/build/linux/trunk/pkg/linux/usr/lib/modules/6.1.0-1/kernel/sound/xen/snd_xen_front.ko.zst
ZSTD /home/daan/build/linux/trunk/pkg/linux/usr/lib/modules/6.1.0-1/kernel/virt/lib/irqbypass.ko.zst
DEPMOD /home/daan/build/linux/trunk/pkg/linux/usr/lib/modules/6.1.0-1
Warning: 'make modules_install' requires /doesnt/exist. Please install it.
This is probably in the kmod package.
rm: cannot remove '/home/daan/build/linux/trunk/pkg/linux/usr/lib/modules/6.2.0-arch1-1/source': No such file or directory
rm: cannot remove '/home/daan/build/linux/trunk/pkg/linux/usr/lib/modules/6.2.0-arch1-1/build': No such file or directory
---

I just compiled 6.2.1-arch1 with EFI_HANDOVER_PROTOCOL set and the kernel boots fine. So this is the solution for me. I'll close this bug report.

Thank you very much for your help sorting this out!!

What bootloader is the system using? Is that bootloader up to date?

# diff /boot/EFI/syslinux/ /usr/lib/syslinux/efi64/
Only in /boot/EFI/syslinux/: syslinux.cfg

So, yeah, it seems to be up to date

Can you reproduce the issue using a different bootloader?

I can test this on different hardware tonight, but with the same bootloader. I screamingly ran away from grub in 2008 and have only ever used syslinux since. I don't know about other bootloaders that are easy to handle...

If it's a bootloader bug I would expect more users would run into this, no?

> If it's a bootloader bug I would expect more users would run into this, no?
Depends how many users running testing use syslinux and EFI boot. syslinux does try to use the EFI handover protocol [1] and was not considered when the protocol was made optional [2].

[1] https://repo.or.cz/syslinux.git/blob/HEAD:/efi/main.c#l1201
[2] https://lore.kernel.org/all/20221122161017.2426828-18-ardb%40kernel.org/.

I just put the SSD back in my old Thinkpad X1 carbon g5 and it turns out to boot just fine with the vanilla 6.2.1-arch1 kernel (and syslinux).

I now remember having had a similar, or maybe even the exact same, booting issue with syslinux and a recent kernel on a HP thin client. Back then I switched to an LTS kernel. Now I wonder if it was caused by the same EFI_HANDOVER_PROTOCOL parameter...

However, I'm afraid I have to give up on testing this bug because I decided to return the X1Cg10 to Lenovo because I can't get the camera to work and it turns out Intel isn't releasing the sources necessary to make it work. It's a shame Lenovo decided to include unsupported hardware in this machine.

I'm sorry!

Um, probably would have been better to mention which bootloader you were using from the outset. syslinux development hasn't been updated in years so it's no wonder there are issues when using it to boot modern hardware. FYI syslinux is considered basically dead by some folks [1]

[1] https://fedoraproject.org/wiki/Releases/37/ChangeSet#BIOS_boot.iso_with_GRUB2

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/Kconfig?h=v6.2&id=cc3fdda2876e58a7e83e558ab51853cf106afb6a specifically suggests people should set this to Y if in doubt. Doesn't seem like this will be removed entirely any time soon. No reason not to just have this enabled in the archlinux kernel configs. This is breaking installs for no good reason right now