Arch Linux

I should make clear that I can log into a 5.10.4.a-1-hardened console without problem.

Possibly relevant: I am attempting to launch a gnome wayland session:

``` XDG_SESSION_TYPE=wayland QT_QPA_PLATFORM=wayland exec dbus-run-session gnome-session > /dev/null 2>&1

Edit
Further, I do not use GDM, but rather a simple bash case statement to choose between desktop or console

@IrvineHimself, see similar effect. I also use a script to start X. The dialog starts and at this point I'm able to reach a different TTY, login and even start an X session(i3) using xinit. If however, I use the script dialog, I'm unable to do anything at all after that, only hard reset:( Didn't really have time to investigate further. Everything was fine on 5.9.16.a(and before that..). It's possible I missed some mailing list article, but, just letting you know Thanks, mark.
edit: the script starts through .zprofile on TTY1(after getty override), if I start the same script on a different TTY the result is the same(need hard reset!)

1. What kind of GPU/CPU are you using
2. Does it work with vanilla 5.10.4.arch2-1
3. Please try to use GDM if that makes any difference compared to lunch from tty?
4. What exactly do you mean by "script dialog"?

Hey Levente, thanks for calling back;)

1. Using Intel GPU/CPU, NUC 10i7FNK
2. Yes, it does work with vanilla, but, I'm on 5.10.3-arch1-1 not on 5.10.4.arch2-1
3. He, I don't want to use GDM I want to use my own script ;) I will try if it work though
4, A simple script using Dialog (see) https://archlinux.org/packages/core/x86_64/dialog/

@mark:
2. Please explicitly try 5.10.4.arch2-1 as its more near to 5.10.4.a-1 as both are based on v5.10.4
3. I understand, but we need to pinpoint the problems here

1) CPU:
. Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz

GPU:
. Intel Corporation Skylake GT2
. NVIDIA Corporation GM108M [GeForce 930M] (rev a2)

Edit: Using Nouveau drivers

2) It appears to work fine with Linux-5.10.4-arch2-1 from testing, for which I have attached the journalctl output.

3) I will install GDM to check, but it will take bit of planning since I need to edit my bash-profile. I checked that it was a login-console with: 'shopt login_shell -> login_shell on'

4) In my case it's just a simple script called from my bash_profile, I have attached both.

Irvine

Okay @ Irvine, same result, without GDM using my own script everything is fine on 5.10.4.arch2-1.
I think there's no deed to try GDM.
@Levente, if you really need results from GDM I will try it ?

If anyone of you has experience with PKGBUILDs and the tooling, it would be cool to help with the following:
5) Compile hardened with using the 'config' file from vanilla
6) If 5. doesn't work, try a git clone of hardened and 'git bisect' between v5.10.4 and hardened tag 5.10.4.a (or simply the 5.10 branch). Should be like max 15 bisect steps. There is a linux-hardened-git AUR package that may help with it, but needs to adjust the source to a local directory that is the bisect target and avoid cleaning the build dir for pretty fast steps.

Compiling right now with vanilla config and hardened patch,so far so good;) (10min)
I made tmpfs 20G(RAM=32G) hope it's enough..
I'll update this message along the process. BTW. for point 6 I would need to learn first;)

I just tried installing GDM: Since it is not pure wayland friendly, I didn't really explore the settings. But, "out of the box", GDM *FAILS* to launch gnome with the hardened version of 10.4, whereas GDM succeeds in launching gnome with vanilla 10.4

With respect to bisecting the commits, I have done it *once* before. So, while I will need to check the procedure, I will give it a go.

Unfortunately, I have other commitments at the moment so I won't be able to get back to you till tomorrow.

Irvine

Right, compiling kernel went well(though I did see some messages about Sphinx and Latex..), there was only 600MB left of tmpfs.
Installing went fine too, though I reached the same point, complete lockup, only hard reset worked:(
Don't know if I'm going to go for bisecting right away, going to read the wiki on it again first and raise tmpfs to 25G.

Bisecting linux-hardened-git should roughly be the following:

$ git clone https://aur.archlinux.org/linux-hardened-git.git
$ cd linux-hardened-git
$ makepkg -Codd #checkout the git source
$ cd src/linux-hardened # if you want to use the standard config copy it .config in this directory now
$ git checkout v5.10.4 # if the tag is not present use commit b1313fe517ca3703119dcc99ef3bbf75ab42bcfb
$ cd ../.. makepkg -ersi # check if 5.4.10 built on your system works

$ cd linux-hardened-git/src/linux-hardened
$ git checkout v5.10.4.a
$ cd ../..
$ makepkg -ersif # check hardened built locally fails

$ cd linux-hardened-git/src/linux-hardened
$ git bisect start
$ git bisect bad
$ git bisect good v5.10.4 # or b1313fe517ca3703119dcc99ef3bbf75ab42bcfb
$ cd ../..
$ makepkg -ersif

$ cd linux-hardened-git/src/linux-hardened
$ git bisect $result
$ cd ../..
$ makepkg -ersif

Thanks for chiming in loqs ;) going ahead right away.
Will probably take a while...
BTW. I need not use a PKGBUILD in this process?

The first git clone gets the PKGBUILD for linux-hardened-git from AUR which the rest of the instructions use.
You do not have to build the kernel using a PKGBUILD for bisection you could replace makepkg with the required make commands.

As a reminder do not clone the git tree to RAM to avoid losing it and the built source during reboots. Later rebuilds will be quicker avoiding having to rebuild unchanged files.

Hey @loqs, running into a wall here..
$ cd hardened
$ git clone https://aur.archlinux.org/linux-hardened-git.git
$ cd linux-hardened-git
$ makepkg -Codd
$ cd /tmp/makepkg/linux-hardened-git/src/linux-hardened
$ git checkout v5.10.4 # or
$ git checkout commit b1313fe517ca3703119dcc99ef3bbf75ab42bcfb

Neither of the last 2 commands yield a result but:
error: pathspec 'v5.10.4' did not match any file(s) known to git
git checkout commit b1313fe517ca3703119dcc99ef3bbf75ab42bcfb

@mark:
You need to pull in the tag objects for vanilla from the stable tree:
$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
$ git fetch stable


I've installed gnome now but I can't reproduce this issue, starts up on my end without an issue.

By commit v5.10.4 is 9701b9dcf8213f8bbd78625eb687f9089c3b5b95 I messed up the commit sorry
Edit:
No was correct originally and the repo without needing to add any more remotes contains and can checkout git b1313fe517ca3703119dcc99ef3bbf75ab42bcfb

Ah, now it's becoming to make some sense, I do however still have some blanks of exactly what it is I'm doing but I think it'll become clear once I'm a little further.
@Levente & @loqs, thanks for the patience..
@loqs, I set my makepkg.conf to build local, so that should be okay now, the first package is building so let's see what happens after this, that's not entirely clear to me (yet)

So, this what I did so far
$ git clone https://aur.archlinux.org/linux-hardened-git.git
$ cd linux-hardened-git
$ makepkg -Codd #checkout the git source
$ cd src/linux-hardened
$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
$ git fetch stable #like Levente said
$ git checkout v5.10.4
$ cd ../.. && makepkg -ersi

If this one is not working already,which it isn't, do I still need to continue to:
$ git checkout v5.10.4.a
or do I need to bisect between a lower version? I'd rather doit right right away;) Thanks

What .config did you use?
If in doubt attach linux-hardened-git/src/linux-hardened/.config
Edit:
Oh and do you use any out of tree modules such as nvidia?

@mark:
If no errors happened in between, that would mean the bug is present in vanilla and not the hardened patchset. Can you try to append the 5 additional patches from https://git.archlinux.org/linux.git/log/?h=v5.10.4-arch2 from "ALSA: hda/hdmi: fix locking in silent_stream_disable" to "Bluetooth: Fix attempting to set RPA timeout when unsupported" (the CLONE_NEWUSER can be left out).
You could add that as another remote and cherry-pick the commits for convenience.
If it still fails then the error must be somewhere in the kconfig, but i thought we already ruled that out by trying the vanilla kernel config.

In this case we are using "bisection" as shorthand for "bisection search algorithm". You keep bisecting until the "good" and "bad" commits are the same. It's pretty obvious when this happens. as "git" will give a nice print out of the details of the offending commit.

You might find these links helpful:
https://en.wikipedia.org/wiki/Bisection_method
https://wiki.archlinux.org/index.php/Bisecting_bugs_with_Git

I won't have time to do it until tomorrow, however, it is worthwhile we both do it to make sure that it is the same commit causing both our problems. (It is also a useful skill to master.)

@loqs, config version is:
# Linux/x86 5.10.4 Kernel Configuration
No out of tree modules that I know of, I'm completely Intel (NUC 10i7FNK)

@Levente, by 'in between' what do you mean by that? as the error already occurred on v5.10.4!
I did not have explicit errors happening, but, it's hard to say what I'm looking for. First timer ;) I know... but if you have a little more patience with this newbie I can do it on my own next time! Thanks

edit:@Irvine, I read the articles but the still are hard to grasp if done for the first time, at least for me it is..

@mark:
The link I have posted shows patches ontop of the vanilla v5.10.4:
https://git.archlinux.org/linux.git/log/?h=v5.10.4-arch2
ALSA: hda/hdmi: fix locking in silent_stream_disable Jan Alexander Steffens (heftig)
iwlwifi: Fix regression from UDP segmentation support Eric Dumazet
btrfs: Fix 500-2000% performance regression w/ 5.10 Josef Bacik
drm/amd/display: Add get_dig_frontend implementation for DCEx Rodrigo Siqueira
HID: quirks: Add Apple Magic Trackpad 2 to hid_have_special_driver list Felix Hädicke
Bluetooth: Fix attempting to set RPA timeout when unsupported Edward Vear

You said it doesn't happen on 5.10.4-arch2, and besides the kconfig those patches are the only difference in 5.10.4-arch2 compared to v5.10.4.

@Levente, yes 5.10.4-arch2 is okay, so if that's the only difference it must be one of those patches, I agree with the logic yes.
I'm done for today, will continue somewhere tomorrow and get the answer I'm sure;)

To cherry-pick those five commits
git cherry-pick -x 0438f3011663ce01a981bb8b3af0e84e73551ecf e5e77ad2223f662e1615266d8ef39a8db7e65a70 88fbde26128dedf0097aa45b8c923973a4e5f843 0128c7314a4d856461caacfdd528a7bffa03d41e 00f09a6a8193b46c83ae1c8ff6623db011f90099

git log --oneline
Check there are those five commits after 5.10.4

Then build that package

My best guess would be trying out this single patch:
20210101083852.12094-1-tiwai@suse.de/"> https://lore.kernel.org/alsa-devel/20210101083852.12094-1-tiwai@suse.de/
which is:
https://git.archlinux.org/linux.git/commit/?h=v5.10.4-arch2&id=00f09a6a8193b46c83ae1c8ff6623db011f90099
so cherry-pick 00f09a6a8193b46c83ae1c8ff6623db011f90099 should possibly fix it?

Well, I went on for a little while longer, but since things weren't working as they should, I did it again..
This is what I did but I'm unable to cherry pick even after a 'fetch -all', I'm sure it.s me doing something wrong:(
The commands I used:

$ git clone https://aur.archlinux.org/linux-hardened-git.git
$ cd linux-hardened-git
$ makepkg -Codd
$ cd src/linux-hardened
$ git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
$ git fetch stable
$ git checkout v5.10.4
$ git cherry-pick -x 00f09a6a8193b46c83ae1c8ff6623db011f90099

not working, so:

$ git fetch --all
$ git cherry-pick -x 00f09a6a8193b46c83ae1c8ff6623db011f90099
$ fatal: bad object 00f09a6a8193b46c83ae1c8ff6623db011f90099
See you tomorrow..

@mark:
that is a commit that is only present in the arch own tree and not in linux-stable or hardened.
You can add another remote:
$ git remote add arch https://git.archlinux.org/linux.git
$ git remote fetch arch

@Levente, Hehe, I guess I ain't getting a free hour today boss :-)

$ git remote add arch https://git.archlinux.org/linux.git
added.
$ git remote fetch arch
$ error: Unknown subcommand: fetch

Should that be 'add'?

git fetch arch
git cherry-pick -x 00f09a6a8193b46c83ae1c8ff6623db011f90099

@loqs, why didn't I try to do that myself:)

$ git fetch arch
$ From https://git.archlinux.org/linux
$ * [new branch] master -> arch/master

but the result is still the same..
$ fatal: bad object 00f09a6a8193b46c83ae1c8ff6623db011f90099

I even checked the commit number myself twice. it's correct..:(

Okay, I have started the bisection. Some points for anybody following:

1) git 'checkout v5.10.4' didn't work, so I used 'b1313fe517ca3703119dcc99ef3bbf75ab42bcfb'
.. After compiling, gnome failed to launch

At this point I looked deeper into 'git tags' running:
.. $ git tag -l --sort=-creatordate --format='%(creatordate:short): %(refname:short)'
.. 2020-12-30: 5.10.4.a
.. 2020-12-30: 5.10.3.a
.. 2020-12-30: 5.4.86.a
.. 2020-12-30: 4.19.164.a
.. 2020-12-30: 4.14.213.a
.. 2020-12-30: 5.10.2.a
.. 2020-12-30: 5.10.1.a
.. 2020-12-30: 5.10.a
.. 2020-12-21: 5.9.16.a
.. ~~~~~~~~~
.. ~~~~~~~~~

This leads me to believe the 'v prefix' was the original problem with the tag.

Anyway, in an effort to find the first good commit, I ran
.. $ git checkout 5.10.a
.. Previous HEAD position was b1313fe517ca Linux 5.10.4
.. HEAD is now at 1a36abfd6b77 Revert "dccp: don't free ccid2_hc_tx_sock struct in dccp_disconnect()"
.. $

I am now compiling this. It will take a while, so do not expect fast reporting. If this also fails, I will bisect between the last known good version, (5.9.16.a,) and 5.10.a Otherwise, I will bisect between 5.10.a and 5.10.4.a


To help speed things up I have made a few customisations to /etc/makepkg.conf

#Edited: CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
CFLAGS="-march=native -O2 -pipe -fno-plt"

#Edited: CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
CXXFLAGS="${CFLAGS}"

#Edited: #MAKEFLAGS="-j2"
MAKEFLAGS="-j$(nproc)"

Any other hints on speeding this up will be greatly appreciated, as will any comments on potential errors, misconceptions or pitfalls related to my method.

Irvine

@IrvineHimself
if b1313fe517ca3703119dcc99ef3bbf75ab42bcfb already fails that means its a vanilla bug and has nothing to do with hardening patches, no need to bisect based on 5.10.a.

To find the offending commit, you can bisect between v5.9.16 and v5.10 however as the arch package seems to work i recommend what i wrote in the previous comments in https://bugs.archlinux.org/task/69171#comment195456 and https://bugs.archlinux.org/task/69171#comment195458.

Please add the arch linux own kernel tree and test the 5 commits on top of v5.10.4 that the arch vanilla kernel has added, one of them needs to be the fix.

@mark
to fetch all tags for 'arch' you can add
```
fetch = +refs/tags/*:refs/tags/*
```
to the .git/config for the [remote "arch"] and then fetch again

@Levente, thank you very much, after your edit to config and editing .gitconfig? I was able to fetch again.
I was only able to 'cherry-pick' after I did:

$ git reset --hard origin/master
$ git cherry-pick -x 00f09a6a8193b46c83ae1c8ff6623db011f90099
$ Auto-merging sound/pci/hda/patch_hdmi.c ....

It did merge. I'm not there yet, I still need to build the kernel..
However I'm hopeful, but, it's clear to me how 'deep' GIT really goes and really need to update my game!

If you boot linux-5.10.4.arch2-1 with the option oops=panic does that produce the issue?

After:
1) git checkout 5.10.4.a
2) git remote add arch https://git.archlinux.org/linux.git
3) editing .git/config to include: fetch = +refs/tags/*:refs/tags/*
4) git fetch arch

I ran:
5) git cherry-pick -x 00f09a6a8193b46c83ae1c8ff6623db011f90099
6) makepkg -ersif

This appears to have fixed the problem. I have attached journalctl output from relevant relevant boot.

Thanks for your patience and I hope this helps
Irvine

Edited: to attach journalctl output

@Irvine, nice , my build failed..

@loqs & Levente, there must be a failure in my build for I get:
$ DEPMOD 5.11.0-rc1-1-hardened-git-r969629
$ rm: cannot remove '/home/mark/build/kernel/hardened/linux-hardened-git/pkg/linuxhardened-git-r969629/source': No such file or directory
$ rm: cannot remove '/home/mark/build/kernel/hardened/linux-hardened-git/pkg/linuxhardened-git-r969629/build': No such file or directory

How can I prevent this from happening next time( since Irvine solved it probably ) I try to build from git for solving issues?
Also, how do I revert to a clean local tree, for I can save this build for future builds? Thanks for all the help, mark

@loqs I added 'oops=panic' to the grub entry for my plain vanilla kernel 5.10.4-arch2-1. It had no problem launching gnome.

As pointed out above, the solution definitely appears to be the patch for "ALSA: hda/hdmi: fix ..." in commit 00f09a6a8193b46c83ae1c8ff6623db011f90099

@Irvinehimself I was wondering if with the addition of 00f09a6a8193b46c83ae1c8ff6623db011f90099 you would encounter  FS#69190 

Found my issue, I forgot to checkout 5.10.4.a after I fetched arch;)

@loqs I don't have 'sleep' enabled. In fact, since I strive to keep my system 'lean & mean', I don't even have the 'acpid' package installed.

As I recall, though I am not certain without checking, another problem I would have is a very small "SWAPFILE" of only 2GB.

In general, for power management, I have a low power warning which, again without checking, I believe I set in the manufacturer's BIOS



@Mark

Check 'makepkg --help' but I think 'makepkg -Codd' should clean the src directory and reset everything.

After which:
1) git checkout 5.10.4.a
2) git remote add arch https://git.archlinux.org/linux.git
3) editing .git/config to include: fetch = +refs/tags/*:refs/tags/*
4) git fetch arch
5) git cherry-pick -x 00f09a6a8193b46c83ae1c8ff6623db011f90099
6) cd ../..
7) makepkg -ersif

Irvine

@Irvine, thank you. I didn't do a 'makepkg Codd' and did just build after checking 5.10.4.a , it build fine, but, I'll keep it in mind for next time;)
Well, I can confirm the same as Irvine, build succeeded and I had no problems using my own script to start X after cherry-pick 00f09a6a8193b46c83ae1c8ff6623db011f90099.

So, now that we know the what I would like to know the why and why only 3 Archers seem to be affected.
I can't imagine that if 2.72% of users run Linux-hardened, which would translate to 2700 users if we look at the number or registered forum members only 3 are affected by this 'bug' (see) https://pkgstats.archlinux.de/packages#query=linux-hardened
The numbers in the previous sentence may or not be ± accurate at all.

What is so different on our 3 systems compared to all other 'hardened' users we run into this bug?
I get the feeling this patch ain't going to be reverted because other people having crashes without it.
Just trying to investigate what's next, do we 'forever' have to patch our system reverting commit 00f09a6a8193b46c83ae1c8ff6623db011f90099 ?, what's next?

Thanks, mark

@mark
you are mixing quote some stuff up, 00f09a6a8193b46c83ae1c8ff6623db011f90099 is cherry-picked which was an early backport, there is no revert.
That commit is scheduled for inclusion:
https://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git/commit/?h=for-linus&id=3d5c5fdcee0f9a94deb0472e594706018b00aa31

and either way, if we apply that in our arch vanilla kernel and it fixes problems people were facing we would also include it in hardened. You don't need to expect to build your own kernels for all eternity, this was purely a debugging workflow to pinpoint the issues which i can't reproduce.

I think your numbers are a bit off:

1) Reading the forums, a lot of users don't update regularly. Further, even worse, while they update regularly, they use hibernate and never reboot.

2) While many users have Linux-hardened installed for testing or as a vanity, in many instances it is not their goto kernel and may not have been used years.

3) Even if they do regularly update and use a kernel, (or pckg,) unless it's something that causes them great personal pain and anguish, many users do not report bugs either here or to the forum. This is especially true if they have an alternative package/kernel available.

4) Not everyone uses Gnome, (you pointed out above that I3 was unaffected)

5) Given the nature of the patch, (and @Anthrax's inability to confirm the bug,) there appears appears to be hardware/peripheral components to an individuals susceptibility.

In summary: Before installing *any* package, (Official, AUR, or some alternative source,) I strive to read bug reports, security advisories and your basic google search. Armed with this qualitative overview of bug reporting, I would not read too much into how many users are reporting problems.

Irvine

@Levente, ah, of course, I read about cherry-pick but I understand it and get the picture now, thanks;)
It's reassuring to know I don't need to revert that commit forever, I rather use the default hardened repo[Extra] kernel.
It's not that I'm afraid to build my own kernel. I have done this quite some time stripping everything with menuconfig by hand and using a short module list.
The point is I don't really see a big advantage in doing that just to get a small kernel, my system is fast enough and plenty RAM.

@Irvine, yeah, I agree my numbers may be 'very' off;)
1 I'm no pro by any means and thus try to 'run a tight ship' update frequently so I notice problems as soon as possible.
2 It's my main kernel, I use it together with firejail/apparmor for I'm running a few secure shells websites etc. and like to 'keep em separated' ;)
3 This is sad:( I would expect a lot of Archers to face their problems 'head on' and don't put it away by using something else, this is bad housekeeping! at least that's what I think about it..

I mostly do what mention in your summary but I don't read every bug report only ones that interest me and I certainly read sec. advices and not only the ones on Arch.

mark

edit: at the post below mine, @Levente, I don't use 'pkgstats' I just looked at the results, but you're right about derailing I leave it here, thanks

@mark:
This now starts to go too much off topic so we shouldn't derail this bug ticket -- However I also wouldn't give too much credit for numbers in pkgstats as they do not provide a uniform slice of the whole community including all sub groups of it. One can expect that security and privacy conscious people who care enough to run a hardened kernel and apps using firejail/apparmor etc tend to less often feel comfortable in using tools that in any way track packages they are using, so i expect the percentage to be higher and way more off compared to looking at alike things not touching such group boundaries (like an editor f.e.).