FS#42332 - [mkinitcpio] [linux] Late firmware loading causes freezes when instructions are being disabled

Attached to Project: Arch Linux
Opened by Steven Honeyman (stevenhoneyman) - Saturday, 11 October 2014, 12:26 GMT
Last edited by Thomas Bächler (brain0) - Sunday, 26 October 2014, 11:17 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Tobias Powalowski (tpowa)
Thomas Bächler (brain0)
Dave Reisner (falconindy)
Architecture All
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:

Intel found problems with TSX instructions on their Haswell and above processors, so with the recent microcode update - they just disabled them.
Arch's current intel-ucode package provides microcode updates through /usr/lib/firmware/intel-ucode/* and the kernel firmware loader. This is a big problem for anyone that decides to update to the latest microcode(1) with an affected processor - as soon as a reload is done, every process that uses libpthread segfaults. On reboot, systemd has already started when the microcode is updated; so startup just... stops.

LKML discussion: https://lkml.org/lkml/2014/9/18/218

Current workarounds/fixes include:
* Remove /usr/lib/firmware/intel-ucode and force early load of microcode through a cpio pre-pended to initramfs
* Find a way to revert the changes made in glibc commit 1cdbe579482c ("Add the low level infrastructure for pthreads lock elision with TSX")
-devel@lists.fedoraproject.org/msg81403.html"> https://www.mail-archive.com/devel@lists.fedoraproject.org/msg81403.html
* Hardware blacklist in glibc for HLE support on the affected CPUs



(1) packages in Arch repo have been flagged as OOD for some time, latest is directly from Intel, dated 2014-09-15
- http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=24290
This task depends upon

Closed by  Thomas Bächler (brain0)
Sunday, 26 October 2014, 11:17 GMT
Reason for closing:  Fixed
Comment by Steven Honeyman (stevenhoneyman) - Saturday, 11 October 2014, 12:29 GMT Comment by Allan McRae (Allan) - Sunday, 12 October 2014, 00:11 GMT
There is a slight issue here... Removing --enable-lock-elision from the glibc build currently leaves some of it enabled. There is a patch to fix that, but Mageia notice issues when applying it. I am chasing these issues upstream.

Adding Thomas as an FYI so that he does not update intel-ucode until I have glibc fixed.
Comment by Thomas Bächler (brain0) - Sunday, 12 October 2014, 07:12 GMT
I already hard-froze my laptop trying to upgrade the ucode and was barely able to restore it to working order.
Comment by Thomas Bächler (brain0) - Sunday, 12 October 2014, 07:20 GMT
Allan, we should also consider early loading of the microcode as discussed in the lkml thread.
Comment by Allan McRae (Allan) - Sunday, 12 October 2014, 08:09 GMT
I agree about the early loading, but I am not so familiar with that area.
Comment by Thomas Bächler (brain0) - Sunday, 12 October 2014, 08:12 GMT
This is the first time I am reading about it, and the documentation is extremely sparse.
Comment by Thomas Bächler (brain0) - Sunday, 12 October 2014, 08:15 GMT
This is the extremely badly written documentation: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/x86/early-microcode.txt

There is no indication whatsoever where to get "microcode.bin" from - intel offers a microcode.dat file which we currently split into cpu-specific firmware files. Anyway, this is a kernel bug, not a glibc one.
Comment by Allan McRae (Allan) - Sunday, 12 October 2014, 08:37 GMT
I'm going to follow Fedora/Debian's lead and temporarily disable lock elision in glibc until this is fixed.
Comment by Steven Honeyman (stevenhoneyman) - Sunday, 12 October 2014, 08:47 GMT
microcode.bin is produced simply by concatenating any number of the split binary blobs together. For example if you only have one affected CPU you can just use the 1 microcode update that applies.
To make a microcode.bin that works for all Intel cpus, then:

cat /lib/firmware/intel-ucode/* >microcode.bin
(and then follow the cpio instructions from the kernel docs)
Comment by Thomas Bächler (brain0) - Sunday, 12 October 2014, 10:30 GMT
Allan, no need to change glibc, I will likely have this problem solved very soon.
Comment by Thomas Bächler (brain0) - Sunday, 12 October 2014, 17:17 GMT
The kernels and intel ucode package in testing now implement early loading. However, you need to configure your boot loader (see https://wiki.archlinux.org/index.php/Microcode#Enabling_Intel_Microcode_Updates).

Loading...