FS#49806 - [intel-ucode] Stuck at black screen

Attached to Project: Arch Linux
Opened by Peter (protake) - Wednesday, 22 June 2016, 14:35 GMT
Last edited by Christian Hesse (eworm) - Tuesday, 12 July 2016, 07:19 GMT
Task Type Bug Report
Category Packages: Testing
Status Closed
Assigned To Thomas Bächler (brain0)
Christian Hesse (eworm)
Bartłomiej Piotrowski (Barthalion)
Architecture All
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

After the latest upgrade, my laptop gets stuck at boot, just showing a black screen. Removing intel-ucode from my systemd-boot entry fixes the issue. Laptop has a Skylake i7-7500U.
This task depends upon

Closed by  Christian Hesse (eworm)
Tuesday, 12 July 2016, 07:19 GMT
Reason for closing:  Fixed
Additional comments about closing:  intel-ucode 20160607-2 in [testing]
Comment by Peter (protake) - Wednesday, 22 June 2016, 14:36 GMT
It's a i7-6500U. Sorry, typo.
Comment by Christian Hesse (eworm) - Wednesday, 22 June 2016, 20:25 GMT
Oh, that's no good. I tested with a m3-6Y30 and i3-6100T, which are Skylake as well.
Microcode has been updated from revision 0x74 to 0x84 (m3) and 0x39 to 0x8a (i3) for me.

Logs should contain something like this:

microcode: microcode updated early to revision 0x8a, date = 2016-04-06
microcode: CPU0 sig=0x506e3, pf=0x2, revision=0x8a

Booting without microcode update you should see the second line at lease. What does it look like for you?
Comment by Peter (protake) - Wednesday, 22 June 2016, 20:36 GMT
[ 1.465137] microcode: CPU0 sig=0x406e3, pf=0x80, revision=0x49
[ 1.465150] microcode: CPU1 sig=0x406e3, pf=0x80, revision=0x49
[ 1.465169] microcode: CPU2 sig=0x406e3, pf=0x80, revision=0x49
[ 1.465189] microcode: CPU3 sig=0x406e3, pf=0x80, revision=0x49
[ 1.465240] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Comment by Peter (protake) - Thursday, 23 June 2016, 12:39 GMT
Works fine on my Haswell E3-1231v3, though:

[ 0.060929] microcode: CPU3 microcode updated early to revision 0x20, date = 2016-03-16
[ 0.194344] microcode: CPU0 sig=0x306c3, pf=0x2, revision=0x20
Comment by Henrique de Moraes Holschuh (hmh) - Tuesday, 28 June 2016, 23:59 GMT
Also tracking a similar regression in Debian.

Cross-bugtracker URL:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=828819

Relevant details on the Debian side:
Processor: i7-6500U, sig. 0x406e3.
Original microcode: 0x6A.
dmidecode available in Debian's bug report.

It looks like pf=0x80 might be relevant.

It looks like microcode being loaded from FIT (secure) is *not* relevant (microcode with odd revision in arch linux report -- loaded from FIT, microcode with even revision in Debian report -- uploaded through WRMSR).
Comment by Henrique de Moraes Holschuh (hmh) - Wednesday, 29 June 2016, 00:04 GMT
@Christian Hesse (eworm):

It would be helpful to know the "pf" of the two processors that updated fine, please...
Comment by Christian Hesse (eworm) - Wednesday, 29 June 2016, 06:07 GMT
Intel(R) Core(TM) i3-6100T CPU @ 3.20GHz:
microcode: microcode updated early to revision 0x8a, date = 2016-04-06
microcode: CPU0 sig=0x506e3, pf=0x2, revision=0x8a

Intel(R) Core(TM) m3-6Y30 CPU @ 0.90GHz:
microcode: microcode updated early to revision 0x8a, date = 2016-04-06
microcode: CPU0 sig=0x406e3, pf=0x80, revision=0x8a
Comment by Henrique de Moraes Holschuh (hmh) - Wednesday, 29 June 2016, 10:56 GMT
Thanks. We still have too few data points, but so far:

Only sig=0x406e3, pf=0x80, rev < 0x74 seems affected by the regression.
Comment by Christian Hesse (eworm) - Wednesday, 29 June 2016, 10:59 GMT
Peter, what system do you own? Is it a "Tuxedo InfinityBook 13v2"?
Comment by Peter (protake) - Wednesday, 29 June 2016, 11:06 GMT
I have read the bug report over at Debian and, in fact, it is. I'm not sure about the v2:

Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: TUXEDO
Product Name: Skylake Platform
Version: 0.1
Family: Skylake System
Comment by Tatsuyuki Ishi (ishitatsuyuki) - Wednesday, 29 June 2016, 11:16 GMT
Just for your reference, I'm using Lenovo ThinkPad X1 Yoga with the following data:
[ 0.125467] smpboot: CPU0: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz (family: 0x6, model: 0x4e, stepping: 0x3)
[ 0.739403] microcode: CPU0 sig=0x406e3, pf=0x80, revision=0x88
Comment by Henrique de Moraes Holschuh (hmh) - Wednesday, 29 June 2016, 11:19 GMT
@Tatsuyuki Ishi: does your Thinkpad X1 update to microcode 0x8a without issues?
Comment by Henrique de Moraes Holschuh (hmh) - Wednesday, 29 June 2016, 11:22 GMT
@Peter: if you are feeling adventurous, you might want to run BIOS BITS on that laptop and check what it complains about (http://www.biosbits.org)... who knows, maybe if you tell BITS to fix all hardware initialization and boot with microcode 0x8a (you will need to have the microcode.dat file from Intel "on hand" to feed BITS), it will work...
Comment by Tatsuyuki Ishi (ishitatsuyuki) - Thursday, 30 June 2016, 12:28 GMT
@hmh: sorry for the slow reply, I'm also suffering from this issue.
Comment by Henrique de Moraes Holschuh (hmh) - Thursday, 30 June 2016, 13:52 GMT
Given Tatsuyuki-san's data point, the problem is not related to a big skip on revisions, or to a minimum revision. The m3-6y30 has absolutely the same microcode signature (same sig and pf), and updated fine. None of the i7-6500U in this report managed to update, regardless of the microcode version they were running at the time of the update.

Does anyone have an i7-6500U that actually updated fine?
Comment by Henrique de Moraes Holschuh (hmh) - Thursday, 30 June 2016, 14:59 GMT
Given Tatsuyuki-san's data point, the problem is not related to a big skip on revisions, or to a minimum revision. The m3-6y30 has absolutely the same microcode signature (same sig and pf), and updated fine. None of the i7-6500U in this report managed to update, regardless of the microcode version they were running at the time of the update.

Does anyone have an i7-6500U that actually updated fine?
Comment by Henrique de Moraes Holschuh (hmh) - Thursday, 30 June 2016, 19:18 GMT
Given Tatsuyuki-san's data point, the problem is not related to a big skip on revisions, or to a minimum revision. The m3-6y30 has absolutely the same microcode signature (same sig and pf), and updated fine. None of the i7-6500U in this report managed to update, regardless of the microcode version they were running at the time of the update.

Does anyone have an i7-6500U that actually updated fine?
Comment by Henrique de Moraes Holschuh (hmh) - Monday, 04 July 2016, 10:44 GMT
(From Debian bug report):
ASUS UX305CA-AS (processor M3-6Y30) fails to update (BIOS 205 - microcodes rev 0x74).
BIOS 300 is reported to work, however since that BIOS has microcode 0x8a, it simply means the microcode is already current and no update is attempted.
Comment by Christian Hesse (eworm) - Monday, 04 July 2016, 18:52 GMT
It becomes even more mysterious... My m3-6Y30 (inside a Lenovo Yoga 700-11) works just fine...
Comment by Henrique de Moraes Holschuh (hmh) - Tuesday, 05 July 2016, 11:18 GMT
Yes, it only happens to sig=0x406e3, pf=0x80 so far, but it doesn't always happen and it is not related to the previous version of the microcode, so it clearly depends on something else: probably either some low-level BIOS component, or PCH firmware level. Might also depend on some system setting (SGX/TXT, TSX, GPU, or power-management related for example).
Comment by Henrique de Moraes Holschuh (hmh) - Friday, 08 July 2016, 21:17 GMT Comment by Henrique de Moraes Holschuh (hmh) - Saturday, 09 July 2016, 16:14 GMT
I've worked around this in Debian by removing the microcode for signature 0x406E3 from the distribution, given that the only available workaround for users hit by the issue is to update to a firmware that has microcode rev. 0x8a or newer in the first place (thus rendering the microcode update useless, anyway). This change will propagate to Ubuntu eventually.

I plan to revert this change the next time this microcode gets updated by Intel, in hope that they fixed the underlying issue.

Arch might want to do the same as I did for Debian. Either use "iucode_tool -s !0x406e3" to generate the initramfs and /lib/firmware/intel-ucode, or, alternatively, do not ship /lib/firmware/intel-ucode/06-4e-03 nor allow it to be added to the early initramfs cpio archive.

Note: the removal of this microcode is going to affect Skylake-U K-1 processors as collateral damage. The K-1 stepping is a lot less broken than the Skylake-U/Y D-1 and might not object to the microcode update, but we have no reports in Debian, Arch or Fedora from owners of such processors to know (Skylake-U K-1 processors report signature=0x406e3, pf=0x40).
Comment by Christian Hesse (eworm) - Saturday, 09 July 2016, 20:16 GMT
Going the same way... Just uploaded intel-ucode 20160607-2 to [testing].
Anybody wants to confirm that the issue is "fixed"?
(Well, it is worked around for now... Let's wait for Intel and their next microcode release.)
Comment by Peter (protake) - Sunday, 10 July 2016, 12:09 GMT
Issue has indeed been "fixed".

Loading...