FS#48230 - [cryptsetup/linux-lts] 4.1.18 luks-root broken

Attached to Project: Arch Linux
Opened by c (c) - Thursday, 18 February 2016, 00:44 GMT
Last edited by Andreas Radke (AndyRTR) - Wednesday, 24 February 2016, 17:01 GMT
Task Type Bug Report
Category Upstream Bugs
Status Closed
Assigned To Tobias Powalowski (tpowa)
Thomas Bächler (brain0)
Andreas Radke (AndyRTR)
Bartłomiej Piotrowski (Barthalion)
Architecture x86_64
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 10
Private No

Details

Description:
Since today's 4.1.18 linux-lts update there's the following error at the password prompt,
and I could only fix it by downgrading to 4.1.17. Tried 4.4.1 and that one works as well.
This task depends upon

Closed by  Andreas Radke (AndyRTR)
Wednesday, 24 February 2016, 17:01 GMT
Reason for closing:  Fixed
Comment by c (c) - Thursday, 18 February 2016, 00:45 GMT
Here's the error. Sorry forgot it in the description.
sda3 is the root dm-crypt partition supplied in the kernel cmdline.

The error:
Failed to setup dm-crypt key mapping for device /dev/sda3
Comment by Al T. (alfmel) - Thursday, 18 February 2016, 02:13 GMT
I can confirm the error. Upgraded to linux-lts-4.1.18-1 and encrypted root partition could not be mounted, even with the password. The error said to verify the appropriate encryption module was supported by the kernel. I assume the modules were missing from the initramfs. Reverting back to 4.1.17-1 and rebuilding the initramfs image fixed the issue.
Comment by Adam Lesperance (lespea) - Thursday, 18 February 2016, 03:03 GMT
This also affects entries in crypttab. Getting the error message: "Failed to activate: Invalid argument"
Comment by Doug Newgard (Scimmia) - Thursday, 18 February 2016, 03:19 GMT Comment by c (c) - Thursday, 18 February 2016, 04:23 GMT
Can we retract the 4.1.18 update and replace it with 4.1.17 so that it's not installed by more people?
Comment by Amarildo (Amarildo) - Thursday, 18 February 2016, 11:52 GMT
I can confirm this too.

linux-lts 4.1.18-1

I have an encrypted LVM. The message that appeared to me was something like:

"Failed to setup something for /dev/sda2. Make sure the Kernel has xts-plain64 support".
Comment by Noel Kuntze (thermi) - Thursday, 18 February 2016, 14:12 GMT
I am impacted by this, too. For me, the error is the following:
"""
Failed to setup dm-crypt key mapping for device /dev/sda4.
Check that kernel supports aes-xts-plain64 cipher (check syslog for more info).
"""
Reverting to 4.1.17 and rebuilding the initcpio fixes the issue.
Comment by Andreas Radke (AndyRTR) - Thursday, 18 February 2016, 15:54 GMT Comment by Philip Müller (philm) - Sunday, 21 February 2016, 12:02 GMT
Followed kernels are affected by this: 3.10.97, 3.14.61, 3.18.27, 4.1.18.
Earlier posted kernels: 3.13.11-ckt34, 3.19.8-ckt34, 4.2.8-ckt3, 4.5-rc1
Kernels not affected by this: 3.12.54, 4.3.6, 4.4.2, 4.5-rc3

Cryptsetup already supports these changes after backporting some patches: https://gitlab.com/cryptsetup/cryptsetup/issues/284
However the kernel API need to be properly patched as well (otherwise all old cryptsetup version will not work anyway).
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 12:09 GMT
Followed patches need to been applied to the current stable release of cryptsetup: 93ed401b, 4dc88e8f. This will avoid more damage users have with encrypted root-partitions.
Comment by John Tyree (hiptobecubic) - Sunday, 21 February 2016, 15:47 GMT
Why are we still allowing users to download this? It renders machines 100% unusable. These are LTS users as well, some effort to maintain a coherent setup is implicit.

Can we not roll back to the previous working release? "Don't use LTS" is not a solution for everyone.
Comment by Andreas Radke (AndyRTR) - Sunday, 21 February 2016, 16:47 GMT
"I'm announcing the release of the 4.1.18 kernel.

All users of the 4.1 kernel series must upgrade."

This means it fixes security related issues. All other users really want that new release.

Please help to find a quick solution what patches need to be applied to this release to make it work again.
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 16:50 GMT
I already pointed them out to you. In Manjaro I fixed it hours ago already.

Simply install the new cryptsetup package: https://projects.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/cryptsetup&id=ea2c8f73c45aa239ed5f356a8ecd01aeba51ef1d

All users who can't access their HDDs anymore have to fix them on their own following for example my guide: https://forum.manjaro.org/index.php?topic=31356.msg257698#msg257698
Comment by John Tyree (hiptobecubic) - Sunday, 21 February 2016, 19:10 GMT
Andreas,

That's all well and good for those other users, *but the system cannot boot*. Worse than that, people with machines that are doing important things are left in a situation where they can't even *access their filesystem*. It's not like we're talking about new drivers for gravis gamepads. Waiting until after the system is rendered unbootable and then expecting the user to come find this bug report and patch things manually is leaving the "maintenance" part out of "maintaining a linux distribution." Maybe no one noticed it when it was released, but this bug report is *days* old now and new people are still installing that package and bricking themselves.

At the very least, slap a giant warning on it at install time.
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 19:44 GMT
Hey John, do you keep track on all issues reported somewhere? Upstream knew about it for sure. Some user space libraries like cryptsetup already fixed it in their git-versions. And it slipped thru. Simply cos not everybody is using encrypted root partitions. It happened. Clearly the user has to "fix" it as his system is otherwise not bootable. As long as there is a way to get the system back it is fine. Nothing is broken. The data is not lost. Simply a function needed to be adopted as there was a long lived bug in the system which got properly fixed now.

So what is the matter here really?
Comment by John Tyree (hiptobecubic) - Sunday, 21 February 2016, 19:48 GMT
The problem is not that something slipped through. That happens. The problem is that it slipped through on *Thursday*. It is now *Sunday* and new people are still updating to this broken package with no hint that anything is wrong until they reboot and are helpless. These situations are exactly what "rollback" is all about.

It doesn't make sense to continuing distributing a *known broken* package. If we're committed to doing that for some reason, at least tell everyone it's broken when they install it so they can do something about it before rebooting.
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 20:03 GMT
Ok, I can only speak for Manjaro. I got notified about a bug regarding kernel 3.19.8.14-1 on the 11th February. So this issue is longer out. The broken package is cryptsetup and not linux-lts. Upstream fixed a long lived bug in their kernels. The patch for cryptsetup exists since two month as upstream is speaking to each other.

I know that some people are angry which faced the issue. However at Manjaro we reacted within hours (stable update: Sun Feb 21 09:38:51 CET 2016; fix-release: Sun Feb 21 11:57:16 CET 2016). Also we released a guide on how to fix it, if you faced the issue before the fix came out.

However I can't speak for Archlinux. The needed fixed package is now in their testing branch. So as soon as this is on stable we are good.

So if you really want to dig deeper on when upstream knew it, you can check also this:

https://lkml.org/lkml/2015/12/29/383
https://github.com/smuellerDD/libkcapi/commit/1d6c3b1b540caea784a95b1ca6e2cf38c174f585
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 21:16 GMT
Ok, followed patches got missed within 4.1 kernel series:

dd504589577d8e8e70f51f997ad487a4cb6c026f
crypto: algif_skcipher - Require setkey before accept(2)

a0fa2d037129a9849918a92d91b79ed6c7bd2818
crypto: algif_skcipher - Add nokey compatibility path

patch: http://www.spinics.net/lists/stable/msg120446.html
Comment by c (c) - Monday, 22 February 2016, 14:36 GMT
The real problem is that cryptsetup is adapted at all. The kernel crypto API shall not break because Linus says so and the kernel cryptsetup are not a combination like OpenBSD's root crypt support in kernel+bootloader. The most important patch is the kernel patch posted on the mailing list. From the mailing list it's also apparent upstream had been notified of this breakage but nothing happened between January and now to fix it. Maybe too much non-critical stuff is merged into patch releases. This is quite an unusually serious regression and highly surprising. It's way worse than the broken i915 atomic code since 4.3 (which is the reason I'm on 4.1-lts).
Comment by Andreas Radke (AndyRTR) - Monday, 22 February 2016, 17:59 GMT
So with patched cryptsetup there's no need for immediately patching the lts kernel with these 2 additional commits? We can wait for 4.1.19 or whenever they get merged to a stable lts kernel release?
Comment by Philip Müller (philm) - Monday, 22 February 2016, 18:09 GMT
You can either patch the linux-lts package with the linked patch by Milan Broz, which need to be verified anyway, or push the patched cryptsetup, like I did on Manjaro to your core repository. Both will fix it. With the patches added to cryptsetup you simply support the new style of crypto API, which is needed to get new users not into the dilemma. All users facing the issue need the new cryptsetup and manual user intervention to get their boxes back.
Comment by Andreas Radke (AndyRTR) - Monday, 22 February 2016, 18:13 GMT
Just moved cryptsetup.

Loading...