Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#31230 - [libcryptsetup] dm-crypt unlocking with systemd-cryptsetup is a lot slower than with the old scripts

Attached to Project: Arch Linux
Opened by Heiko Baums (cyberpatrol) - Tuesday, 21 August 2012, 21:21 GMT
Last edited by Dave Reisner (falconindy) - Saturday, 20 October 2012, 16:56 GMT
Task Type Bug Report
Category Upstream Bugs
Status Closed
Assigned To Thomas Bächler (brain0)
Dave Reisner (falconindy)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
If the harddisk is encrypted with dm-crypt/LUKS the unlocking of the containers done by /etc/rc.d/functions at boot time is a lot slower with the new syntax in /etc/crypttab (systemd-cryptsetup) than with the old syntax (the old scripts).

I have 2 encrypted partitions which are unlocked at boot time by initscripts. The keys are written rawly on a USB stick and have to be read rawly. (See: do_unlock_legacy() -> /dev*) -> *))

With the new syntax it takes between 42 and 43 seconds to read the 2 keys from the USB stick and to unlock both partitions. Every single partition takes about 21 seconds.
With the old syntax it takes between 11 and 13 seconds to read the 2 keys from the USB stick and to unlock both partitions. Every single partition takes about 6 seconds.

This long delay with the new syntax (systemd-cryptsetup) is totally unacceptable and needs to be fixed or removed in favor of the old syntax and scripts.

This is the old syntax in /etc/crypttab:
home /dev/sdaX /dev/usbstick:12345678:1024

This is the new syntax in /etc/crypttab:
home /dev/sdaX /dev/usbstick keyfile-size=1024,keyfile-offset=12345678
This task depends upon

Closed by  Dave Reisner (falconindy)
Saturday, 20 October 2012, 16:56 GMT
Reason for closing:  Fixed
Additional comments about closing:  testing/cryptsetup 1.5.1-1
Comment by Dave Reisner (falconindy) - Tuesday, 21 August 2012, 21:50 GMT
strace'ing this outside of the boot process to see where it actually spends a lot of time would be useful.
Comment by Heiko Baums (cyberpatrol) - Tuesday, 21 August 2012, 22:37 GMT
Tell me how to do this (the strace'ing), and I see if I can find it out.
Comment by Dave Reisner (falconindy) - Tuesday, 21 August 2012, 22:44 GMT
strace /usr/lib/systemd/systemd-cryptsetup attach <name> <device> <keyfile> <options>

essentially, each of columns in order from /etc/crypttab. It'll probably be fairly noisy. You can pass the -o flag with a filename to get the output logged to a file, but I assume you'd want to look at it as it goes by to see if there's some sort of crazy loop, or some syscall that it's hanging on.
Comment by Heiko Baums (cyberpatrol) - Wednesday, 22 August 2012, 11:50 GMT
It will take a while until I can do the strace. After the latest update of my Live USB stick I can't boot it anymore. So I have to wait until  FS#31240  is fixed.
Comment by Tom Gundersen (tomegun) - Wednesday, 22 August 2012, 18:45 GMT
Thanks for the report. Very good to know. This should be fixed.

My guess is that libcryptsetup does something crazy, but a strace would tell us for certain (hopefully).

Notice how the legacy stuff does dd with "bs=1". If I understand correctly, that's about as inefficient as it is possible to get, so I'm rather confused how we managed to be slower than that...
Comment by Dave Reisner (falconindy) - Thursday, 23 August 2012, 03:33 GMT
libcryptsetup reads a byte at a time as well. IMO, this isn't something that needs fixing in systemd. strace attached.
   strace.log (197.1 KiB)
Comment by Dave Reisner (falconindy) - Thursday, 23 August 2012, 03:41 GMT
Adding an strace for cryptsetup as well, which looks pretty much identical (as I would expect). Glancing over the libcryptsetup internals, these 2 binaries take different front end code paths but it all converges on some extremely general low level methods which do all the real work.

Nothing else that I can test myself here.
Comment by Heiko Baums (cyberpatrol) - Friday, 24 August 2012, 00:00 GMT
Here are the times again measured by the time command:

# time uncrypt-initscripts
unlock swap
unlock home

real 0m13.498s
user 0m12.942s
sys 0m0.100s

# time uncrypt-systemd
unlock swap
unlock home

real 0m45.571s
user 0m15.806s
sys 0m28.248s

uncrypt-* are scripts that do the same as the corresponding commands in /etc/rc.d/functions but without all those cases and variables.

Doing an strace of systemd-cryptsetup is almost impossible if not totally impossible. After 5 minutes of seeing a lot of "read..." I cancelled it.

Is it possible that systemd-cryptsetup first reads the whole keyfile device, which would be a total 4 GB stick, and a slow one, in my case, and cuts the relevant part out of this whole output afterwards? Just a wild guess. If this is the case then this needs to be fixed to the way dd reads it.
Comment by Dave Reisner (falconindy) - Friday, 24 August 2012, 03:23 GMT
the fact that the strace is lengthy does not make it 'almost impossible'. If you extract the key and use it as a keyfile rather than an offset, is it any faster?
Comment by Tom Gundersen (tomegun) - Friday, 24 August 2012, 19:37 GMT
@heiko: do you happen to remember why you chose "bs=1" when you read the key from the device in the original code? apparently cryptsetup does the same, and it makes no sense to me.
Comment by Heiko Baums (cyberpatrol) - Friday, 24 August 2012, 23:08 GMT
I think just because to be sure that exactly the key (the given amount of bytes) and not more is read and passed to cryptsetup. At that time I haven't tested if it works with a bs>1.

I mean, bs=512 would be faster, but what if the keyfile size is just 1011? Are then only the 1011 bytes read or would it read 1024 bytes? With bs=1 I'm sure that it will only read the 1011 bytes.
Comment by Dave Reisner (falconindy) - Friday, 24 August 2012, 23:41 GMT
Who said anything about 512?

bs=$keyfile_size count=1

Anyways, the issue is that cryptsetup "seeks" to the keyfile_offset 1 byte at a time, which means a syscall for every byte. It can't actually use lseek because there's no guarantee of seekable input (cryptsetup can read the file from stdin which could be a pipe). We never saw this issue with initscripts because we guaranteed a 0 byte offset by extracting the key from the containing file to its own temporary file.

I've put together a patch which reads up to 4k at a time, and it's much faster. I'll submit it upstream after I do the same optimization for reading the actual key.
Comment by Heiko Baums (cyberpatrol) - Friday, 24 August 2012, 23:48 GMT
> bs=$keyfile_size count=1
If this is really faster and working as expected, I don't mind.

Is it really necessary to seek to the keyfile_offset? Is dd doing this the same way? If yes, wouldn't it be better then to seek the same way as reading the keyfile itself, with something like bs=$keyfile-offset-1 count=1?
Comment by Heiko Baums (cyberpatrol) - Saturday, 25 August 2012, 00:13 GMT
Now I've tested it. Reading the keyfile with bs=$keyfile_size count=1 doesn't work. Gives this error message:
No key available with this passphrase.

So bs=1 count=$keyfile-size is necessary.
Comment by Dave Reisner (falconindy) - Saturday, 25 August 2012, 00:15 GMT
It really doesn't matter. dd isn't being used anymore.
Comment by Heiko Baums (cyberpatrol) - Saturday, 25 August 2012, 00:19 GMT
In the old scripts dd is still used. Nevertheless systemd-cryptsetup needs to be at least as fast as dd, if not faster, and systemd-cryptsetup has to give the exact same result (read the exact same key) as dd. So it really does matter.
Comment by Heiko Baums (cyberpatrol) - Saturday, 25 August 2012, 00:41 GMT
Now tried a combination between the old scripts and systemd-cryptsetup as Dave suggested. I read the keyfile with dd and ran systemd-cryptsetup with:
/usr/lib/systemd/systemd-cryptsetup attach "name" "device" "keyfile"

These are the results:
# time uncrypt-file
unlock swap
unlock home

real 0m14.486s
user 0m13.932s
sys 0m0.067s

# time uncrypt-file
unlock swap
unlock home

real 0m15.066s
user 0m14.516s
sys 0m0.057s

So the main problem is how the keyfile is read by systemd-cryptsetup. But unlocking the container by systemd-cryptsetup is also slightly slower, even if it's only 2 or 3 seconds.
Comment by Dave Reisner (falconindy) - Saturday, 25 August 2012, 00:55 GMT
No, the problem is how _libcryptsetup_ reads the keyfile. You'll find that this is just as slow using cryptsetup directly with the --keyfile-offset option.

You're on your own to figure out those "2-3 seconds". I'm only willing to fix the obvious problem here. Based on your constant abuse of Arch developers, I think you should show some gratitude that I'm even going out of my way to do this. It's because of everyone but yourself that this option even exists with systemd, and yet I suspect you're one of the extraordinarily few users of it.

http://code.google.com/p/cryptsetup/issues/detail?id=135
Comment by Heiko Baums (cyberpatrol) - Saturday, 25 August 2012, 01:11 GMT
What "constant abuse" of which Arch developers are you talking about?
Comment by Heiko Baums (cyberpatrol) - Saturday, 25 August 2012, 01:19 GMT
The 2-3 seconds are also part of this bug. And they affect not only this one use case, but every unlocking of encrypted containers. They simply make the boot process of everyone who uses dm-crypt/LUKS slower. And I'm definitely not the only one who uses dm-crypt/LUKS. And I bet you will get other bug reports from other people about that, if this won't be fixed.

And this option (reading a keyfile raw from a device) doesn't exist in the initscripts (/etc/rc.d/functions) just because of me, but because it already existed in the encrypt hook long before and was documented in the wiki. And if you change the initscripts and use different mechanisms and tools, those changes and tools simply must cover every use case which has been covered before. Regressions in such an important part of the system is a no-go, regardless of how many people are using it.

Like I said on the mailing list, even if Tom was ranting about that, we're not talking about some fancy stuff like a GTK theme, we're talking about something very serious.
Comment by Dave Reisner (falconindy) - Monday, 27 August 2012, 12:34 GMT
Fixed upstream:

http://code.google.com/p/cryptsetup/source/detail?r=961682aa6b228

I'll leave it up to the cryptsetup maintainer to figure out if they want to backport this or just wait for the release. Doesn't bother me either way.

Loading...