Arch Linux

The kernel rollback showed the absence of the random string: fast init done at the beginning of the download

from https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.14.36 the following appear the most likely to be related to the change you observed
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=v4.14.36&id=90936d903c2f34663cffe68d9845debdeb85174c
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=v4.14.36&id=d152fcc173149a99d6f707a5b8a80d83d906755b
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=v4.14.36&id=7b6b1f3a192372937164d1293b432c640ffc7c8f

If the system has a TPM is the issue still present in 4.14.37?

I also have this problem, crng init is delayed 45 secs. Still affects 4.14.38.

Please bisect between 4.14.35 and 4.14.36 and report the bad commit upstream.

4.16.5 kernel also have this bug

The problem is not very well resolved:
# pacman -S haveged
# systemctl enable haveged

I had noticed that pressing a few keys to provide some entropy, quickly initialized the random generator.
There are also several messages such as this:
random: systemd: uninitialized urandom read (16 bytes read)
Could it be that the recent kernel changes broke systemd? And why does this only affect a few people?

Once you have located which commit is the cause you could discuss that commit upstream with the kernel and systemd developers.

@hetfig and @AndyRTR do you want to revert
for linux https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=cd8d7a5778a4abf76ee8fe8f1bfcf78976029f8d
for linux-lts https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=6e513bc20ca63f594632eca4e1968791240b8f18
as fedora has done while waiting on upstream to provide a permanent solution?
https://lkml.org/lkml/2018/4/26/103

Only few people seem to be affected. Better use custom builds or stay with older pkg versions until upstream solution is available.

Same here with 4.16.12-1-ARCH

It looks like, that only systems with no TPM are affected:

- Fujitsu PC, Intel Core i3-4150, no TPM built-in ( delayed up to 30s - Kernel 4.17.2 )
- Acer Swift 3, Intel Core i5-8250U, TPM built-in ( no delay during boot - Kernel 4.17.2 )

Regards from Germany

For me and without TPM, only linux-lts is affected. linux-4.17.2 is not affected. Also, a custom 4.14 build with a configuration based on linux-stable, is not affected.
Reverting the patch as suggested, gave me stability problems with suspend/resume.

I'm also affected on 4.17.2.

I'm also affected on two of my machines (on 4.14 and 4.17), none of them has tpm.

Upstream proposed fix for systems that have rdrand https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=81e69df38e2911b642ec121dec319fad2a4782f3

Queued for 4.17.12 https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/queue-4.17/random-mix-rdrand-with-entropy-sent-in-from-userspace.patch?id=377beaad2a08ee1ca50277c290d17efd60a5f5a0
Queued for 4.14.60 https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/queue-4.14/random-mix-rdrand-with-entropy-sent-in-from-userspace.patch?id=b77986f577d6c5bcea7d5d075b626c0752e03ec1

FYI, that doesn't fix the issue at all. It only mixes in rdrand entropy to any entropy provided by userspace. Which is funny when you use rng-tools to work around this: mixing rdrand entropy into rdrand entropy. Why not just give us a kernel command line option to tell the kernel to trust rdrand? Intel has much better options available to fuck us over rather than surreptitiously tamper their hardware rng.

Looks like this is the solution: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9b25436662d5fb4c66eb527ead53cab15f596ee0

Can those affected test linux 4.19.arch1-1 which has CONFIG_RANDOM_TRUST_CPU=y

4.19.arch1-1 works for me. Though, should this really be enabled by default considering a lot of people distrust their CPU vendors?

I'll probably revert the config change so you will have to boot with the parameter.

I haven't yet tried 4.19, however if you do revert the option please keep it as a boot time parameter. Without it, the system appears to hang until I press a few keys and I think this is unacceptable as default behaviour. Since the kernel has several sources of entropy and the lack of trust for the CPU is more of a problem in virtual machines and such, maybe you should consider keeping CONFIG_RANDOM_TRUST_CPU=y as default and letting people change it if they so wish.

"please keep it as a boot time parameter."

This isn't a distro choice. Boot parameter will always exist.

"maybe you should consider keeping CONFIG_RANDOM_TRUST_CPU=y as default and letting people change it if they so wish."

The problem with this is that people who may want to switch it off, won't be aware that such thing exist in the first place as there won't be any visual changes in their system unless you look under the hood. On the other hand in case CONFIG_RANDOM_TRUST_CPU=N people who may want to switch it on, will have to be aware of it otherwise they won't boot their system. CONFIG_RANDOM_TRUST_CPU=N is also the same behaviour as before Linux 4.19 so having to switch something on or use other tools like haveged won't be a regression.

The behaviour introduced in 4.18 was causing problems for some people. I think it was mentioned that Fedora went as far as reversing the patch. We'll have to wait and see how other distros are handling this in 4.19, but I am guessing that whey will default in =Y, because it will cause fewer problem reports for them.
Also, not trusting the CPU and its RNG really only affects the first seconds of booting: after that, network activity, keyboard etc. mix in more entropy. This option was introduced for certain low entropy situations, such as virtual machines. If someone wants to be extra cautious and it doesn't cause problems, then they can enable it (and it would be advertised in wiki/Security), but imho it's not a "must have" feature for everyone.

"Also, not trusting the CPU and its RNG really only affects the first seconds of booting"

That's right but that's the actual concern. CPU RNG was always added to entropy mix AFTER boot and that wasn't controversial at all. CONFIG_RANDOM_TRUST_CPU=Y allows to use CPU RNG as seed for entropy in early boot which theoretically can affect further entropy mix in deterministic way.

"If someone wants to be extra cautious and it doesn't cause problems, then they can enable it"

Reading from context I think you meant "disable" not "enable".

"it's not a "must have" feature for everyone"

I know you mean the opposite but this perfectly fits for: CONFIG_RANDOM_TRUST_CPU=y it's not a "must have" feature for everyone. I bet it's needed for 1% usecases.

https://lkml.org/lkml/2018/7/17/1279

A discussion about the original patch, its intention, pros and cons.

With 4.19 kernels, CONFIG_RANDOM_TRUST_CPU=n doesn't cause too much a delay any more.
[ 4.206676] random: crng init done
Even if it did, the solution is to use random.trust_cpu=on

@dimosd see  FS#61233  for a recent case of the delay being in the tens of seconds range and random.trust_cpu=on having no effect due to the absence of RDRAND and RDSEED haveged did resolve the issue.
Systemd 420 will prefer using the RDRAND processor instruction over /dev/urandom whenever it requires randomness that neither has to be crypto-grade nor should be reproducible.
Possibly close as Not a bug / Won't Fix as this is what upstream has chosen to do?

With systemd 240, linux 4.19.14, cpu supports RDRAND: I get delays of 4-24 secs and no delay if I use random.trust_cpu=on
If the cpu didn't support RDRAND then I would have to use haveged.
These are the two known workarounds and I don't think there's anything more we can do for now.

https://www.phoronix.com/scan.php?page=news_item&px=Systemd-RdRand-Direct also mentions a "high_quality_required" systemd option.

With recent systemd 243.0 release the issue gets worse. LTS-kernel hangs on both of my systems now.

I guess this is related to this bug here and can be resolved updating haveged and fixing its service file.
https://github.com/systemd/systemd/issues/13252 and

@eworm - can you please have a look?

@mtorromeo I think the rngd.service unit is also lacking. It specifies WantedBy=sysinit.target but that doesn't do what you want without a few more directives:

DefaultDependencies=no
Before=sysinit.target shutdown.target

(Technically also Conflicts=shutdown.target but I'm not sure shutting down the entropy gatherer while systemd still needs bits is a good idea)

Fixed for me with haveged 1.9.6-1 using 4.19 lts kernel. Please test it.

Published rng-tools-6.7-2 with the changes to dependency resolution.

Is this still an issue? (Works for me.)

5.4 is adding https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=50ee7529ec4500c88f8664560770a7a1b65db72b which should reduce the issue.
It is still reported as an issue on the forums for new installs with GDM without rotational media or requiring keyboard input before GDM starts.
Edit:
https://lore.kernel.org/lkml/alpine.DEB.2.21.1909290010500.2636%40nanos.tec.linutronix.de/

I still have problems with it so I decided to enable random.trust_cpu=on in boot options. I would like it to point out that
- haveged is not necessarily a good alternative to RDRAND because it has the same problem: it cannot be verified
- The following distributions have now enabled CONFIG_RANDOM_TRUST_CPU: Debian, Ubuntu, Fedora, Alpine. In particular Ubuntu and Alpine had initially disabled it but later reverted it because of bug reports.
Any decision on the default would be a compromise, CONFIG_RANDOM_TRUST=y deals with the known while =n deals with the unknown.

From a technical reasoning standpoint its totally not important what other distros do, other distros also heavily patch their software, so we do that as well?
Back to technical reasoning:
Right now its an opt-in, as it should be. People who really need to trust the cpu on RNG because they are affected can opt-in to do so, while not exposing the majority of people who are not affected. The only thing that really is unknown is random.trust_cpu itself, i fail to see where all the blind trust comes from. Hardware vendors mostly choose performance over security to compete on the market. With all the lately discovers of spectre/meltdown/L1TF/MDS/TAA/iTLB i fail to see why we want to blindly trust on rng on a global scale instead of opt-in who _really_ needs it. The CPU rngs are closed spec knowledge, not audit-able in the classical sense, non-blocking infinite source of numbers and are purely considered safe from the vendors themselves because "its surely too complicated for anyone to ever understand what influences it so it must be safe".

The current setting of disabling it by default is the only sane option to do for a user base that is considered to be technically competent to enable it themselves if they really are affected and need to plus decided themselves its fine for them to trust it. It could surely be documented better in the wiki (contributions welcome) but the setting shall not be changed.

The good thing about Arch Linux is that it gives users a lot of flexibility to make their own decisions, in case they disagree with the decisions made by the distribution for them. But sane defaults are still important.
>From a technical reasoning standpoint its totally not important what other distros do
I very much disagree here. Arch Linux maintainers are highly capable, but so are the maintainers of other respected distributions. There have been discussions elsewhere on the subject and the current consensus is that disabling CONFIG_RANDOM_TRUST_CPU causes harm, most often in low entropy situations such as virtual machines but in real hardware as well.
For me the sane option would be to ship bug-free. An extra security conscious user should disable RDRAND (if possible) and also disable hyper-threading for instance, but we don't do it by default because it causes a large performance penalty. There has been no demonstration of exploiting RDRAND to my knowledge, unlike HT.

security is not something that comes by opt-outing of thousands of switches. security must come by default and allow users who do not want something because of some reason to opt-out that's literally the only sane approach to security.
Users have the flexibility to turn it on, like you did. no insecure default needed no matter what you claim. Again: Why on earth should something be considered a good source of entropy for early boot (which fundamentally is implicitly how secure KASLR will be) if its purely based on closed-spec and by its creators only deemed secure because "its surely too complicated for anyone to ever understand what influences it so it must be safe".
The good thing about our user base is that we can expect competence, we neither enable, start or restart systemd units and we expect users to configure their systems how they like it. we are neither debian nor ubuntu. You are mixing my statement, its not about the package maintainers of the distros its about the expectations related to the user base, which frankly is fundamentally different to distros like ubuntu and debian.

Fine. I understand your concerns about security and I am not suggesting that we should be sloppy. However, I have been banging on the keyboard to get the system to boot for the past few months, exactly to avoid turning this option on, until it occurred to me that the situation is slightly ridiculous.

That's why i suggested we improve the documentation in the wiki and make this use-case easier to spot and address with a small sub section of the implications just to raise tiny awareness of what people may turn on :)

> >From a technical reasoning standpoint its totally not important what other distros do

> I very much disagree here. Arch Linux maintainers are highly capable, but so are the maintainers of other respected distributions. There have been discussions elsewhere on the subject and the current consensus is that disabling CONFIG_RANDOM_TRUST_CPU causes harm, most often in low entropy situations such as virtual machines but in real hardware as well.

Then do not defend your stance by saying "but some other distros did it". Defend your stance by saying "the well-respected maintainers of X distro had the following observation to make on the pros/cons of it, and I agree with their analysis".

Levente is saying, let's focus on merit-based arguments rather than simply blindly trusting another distro's judgment calls. Based purely on this argument, I have no idea why those distros made the decision they did.

Given one of the example distros is Ubuntu, it is plausible to me that the rationale was "users know nothing about computers and most of them don't have anything secure, but the distro comes preinstalled with gdm so we should optimize for this use case".

This fails to apply to archlinux for a whole bunch of reasons, including the fact that gdm is not preinstalled and the archlinux user base is explicitly targeted at people who tend to have biased ideas like "gnome is evil and DEs in general sort of suck, let me use this niche tiling WM that really enhances my personal use". I somehow doubt the i3 and sway users use gdm!

The logical conclusion here is that *iff* the tradeoff by Ubuntu and others was "our gdm users are sufficiently problematic that we're willing to make security sacrifices on their behalf", we should be definitively doing the opposite.

So: what technical arguments did these other distros use? Maybe something from them, applies to us as well.

What I am trying to do is summarize 1.5 year's experience of others on the subject. It's not a black and white issue, because it's a policy rather than pure technical decision. It introduces bugs, therefore it has a cost.
Some use cases that may trigger the bug:
- Encrypted swap (read from /dev/urandom, block booting early)
- Losing connection to a remote server after reboot for some time (booting blocks before the network is initialized)
- Initializing encrypted LVM on a low entropy system leads to reduced security

Some comments:
https://lists.debian.org/debian-devel/2018/12/msg00204.html (the whole thread brings several pros-and-cons)
https://gitlab.alpinelinux.org/alpine/aports/issues/9960

The above are mostly server oriented and thus security minded distros with a technical user base.

Please don't derail this issue; it's not just a problem for GDM.

As far as my opinion goes, I would enable RANDOM_TRUST_CPU. Stalling boots are quite painful (especially if it's not obvious why) and I think we're better served with the smoother experience than satisfying our paranoia about attacks on the RNG that haven't been demonstrated (especially if it would have to be pre-boot or early-boot).

But I also agree that our users should be competent enough to discover "if boot is stalling, try random.trust_cpu=1".

What about enabling it after the active entropy generation lands in our kernel? Theodore Ts'o (who added the above config) thinks the HWRNG is trustworthier than the jitter entropy and I'm inclined to agree with him.

No attacks? Well for the bugs already have been appeared no "attack" was needed, the rng always constantly returns the same value and even systemd needed to work around it:
https://github.com/systemd/systemd/commit/b62bc66018fa1ada09554e7ee46abbbfc8e6b3ad
And yet another set of kernel workaround patches to handle this borked hardware/firmware combination: https://lore.kernel.org/patchwork/patch/1115413/.
And yes, this is a CVE worthy hardware issue with the RNG, so please stop dragging security discussions always down to "paranoia", you can leave that part out and still be technically reasonable in your arguments

The AMD RDRAND bugs are well known and fairly recent, but they are not the reason the kernel stopped trusting RDRAND. And I am not saying it should provided there was an alternative, but there isn't in all cases (yet?). So disabling RDRAND will cause breakage for some for the time being. This is why I think this should be a "hardening" option rather than the default. Besides it should be mentioned that even when RDRAND is enabled it is mixed with other (lower quality) sources of entropy.

@dimosd: You are repeating yourself so we are doing a full round now. sane hardening must come per default that's how security must work. Lets not repeat the same arguments over and over again, heftig provided new ones and i gonna think about it, but we are wasting everyone's time if we do full circles all over again.
PS: Read the kernel docs, the kernel itself neither blindly trusts nor blindly mistrusts it, its a matter of downstream choice
PPS: an alternative is using the TPMs RNG in case you have a TPM, if someone decides to trust that more than the CPU, but that requires defining the trust in terms of rng_core.default_quality
PPPS: I wouldn't call a bug known since 2014 fairly recent: https://bugzilla.kernel.org/show_bug.cgi?id=85911

Not sure if those uninitialized urandom reads are safe.

I'm for staying on the safe side. The workaround to add random.trust_cpu=1 to the kernel boot prompt is well documented.

Maybe someone is willing to write some note to our wiki so it can be easily found how to solve.
Best place should be either https://wiki.archlinux.org/index.php/Arch_boot_process or https://wiki.archlinux.org/index.php/Random_number_generation
maybe with links to each other.

Then we should close this issue.