FS#78892 - [linux-firmware] Please switch to zstd compressed firmware
Attached to Project:
Arch Linux
Opened by Emil (xexaxo) - Monday, 26 June 2023, 12:17 GMT
Last edited by Laurent Carlier (lordheavy) - Sunday, 09 July 2023, 09:56 GMT
Opened by Emil (xexaxo) - Monday, 26 June 2023, 12:17 GMT
Last edited by Laurent Carlier (lordheavy) - Sunday, 09 July 2023, 09:56 GMT
|
Details
Description:
With v20230625 my firmware compression patches have landed. Now we have upstream support for xz and zstd compressed firmware and can drop our local patch. The kernel had support for zstd compressed firmware since 5.19 and all the kernels supported in Arch are much newer. In terms of numbers zstd default vs xz - size has slightly gone slightly up - 408 vs 358 - compression speed (in case we care) - 27.27s user 7.31s system 104% cpu 33.218 total, vs 214.19s user 26.09s system 99% cpu 4:00.43 total - decompression speed - should be much faster, but didn't measure it If the increased size is a concern, we can use ZSTD_CLEVEL=19. Then the numbers change to: - size is down to 378M - compression speed is up (but still faster than xz) to 166.09s user 14.83s system 100% cpu 3:00.03 total Note: the env. variable does not support ultra levels & I'm not sure if the kernel supports those. Additional info: * package version(s) 20230404.2e92a49f-1 Steps to reproduce: - update the PKGBUILD to the v20230625 release or later, drop out xz patch - update the PKGBUILD to switch to zst compression - mkinitcpio -P - enjoy the faster boot times - lolz much speed, in practise a few us |
This task depends upon
Closed by Laurent Carlier (lordheavy)
Sunday, 09 July 2023, 09:56 GMT
Reason for closing: Implemented
Additional comments about closing: linux-firmware-20230625.ee91452d-4
Sunday, 09 July 2023, 09:56 GMT
Reason for closing: Implemented
Additional comments about closing: linux-firmware-20230625.ee91452d-4
Rolling back to -1 fixes the issue.
```
amdgpu 0000:04:00.0: amdgpu: Fetched VBIOS from VFCT
amdgpu: ATOM BIOS: 113-REMBRANDT-X37
amdgpu 0000:04:00.0: Direct firmware load for amdgpu/yellow_carp_toc.bin failed with error -2
[drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <psp> failed -19
amdgpu 0000:04:00.0: Direct firmware load for amdgpu/yellow_carp_dmcub.bin failed with error -2
[drm:dm_early_init [amdgpu]] *ERROR* DMUB firmware loading failed: -19
[drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <dm> failed -19
amdgpu 0000:04:00.0: Direct firmware load for amdgpu/yellow_carp_pfp.bin failed with error -2
[drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <gfx_v10_0> failed -19
[drm] VCN(0) decode is enabled in VM mode
[drm] VCN(0) encode is enabled in VM mode
amdgpu 0000:04:00.0: Direct firmware load for amdgpu/yellow_carp_vcn.bin failed with error -2
[drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <vcn_v3_0> failed -19
[drm] JPEG decode is enabled in VM mode
amdgpu 0000:04:00.0: amdgpu: Fatal error during GPU init
amdgpu 0000:04:00.0: amdgpu: amdgpu: finishing device.
```
https://github.com/dracutdevs/dracut/commit/9d8387ed803dfc3e8b97d2e415a15083774d7ac6
Edit:
Should also note booster appears to only support xz compression https://github.com/anatol/booster/blob/0.10/generator/generator.go#L309
The kernel can directly decompress zstd firmware since v5.19, as such mkinitcpio/dracut/booster do not need to decompress it. At a glance neither of them do, they just need suffix every `modinfo -F firmware amdgpu` entry with .zst when adding into the initrd. There is one caveat - the amd cpu microcode cannot be decompressed early (most likely a kernel bug, but it can be when loaded "late". As such a) those are not compressed and b) at least for mkinitcpio, we use separate "early" initrd image.
Looking at the `failed with error -2` messages, those are "file is missing". Although if you can share more details (below) that would be appreciated:
- kernel package and version (FS78939 shows 6.4.0-stable-1 which is not an Arch pkg IIRC)
- initrd tool package and version - mkinitcpio, dracut, booster, other
- are you seeing other non "failed with error -2" firmware messages - please attach full journal/dmesg
- drop drm/amdgpu from the initrd, does things work - for mkinitcpio omit the kms hook, for other see their documentation
- usually you don't need drm modules in early boot - the UEFI splash is displayed, until X/Wayland session kicks in
Thanks in advance
https://github.com/dracutdevs/dracut/issues/1850#issuecomment-1364554967
https://github.com/dracutdevs/dracut/issues/1850#issuecomment-1364692255
it apparently is not the key that Arch expects.
https://github.com/dracutdevs/dracut/issues/1850#issuecomment-1428280936
Arch is waiting for a closed bug to be dealt with - which in my experience won't happen because it is closed.
https://github.com/dracutdevs/dracut/pull/2101
That said, Laurent has already backported the patch to dracut 056-3. Thank you
For booster - I've opened upstream MR and an Arch tracking bug https://bugs.archlinux.org/task/78947. Please give it a test and comment.
To run give it a list of files or a directory.
The tradeoff, as usual, is disk space vs speed. If compression is significant enough, then reading compressed files and decompressing (in memory) can even be faster than reading uncompressed file. The latter is the win/win scenario. I did not see this latter case in my tests.
The numbers speak for themselves. It seems clear that zst is a win for both kernel modules and firmware. When there are a few files, like firmware, then the size benefit seems to outweigh the time penalty for xz as well. When loading many files zst still looks decent. We're already using zst for kernel modules.
Here's some results from the test program to compare time to read or read plus decompress when required:
Test1: 1
--------
single file - iwlwifi-ty-a0-gf-a0-81.ucode
This is only 1 file and 1 run so numbers less precise of course:
Summary
no : 0.001 secs 1.55 MB
xz : 0.037 secs 522.74 KB
zst : 0.002 secs 632.96 KB
Test 2:
--------
Copy all files from /usr/lib/firmware and repeat test on all files (I have about 250 files)
Summary
no : 0.050 secs 324.46 MB
xz : 3.471 secs 50.13 MB
zst : 0.187 secs 60.00 MB
As an executive summary - at the highest levels zstd compressed files are within 1-2% margin of xz compressed ones. While being 4x-18x times faster to decompress.
Some notable caveats, that not many people are aware of:
- the decompression happens in the kernel - plus I'm involved in some WIP work, to also handle the modules
- kernel supports up-to 19, no ultra levels - size is 5-7% larger than xz, decompression speed is unchanged for all levels, decompression memory is identical as xz
- the in-kernel decompression is not identical to the userspace ones - although userspace numbers are close enough representation
- drivers issue multiple _sequential_ _blocking_ firmware requests, directly affecting the driver load and ultimately boot times - amdgpu, nouveau and intel GPU drivers are the most prominent examples
[1] https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html
I have to revert to "linux-firmware-20230404.2e92a49f-1-any.pkg.tar.zst" to get X loading.
I'm on EndeavourOS with kernel 6.4.1-arch1-1 (but 6.3.9 has the same problem).
- intel cpu and gpu (i.e. not using amdgpu) - meaning the revert from git head doesn't impact me
- dracut 059 (thanks for getting this one out @grazzolini :) ) t
The zst compressed firmware are working fine.
Added ack to testing signoffs.
Thanks!