Arch Linux

Running pacstrap too quickly after booting archiso fails and corrupts keyring due to pacman-init.service / 'pacman-key --populate' still running. No way to recover other than reboot archiso and re-run since rootfs keyring is now corrupt.

Fails with other "unable to import key" (if run with -G) or with permission errors for keyring.

Additional info:
* archlinux-2022.09.03-x86_64.iso
* takes about a minute for pacman-key --populate (pacman-init.service) to finish after boot (on KVM on Intel N6005)
* even if pacstrap -G is used, it fails trying to import keys from host and pacman-init.service fails and corrupts database
* easy to reproduce if running pacstrap from cloud init script quickly after boot
* also possible to reproduce manually (partition and pacstrap within a minute is not unreasonable with experience), especially on slower hardware or a small VM

Steps to reproduce:
* boot archlinux-2022.09.03-x86_64.iso on slow/modest hardware
* login
* mkdir /tmp/1
* pacstrap /tmp/1 base (as quickly as possible, while pacman-key --populate -> gpg is still running)

Possible solution
* pacman-key + pacstrap could flock on /etc/pacman.d

Output not always consistent, but here are examples:

pacstrap -G from cloud-init:

[ 52.056675] cloud-init[667]: :: Import PGP key C06086337C50773E, "Jelle van der Waa <jelle@archlinux.org>"? [Y/n]
[ 52.555798] cloud-init[667]: :: Import PGP key 4A1AFC345EBE18F8, "Sébastien Luttringer <seblu@seblu.net>"? [Y/n]
[ 52.711634] cloud-init[667]: :: Import PGP key 9D4C5AA15426DA0A, "Frederik Schwan <freswa@archlinux.org>"? [Y/n]
[ 52.862281] cloud-init[667]: :: Import PGP key 94657AB20F2A092B, "Andreas Radke <andyrtr@archlinux.org>"? [Y/n]
[ 53.021358] cloud-init[667]: :: Import PGP key 7258734B41C31549, "David Runge <dvzrv@archlinux.org>"? [Y/n]
[ 53.196925] cloud-init[667]: :: Import PGP key 786C63F330D7CB92, "Felix Yan <felixonmars@archlinux.org>"? [Y/n]
[ 53.352372] cloud-init[667]: :: Import PGP key 7A4E76095D8A52E4, "Antonio Rojas <arojas@archlinux.org>"? [Y/n]
[ 53.506727] cloud-init[667]: :: Import PGP key 771DF6627EDF681F, "Tobias Powalowski <tpowa@archlinux.org>"? [Y/n]
[ 53.669730] cloud-init[667]: :: Import PGP key 3B94A80E50A477C7, "Jan Alexander Steffens (heftig) <heftig@archlinux.org>"? [Y/n]
[ 53.833889] cloud-init[667]: :: Import PGP key FC1B547C8D8172C8, "Levente Polyak <anthraxx@archlinux.org>"? [Y/n]
[ 53.850922] cloud-init[667]: :: Import PGP key 139B09DA5BF0D338, "David Runge <dvzrv@archlinux.org>"? [Y/n]
[ 54.009229] cloud-init[667]: :: Import PGP key 51E8B148A9999C34, "Evangelos Foutras <foutrelis@archlinux.org>"? [Y/n]
[ 54.314243] cloud-init[667]: :: Import PGP key 7F2D434B9741E8AC, "Pierre Schmitz <pierre@archlinux.de>"? [Y/n]
[ 54.335775] cloud-init[667]: :: Import PGP key E5BB298470AD4E41, "Sébastien Luttringer <seblu@seblu.net>"? [Y/n]
[ 54.499028] cloud-init[667]: :: Import PGP key F99FFE0FEAE999BD, "Allan McRae <allan@archlinux.org>"? [Y/n]
[ 54.676846] cloud-init[667]: :: Import PGP key 686B063AC4BC0EC9, "Jonas Witschel <diabonas@archlinux.org>"? [Y/n]
[ 55.768419] cloud-init[667]: :: Import PGP key 6D1655C14CE1C13E, "Florian Pritz <bluewind@xinu.at>"? [Y/n]
[ 55.939175] cloud-init[667]: :: Import PGP key F22FB1D78A77AEAB, "Giancarlo Razzolini <grazzolini@archlinux.org>"? [Y/n]
[ 55.940452] cloud-init[667]: checking package integrity...
[ 60.370392] cloud-init[667]: error: iana-etc: signature from "Jelle van der Waa <jelle@archlinux.org>" is unknown trust
[ 60.370514] cloud-init[667]: :: File /mnt/var/cache/pacman/pkg/iana-etc-20220715-1-any.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
[ 60.370882] cloud-init[667]: Do you want to delete it? [Y/n] error: filesystem: signature from "Sébastien Luttringer <seblu@seblu.net>" is unknown trust
[ 60.371081] cloud-init[667]: :: File /mnt/var/cache/pacman/pkg/filesystem-2021.12.07-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
[ 60.371326] cloud-init[667]: Do you want to delete it? [Y/n] error: linux-api-headers: signature from "Frederik Schwan <freswa@archlinux.org>" is unknown trust
[ 60.371560] cloud-init[667]: :: File /mnt/var/cache/pacman/pkg/linux-api-headers-5.18.15-1-any.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
[ 60.377834] cloud-init[667]: Do you want to delete it? [Y/n] error: tzdata: signature from "Andreas Radke <andyrtr@archlinux.org>" is unknown trust
[ 60.377929] cloud-init[667]: :: File /mnt/var/cache/pacman/pkg/tzdata-2022c-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
[ 60.380334] cloud-init[667]: Do you want to delete it? [Y/n] error: glibc: signature from "Frederik Schwan <freswa@archlinux.org>" is unknown trust
[ 60.380411] cloud-init[667]: :: File /mnt/var/cache/pacman/pkg/glibc-2.36-3-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
[ 60.383979] cloud-init[667]: Do you want to delete it? [Y/n] error: gcc-libs: signature from "Frederik Schwan <freswa@archlinux.org>" is unknown trust
...


pacstrap (no -G), from cloud-init:

[ 65.875665] cloud-init[659]: :: Import PGP key C06086337C50773E, "Jelle van der Waa <jelle@archlinux.org>"? [Y/n] error: key "Jelle van der Waa <jelle@vdwaa.nl>" could not be imported
[ 66.710534] cloud-init[659]: :: Import PGP key 4A1AFC345EBE18F8, "Sébastien Luttringer <seblu@seblu.net>"? [Y/n] error: key "Sébastien Luttringer <seblu@seblu.net>" could not be imported
[ 67.305448] cloud-init[659]: :: Import PGP key 94657AB20F2A092B, "Andreas Radke <andyrtr@archlinux.org>"? [Y/n] error: key "Andreas Radke <andyrtr@archlinux.org>" could not be imported
[ 67.896757] cloud-init[659]: :: Import PGP key 7258734B41C31549, "David Runge <dvzrv@archlinux.org>"? [Y/n] error: key "David Runge <dvzrv@archlinux.org>" could not be imported
[ 69.532329] cloud-init[659]: :: Import PGP key 786C63F330D7CB92, "Felix Yan <felixonmars@archlinux.org>"? [Y/n] error: key "Felix Yan <felixonmars@archlinux.org>" could not be imported
[ 70.142665] cloud-init[659]: :: Import PGP key 7A4E76095D8A52E4, "Antonio Rojas <arojas@archlinux.org>"? [Y/n] error: key "Antonio Rojas <arojas@archlinux.org>" could not be imported
[ 70.721142] cloud-init[659]: :: Import PGP key 771DF6627EDF681F, "Tobias Powalowski <tpowa@archlinux.org>"? [Y/n] error: key "Tobias Powalowski <tobias.powalowski@googlemail.com>" could not be imported
[ 71.338407] cloud-init[659]: :: Import PGP key 3B94A80E50A477C7, "Jan Alexander Steffens (heftig) <heftig@archlinux.org>"? [Y/n] error: key "Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>" could not be imported
[ 72.130304] cloud-init[659]: :: Import PGP key FC1B547C8D8172C8, "Levente Polyak <anthraxx@archlinux.org>"? [Y/n] error: key "Levente Polyak (anthraxx) <levente@leventepolyak.net>" could not be imported
[ 72.710881] cloud-init[659]: :: Import PGP key 139B09DA5BF0D338, "David Runge <dvzrv@archlinux.org>"? [Y/n] error: key "David Runge <dvzrv@archlinux.org>" could not be imported
[ 73.341467] cloud-init[659]: :: Import PGP key 51E8B148A9999C34, "Evangelos Foutras <foutrelis@archlinux.org>"? [Y/n] error: key "Evangelos Foutras <evangelos@foutrelis.com>" could not be imported
[ 74.087638] cloud-init[659]: :: Import PGP key 7F2D434B9741E8AC, "Pierre Schmitz <pierre@archlinux.de>"? [Y/n] error: key "Pierre Schmitz <pierre@archlinux.de>" could not be imported
[ 74.884594] cloud-init[659]: :: Import PGP key E5BB298470AD4E41, "Sébastien Luttringer <seblu@seblu.net>"? [Y/n] error: key "Sébastien Luttringer <seblu@seblu.net>" could not be imported
[ 75.474338] cloud-init[659]: :: Import PGP key F99FFE0FEAE999BD, "Allan McRae <allan@archlinux.org>"? [Y/n] error: key "Allan McRae <me@allanmcrae.com>" could not be imported
[ 76.489611] cloud-init[659]: :: Import PGP key 686B063AC4BC0EC9, "Jonas Witschel <diabonas@archlinux.org>"? [Y/n] error: key "Jonas Witschel <jonas.witschel@diabonas.de>" could not be imported
[ 77.221496] cloud-init[659]: :: Import PGP key 6D1655C14CE1C13E, "Florian Pritz <bluewind@xinu.at>"? [Y/n] error: key "Florian Pritz <bluewind@xinu.at>" could not be imported
[ 77.802193] cloud-init[659]: :: Import PGP key F22FB1D78A77AEAB, "Giancarlo Razzolini <grazzolini@archlinux.org>"? [Y/n] error: key "Giancarlo Razzolini (grazzolini) <grazzolini@archlinux.org>" could not be imported
[ 77.804147] cloud-init[659]: error: required key missing from keyring
[ 77.804835] cloud-init[659]: error: failed to commit transaction (unexpected error)
[ 77.854866] cloud-init[659]: Errors occurred, no packages were upgraded.

journalctl -u pacman-init.service:

Sep 09 10:59:51 archiso pacman-key[940]: gpg: [don't know]: invalid packet (ctb=00)
Sep 09 10:59:51 archiso pacman-key[940]: gpg: keydb_search failed: Invalid packet
Sep 09 10:59:51 archiso pacman-key[940]: gpg: [don't know]: invalid packet (ctb=00)
Sep 09 10:59:51 archiso pacman-key[940]: gpg: keydb_search failed: Invalid packet
Sep 09 10:59:51 archiso pacman-key[940]: gpg: [don't know]: invalid packet (ctb=00)
Sep 09 10:59:51 archiso pacman-key[940]: gpg: keyring_get_keyblock: read error: Invalid packet
Sep 09 10:59:51 archiso pacman-key[940]: gpg: keyring_get_keyblock failed: Invalid keyring
Sep 09 10:59:51 archiso pacman-key[940]: gpg: failed to rebuild keyring cache: Invalid keyring
Sep 09 10:59:51 archiso pacman-key[940]: gpg: marginals needed: 3 completes needed: 1 trust model: pgp
Sep 09 10:59:51 archiso pacman-key[940]: gpg: [don't know]: invalid packet (ctb=00)
Sep 09 10:59:51 archiso pacman-key[940]: gpg: keyring_get_keyblock: read error: Invalid packet
Sep 09 10:59:51 archiso pacman-key[940]: gpg: keydb_get_keyblock failed: Invalid keyring
Sep 09 10:59:51 archiso pacman-key[940]: gpg: validate_key_list failed
Sep 09 10:59:51 archiso pacman-key[333]: ==> ERROR: Trust database could not be updated.
Sep 09 10:59:51 archiso systemd[1]: pacman-init.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 10:59:51 archiso systemd[1]: pacman-init.service: Failed with result 'exit-code'.
Sep 09 10:59:51 archiso systemd[1]: Failed to start Initializes Pacman keyring.