FS#61609 - Pacman should upgrade archlinux-keyring first
|
Details
Hi, I run into a problem with "pacman -Syu" that appears to
be well-known:
`Signature from "User <email@example.org>" is unknown trust` A friend tipped me with "pacman -Sy archlinux-keyring". Later on, I found the solution in pacman's Wiki, also. I understand how the upgrade failed without archlinux-keyring being upgraded. Specifically, I understand that, after a simple "pacman -Syu" command, archlinux-keyring was among the packages about to be upgraded; but, without the keyring being up-to-date first, the whole upgrade of the whole bunch failed -- including the upgrade of archlinux-keyring itself. This should not have been a Catch-22 issue. May I please suggest a two-tiered upgrade, sending in first the packages that are prerequisite to a healthy overall upgrade? At the very least, if the upgrade fails because of this "unknown trust" issue, I would greatly appreciate being informed, by the tool, about what to do next. |
This task depends upon
I would like to point out that I did not "fail to keep" the system up to date. I did not boot the Arch Linux machine in a long time, that is correct. Maybe because I was in a different country for a while, or on a different continent, or in a hospital, or maybe just stayed at home and was over-my-head busy with real work and not much time for hobbies.
I was responsible enough to "pacman -Syu" first thing after I booted the machine. Let's stick to that.
You may want to fix it, or you may not, that is your choice. As for myself, I was also responsible to give you this insight, take the time to file this bug report, although my problem has already been solved. If I use other systems (including other Linux systems) in my mission-critical work, I hope I'm excused. If I'm not used to seeing tools (especially core system tools) failing on me for the wrong reasons, I hope I'm excused for that, also. I still maintain that a two-tier installation procedure is a no-brainer, and that real-world issues happen to real-world users. Software should not be brittle on purpose.
Remember, Pacman is not Arch specific, so not much will happen here. The only way this could work is with the old SyncFirst, which was removed and has been stated MANY times that it will not return.
You have chosen to use a distro that presupposes the user has time for regularly scheduled maintenance, and failed to provide that regularly scheduled maintenance.
Despite this, your system is not broken -- it merely requires that you follow some simple instructions documented on the wiki in order to recover. Or, quite simply, refresh your GnuPG keyring using pacman-key --refresh-keys. Actually, the archlinux-keyring case is by far the absolute easiest to solve -- it is much much harder to resolve cases where packages first drop reverse dependencies, then get replaced by something else, as that gets you into a situation where pacman cannot even calculate a valid dependency tree regardless of signature trust, and the only solution is to know the complete state of the repository every step of the way for the last however many months you didn't upgrade. The other solution to this is to use the Arch Linux Archive to upgrade the system in stages, one month of updates at a time. It's not fun to do, by the way.
...
Just to be perfectly clear: there are *no* packages that are "prerequisite to a healthy overall upgrade". All packages are equally important -- any package can depend on any other package that provides some shared library dependency with useful functionality (though the [core] repository is standalone for booting, so hey, there's only potentially 233 packages that need to be updated in utter lockstep), we don't version dependencies due to our policy that we refuse to support partial upgrades, and even with the SyncFirst keyword we used to have, the only thing which is actually safe to update on its own is the keyring, because the keyring is simply an offline cache of some GnuPG data (which anyways you can create entirely on your own using --refresh-keys and pacman will even download it for you the first time, as long as you have the six master keys to verify the rest of the distro).
As for having the pacman tell the user what to do to resolve the issue, how is pacman supposed to know that? Pacman has no way of associating that signature with a package or even knowing if it's provided by a package at all.
FYI: I'm brand new to Arch Linux, accustomed to other Linux distributions, and just joined this community earlier today. I did not read past announcements, I do not know what SyncFirst, and no matter how "MANY" times it had been stated that it would not return, I would not have known.
I am just learning about SyncFirst from your follow-up explanations, and I will read more about it, later. But from what I've understood so far, I don't think this is what I originally had in mind.
Importantly, I did not suggest changing the dependency handling. Let us put SyncFirst behind us.
What I did suggest is a two-stage update. Two full stages. As if somebody would run `pacman -Sy archlinux-keyring` first (if it's on the list), and then `pacman -Syu` afterwards. As if it were driven from an external shell program (but still driven from the inside).
Consider, the example, Linux Mint. I do not have intimate knowledge of their internals, but from my end-user P.O.V., I can see the Mint Update program updating itself from time to time, as needed. If Mint Update needs updating, it updates itself first, and it updates itself fully. The same thing happens if any Mint mirror repositories get changed. All of the other updates are coming after that. (I can actually see Mint Update restarting itself. Aside from this visual glitch, which doesn't really bother me, it works great.)
Coming back to pacman, the simplest implementation that I'm envisioning is breaking the list of candidate packages in two smaller lists. In the first list, there should only be essential components, like archlinux-keyring, and (perhaps) pacman itself, and not much else. Most of the times, the first list is expected to be empty. And for the second list, everything else is in.
Pacman itself can be 100% agnostic to that. The actual distribution can pick and choose what to send to pacman in the 1st list, and what to send in the 2nd list. What pacman can do is guarantee that no item in the 2nd list will be installed, until all of the 1st list is fully installed.
The ongoing distro maintenance efforts should be small (notwithstanding this architectural change, of course). The benefits for end users are (in my opinion) big, with the tool doing the right job in the right way, and without any fuss.
"pacman -Sy <list-of-packages>"
should work as if
"pacman -Sy archlinux-keyring && pacman -Sy <list-of-packages-except-archlinux-keyring>"
or,
"pacman -Syu"
should work as if
"pacman -Sy archlinux-keyring && pacman -Syu"
This ^^
Can I write a shell script for myself? Of course I can. But pacman should be a bit smarter than that, and it should know what the user probably doesn't. It should know what ought to be in the first list of packages, and what ought to be in the second list of packages. It should know when the first list of packages is empty, also. Everything that pacman needs to know regarding these two distinct (and disjoint) lists, can and should be given by the Linux distribution.
For some technical context, you might want to look at the original proposal and patch to remove it: https://lists.archlinux.org/pipermail/pacman-dev/2012-February/015123.html
If you look at the code you'll also see how it was implemented.
...
Ultimately, there doesn't seem to be any way to handle this in a generic manner, the only safe updates are in very limited cases -- basically just the keyring, since we know the keyring is safe to update at any time but don't know about anything else. Bringing back SyncFirst but hardcoding the package list which is considered critical, could partially alleviate these concerns -- but that's both inelegant and hard to track.
It's been implemented, though, in MSYS2: https://github.com/Alexpux/MSYS2-pacman/pull/26/commits/6e0ff8324dca152b312a2d39ab025f9888896b9c
They hardcode a list of seven packages into the binary, which are required to fully bring up the MSYS2 runtime shell on Windows. We have a lot more packages required to bring up a bootable Linux OS, and those dependencies are intertwined with the rest of the system which means if you update curl and pacman, you break half the userland that needs the old curl. Just see the recent issues people had with partially updating libidn2, for another example.
Back in Ye Olden Daye (2012) we had SyncFirst, and we used it, and what we used it for was pacman -- to guarantee that the updater itself was updated in order to handle more cases better. As said, it wasn't working out very well -- and of course not, since it was a binary package linking to shared libraries that could break the system via partial upgrades, which was exactly what it tried to do, by updating itself and its own direct dependencies in a first run. We don't really have issues where that is a concern anymore, to my knowledge, the last such case being the introduction of hooks where we gave everyone 2 months notice to upgrade before making use of the new features: https://www.archlinux.org/news/required-update-to-pacman-501-before-2016-04-23/
Should we cherry-pick the keyring and only try to protect people against keyring issues? Keyring issues aren't even common -- as a general rule, keys tend to be good for quite some time, and when new keys cycle into the system, pacman already knows how to download those for you without fuss or interruption.
[1] assuming the gpg infrastructure works...
The problem is this;
when I update after coming back from holidays (we have a month here), I get questions about importing a PGP key. The ones with an email address (the primary ones) are fine. Stuff works.
But sometimes an pacman -Su leads to something like this;
```
error: libinih: key "95220BE99CE6FF778AE0DC670F65C7D881506130" is unknown
:: Import PGP key 95220BE99CE6FF778AE0DC670F65C7D881506130? [Y/n] n
:: File /var/cache/pacman/pkg/libinih-55-2-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] ^C
```
Now, if you look at https://archlinux.org/master-keys/, you won't find this key. So what gives?
Turns out, its a subkey of Maxime Gauduin, which has as the main key: 0x81506130
Pacman seems to decide it needs that key from a signature it found on a package. Pacman doesn't know the main pgp identity and experience shows you can end up with a rather bad experience updating after that (as other posters showed).
So, the bugreport I'd word like this;
Pacman has a feature to import into the keyring a gpg key when it encounters it on a package. This key is not always the main key, does not have an email address attached and as such this feature is buggy because importing it doesn't always leave the keyring in a good state. Currently we make the user fix up after pacman. But this is not a user bug, this is a pacman bug, it should not import a gpg key which is not the primary key.
It might be nice to detect this situation and use the keyserver search to find the main key or something, which is possible for instance on: https://keyserver.ubuntu.com/pks/lookup?search=0x95220BE99CE6FF778AE0DC670F65C7D881506130&fingerprint=on&op=index
I'd be more than happy to avoid the corruption by pacman bailing out. Would be nice to suggest to the user to find and install the appropriate keyring package in that case.
Thanks!
This is not an isolated incident. I've had it half a dozen times in the last year. Either I'm most unlucky or the issue isn't as simple as you say.
Naturally the 'Syu' form is always used in actual production.
Its the same issue indeed. The issue is that pacman imports the wrong gpg key. It never resolves the gpg public key. Which is what I wrote above.
With the same outcome, pacman is doing something wrong with the keyring. (which then leads pacman to reject a perfectly fine downloaded package).
https://gitlab.archlinux.org/pacman/pacman/-/commit/e1246baddd14ec6f4b6270b59bea0e1b639472a7
It was required to first install the archlinux-keyring package, before
updating the other packages was possible [1].
Since the bug report is not closed I suspect pacman should handle this automatically?
[1]
[rocketmouse@archlinux ~]$ sudo pacman -Syu
[snip]
Packages (14) archlinux-keyring-20220727-1 cmake-3.23.3-1 cython-0.29.32-1 cython2-0.29.32-1 gedit-42.2-1 lib32-libva-2.15.0-1 lib32-wayland-1.21.0-1 osinfo-db-20220727-1
plan9port-20220720-1 qt5-base-5.15.5+kde+r174-1 shellharden-4.3.0-1 sndio-1.9.0-1 webkit2gtk-2.36.5-1 webkit2gtk-4.1-2.36.5-1
[snip]
(14/14) checking package integrity [snip] 100%
error: plan9port: signature from "Alexander F. Rødseth <xyproto@archlinux.org>" is marginal trust
:: File /var/cache/pacman/pkg/plan9port-20220720-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: shellharden: signature from "Alexander F. Rødseth <xyproto@archlinux.org>" is marginal trust
:: File /var/cache/pacman/pkg/shellharden-4.3.0-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.
[rocketmouse@archlinux ~]$ sudo pacman -S archlinux-keyring
[snip]
[rocketmouse@archlinux ~]$ sudo pacman -Syu
[snip]
Packages (13) cmake-3.23.3-1 cython-0.29.32-1 cython2-0.29.32-1 gedit-42.2-1 lib32-libva-2.15.0-1 lib32-wayland-1.21.0-1 osinfo-db-20220727-1 plan9port-20220720-1
qt5-base-5.15.5+kde+r174-1 shellharden-4.3.0-1 sndio-1.9.0-1 webkit2gtk-2.36.5-1 webkit2gtk-4.1-2.36.5-1
[snip]
(13/13) checking package integrity [snip] 100%
(13/13) loading package files [snip]