FS#61609 - Pacman should upgrade archlinux-keyring first

Attached to Project: Pacman
Opened by Cosmin Truta (ctruta) - Saturday, 02 February 2019, 22:56 GMT
Task Type Feature Request
Category General
Status Assigned
Assigned To Andrew Gregory (andrewgregory)
Architecture All
Severity Low
Priority Normal
Reported Version 5.1.2
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 4
Private No

Details

Hi, I run into a problem with "pacman -Syu" that appears to be well-known:

`Signature from "User <email@example.org>" is unknown trust`

A friend tipped me with "pacman -Sy archlinux-keyring". Later on, I found the solution in pacman's Wiki, also.

I understand how the upgrade failed without archlinux-keyring being upgraded. Specifically, I understand that, after a simple "pacman -Syu" command, archlinux-keyring was among the packages about to be upgraded; but, without the keyring being up-to-date first, the whole upgrade of the whole bunch failed -- including the upgrade of archlinux-keyring itself.

This should not have been a Catch-22 issue.

May I please suggest a two-tiered upgrade, sending in first the packages that are prerequisite to a healthy overall upgrade?

At the very least, if the upgrade fails because of this "unknown trust" issue, I would greatly appreciate being informed, by the tool, about what to do next.
This task depends upon

Comment by Doug Newgard (Scimmia) - Saturday, 02 February 2019, 22:58 GMT
This is a failure to keep the system up to date more than anything.
Comment by Cosmin Truta (ctruta) - Saturday, 02 February 2019, 23:42 GMT
Blame the user much, Doug Newgard?

I would like to point out that I did not "fail to keep" the system up to date. I did not boot the Arch Linux machine in a long time, that is correct. Maybe because I was in a different country for a while, or on a different continent, or in a hospital, or maybe just stayed at home and was over-my-head busy with real work and not much time for hobbies.

I was responsible enough to "pacman -Syu" first thing after I booted the machine. Let's stick to that.

You may want to fix it, or you may not, that is your choice. As for myself, I was also responsible to give you this insight, take the time to file this bug report, although my problem has already been solved. If I use other systems (including other Linux systems) in my mission-critical work, I hope I'm excused. If I'm not used to seeing tools (especially core system tools) failing on me for the wrong reasons, I hope I'm excused for that, also. I still maintain that a two-tier installation procedure is a no-brainer, and that real-world issues happen to real-world users. Software should not be brittle on purpose.
Comment by Doug Newgard (Scimmia) - Saturday, 02 February 2019, 23:55 GMT
Blame the user when it's the user that broke it? Yes, yes I do.

Remember, Pacman is not Arch specific, so not much will happen here. The only way this could work is with the old SyncFirst, which was removed and has been stated MANY times that it will not return.
Comment by Eli Schwartz (eschwartz) - Sunday, 03 February 2019, 00:15 GMT
We once tried to do this, it was called SyncFirst, and we removed it because it was a tremendously hacky, broken mess that was more trouble than it was worth. I assure you we did not remove the pre-existing feature that did exactly what you are asking for, out of a desire to "be brittle on purpose" or "blame the user", and instead we had some pretty good reasons for not liking its practical effects.

You have chosen to use a distro that presupposes the user has time for regularly scheduled maintenance, and failed to provide that regularly scheduled maintenance.

Despite this, your system is not broken -- it merely requires that you follow some simple instructions documented on the wiki in order to recover. Or, quite simply, refresh your GnuPG keyring using pacman-key --refresh-keys. Actually, the archlinux-keyring case is by far the absolute easiest to solve -- it is much much harder to resolve cases where packages first drop reverse dependencies, then get replaced by something else, as that gets you into a situation where pacman cannot even calculate a valid dependency tree regardless of signature trust, and the only solution is to know the complete state of the repository every step of the way for the last however many months you didn't upgrade. The other solution to this is to use the Arch Linux Archive to upgrade the system in stages, one month of updates at a time. It's not fun to do, by the way.

...

Just to be perfectly clear: there are *no* packages that are "prerequisite to a healthy overall upgrade". All packages are equally important -- any package can depend on any other package that provides some shared library dependency with useful functionality (though the [core] repository is standalone for booting, so hey, there's only potentially 233 packages that need to be updated in utter lockstep), we don't version dependencies due to our policy that we refuse to support partial upgrades, and even with the SyncFirst keyword we used to have, the only thing which is actually safe to update on its own is the keyring, because the keyring is simply an offline cache of some GnuPG data (which anyways you can create entirely on your own using --refresh-keys and pacman will even download it for you the first time, as long as you have the six master keys to verify the rest of the distro).
Comment by Andrew Gregory (andrewgregory) - Sunday, 03 February 2019, 00:54 GMT
How would either of the proposed features work? The reason SyncFirst was removed is because it makes proper dependency handling extremely difficult. Ignore dependencies and you end up with broken software, take dependencies into account and there's a very good chance you end up bringing the very packages you were trying to delay into the initial transaction.

As for having the pacman tell the user what to do to resolve the issue, how is pacman supposed to know that? Pacman has no way of associating that signature with a package or even knowing if it's provided by a package at all.
Comment by Cosmin Truta (ctruta) - Sunday, 03 February 2019, 02:21 GMT
Ok, Eli, Andrew, thank you for your responses. I'm happy to see that we're finally getting to the technical issues.

FYI: I'm brand new to Arch Linux, accustomed to other Linux distributions, and just joined this community earlier today. I did not read past announcements, I do not know what SyncFirst, and no matter how "MANY" times it had been stated that it would not return, I would not have known.

I am just learning about SyncFirst from your follow-up explanations, and I will read more about it, later. But from what I've understood so far, I don't think this is what I originally had in mind.

Importantly, I did not suggest changing the dependency handling. Let us put SyncFirst behind us.

What I did suggest is a two-stage update. Two full stages. As if somebody would run `pacman -Sy archlinux-keyring` first (if it's on the list), and then `pacman -Syu` afterwards. As if it were driven from an external shell program (but still driven from the inside).

Consider, the example, Linux Mint. I do not have intimate knowledge of their internals, but from my end-user P.O.V., I can see the Mint Update program updating itself from time to time, as needed. If Mint Update needs updating, it updates itself first, and it updates itself fully. The same thing happens if any Mint mirror repositories get changed. All of the other updates are coming after that. (I can actually see Mint Update restarting itself. Aside from this visual glitch, which doesn't really bother me, it works great.)

Coming back to pacman, the simplest implementation that I'm envisioning is breaking the list of candidate packages in two smaller lists. In the first list, there should only be essential components, like archlinux-keyring, and (perhaps) pacman itself, and not much else. Most of the times, the first list is expected to be empty. And for the second list, everything else is in.

Pacman itself can be 100% agnostic to that. The actual distribution can pick and choose what to send to pacman in the 1st list, and what to send in the 2nd list. What pacman can do is guarantee that no item in the 2nd list will be installed, until all of the 1st list is fully installed.

The ongoing distro maintenance efforts should be small (notwithstanding this architectural change, of course). The benefits for end users are (in my opinion) big, with the tool doing the right job in the right way, and without any fuss.
Comment by Cosmin Truta (ctruta) - Sunday, 03 February 2019, 02:36 GMT
I wrote the previous long message, now please allow me to rewrite it in a shorter way:

"pacman -Sy <list-of-packages>"
should work as if
"pacman -Sy archlinux-keyring && pacman -Sy <list-of-packages-except-archlinux-keyring>"

or,

"pacman -Syu"
should work as if
"pacman -Sy archlinux-keyring && pacman -Syu"

This ^^

Can I write a shell script for myself? Of course I can. But pacman should be a bit smarter than that, and it should know what the user probably doesn't. It should know what ought to be in the first list of packages, and what ought to be in the second list of packages. It should know when the first list of packages is empty, also. Everything that pacman needs to know regarding these two distinct (and disjoint) lists, can and should be given by the Linux distribution.
Comment by Eli Schwartz (eschwartz) - Sunday, 03 February 2019, 03:06 GMT
This is exactly what the SyncFirst keyword did. I personally am not blaming you for not knowing about old history -- just trying to head off the possible implication that if we actually had it and removed it, then it is user-unfriendly to remove it.

For some technical context, you might want to look at the original proposal and patch to remove it: https://lists.archlinux.org/pipermail/pacman-dev/2012-February/015123.html

If you look at the code you'll also see how it was implemented.

...

Ultimately, there doesn't seem to be any way to handle this in a generic manner, the only safe updates are in very limited cases -- basically just the keyring, since we know the keyring is safe to update at any time but don't know about anything else. Bringing back SyncFirst but hardcoding the package list which is considered critical, could partially alleviate these concerns -- but that's both inelegant and hard to track.

It's been implemented, though, in MSYS2: https://github.com/Alexpux/MSYS2-pacman/pull/26/commits/6e0ff8324dca152b312a2d39ab025f9888896b9c
They hardcode a list of seven packages into the binary, which are required to fully bring up the MSYS2 runtime shell on Windows. We have a lot more packages required to bring up a bootable Linux OS, and those dependencies are intertwined with the rest of the system which means if you update curl and pacman, you break half the userland that needs the old curl. Just see the recent issues people had with partially updating libidn2, for another example.

Back in Ye Olden Daye (2012) we had SyncFirst, and we used it, and what we used it for was pacman -- to guarantee that the updater itself was updated in order to handle more cases better. As said, it wasn't working out very well -- and of course not, since it was a binary package linking to shared libraries that could break the system via partial upgrades, which was exactly what it tried to do, by updating itself and its own direct dependencies in a first run. We don't really have issues where that is a concern anymore, to my knowledge, the last such case being the introduction of hooks where we gave everyone 2 months notice to upgrade before making use of the new features: https://www.archlinux.org/news/required-update-to-pacman-501-before-2016-04-23/

Should we cherry-pick the keyring and only try to protect people against keyring issues? Keyring issues aren't even common -- as a general rule, keys tend to be good for quite some time, and when new keys cycle into the system, pacman already knows how to download those for you without fuss or interruption.
Comment by Allan McRae (Allan) - Sunday, 03 February 2019, 04:27 GMT
What is causing this in the Arch repos. Pacman should[1] handle a new key being added - it just requests and downloads it. I guess a key with an expiry dates set that then gets extended would not be detected? Any other case?

[1] assuming the gpg infrastructure works...
Comment by Tom (TomCat) - Monday, 20 June 2022, 15:57 GMT
I want to put attention on this and instantly point out that this feature request is talking about a solution without describing the problem. This causes people to consider only the solution instead of any solution that solves it. A thinking pattern that many technical people have. I think we might find a different solution if we don't start with a solution (to upgrade archlinux-keyring) but start with the actual problem.

The problem is this;

when I update after coming back from holidays (we have a month here), I get questions about importing a PGP key. The ones with an email address (the primary ones) are fine. Stuff works.

But sometimes an pacman -Su leads to something like this;

```
error: libinih: key "95220BE99CE6FF778AE0DC670F65C7D881506130" is unknown
:: Import PGP key 95220BE99CE6FF778AE0DC670F65C7D881506130? [Y/n] n
:: File /var/cache/pacman/pkg/libinih-55-2-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] ^C
```

Now, if you look at https://archlinux.org/master-keys/, you won't find this key. So what gives?
Turns out, its a subkey of Maxime Gauduin, which has as the main key: 0x81506130

Pacman seems to decide it needs that key from a signature it found on a package. Pacman doesn't know the main pgp identity and experience shows you can end up with a rather bad experience updating after that (as other posters showed).

So, the bugreport I'd word like this;

Pacman has a feature to import into the keyring a gpg key when it encounters it on a package. This key is not always the main key, does not have an email address attached and as such this feature is buggy because importing it doesn't always leave the keyring in a good state. Currently we make the user fix up after pacman. But this is not a user bug, this is a pacman bug, it should not import a gpg key which is not the primary key.

It might be nice to detect this situation and use the keyserver search to find the main key or something, which is possible for instance on: https://keyserver.ubuntu.com/pks/lookup?search=0x95220BE99CE6FF778AE0DC670F65C7D881506130&fingerprint=on&op=index

I'd be more than happy to avoid the corruption by pacman bailing out. Would be nice to suggest to the user to find and install the appropriate keyring package in that case.

Thanks!
Comment by Amin Vakil (aminvakil) - Monday, 20 June 2022, 16:56 GMT
Tom, the problem you have faced was because of a faulty package or faulty archlinux-keyring I don't remember, but it was fixed quickly, why are you running `pacman -Su` instead of `pacman -Syu`?
Comment by Tom (TomCat) - Monday, 20 June 2022, 17:10 GMT
Hi Amin.

This is not an isolated incident. I've had it half a dozen times in the last year. Either I'm most unlucky or the issue isn't as simple as you say.

Naturally the 'Syu' form is always used in actual production.
Comment by Amin Vakil (aminvakil) - Monday, 20 June 2022, 17:13 GMT
I'm not sure, but the problem you have stated and the package is exactly these bugs: https://bugs.archlinux.org/task/74688 https://bugs.archlinux.org/task/74588
Comment by Tom (TomCat) - Monday, 20 June 2022, 17:20 GMT
Oh, but if you read the bugs you link then you'd see that the package was likely never corrupt, just that pacman thought it was due to exactly this issue.
Its the same issue indeed. The issue is that pacman imports the wrong gpg key. It never resolves the gpg public key. Which is what I wrote above.

With the same outcome, pacman is doing something wrong with the keyring. (which then leads pacman to reject a perfectly fine downloaded package).
Comment by Morten Linderud (Foxboron) - Monday, 20 June 2022, 17:31 GMT
The key is/was fine. The issue was that the WKD handling in `pacman` was broken and a patch has recently been backported. The case above would have been solved with this patch.


https://gitlab.archlinux.org/pacman/pacman/-/commit/e1246baddd14ec6f4b6270b59bea0e1b639472a7
Comment by Ralf Mardorf (Ralf) - Friday, 29 July 2022, 13:20 GMT
Today an update of archlinux-keyring and other packages was available.
It was required to first install the archlinux-keyring package, before
updating the other packages was possible [1].

Since the bug report is not closed I suspect pacman should handle this automatically?

[1]
[rocketmouse@archlinux ~]$ sudo pacman -Syu
[snip]

Packages (14) archlinux-keyring-20220727-1 cmake-3.23.3-1 cython-0.29.32-1 cython2-0.29.32-1 gedit-42.2-1 lib32-libva-2.15.0-1 lib32-wayland-1.21.0-1 osinfo-db-20220727-1
plan9port-20220720-1 qt5-base-5.15.5+kde+r174-1 shellharden-4.3.0-1 sndio-1.9.0-1 webkit2gtk-2.36.5-1 webkit2gtk-4.1-2.36.5-1

[snip]
(14/14) checking package integrity [snip] 100%
error: plan9port: signature from "Alexander F. Rødseth <xyproto@archlinux.org>" is marginal trust
:: File /var/cache/pacman/pkg/plan9port-20220720-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: shellharden: signature from "Alexander F. Rødseth <xyproto@archlinux.org>" is marginal trust
:: File /var/cache/pacman/pkg/shellharden-4.3.0-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.
[rocketmouse@archlinux ~]$ sudo pacman -S archlinux-keyring
[snip]
[rocketmouse@archlinux ~]$ sudo pacman -Syu
[snip]

Packages (13) cmake-3.23.3-1 cython-0.29.32-1 cython2-0.29.32-1 gedit-42.2-1 lib32-libva-2.15.0-1 lib32-wayland-1.21.0-1 osinfo-db-20220727-1 plan9port-20220720-1
qt5-base-5.15.5+kde+r174-1 shellharden-4.3.0-1 sndio-1.9.0-1 webkit2gtk-2.36.5-1 webkit2gtk-4.1-2.36.5-1

[snip]
(13/13) checking package integrity [snip] 100%
(13/13) loading package files [snip]

Loading...