FS#63171 - Use a Web Key Directory (WKD) to publish package signing keys

Attached to Project: Pacman
Opened by Jonas Witschel (diabonas) - Friday, 12 July 2019, 11:05 GMT
Last edited by Allan McRae (Allan) - Friday, 11 October 2019, 01:30 GMT
Task Type Bug Report
Category Backend/Core
Status Closed
Assigned To Allan McRae (Allan)
Florian Pritz (bluewind)
Architecture All
Severity High
Priority Normal
Reported Version 5.1.3
Due in Version 5.2.0
Due Date Undecided
Percent Complete 100%
Votes 2
Private No


To verify package signatures, pacman currently uses the SKS keyserver network to fetch unknown PGP keys or to refresh expired keys. This keyserver network has recently been attacked by spamming some high-profile keys with so many signatures that they can't be imported any more into GnuPG [1]. If one of the Arch Linux developer keys were spammed in this way, it would break the keyring of users running "pacman-key --refresh-keys" or that don't have this key in their local keyring yet. It is unclear how likely such an attack is, but since the Tor Browser Developers signing key is also affected [2], there seem to be some attackers intent on breaking as many user keyrings as possible, making the Arch Linux developers a possible target as well.

To overcome this problem, I suggest hosting the package signing keys in a trustworthy, read-only location to where such an attack is impossible. While it is possible to host a dedicated keyserver for this purpose, this is a relatively complex process that requires running additional server software. Instead I recommend hosting the keys in a Web Key Directory (WKD) [3]: this is simply a static list of files in a well-known directory of a server from where they can be retrieved using GnuPG. Setting up such a directory is as easy as running [4]

export GNUPGHOME=/etc/pacman.d/gnupg/
gpg --list-options show-only-fpr-mbox -k '@archlinux.org' | /usr/lib/gnupg/gpg-wks-client -v --install-key

and then copying the ./openpgpkey/archlinux.org/hu/ directory to the Arch Linux server, serving it as https://openpgpkey.archlinux.org/.well-known/openpgpkey/archlinux.org/hu/.

No further maintenance apart from occasionally adding new keys or refreshing expired ones is needed. Other projects like kernel.org [5], Debian [6] or Gentoo [7] are also using a WKD to publish their keys.

After this is implemented, the next step is to change pacman to lookup keys using WKD first (falling back to the keyservers if no key is found this way). This requires changes to "lib/libalpm/signing.c" (replacing "GPGME_KEYLIST_MODE_EXTERN" by "GPGME_KEYLIST_MODE_LOCATE"), "scripts/pacman-key.sh.in" (replacing "--search-keys" by "--locate-key") and possibly other places. I am happy to provide appropriate patches, but I think the WKD should be set up first so that the changes can be properly tested.

[1] https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f
[2] https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f#gistcomment-2959168
[3] https://wiki.gnupg.org/WKD
[4] https://wiki.gnupg.org/WKDHosting
[5] https://www.kernel.org/doc/html/latest/process/maintainer-pgp-guide.html
[6] https://dkg.fifthhorseman.net/blog/wkd-for-debian.org.html
[7] https://www.gentoo.org/news/2019/07/03/sks-key-poisoning.html
This task depends upon

Closed by  Allan McRae (Allan)
Friday, 11 October 2019, 01:30 GMT
Reason for closing:  Implemented
Additional comments about closing:  git commit 48752f1b (and others)
Comment by Allan McRae (Allan) - Friday, 12 July 2019, 11:39 GMT
Assigning to devops for the setup of the WKD. Ping it back to the pacman project once it is set up.

Comment by Levente Polyak (anthraxx) - Friday, 12 July 2019, 12:16 GMT Comment by Eli Schwartz (eschwartz) - Friday, 12 July 2019, 13:20 GMT
If I understand correctly, this only helps us publish keys that use an @archlinux.org uid for lookup (and in $PACKAGER). Which is still useful, but I think that we may need some changes to the TU admission process (that is, the administrivium portion) as well for full effect.
Comment by Jonas Witschel (diabonas) - Friday, 12 July 2019, 13:42 GMT
Correct, WKD works with email addresses instead of key fingerprints for lookup, so for pacman-key needs to lookup the @archlinux.org UIDs in the keyring and refresh these, for new key discovery pacman needs to extract the email address from the packager field and do a lookup with that. Putting everything in place will certainly take a while, but I think it is wise to start the process now when nothing bad has happened yet. The idea would be to keep the keyserver lookup as a fallback for problematic packages and keys, but getting WKD set up to a point where we can pull the plug on keyserver lookup if the need should arise.
Comment by Pol Van Aubel (MacGyver) - Friday, 12 July 2019, 13:54 GMT
If signatures are made using --local-user (-u) with an @archlinux.org UID, the UID is added to the signature and the WKD lookup should, according to the manpage, automatically resolve. There is no need for that UID to actually be e-mail-backed, but it would require all signers to add such a UID to the key they use for signing. (I haven't tested this because I'm not running a WKD myself (yet)). There would then be no need for pacman to implement key discovery based on the packager-field, but it does require the currently used tooling to be changed to include this UID in all future signatures.
Comment by Jonas Witschel (diabonas) - Thursday, 18 July 2019, 08:32 GMT
I've submitted a patch series to support WKD in pacman: https://lists.archlinux.org/pipermail/pacman-dev/2019-July/023493.html

Adding the UID directly to the signature instead of relying on the packager field would be more elegant indeed. Support for this wouldn't even necessarily need changes to pacman and devtools: if you set the environment variable $GPGKEY to a UID (as opposed to using a key ID/fingerprint or leaving it unset and relying on GnuPG to choose the key), this works out of the box, see https://git.archlinux.org/pacman.git/tree/scripts/libmakepkg/integrity/generate_signature.sh.in?id=39c20ad4f1d5f6e915b5be8976b6a94885ca3b0c#n34 and https://git.archlinux.org/devtools.git/tree/commitpkg.in?id=f522ce2277100a147baa2656753231b14cf1f71d#n156 Unfortunately we cannot rely on GnuPG to do the automatic key discovery (using the --auto-key-retrieve option) though, because this would directly import the key into the keyring without giving the user a chance to confirm it first.

Since this solution would require similar policy changes (setting the correct $GPGKEY instead of setting the correct $PACKAGER) and parsing the packager is simpler than extracting the user ID subpacket from the signature, I opted for the former solution in my patch series. Using the UID from the signature would be entirely feasible as well though, this should be integrated in parse_subpacket (https://git.archlinux.org/pacman.git/tree/lib/libalpm/signing.c?id=39c20ad4f1d5f6e915b5be8976b6a94885ca3b0c#n1002). I think this change would be orthogonal to my patches, you would retrieve the UID from the signature if available and try to fetch the key from the WKD with this email address, falling back to the packager email address if unsuccessful.
Comment by Jonas Witschel (diabonas) - Saturday, 20 July 2019, 09:22 GMT
I just realised that gpg-wks-client uses "gpg --export-options export-minimal" to strip away all signatures from the published keys. This isn't helpful for this use case at all because we explicitly want to keep the master key signatures for building the trust database. Note that this is not a limitation of the WKD protocol, but just the way GnuPG chose to implement the publishing process: GnuPG will happily import keys with signatures as long as the whole key is smaller than 64 kilobytes (which all of the keys in archlinux-keyring are), see https://dev.gnupg.org/T4607#127792

Therefore the publishing process for the full keys including signatures should instead be something like

export GNUPGHOME=/etc/pacman.d/gnupg
mkdir -p openpgpkey/archlinux.org/hu
for email in $(gpg --list-options show-only-fpr-mbox --list-keys | grep '@archlinux.org' | cut -d' ' -f2)
wkd_hash="$(/usr/lib/gnupg/gpg-wks-client --print-wkd-hash "$email" | cut -d' ' -f1)"
gpg --export "$email" > "openpgpkey/archlinux.org/hu/$wkd_hash"

and then publishing ./openpgpkey/archlinux.org/hu/ as https://openpgpkey.archlinux.org/.well-known/openpgpkey/archlinux.org/hu/.
Comment by Florian Pritz (bluewind) - Sunday, 04 August 2019, 09:13 GMT
We now have a WKD available at https://openpgpkey.archlinux.org/.well-known/openpgpkey/archlinux.org/hu/

I do see some keys that are bigger than 64kb. Please check if those work with gpg. From what I read here, they probably will not.

The relevant ansible code can be found here: https://git.archlinux.org/infrastructure.git/commit/?id=420d0717be0cc1acd71c6c3293e6e56e64220124

If there are issues, patches are welcome. Just post/link them here or ping someone from the devops team in IRC.
Comment by Jonas Witschel (diabonas) - Monday, 05 August 2019, 10:30 GMT
Cool stuff! I can successfully import all 48 keys published in the WKD using

mkdir /tmp/arch-wkd-test
for email in $(gpg --homedir /etc/pacman.d/gnupg --list-options show-only-fpr-mbox --list-keys | grep '@archlinux.org' | cut -d' ' -f2)
gpg --homedir /tmp/arch-wkd-test --locate-keys "$email"
gpg --homedir /tmp/arch-wkd-test --list-keys --with-key-origin

In particular the keys larger than 64kb work fine: apparently I misread the bug report referred to above, the limit is for publishing the key, not for receiving it from the WKD, see https://dev.gnupg.org/T4607#127880

Some things I noticed with the current state of the WKD:
* archlinux-keyring includes old keys that are no longer trusted, which are consequently also published in the WKD. We should make sure not to publish these in the WKD by filtering out the keys listed in /usr/share/pacman/keyrings/archlinux-revoked
* As a consequence of the first point, the WKD includes the wrong key for Gaetan Bisson (the old revoked one instead of the new one). Fixing the first point will also take care of this.
* According to the WKD specification (https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-4.5), the file https://openpgpkey.archlinux.org/.well-known/openpgpkey/archlinux.org/policy must be present. It can be zero-length, so touch-ing "$workdir/openpgpkey/archlinux.org/policy" is enough.
* It might be a good idea to turn of directory listings for the key directory to stop automatic email harvesting. Given that WKDs are not very common yet and the developer email addresses are already published elsewhere, I don't think this is a very high priority though.
* Optionally, it might be nice to publish the master signing keys (username@master-key.archlinux.org) as well. This is not necessary for normal operation, but could be convenient to import the root of trust.

I will submit patches for these issues, but I think the current implementation already works well enough that we can start thinking about getting WKD support into pacman and modifying the existing developer keys and/or PACKAGER variables where necessary so that they can be used with WKD.
Comment by Jonas Witschel (diabonas) - Monday, 05 August 2019, 10:34 GMT
Regarding the existing developer keys, I have compiled a listing indicating WKD support for every developer at https://wiki.archlinux.org/index.php/User:Diabonas/WKD_support_by_developer_key The good news is that out of 75 developer keys, 25 already work out of the box with WKD. The rest of the developers needs to add an @archlinux.org user ID to their PGP key or modify the PACKAGER environment variable to use their '@archlinux.org' UID (or do both). Note that after adding an @archlinux.org UID to the key, it needs to be re-signed by at least three master keys. This is because during import into the WKD, all UIDs except the @archlinux.org one are stripped away by GnuPG, so signatures on the old, non-Arch UID don't count for establishing trust in the key.
Comment by Jonas Witschel (diabonas) - Monday, 05 August 2019, 10:44 GMT
Regarding pacman support, I have modified my submitted patches a little, they work now with the existing WKD. I invite other people to test these patches to see if everything works as expected, see https://lists.archlinux.org/pipermail/pacman-dev/2019-August/023519.html
Comment by Christian Hesse (eworm) - Monday, 05 August 2019, 20:27 GMT
I would suggest generating the wkd from our archlinux-keyring repository. We have clean keys there, without useless extra signatures, images, whatever.

Can anybody verify that this generates a valid wkd, please? Should be a lot smaller, but still contain anything we need.
Comment by Jonas Witschel (diabonas) - Monday, 05 August 2019, 20:54 GMT
Nice, that makes more sense indeed! The generated WKD contains the same keys as the current one and these keys contain the @archlinux.org UID as well as the master key signatures, so this should work perfectly. Not having all those extra signatures is great as it brings down the size of the keys and should result in faster imports as well.
Comment by Eli Schwartz (eschwartz) - Monday, 05 August 2019, 20:57 GMT
Since the current implementation uses the pacman-key keyring, and that is directly seeded from the keyring, it should not be dirty at all -- the keyring is already clean....

The advantage of the current method is it is automatically applied whenever archlinux-keyring is updated, rather than requiring to be manually triggered in order to git clone an unauthenticated git repository (and maybe adding goop to the script in order to verify a git tag by hand).

The only advantage of using archlinux-keyring.git would be to utilize an unreleased keyring.
Comment by Christian Hesse (eworm) - Monday, 05 August 2019, 20:59 GMT
The pacman keyring is clean as long as you do not refresh the keys. Running "pacman-key --refresh-keys" pulls in all the unwanted things.
Comment by Christian Hesse (eworm) - Monday, 05 August 2019, 21:03 GMT
Well... If you want to keep it separated from archlinux-keyring repository I would still suggest creating a temporary keyring, importing /usr/share/pacman/keyrings/archlinux.gpg as a clean source and generating the wkd from there.
Comment by Christian Hesse (eworm) - Monday, 05 August 2019, 21:10 GMT
Speaking about clean source... The current wkd on orion has 1.5BMB, a clean wkd has just 536KB.
Comment by Jonas Witschel (diabonas) - Monday, 05 August 2019, 21:28 GMT
To generate the WKD from /usr/share/pacman/keyrings/archlinux.gpg like Christian suggested, the attached script can be used as a replacement for the current update-wkd.sh. It also takes care of not publishing revoked keys and adding the required policy file.
Comment by Eli Schwartz (eschwartz) - Monday, 05 August 2019, 21:46 GMT
... or just keep the current script, but change:

gpg --export "$email"


gpg --export-options export-clean,no-export-attributes --export "$email"

The tmpdir does *nothing* here, its only purpose in archlinux-keyring.git is to prevent *modifying* /etc/pacman.d/gnupg with incorrect data. Holding extra state and especially cloning git repositories first just seems like it is totally unnecessary.
Comment by Jonas Witschel (diabonas) - Monday, 05 August 2019, 22:07 GMT
In my case, I use the temporary keyring to delete the revoked keys to prevent them from being published in the WKD. I agree these could also be filtered in the loop, e.g. as in the attached file.
Comment by Florian Pritz (bluewind) - Wednesday, 07 August 2019, 17:34 GMT
I've updated our scripts and recreated the wkd directory Thanks!