Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#41909 - [ca-certificates] Some concerns regarding the "system trust" change

Attached to Project: Arch Linux
Opened by Mantas Mikulėnas (grawity) - Wednesday, 10 September 2014, 11:21 GMT
Last edited by Jan Alexander Steffens (heftig) - Thursday, 11 December 2014, 12:09 GMT
Task Type General Gripe
Category Packages: Core
Status Closed
Assigned To Jan de Groot (JGC)
Pierre Schmitz (Pierre)
Ionut Biru (wonder)
Jan Alexander Steffens (heftig)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No


I noticed that recently nss has been changed to use the system CA certificate store instead of Firefox's built-in one.

On the one hand, I _really like_ being able to define CAs in just one place.

On the other hand, I have concerns about the security of such changes:

1. Who approves the certificate authorities in "ca-certificates"? Is it based on Mozilla's CA list, or is it entirely decided upon by Debian?

1a. If Mozilla approves or distrusts a CA, will this propagate to the new system store? How often are the updates pulled in?

1b. If Google approves or distrusts a CA, will this get ignored completely?

2. Are the trust bits set by Mozilla completely ignored? The NSS database has separate flags for trusting a CA for sites (TLS); email (S/MIME); and code signing (.xpi). Meanwhile, as far as I know, the /etc/ssl/certs symlink-farm does not have any other mode than "fully trusted".

3. What happens to explicitly distrusted certificates that were bundled with the browsers, such as the infamous "MD5 Collisions, Inc." or the several actually-malicious certificates that were revoked? There doesn't seem to be any way to express this in "ca-certificates", and neither Chromium nor Firefox list those anymore.
This task depends upon

Closed by  Jan Alexander Steffens (heftig)
Thursday, 11 December 2014, 12:09 GMT
Reason for closing:  Fixed
Additional comments about closing:  ca-certificates 20140923-7.2
Comment by Jan Alexander Steffens (heftig) - Thursday, 11 September 2014, 09:56 GMT

We decide upon providers. ca-certificates is a metapackage that will gain depends or conflicts as providers are added or removed. Currently, we have -mozilla which is split from NSS while -cacert is from

You can remove ca-certificates if you want to manage providers yourself. Since -utils provides ca-certificates this won't break depends.

1a. Yes. As often as NSS releases, so this is no change from before.
1b. Yes. Also, no change from before. Chromium might do something custom, but that's not in the scope of ca-certificates or NSS.

Comment by Jan Alexander Steffens (heftig) - Monday, 22 September 2014, 23:29 GMT
[testing] now contains another iteration, which uses Fedora's scripts (which really just call p11-kit extract).

The -mozilla package now contains files readable by p11-kit that retain the trust flags, including distrust.

The update-ca-trust script has p11-kit produce several files for legacy apps, such as openssl or java. These files are split into trust for TLS, email and object signing.
Comment by Mantas Mikulėnas (grawity) - Wednesday, 24 September 2014, 07:23 GMT
Reporting in. Currently I'm using ca-certificates-20140923-4 from heftig, and it seems to be working quite well; I see the distrusts being picked up by Chromium and Firefox, etc., as well as my custom CAs which I've added to 'anchors'.

(The release that got removed from [testing] had been missing the "ca-certificates.crt" bundle that both curl and gnutls required.)

Also I just learned about OpenSSL's "-addtrust" and its trusted certificate format and the oddly-undocumented `p11-kit export` command; thanks.

(Now I just hope distros will eventually decide on a single directory layout for this...)
Comment by Jan Alexander Steffens (heftig) - Wednesday, 24 September 2014, 07:33 GMT
Gonna repost roughly what I said in the arch-dev emails (which didn't get any comment) here:

The Debian/OpenSSL way of handling certificates (one cert bundle without trust bits) ends up trusting too much; the trust categories and explicit distrust in the NSS database is ignored. The Fedora implementation uses p11-kit to handle the system trust; its storage format is more expressive and NSS (which speaks PKCS#11) has its increased security restored. For other apps like openssl, curl and java, p11-kit supports extracting cert bundles from its database. That's what the update-ca-trust script does. This also makes ca-certificates-java obsolete. It supports extracting an OpenSSL TRUSTED CERTIFICATE bundle that has trust bits OpenSSL understands, so it can benefit of the improved security. I haven't yet found out how to get OpenSSL to read it instead of /etc/ssl/cert.pem. GnuTLS likewise can use PKCS#11 instead of a cert bundle. I think these advantages make it worth switching over to the Fedora implementation.

Certificates are installed either into /etc/ca-certificates/trust-store (by the admin) or /use/share/ca-certificates/trust-store (by pacman) as annotated X.509 bundles, like -mozilla does. Explicit distrusts use a p11-kit-specific format. It's also possible to install individual certificates into the anchors subdirectory (to trust them) or the blacklist subdirectory (to distrust them), as -cacert does. Once the certificates are there, p11-kit can read them. The short update-ca-trust script that is run post_install exports the trust store into formats for other applications. I think it's better than the symlink mess maintained by update-ca-certificates.

Compared to the Fedora setup I've now inverted two symlinks (/etc/ssl/certs/ca-certificates.crt and /etc/ssl/certs/java/cacerts) so pacman does not complain anymore. Though I do like having all the extracted output written to a single path. update-ca-trust and its manpage are now carried locally, with the manpage modified to use our pathnames. The paths I chose (/etc/ca-certificates and /etc/ssl instead of /etc/pki and /etc/pki/tls) are not set in stone and up for debate. update-ca-trust is really small in comparison to update-ca-certificates: we already had most of the logic needed to maintain the trust-store the way Fedora does, namely p11-kit. If this setup is put into place we can look into configuring GnuTLS and OpenSSL to make use of the advanced trust stores (PKCS#11 for the former, for the latter).

If you want to test the Fedora-like setup, I have the current packages at . Upgrade and downgrade should be smooth.
Comment by Jan Alexander Steffens (heftig) - Sunday, 12 October 2014, 11:09 GMT
I moved the trunks back to core/extra state. The new PKGBUILDs are now also at
Comment by Felix Yan (felixonmars) - Wednesday, 15 October 2014, 09:48 GMT
I have tried the packages from, and so far they work well.
Comment by Mantas Mikulėnas (grawity) - Wednesday, 15 October 2014, 11:20 GMT
Reporting in. Been using the packages for several weeks without problems.

Since the new method has some advantages, and nobody on the mailing list gives a damn, might as well push it?
Comment by Pierre Schmitz (Pierre) - Wednesday, 15 October 2014, 11:31 GMT
I'll look into it. It's quite a different implementation so it cannot simply replace the current packages
Comment by Jan Alexander Steffens (heftig) - Thursday, 30 October 2014, 13:32 GMT
Any progress yet, Pierre?
Comment by Pierre Schmitz (Pierre) - Sunday, 02 November 2014, 07:47 GMT
I feel bad I did not had time to look into this earlier. Thanks for the implementation and testing.

In general I am in favor of the new way to manage the certs.
* We need some documentation which briefly describes the migration from the Debian based scripts. E.g. How do I add or blacklist certs now. This might be short and can refer to the man page, but we should at least give these two examples. There is no need to automate anything though.
* I am not a huge fan of the amount of code or man pages we would the need to maintain locally (in ca-certificates or nss). I would love to see i we could team up with other distros and find a generic solution. E.g. it shouldn't be hard to make the Fedora solution configurable. I wont have time to push this myself, but maybe we can bring this to the mailing list.

I assume if we drop in the Fedora package as-is, we'd need to update a couple of packages to the new cert paths etc? That could make sense if it would reduce our maintenance cost in the future.

In short: I am fine if you like to proceed with this. A migration guide is a requirement though and if we can reduce the Arch specific code in the long run that'd be great.
Comment by Jan Alexander Steffens (heftig) - Sunday, 02 November 2014, 11:07 GMT
Well, we only *need* to update the packages I rebuilt. Notice that /etc/ssl/certs/ca-certificates.crt still exists, so anything else can keep running as-is.

However, code that can either connect to p11-kit or use /etc/ssl/certs/ should be preferred, as those methods take the blacklists into account.

Thanks for taking a look.
Comment by Mantas Mikulėnas (grawity) - Sunday, 02 November 2014, 11:41 GMT
The upgrade instructions might be:

1. Move /usr/local/share/ca-certificates/*.crt to /etc/ca-certificates/trust-source/anchors/
2. Do the same with all manually-added /etc/ssl/certs/*.pem files (and rename those to *.crt)
3. Instead of `update-ca-certificates`, run `update-ca-trust`

Also, this might be necessary, since, AFAIK, OpenSSL relies on the "<hash>.0" symlink farm if ca_dir is used:

4. In programs that have settings like "ca_dir = /etc/ssl/certs", change them to "ca_file = /etc/ssl/certs/ca-certificates.crt"
Comment by Jan Alexander Steffens (heftig) - Thursday, 13 November 2014, 15:36 GMT
Should I add some upgrade instructions to ca-certificates.install and drop this stuff in [testing]?
Comment by Jan Alexander Steffens (heftig) - Monday, 01 December 2014, 00:23 GMT
I'm still debating switching from /etc/ca-certificates to the same /etc/ca as in Fedora, before this lands in [extra]. That would save a bit of patching on our side. Comments?
Comment by Jan Alexander Steffens (heftig) - Monday, 01 December 2014, 01:42 GMT
Another thing I've played around is "trust extract --format=openssl-directory" which generates a directory filled with certs and hash-symlinks, similar to what [core] currently does.

Currently, some symlink directions between /etc/ssl and /etc/ca-certificates/extracted are inverted to make the upgrade painless. Is it worth hanging onto?

ca-certificates-utils pkgrel 6 (currently at ) does the following differently:

- /etc/ssl/certs is a symlink to /etc/ca-certificates/extracted/openssl/certs , which doesn't exist before post_upgrade
- update-ca-trust now only touches /etc/ca-certificates/extracted, also building the above directory for maximum compatibility

I think this is even better, although the upgrade is a bit more painful now, since /etc/ssl/certs has to be deleted first.
Comment by Jan Alexander Steffens (heftig) - Monday, 01 December 2014, 01:58 GMT
Interestingly, the certificates extracted in openssl-directory format also have trust bits (BEGIN TRUSTED CERTIFICATE), so that's a plus over [core], too.

pkgrel 6 is now 6.1 to solve a problem that made update-ca-trust fail after working once.
Comment by Jan Alexander Steffens (heftig) - Monday, 01 December 2014, 03:03 GMT
ca-certificates-utils pkgrel 7 I just released provides painless upgrade. update-ca-trust rewrites /etc/ssl/certs (which is not a symlink) and puts some TLS bundles in /etc/ca-certificates/extracted (instead of extracted/pem).