Arch Linux

The desc file in the repo dbs is generated by repo-add which is part of pacman. So you will need to request adding shaxxx sums to the pacman team...

But what I see as the main issue here is what signs the database and where it is signed? Is there a single master key that resides on the master server and is used to sign everything? Or do developers/TUs sign the database themselves?

I find a single key and thus a single point of weakness disconcerting as any compromise of the server (which by necessity has to be accessible to the outside work for packages to be uploaded) results in that key being fully compromised. I know the attacker would still require the passphrase... but think about that too. Would the developers all need to know the passphrase or would the database reside in a temporary location until it is signed (e.g.) by a cron job and then the database moved. The former involves handing out the passphrase to many people, while the later requires the passphrase sitting in some file on the server. Both are sub optimal.

So that suggests that the database should be generated, transferred to the developers machine where it is signed and the signature transferred back. That requires some complex key management for users to know what keys are acceptable and to allow the quick revoking of a key if necessary.

So I believe this issue is far more complex than stated, and certainly not just adding a single line to sign the database.

> The desc file in the repo dbs is generated by repo-add which is part of pacman. So you will need to request adding shaxxx sums to the pacman team...

Thanks - I'll file that.

As for the rest, I think you are making this too complicated. As an interim solution, a single key to sign the databases will vastly improve the security situation until you have the finer points of your key management system worked out.

I'm not that familiar with how the databases are produced. I think signing individual packages at this point is unnecessary - just sign the databases. Wherever the database is tarred before it is uploaded to the primary mirror, have it signed there by a generic 'Arch Database' key. If multiple packagers need a copy of the key for this, then so be it. It is still a HUGE security improvement.

As I pointed out, with a single key, the passprahse would need to either be known by every single developer and TU or sitting on the server where the database is created (which is the primary mirror). As far as I am concerned, that is so full of holes that it only provides an illusion of extra security, which is far more dangerous.

I think it sitting on the server where the database is created is fine, and is a huge improvement to security. It ensures that this server signed the packages, which means that they can be authenticated on all the other 150+ mirrors, rather than having no protection at all against being modified and re-tarred on any mirror. It focuses the security to the primary server, instead of having no mirror security at all.

Allan, you seem to be against any interim solution to vastly improve security, and very motivated to fight against such measures, while at the same time claiming this issue is of no interest to you and your being unwilling to help with it. If you're not willing to help, then please stand aside so the devtools devs can add this important element to Arch's security.

I'm not against implementing this. I against implementing it in what I see as such a poor fashion that only gives an illusion of improved security. If we provide an illusion of security that is then shattered, that will do much more damage than not providing that illusion in the first place. The two options I can see for a single key (having both a key and passphrase on a single computer, or passing the passphrase out to every dev/TU) are not secure and I do not hear another suggestion on how to implement signing with a single key.

@Allan: You sure seem motivated to interrupt this improvement. It doesn't just provide an illusion of security, it provides a huge improvement to security, as I explained and you did not address. Having the arch server holding a key is better than no key at all with hundreds of vulnerable mirrors. In terms of security, it's like having everyone downloading from Arch rather than from independently run servers. I honestly don't think you understand the security situation. Tell me, what illusion of security does it provide to have people downloading from mirrors without a warning that Arch provides no authentication at all? Many of the people reading my blog say they didn't know Arch had no authentication, and since you instantly remove all posts about it on the forum, it seems you are deliberately setting them up for this. If you're all for no illusions, how about putting a warning about this on the Arch download page.

Signatures by the arch server will provide a significant boost to package distribution security, with limitations, but not an illusion at all. And since pacman won't be doing the signature checks, it is up to users to understand what these signatures are, their limitations, and how to make use of them. The signature will simply imply 'This package was signed by the arch server'. It need make no claims that it was signed by the packagers or developers - one step at a time. Yet this is an important step.

You suggested I submit patches - I submitted one for repo-add. We'll see what's done with it.

Seem we are getting direct... I think you also have no understanding of security. e.g. providing a script to check databases on mirror and advertising it as a security improvement while very naively not covering the number one point of software attack by not providing a signature for your source code. That has kept me quite amused the last couple of weeks. And I have not removed any forum posts so be careful about your use of "you" when making a reply directed at me. But I expect no more given your selective quoting and misinformation in your blog posts. Finally, if the people reading your blog do not realise there is no signing now, how are they going to know what the limitations are with your proposed signing scheme? Short answer is that they are not...


And I will state again, I am not against implementing this. I just want to see it implemented properly if it is going to be implemented at all. Doing a half-assed job only results in the proper solution being greatly delayed, and for the reasons I pointed out above I consider the single key ideas weak. Note I am still talking about the proper solution just for getting the repo dbs signed and nothing regarding pacman.

My proposal on how to do this is:
1) package is uploaded by the developer
2) the current repo signature is checked
3) the new package is added to the repo
4) the repo is transferred to the developers computer for signing
5) the new signature is uploaded
6) the new signature is verified against a whitelist of valid signatures
7) the updated repo and signature is pushed to the mirror

That still leaves areas to figure out... How are the developers/TUs keys going to be distributed to users? Will there be a master key which signs all developers keys? That would be simplest as it allows all users to only have to verify one key manually and then trust its signatures. But then how do we revoke a key when a dev/TU requires removed? We can readily stop new database signing via the whitelist in step #6, but users still trust that key.

> providing a script to check databases on mirror and advertising it as a security improvement while very naively not covering the number one point of software attack by not providing a signature for your source code.

Off-topic, but that did not escape my attention. I have been considering how to do this since the AUR provides no mechanism, and I don't control the servers. Here I will agree with you to an extent, but paccheck is a script which is transparent - it's not a binary. And the PKGBUILD (with SHA256 sum - imagine that) is located on a different server than the source. So your binaries have some catching up to do with my script security.

The thousands of people who have been reading my blog post on Arch's mirror issues daily since it was picked up by Linux Today almost exclusively agree. As Thomas Bächler, apparently an Arch dev himself, wrote:
"Sign the sync database on each update with a key that is stored on the server. This has many flaws, but it would guarantee that the sync db file originates on our server."

Sound familiar? Nothing to "figure out".

At any rate, I realize you think you ARE Arch Linux, and the other devs seem to allow this, but since you have done nothing but attempt to interfere with any suggestion for improving Arch's serious security lapse in this area, I would appreciate it if other devtools devs would consider adding this simple signature to say "this file was signed by the arch server". Thank you.

Once that is done (increasing security by several orders of magnitude for thousands of users with one line of code), we can consider Allan's convoluted proposal.

I see no criticism of my proposal apart from it being convoluted (which is different from incorrect or flawed), where as the single key signing method has "many flaws" as you quoted. I also pointed out why doing a half-assed job for the short term was detrimental in the long term. Is there any actual flaw in what I proposed other than it not being what you want now?


Finally, I do not consider myself Arch Linux. My vote counts for as much as any other dev that care to cast theirs. And I have not interfered with your suggestion other than pointing out its flaws and proposing a better way to achieve it. Is it the case that you can criticise but not take the criticism yourself? Then again, you are prone to exaggeration as your conclusions of 1000s of people agreeing with you based on 10s (being generous) of comments on your blog.

And finally, I will congratulate you on the number of replies to this bug in the first few hours. Experience around here shows that bugs with such controversy at the start result in bike-shedding for several months or years before anything actually gets done. This is mainly because no-one wants to read all the comments to figure out what actually needs done. So each long and not to the point reply effectively delays this being implemented. Something to consider...

> I see no criticism of my proposal apart from it being convoluted (which is different from incorrect or flawed), where as the single key signing method has "many flaws" as you quoted.

I would not say it has "flaws", merely limitations, as any security system does. It's limitations are reasonable and an improvement considering the limitations of the current (non-existent) arrangement for package security. I didn't respond to your proposal because I consider it a simple attempt on your part to derail this security improvement by filling this simple feature request with unnecessary complications.

> Is there any actual flaw in what I proposed other than it not being what you want now?

How about you submit your own feature request on that and I will be happy to discuss it with you, rather than derailing mine?

> Experience around here shows that bugs with such controversy at the start result in bike-shedding for several months or years before anything actually gets done.

So this was your purpose in creating controversy, apparently, since you're the only source of it. I'm sure you'll be happy to see any security improvements to Arch's distribution system bike-shedded, for whatever bizarre reason. You've accomplished that well with pacman for several years now. I'm trying to get around YOU, basically. So please step aside if you're not willing to help.

I did only read falf of the comments here, but there are two different problems here; similar to HTTPS-encryption.

1) Ensure that packages are not modified on their way from the main server to the end user with probably several mirrors in between.
2) Ensure that the package was indeed build by an authorized Arch Linux developer.

Number one can easily be implemented by having the private key on the server and just sign the db files. (In that case we indeed need a better hash algorithm and my comment at https://bugs.archlinux.org/task/23103 is no longer valid then. Of course this does not protect us when the main server is being hacked.

Part two is more complicated and is what is currently implemented by some people on pacman-dev under the term "package signing".

In genereal I wouldn't mind implementing part one if we communitcate correctly which level of security is reached by this approcach and what issues remain. (Afaik this approach was reject for some reason a long time ago; thread can probably found on the ml (afaik it was even started by me)

PS: Here is the thread I mentioned: http://mailman.archlinux.org/pipermail/arch-dev-public/2008-November/009089.html

Looks like that idea was put down because "signing packages" was mostly implemented already.

> How about you submit your own feature request on that and I will be happy to discuss it with you, rather than derailing mine

I will not open a new feature request as it would be a duplicate of this one. And I think have a bug report saying "implement  FS#23101  properly" and spreading discussion of a single issue across bugs is a waste.

> So this was your purpose in creating controversy,,,

Me pointing out useless and offtopic comments derail a discussion was not an ironic attempt at derailment, It was an attempt at keeping you on topic. I have more than shown my willingness to help in this area, including proposing what appears to be the most robust solution to this issue and maintaining the collection of all patches submitted for package signing so far. Pointing out flaws in an idea is not derailing if those flaws are genuine... I will not stand aside an let flawed approaches be implemented when a better approach is available.

> Number one can easily be implemented by having the private key on the server and just sign the db files. (In that case we indeed need a better hash algorithm and my comment at https://bugs.archlinux.org/task/23103 is no longer valid then. Of course this does not protect us when the main server is being hacked.

I agree fully - it merely certifies the packages as originating from the Arch server. And that was my reason for the SHA256 sum request in repo-add, as I just added a comment to explain.

> Part two is more complicated and is what is currently implemented by some people on pacman-dev under the term "package signing".

Right, I am not addressing that here. This is an interim boost to mirror security.

> In genereal I wouldn't mind implementing part one if we communitcate correctly which level of security is reached by this approcach and what issues remain.

If you do so, I will immediately update my paccheck script to check the signatures on the signed databases. A number of users are already using this script to check package authenticity, but it is just a brute approach. I will write a careful explanation of the limitations of the database server signature - that it is merely automatically signed by the server, not the developers directly. This will still improve authentication and reduce mirror load. And as the pacman signing is rolled out and being debugged, these database signatures can provide a second check. Users who check the database signatures manually will be doing so of their deliberate volition, so they can assess its usefulness and limitations. Users can download the key from a keyserver.

I would say the level of security reached is that the packages on the many mirrors are cryptographically assured to be identical to the packages on the primary Arch server, no more no less. Users who use paccheck or a similar signature check with gpg can assure themselves of this, regardless of what mirror they download the packages from. The vulnerabilities are the vulnerabilities of the Arch server itself, since the hashes and keys will be modern.

I think it's a good first step with easy room for growth if desired, as well as immediate usefulness, so thanks for considering it.

Also, one other benefit to consider - even if most users are not checking the database signatures (since it will require using a special tool or gpg), even having a limited set of security-conscious users checking the database signatures and packages will reveal tampering, which can then be brought to the attention of the community more quickly. This provides a kind of mirror monitoring, the alternative being a mirror system that can be compromised silently (the current situation).

> PS: Here is the thread I mentioned: http://mailman.archlinux.org/pipermail/arch-dev-public/2008-November/009089.html
>Looks like that idea was put down because "signing packages" was mostly implemented already.

Yes, "mostly implemented", and now three more years have passed. Your proposal was and is a sound one. The question is, who has to sign off on this to make it happen? Unlike other proposals (Allan's for example, patches welcome!), it doesn't require much of anything to implement - just a key generation and a line to add the sigs. So this is merely a cooperation issue.

IgnorantGuru, please read Pierre's ML post -- it is the same idea as yours, your posts here contain no new information.

Allan, why is it impossible to store the passphrase for a single key on the server in an encrypted file? Then, whoever needs to sign a db will use to use his/her own (dev's) key. Of course, there will have to be an instance of gpg-agent running, but overall, it'll be similar to msmtp's passwordeval option. Then, each dev signs his/her package and a db is signed on-server, only if dev's sig is OK. This way, Pierre's steps 1 & 2 are decoupled, and dev's signatures are invisible to the end user.

I am not suggesting that you haven't thought about this, but I want to know counter-arguments...

@lisaev: My main concern is having the key and its passphrase on the same system. Having the passphrase encrypted makes obtaining it a bit more convoluted, but won't it need unencrytped to do the actual signing?

I think I am missing how that makes the setup more secure apart from having the additional step to obtain the key. Can you clarify that a bit further?

> IgnorantGuru, please read Pierre's ML post -- it is the same idea as yours, your posts here contain no new information.

How true. Yet you seem to lack understanding, so some discussion was deemed necessary. The fact that you have left a veritable root exploit in your package management system unaddressed for years is not something to feel self-satisfied about. First address the dysfunction in your development process, then you can address the issue itself.

Regarding Allan's proposal, the biggest problem with it is that no one has expressed any willingness to promptly code it, including Allan. It's fine to discuss theoretical security plans which involve the cooperation of many people to be successful, but this is a practical matter, and he is offering no practical solution (just as he hasn't for years). As he likes to say, "patches welcome". The proposal here is to have the server sign the database in order to immediately secure the package management system to a large and valuable extent. Pierre and others have expressed an immediate willingness to implement this, and thus immediately address a serious flaw in your package management system.

Yet I can see this is not a technical issue at its core, as the technical solution is merely waiting approval by a dysfunctional development team. There seems to be a lack of understanding and education among the primary Arch developers in the area of security, as well as a lack of will to address a serious security problem, for whatever reason.

If the Arch dev team is unable to address this issue promptly, as apparently they are, I believe they should issue a warning on the main download page and elsewhere, letting users know that the distribution suffers from a major security problem which you are unable to resolve.

@Allan
I'm sorry for being unclear. What I meant is simply the following.

1. Authenticity of the packages.
Each dev has a private/public key. The db file is being signed automatically by a signing server (at least, this is how it's done in other distros). Of course, one needs a server keypair and a passphrase, which is stored on-site in an encrypted file, encrypted with "-r <all devs>". The (properly configured, e.g. with large caching timeout via --default-cache-ttl/--max-cache-ttl) instance of gpg-agent is supposed to make signing automatic. Clearly, the gpg-agent must be started manually (each dev knows a passphrase). The only possibility for a successful attack is to silently hijack the server, and/or use those memory scraping techniques to pull the passphrase from gpg-agent. Alternatively, the server key can be regenerated without a passphrase, but frequently (expiration date = 1 day, for instance). So, even if the server is compromised, the damage will be minimal.
2. Authenticity of devs.
Apparently, this depends on the packaging process itself. Is there a standard way of doing things?

I agree with you: it goes beyond a single-line patch... security is a process after all :)

We seem to have 2 camps here. One camp wants to get a perfect solution and has been working on it for years with no functional implementation. Whenever someone brings up the subject his code is supposed to cover he suggests they provide a patch or stop complaining.

(As a computer security professional his understanding of exactly what a man in the middle attack is and how to defend against it seems off. He claims that a fix that would remove the possibility for man in the middle attacks and mitigate hacked mirrors wouldn't, but I don't want to get all ad hominem-y. I could be misunderstanding his stance entirly.)

The other camp has a couple of lines of code that would would solve a whole bunch of problems but if the main server got hacked or used improperly it could be an issue. It's not quite ideal and we'd want to expand upon it or replace it if the first camp ever completes their code, but it increases security tenfold.

Why are we debating this?

If it's a matter of no one implementing it then I will. It can't be that hard. bash script wrapper that creates the DB however it's normally created then signs it. The hardest part would be changing the DB creation script from md5 to sha256.

EDIT: apparently allans gpg patch is coming along quite nicely. if it's done very soon then he could be right.

http://projects.archlinux.org/users/allan/pacman.git/log/?h=gpg

@Alex
I agree fully of course - just want to drop a link to my db creation script request for SHA256. It can be added in addition to MD5 without upsetting the current pacman, and it's trivial:
https://bugs.archlinux.org/task/23103?project=1

Dan never got back to me so I didn't know if he was done ridiculing me or if he was serious about wanting a patch. It's just a few lines to add in the script, just as I provided. Thanks.

I'll close this issue for now. As long as pacman cannot distinguish between keys used for dbs and packages this cannot be implemented in dbscripts.