Random but important

Security in computer programs is often based on some kind of secret. It might be a key, passphrase or the popular password. It is also equally important that this key is not only secret but also random and cannot be easily guessed.

The same concept applies to our package signing method. The security model which is used to ensure validity of our packages is directly based on PGP. In this trust model you mark a certain key as trusted by signing it with your very own private key.

On Arch Linux we ship the archlinux-keyring package to set up and maintain a set of packager keys and their web of trust. On install a new key pair is generated and the five master keys are locally signed. Any package that is signed by a key that itself is signed by at least three of the master keys is considered valid. This also implies that anything signed directly by that private key is automatically valid as well.

The keyring pacman uses is stored in /etc/pacman.d/gnupg. There is no magic involved here and you can even directly use it with the gpg command by appending the --homedir /etc/pacman.d/gnupg parameter.

It is important to underline that the PGP key pair stored there is unique to your machine. It is not part of any package but generated on install and random. An attacker who is able to obtain or guess that key can sign packages that pacman will happily accept as valid. Be aware of this when you create backups, clone your machines or setup virtual machines. It might be a bad idea to share the same private key on different machines.

For our live system image we ship a service that initializes a new random key on every boot. It is stored in RAM using tmpfs and therefor not persistent. We make use of the haveged service to create enough random data to not slow down the boot process.

Sadly the problem does not seem to be a theoretical one and it seems people are not aware of the dangers a shared /etc/pacman.d/gnupg directory implies. This especially affects system images, virtual machines and installer images you can download.

For example the Arch Linux fork Manjaro ships a static private key on their installer image which also gets copied to the target system. The result is that all Manjaro users have the same "private" key and anybody can easily use this to sign any package that the package manager will accept. It might be even worse as users expect package verification is set up correctly and care less about the source they install packages from.

If you just realized you made a terrible mistake, you can reinitialize your keyring with a new random and secret key using these commands:

$ rm -rf /etc/pacman.d/gnupg
$ pacman-key --init
$ pacman-key --populate archlinux

Keep in mind that package verification was essentially unavailable to this point and it’s up to you what consequences you draw from this. They only real solution is to re-install everything from a trusted source.

tl;dr: Make sure pacman’s PGP keys in /etc/pacman.d/gnupg are random and remain secret.

PHP 5.4.1 in – Suhosin out

Today I finally moved PHP 5.4.1 into our [testing] repository. Our previous tests of 5.4.0 were quite successful. In short PHP 5.4 is ready for production now and will be in our [extra] repository in a couple of days.

When I first packaged PHP 5.4.0 one of the most concerning issues was the lack of a compatible Suhosin patch and extension. Unfortunately this situation has not changed within the last two months: there is still no information if or when Suhosin will be released for recent versions of PHP. Therefor I decided to remove the Suhosin patch and its extension from our repository. To not break compatibility on minor updates the current version in [extra], PHP 5.3.11, includes a patched version of Suhosin; the last official one was dedicated for 5.3.9.

Suhosin is a safe guard for scripts and PHP itself. It consists of  patch to PHP and an extension which can be used independently. The patch aims to protect the PHP core against buffer overflows and string format vulnerabilities. Scripts written in PHP are protected by the Suhosin extension. It can be configured to disallow or limit certain user input. There is also a feature to transparently encrypt cookie and session data. More of Suhosin’s features are described on its website.

Certainly some of Suhosin’s features are a good idea as they might protect you in case PHP itself or scripts fail. This is why it was added years ago when I took over maintainership of PHP. Fortunately some of its features got added to PHP itself over time. But today we are confronted with several problems:

  • There is no Suhosin for PHP 5.4. This means we would be stuck at 5.3 and cannot ship the most recent packages which is one of the goals of Arch Linux.
  • The upstream project has become less and less active in the past years. There are certain indicators which make Suhosin an unhealthy open source project: There is still only one main contributor, the development process is mostly closed, there is no public repository for the patch, no bug tracker, the forums are in maintenance mode for years and the latest news entry is from 2007. Of course things might be different in the background, but this is the situation which is publicly visible.
  • The delays between PHP releases and their Suhosin counterparts are increasing. I often have to patch the Suhosin patch itself to use it with latest minor updates of PHP. This leaves us with a PHP version that is neither supported by the PHP project nor the Suhosin authors. While these modifications are mostly trivial I cannot exclude the possibility that I break PHP in a dangerous way. The situation is even worse on major update like we have now. Adjusting Suhosin to PHP 5.4 is not trivial. In addition to this testing development releases or release candidates often means that you have to disable Suhosin.
  • There are no plans of the project to ever move any of Suhosin’s features into PHP itself. In fact the Suhosin author is strictly against such a move.

In general we have a “Do not patch” policy in Arch and try to keep our packages as close to upstream as possible. In its current state I no longer have any excuse to break this rule for Suhosin; in fact I now have a lot of arguments which support this policy. So even if a new version of Suhosin was released tomorrow, next week or in a few months, it wont be re-added to our PHP packages. The upstream projects would need some changes to its process to be reliable again.The relationship with the PHP project itself has to improve and it should at least be tried to apply certain features into the PHP core project.

Verify all the packages

Once you have verified the finger prints of the five master keys you might want to actually configure pacman to verify packages. While this was already possible in the past, it is now way easier to get started thanks to pacman 4.0.3 and the keyring package.

If you already have played around with signature verification you could start from scratch by (re)moving the /etc/pacman.d/gnupg directory. I also assume that you have disabled signature checking by setting SigLevel = Never in /etc/pacman.conf.

All you need to do now is to install the archlinux-keyring package, initialize your local pacman keyring, finally import all the packager keys and set the trust level for the master keys. This is achieved by the following commands:

# pacman -Syu archlinux-keyring
# pacman-key --init
# pacman-key --populate archlinux

The last command will ask you to confirm each of the master keys. Now you can set SigLevel to PackageRequired for all official repositories in /etc/pacman.conf. For example:

SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

From now on pacman will check the signatures of all packages before installation and the keys are automatically managed by the keyring package.

Note: You can also set the SigLevel globally in the [options] section. The downside is that you wont be able to install your own packages using pacman -U anymore if they are not signed by a trusted key. Future versions of pacman will have a separate SigLevel option for packages that are installed from you local hard drive.

Trust the Master Keys

All our packages are finally signed, pacman itself and our development tools are able to create and verify signatures for some time now. There are still some bits and pieces missing untill we can have package verification enabled by default, but we are almost there.

At some point in time you might want to switch from an untrusted setup to one that magically verifies every package. Sadly it wont be that easy and to be entirely correct paranoid users will even need to reinstall from a signed medium (which does not exist yet) as it is hard to tell if your system was already compromised. I assume most users will instead hope for the best and skip this step.

While pacman can tell you if a package was signed by a private key to a given public key, it will be up to you to decide whether that key can be trusted or not. Due to the nature of the web of trust used in PGP you wont need to verify the key of every single developer. Instead you will only need to trust five of us who have created so called master keys.

Graph of the Arch Linux web of trust

These master keys wont be used to sign packages directly but to sign the keys of every packager. If a packager key is signed by at least three of those master keys PGP will automatically trust this key as well. To sum things up, you will only need to verify five keys once and you are done. You neither need to care about verifying each key nor to kee track of developers joining or leaving the team.

You will find the finger prints of all five master keys on our website. However, it is certainly a good idea to check other sources as well. For all what we know that site could already be compromised. The more clues you can gather to verify a key the better. Here are some hints about what you could verify. But don’t stick to that list, be creative.

  1. Make sure you are accessing that master key website via https.
  2. Verify that the SSL certificate is valid.
  3. You may also check that the certificate is issued by StartCom Ltd. to Aaron Griffin.

Each key holder has also signed their master key with their packager key. Therefore it is useful (and probably easier) to verify these packager keys as well. Let’s see what we can find out about these keys

  1. Some packager keys are verified by CAcert. If you trust those you now know that their email address and name match a real person. Keep in mind that this does not mean that this person is authorized to create packages for Arch Linux.
  2. Some of us have their own homepage where we also publish our keys.:
  3. In that case you can check whether the whois entry matches the person’s name.
  4. If they provide https access check their certificate and the name it was issued to.
  5. Check if that homepage is the one that these developers advertise in their signatures, forum profiles etc..
  6. If they sign their mails to the mailing list you can verify these as well. See if you can find mails from different time periods.
  7. Be a stalker, use your favorite search engine to look up their names, mail address, homepage addresses but also key fingerprints and ids.

These are some basic checks you can quickly do online with a reasonable amount of effort. Of course none of these checks are absolute and even if you do them all you cannot be certain. At some point you will have to ask yourself how much effort and money an attacker would invest in compromising Arch Linux users. This is a general rule for security related topics: you may not be able to make an attack impossible, but you can try to make it too expensive.

Last but not least, if you are too lazy to verify all this yourself, ask someone you trust instead. It is a web of trust after all. This also means that if you went ahead and verified all master keys as good as possible, tell people about it. You may just write a brief blog article about it and quote the key finger prints.

Of course the best way to let everybody know that you have verified someones key is to sign it using your own. But I would advice you to only do so if you have met that key owner in person. That also means: if you ever meet a developer or trusted user: ask them to sign each others key.

I am leaving you with a list of short cuts to start with:

Arch before it was cool

Ten years ago Judd released the first version of Arch Linux. This is quite an age for a Linux distribution and we are still rolling. Allan just posted a brief history of mile stones that were important to him. Inspired by this nice idea…I am going to do exactly the same.

It has been a long time but it seems I started a more serious look at Linux back in April 2003. That time I joined linuxforen.de.

In June 2004 I came in contact with Arch when I was looking for a lightweight and technical easy distribution with rolling releases.

Here is the Arch Linux history from my perspective:

2004-06-26 I asked for a new distribution at linuxforen.de and tried Arch for the first time.
2004-07-05 I registered at our great forums.
2004-12-21 My first contribution was setting up a forum for German users. I used to code my own forum system back then and hosted several boards.
2006-10-13 I send my application as trusted user to tur-users. Eric was so kind to sponsor me.
2006-10-29 My application was accepted by most Trusted Users and I joined the team. I started contributing the first set of lib32 package to be able to play games like Doom 3 or Quake 4 on the x86_64 port of Arch.
2007-05-12 I sent my application to the Arch developers in order to help out with x86_64 and KDE.
2007-05-13 I was accepted as a developer along with Eric.
2007-06-30 I wrote an article for the Linux User magazine about Arch Linux. People could actually buy this along with a DVD at their kiosk.
2007-07-10 We started to reorganize our repositories. The [core] repository and its signoff policy was born.
2007-08-26 First meetup of some developers at FrOSCon. Most people didn’t know about Arch yet and only very few found us at a distant room within the conference building.
2007-09-26 I started breaking PHP.
2007-10-01 Judd resigned and handed the leadership over to Aaron.
2007-12-19 I resigned as a Trusted User; in order to concentrate on my role as developer but also to keep the TU group independent from the developers.
2008-07-27 I replaced KDE 3 with KDE 4 and broke everything. I got a lot of mails from people who were really happy about that.
2008-08-24 Again, it was time for FrOSCon. We now had our own booth and a developer room. Thomas and I even gave an interview.
2008-11-06 I launched the pkgstats project. Statistics are still available at a “temporary page“.
2009-02-05 I started maintaining our wiki setup. I already did this for archlinux.de, so I made it independent from the language and the site it was running on.
2009-08-03 I introduced a split up version of KDE.
2009-08-26 We met again at FrOSCon.
2009-09-22 I also split PHP packages.
2010-02-10 “I am done with packaging” was the subject of a mail I sent to the arch-dev mailing list. I wanted to concentrate on our scripts and workflow in general. As such I took over maintainership of dbscripts and devtools. In the process I mainly dropped KDE and Andrea took the lead here.
2010-03-23 Packages could be compressed with xz to reduce their size significantly by about 30%.
2010-06-28 I took over maintainer ship of our forums.
2010-08-11 I started working on the concept of a [staging] repository again. I finally got enough positive feedback to get this implemented a long with a package pool.
2010-08-30 FrOSCon 2010 and we were there.
2010-09-06 I finished support for a package pool which makes moving packages between repositories a matter for adjusting some symlinks and updateing the db files. This improved the usage of the [testing] repository a lot.
2011-05-11 I uploaded the first signed package to the repository.

These are some information about my history at Arch I could gather in a reasonable amount of time. It has been fun to dig through all these old mails. Some information are hard to get or lost forever as we changed our mailing lists, forums, wiki and even switched from cvs to svn in the meantime.

I probably could have added dates to the timeline forever. But I’ll just stop here to keep it less boring. If I had missed anything important, let me know. Also I didn’t add other conference we had attended like last years FrOSCon, OpenRheinRuhr or FOSDEM as I simply have no pictures of those.

At this point I am still working on getting PHP 5.4 ready for action and helping to finalize the package signing process. I am also working on repository management and on the package building process in general. There will be separate articles about these ideas.

Arch Linux 0.1 and pacman 1.1

If you like to have a look at the very first version of Arch: I have uploaded arch-0.1-full-i686.iso to our mirrors. It’s md5sum is: