Trust the Master Keys

All our packages are finally signed, pacman itself and our development tools are able to create and verify signatures for some time now. There are still some bits and pieces missing until we can have package verification enabled by default, but we are almost there.

At some point in time you might want to switch from an untrusted setup to one that magically verifies every package. Sadly it wont be that easy and to be entirely correct paranoid users will even need to reinstall from a signed medium (which does not exist yet) as it is hard to tell if your system was already compromised. I assume most users will instead hope for the best and skip this step.

While pacman can tell you if a package was signed by a private key to a given public key, it will be up to you to decide whether that key can be trusted or not. Due to the nature of the web of trust used in PGP you wont need to verify the key of every single developer. Instead you will only need to trust five of us who have created so called master keys.

Graph of the Arch Linux web of trust

These master keys wont be used to sign packages directly but to sign the keys of every packager. If a packager key is signed by at least three of those master keys PGP will automatically trust this key as well. To sum things up, you will only need to verify five keys once and you are done. You neither need to care about verifying each key nor to keep track of developers joining or leaving the team.

You will find the finger prints of all five master keys on our website. However, it is certainly a good idea to check other sources as well. For all what we know that site could already be compromised. The more clues you can gather to verify a key the better. Here are some hints about what you could verify. But don’t stick to that list, be creative.

  1. Make sure you are accessing that master key website via https.
  2. Verify that the SSL certificate is valid.
  3. You may also check that the certificate is issued by StartCom Ltd. to Aaron Griffin.

Each key holder has also signed their master key with their packager key. Therefore it is useful (and probably easier) to verify these packager keys as well. Let’s see what we can find out about these keys

  1. Some packager keys are verified by CAcert. If you trust those you now know that their email address and name match a real person. Keep in mind that this does not mean that this person is authorized to create packages for Arch Linux.
  2. Some of us have their own homepage where we also publish our keys.:
  3. In that case you can check whether the whois entry matches the person’s name.
  4. If they provide https access check their certificate and the name it was issued to.
  5. Check if that homepage is the one that these developers advertise in their signatures, forum profiles etc..
  6. If they sign their mails to the mailing list you can verify these as well. See if you can find mails from different time periods.
  7. Be a stalker, use your favorite search engine to look up their names, mail address, homepage addresses but also key fingerprints and ids.

These are some basic checks you can quickly do online with a reasonable amount of effort. Of course none of these checks are absolute and even if you do them all you cannot be certain. At some point you will have to ask yourself how much effort and money an attacker would invest in compromising Arch Linux users. This is a general rule for security related topics: you may not be able to make an attack impossible, but you can try to make it too expensive.

Last but not least, if you are too lazy to verify all this yourself, ask someone you trust instead. It is a web of trust after all. This also means that if you went ahead and verified all master keys as good as possible, tell people about it. You may just write a brief blog article about it and quote the key finger prints.

Of course the best way to let everybody know that you have verified someones key is to sign it using your own. But I would advice you to only do so if you have met that key owner in person. That also means: if you ever meet a developer or trusted user: ask them to sign each others key.

I am leaving you with a list of short cuts to start with:

Arch before it was cool

Ten years ago Judd released the first version of Arch Linux. This is quite an age for a Linux distribution and we are still rolling. Allan just posted a brief history of mile stones that were important to him. Inspired by this nice idea…I am going to do exactly the same.

It has been a long time but it seems I started a more serious look at Linux back in April 2003. That time I joined

In June 2004 I came in contact with Arch when I was looking for a lightweight and technical easy distribution with rolling releases.

Here is the Arch Linux history from my perspective:

2004-06-26 I asked for a new distribution at and tried Arch for the first time.
2004-07-05 I registered at our great forums.
2004-12-21 My first contribution was setting up a forum for German users. I used to code my own forum system back then and hosted several boards.
2006-10-13 I send my application as trusted user to tur-users. Eric was so kind to sponsor me.
2006-10-29 My application was accepted by most Trusted Users and I joined the team. I started contributing the first set of lib32 package to be able to play games like Doom 3 or Quake 4 on the x86_64 port of Arch.
2007-05-12 I sent my application to the Arch developers in order to help out with x86_64 and KDE.
2007-05-13 I was accepted as a developer along with Eric.
2007-06-30 I wrote an article for the Linux User magazine about Arch Linux. People could actually buy this along with a DVD at their kiosk.
2007-07-10 We started to reorganize our repositories. The [core] repository and its signoff policy was born.
2007-08-26 First meetup of some developers at FrOSCon. Most people didn’t know about Arch yet and only very few found us at a distant room within the conference building.
2007-09-26 I started breaking PHP.
2007-10-01 Judd resigned and handed the leadership over to Aaron.
2007-12-19 I resigned as a Trusted User; in order to concentrate on my role as developer but also to keep the TU group independent from the developers.
2008-07-27 I replaced KDE 3 with KDE 4 and broke everything. I got a lot of mails from people who were really happy about that.
2008-08-24 Again, it was time for FrOSCon. We now had our own booth and a developer room. Thomas and I even gave an interview.
2008-11-06 I launched the pkgstats project. Statistics are still available.
2009-02-05 I started maintaining our wiki setup. I already did this for, so I made it independent from the language and the site it was running on.
2009-08-03 I introduced a split up version of KDE.
2009-08-26 We met again at FrOSCon.
2009-09-22 I also split PHP packages.
2010-02-10 “I am done with packaging” was the subject of a mail I sent to the arch-dev mailing list. I wanted to concentrate on our scripts and workflow in general. As such I took over maintainership of dbscripts and devtools. In the process I mainly dropped KDE and Andrea took the lead here.
2010-03-23 Packages could be compressed with xz to reduce their size significantly by about 30%.
2010-06-28 I took over maintainer ship of our forums.
2010-08-11 I started working on the concept of a [staging] repository again. I finally got enough positive feedback to get this implemented a long with a package pool.
2010-08-30 FrOSCon 2010 and we were there.
2010-09-06 I finished support for a package pool which makes moving packages between repositories a matter for adjusting some symlinks and updateing the db files. This improved the usage of the [testing] repository a lot.
2011-05-11 I uploaded the first signed package to the repository.

These are some information about my history at Arch I could gather in a reasonable amount of time. It has been fun to dig through all these old mails. Some information are hard to get or lost forever as we changed our mailing lists, forums, wiki and even switched from cvs to svn in the meantime.

I probably could have added dates to the timeline forever. But I’ll just stop here to keep it less boring. If I had missed anything important, let me know. Also I didn’t add other conference we had attended like last years FrOSCon, OpenRheinRuhr or FOSDEM as I simply have no pictures of those.

At this point I am still working on getting PHP 5.4 ready for action and helping to finalize the package signing process. I am also working on repository management and on the package building process in general. There will be separate articles about these ideas.

Arch Linux 0.1 and pacman 1.1

If you like to have a look at the very first version of Arch: I have uploaded arch-0.1-full-i686.iso to our mirrors. It’s md5sum is:


PHP 5.4.0 – Try this at home

You might have noticed that we still do not have the latest version of PHP in our repositories. Unfortunately there are still several issues which prevents me from updating to 5.4.0 right away.

Concerning the php package itself the updated version of the Suhosin patch is not available yet.

In addition to this, several modules do not work with the new PHP yet. Among them is of course the suhosin extension but also APC and xdebug. The state of other extensions still needs to be researched; if you are lucky simply recompiling will do it.

While I obviously cannot recommend the use of PHP 5.4.0 on any server or production system yet, I would consider it a good idea to start testing your setup and scripts for compatibility. Also at Arch we like to play with the newest features and get hit by bugs nobody has seen before.

Therefore I set up a repository for all administrators and developers who like their packages broken. These come without suhosin (patch nor extension) and are available for both i686 and x86_64. I also packaged some modules like memcache(d), apc and geoip. Note that I will base these builds upon the [testing] repository. This might not be a problem right now, but once you see a our stock version of PHP in [testing] these package wont work without the usage of [testing] as well.

To use this repo put the following lines on top of all other repositories (including [testing]) in pacman.conf:

Server =$repo/os/$arch

For those who are all over the brand new signing stuff: Yes, all these packages are signed using my Arch packager key.

Before you get started have a brief look at the changelog and migration guide which are provided upstream. Then make sure you get all changes in php.ini merged. The easiest was to do so is probably starting with the packaged one and adjust the changes you need. For testing properly it is important to enable error reporting:

error_reporting = E_ALL
display_errors = On
display_startup_errors = On

Note that E_ALL now also includes E_STRICT by default. If you are testing third party code make sure it does not override these settings by either .htaccess or calling error_reporting(). If you find any error this way: You know the drill, report it to the upstream projects.

If you have any suggestions about the packages or configuration, let me know. Especially drop a line if you found any issues regarding extensions or web apps. It’ll help me a lot to know how many breakage such an update will cause.

Hello World

You must be very desperate to read a “Hello World” post on an empty blog using the default WordPress theme. However, I am glad you are still reading. In future you will find articles about Arch Linux development, PHP and anything related here.

There is nothing more to say here, but an article with just one paragraph looks kind of lame, don’t you agree? If you are still with me you should subscribe to the RSS feed which is hidden somewhere in order to get notified about more exciting articles.