informaton – Page 2 – Pierre Schmitz

2013-04-282021-09-01

Farewell i686!

The first 64 Bit x86 CPU was released ten years ago: the AMD Opteron. Intel followed with compatible processors about one year later. Another two years later, in 2006, a first port of Arch Linux to the x86_64 platform was available. Starting outside the official repositories, the new architecture was merged into the Arch project and became a first class citizen within the following years. Right from the start it was designed as a pure 64 Bit system and kept in sync with the i686 tree. That also meant no multilib support at first and every package was compiled for x86_64.

Of course getting there was not that easy. Our repositories and tools like pacman did not have any concept of different architectures. While most software would just work fine when compiled for x86_64, we occasionally ran into issues that were specific to the 64 Bit architecture.

Years later the relation between 32 and 64 bit architecture seem to have switched roles. Developers have chosen x86_64 as their primary platform, our i686 packages seem to get tested by less users and we see issues specific to the 32 Bit architecture. We recently relaxed our signoff policy for i686 packages in order to counteract this trend and avoid holding back development due to lack of testing.

To back these observations with some numbers, I analyzed the architecture submitted by users via pkgstats over the past few years. Even if we assume that these data are not absolutely accurate and I could not find all the data gathered between 2009 and 2010, the results are clear: While the x86_64 version of Arch Linux was installed by only 20% of our users in 2008, its usage grew to 80% in 2013. It is fair to say that within the next years i686 usage will fade away.

For a few month we also collect data about the actual CPU architecture users have. Right now about 93% operate a CPU that would be capable of running a 64 Bit system. There is not much data yet, but it seems to be a slowly rising number.

We are pretty close to the point where i686 Arch Linux installations have been almost completely replaced by x86_64. This of course means that it might no longer be practical to provide support for an architecture that is barely used. The question is not if 32 bit support might be dropped, but when.

Don’t Panic! I don’t see us dropping i686 support this year, but I would not dare to predict anything beyond that date. As always these decision are not based on business or politics. We will see when i686 usage will drop below a critical number and developers loose interest in packaging and supporting that architecture.

2013-04-192021-09-03

Random but important

Security in computer programs is often based on some kind of secret. It might be a key, passphrase or the popular password. It is also equally important that this key is not only secret but also random and cannot be easily guessed.

The same concept applies to our package signing method. The security model which is used to ensure validity of our packages is directly based on PGP. In this trust model you mark a certain key as trusted by signing it with your very own private key.

On Arch Linux we ship the archlinux-keyring package to set up and maintain a set of packager keys and their web of trust. On install a new key pair is generated and the five master keys are locally signed. Any package that is signed by a key that itself is signed by at least three of the master keys is considered valid. This also implies that anything signed directly by that private key is automatically valid as well.

The keyring pacman uses is stored in /etc/pacman.d/gnupg. There is no magic involved here and you can even directly use it with the gpg command by appending the --homedir /etc/pacman.d/gnupg parameter.

It is important to underline that the PGP key pair stored there is unique to your machine. It is not part of any package but generated on install and random. An attacker who is able to obtain or guess that key can sign packages that pacman will happily accept as valid. Be aware of this when you create backups, clone your machines or setup virtual machines. It might be a bad idea to share the same private key on different machines.

For our live system image we ship a service that initializes a new random key on every boot. It is stored in RAM using tmpfs and therefor not persistent. We make use of the haveged service to create enough random data to not slow down the boot process.

Sadly the problem does not seem to be a theoretical one and it seems people are not aware of the dangers a shared /etc/pacman.d/gnupg directory implies. This especially affects system images, virtual machines and installer images you can download.

For example the Arch Linux fork Manjaro ships a static private key on their installer image which also gets copied to the target system. The result is that all Manjaro users have the same “private” key and anybody can easily use this to sign any package that the package manager will accept. It might be even worse as users expect package verification is set up correctly and care less about the source they install packages from.

If you just realized you made a terrible mistake, you can reinitialize your keyring with a new random and secret key using these commands:

$ rm -rf /etc/pacman.d/gnupg
$ pacman-key --init
$ pacman-key --populate archlinux

Keep in mind that package verification was essentially unavailable to this point and it’s up to you what consequences you draw from this. They only real solution is to re-install everything from a trusted source.

tl;dr: Make sure pacman’s PGP keys in /etc/pacman.d/gnupg are random and remain secret.

2012-04-292021-09-01

PHP 5.4.1 in – Suhosin out

Today I finally moved PHP 5.4.1 into our [testing] repository. Our previous tests of 5.4.0 were quite successful. In short PHP 5.4 is ready for production now and will be in our [extra] repository in a couple of days.

When I first packaged PHP 5.4.0 one of the most concerning issues was the lack of a compatible Suhosin patch and extension. Unfortunately this situation has not changed within the last two months: there is still no information if or when Suhosin will be released for recent versions of PHP. Therefor I decided to remove the Suhosin patch and its extension from our repository. To not break compatibility on minor updates the current version in [extra], PHP 5.3.11, includes a patched version of Suhosin; the last official one was dedicated for 5.3.9.

Suhosin is a safe guard for scripts and PHP itself. It consists of patch to PHP and an extension which can be used independently. The patch aims to protect the PHP core against buffer overflows and string format vulnerabilities. Scripts written in PHP are protected by the Suhosin extension. It can be configured to disallow or limit certain user input. There is also a feature to transparently encrypt cookie and session data. More of Suhosin’s features are described on its website.

Certainly some of Suhosin’s features are a good idea as they might protect you in case PHP itself or scripts fail. This is why it was added years ago when I took over maintainership of PHP. Fortunately some of its features got added to PHP itself over time. But today we are confronted with several problems:

There is no Suhosin for PHP 5.4. This means we would be stuck at 5.3 and cannot ship the most recent packages which is one of the goals of Arch Linux.
The upstream project has become less and less active in the past years. There are certain indicators which make Suhosin an unhealthy open source project: There is still only one main contributor, the development process is mostly closed, there is no public repository for the patch, no bug tracker, the forums are in maintenance mode for years and the latest news entry is from 2007. Of course things might be different in the background, but this is the situation which is publicly visible.
The delays between PHP releases and their Suhosin counterparts are increasing. I often have to patch the Suhosin patch itself to use it with latest minor updates of PHP. This leaves us with a PHP version that is neither supported by the PHP project nor the Suhosin authors. While these modifications are mostly trivial I cannot exclude the possibility that I break PHP in a dangerous way. The situation is even worse on major update like we have now. Adjusting Suhosin to PHP 5.4 is not trivial. In addition to this testing development releases or release candidates often means that you have to disable Suhosin.
There are no plans of the project to ever move any of Suhosin’s features into PHP itself. In fact the Suhosin author is strictly against such a move.

In general we have a “Do not patch” policy in Arch and try to keep our packages as close to upstream as possible. In its current state I no longer have any excuse to break this rule for Suhosin; in fact I now have a lot of arguments which support this policy. So even if a new version of Suhosin was released tomorrow, next week or in a few months, it wont be re-added to our PHP packages. The upstream projects would need some changes to its process to be reliable again.The relationship with the PHP project itself has to improve and it should at least be tried to apply certain features into the PHP core project.

2012-04-152021-09-01

Verify all the packages

Once you have verified the finger prints of the five master keys you might want to actually configure pacman to verify packages. While this was already possible in the past, it is now way easier to get started thanks to pacman 4.0.3 and the keyring package.

If you already have played around with signature verification you could start from scratch by (re)moving the /etc/pacman.d/gnupg directory. I also assume that you have disabled signature checking by setting SigLevel = Never in /etc/pacman.conf.

All you need to do now is to install the archlinux-keyring package, initialize your local pacman keyring, finally import all the packager keys and set the trust level for the master keys. This is achieved by the following commands:

# pacman -Syu archlinux-keyring
# pacman-key --init
# pacman-key --populate archlinux

The last command will ask you to confirm each of the master keys. Now you can set SigLevel to PackageRequired for all official repositories in /etc/pacman.conf. For example:

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

From now on pacman will check the signatures of all packages before installation and the keys are automatically managed by the keyring package.

Note: You can also set the SigLevel globally in the [options] section. The downside is that you wont be able to install your own packages using pacman -U anymore if they are not signed by a trusted key. Future versions of pacman will have a separate SigLevel option for packages that are installed from you local hard drive.

2012-04-152021-09-03

Trust the Master Keys

All our packages are finally signed, pacman itself and our development tools are able to create and verify signatures for some time now. There are still some bits and pieces missing until we can have package verification enabled by default, but we are almost there.

At some point in time you might want to switch from an untrusted setup to one that magically verifies every package. Sadly it wont be that easy and to be entirely correct paranoid users will even need to reinstall from a signed medium (which does not exist yet) as it is hard to tell if your system was already compromised. I assume most users will instead hope for the best and skip this step.

While pacman can tell you if a package was signed by a private key to a given public key, it will be up to you to decide whether that key can be trusted or not. Due to the nature of the web of trust used in PGP you wont need to verify the key of every single developer. Instead you will only need to trust five of us who have created so called master keys.

These master keys wont be used to sign packages directly but to sign the keys of every packager. If a packager key is signed by at least three of those master keys PGP will automatically trust this key as well. To sum things up, you will only need to verify five keys once and you are done. You neither need to care about verifying each key nor to keep track of developers joining or leaving the team.

You will find the finger prints of all five master keys on our website. However, it is certainly a good idea to check other sources as well. For all what we know that site could already be compromised. The more clues you can gather to verify a key the better. Here are some hints about what you could verify. But don’t stick to that list, be creative.

Make sure you are accessing that master key website via https.
Verify that the SSL certificate is valid.
You may also check that the certificate is issued by StartCom Ltd. to Aaron Griffin.

Each key holder has also signed their master key with their packager key. Therefore it is useful (and probably easier) to verify these packager keys as well. Let’s see what we can find out about these keys

Some packager keys are verified by CAcert. If you trust those you now know that their email address and name match a real person. Keep in mind that this does not mean that this person is authorized to create packages for Arch Linux.
Some of us have their own homepage where we also publish our keys.:
In that case you can check whether the whois entry matches the person’s name.
If they provide https access check their certificate and the name it was issued to.
Check if that homepage is the one that these developers advertise in their signatures, forum profiles etc..
If they sign their mails to the mailing list you can verify these as well. See if you can find mails from different time periods.
Be a stalker, use your favorite search engine to look up their names, mail address, homepage addresses but also key fingerprints and ids.

These are some basic checks you can quickly do online with a reasonable amount of effort. Of course none of these checks are absolute and even if you do them all you cannot be certain. At some point you will have to ask yourself how much effort and money an attacker would invest in compromising Arch Linux users. This is a general rule for security related topics: you may not be able to make an attack impossible, but you can try to make it too expensive.

Last but not least, if you are too lazy to verify all this yourself, ask someone you trust instead. It is a web of trust after all. This also means that if you went ahead and verified all master keys as good as possible, tell people about it. You may just write a brief blog article about it and quote the key finger prints.

Of course the best way to let everybody know that you have verified someones key is to sign it using your own. But I would advice you to only do so if you have met that key owner in person. That also means: if you ever meet a developer or trusted user: ask them to sign each others key.

I am leaving you with a list of short cuts to start with: