Farewell i686!

The first 64 Bit x86 CPU was released ten years ago: the AMD Opteron. Intel followed with compatible processors about one year later. Another two years later, in 2006, a first port of Arch Linux to the x86_64 platform was available. Starting outside the official repositories, the new architecture was merged into the Arch project and became a first class citizen within the following years. Right from the start it was designed as a pure 64 Bit system and kept in sync with the i686 tree. That also meant no multilib support at first and every package was compiled for x86_64.

Of course getting there was not that easy. Our repositories and tools like pacman did not have any concept of different architectures. While most software would just work fine when compiled for x86_64, we occasionally ran into issues that were specific to the 64 Bit architecture.

Years later the relation between 32 and 64 bit architecture seem to have switched roles.  Developers have chosen x86_64 as their primary platform, our i686 packages seem to get tested by less users and we see issues specific to the 32 Bit architecture. We recently relaxed our signoff policy for i686 packages in order to counteract this trend and avoid holding back development due to lack of testing.

To back these observations with some numbers, I analyzed the architecture submitted by users via pkgstats over the past few years. Even if we assume that these data are not absolutely accurate and I could not find all the data gathered between 2009 and 2010, the results are clear: While the x86_64 version of Arch Linux was installed by only 20% of our users in 2008, its usage grew to 80% in 2013. It is fair to say that within the next years i686 usage will fade away.

Architecture Usage by Date
Architecture Usage by Date

For a few month we also collect data about the actual CPU architecture users have. Right now about 93% operate a CPU that would be capable of running a 64 Bit system. There is not much data yet, but it seems to be a slowly rising number.

CPU Architecture Usage by Date
CPU Architecture Usage by Date

We are pretty close to the point where i686 Arch Linux installations have been almost completely replaced by x86_64. This of course means that it might no longer be practical to provide support for an architecture that is barely used. The question is not if 32 bit support might be dropped,  but when.

Don’t Panic! I don’t see us dropping i686 support this year, but I would not dare to predict anything beyond that date. As always these decision are not based on business or politics. We will see when i686 usage will drop below a critical number and developers loose interest in packaging and supporting that architecture.

Random but important

Security in computer programs is often based on some kind of secret. It might be a key, passphrase or the popular password. It is also equally important that this key is not only secret but also random and cannot be easily guessed.

The same concept applies to our package signing method. The security model which is used to ensure validity of our packages is directly based on PGP. In this trust model you mark a certain key as trusted by signing it with your very own private key.

On Arch Linux we ship the archlinux-keyring package to set up and maintain a set of packager keys and their web of trust. On install a new key pair is generated and the five master keys are locally signed. Any package that is signed by a key that itself is signed by at least three of the master keys is considered valid. This also implies that anything signed directly by that private key is automatically valid as well.

The keyring pacman uses is stored in /etc/pacman.d/gnupg. There is no magic involved here and you can even directly use it with the gpg command by appending the --homedir /etc/pacman.d/gnupg parameter.

It is important to underline that the PGP key pair stored there is unique to your machine. It is not part of any package but generated on install and random. An attacker who is able to obtain or guess that key can sign packages that pacman will happily accept as valid. Be aware of this when you create backups, clone your machines or setup virtual machines. It might be a bad idea to share the same private key on different machines.

For our live system image we ship a service that initializes a new random key on every boot. It is stored in RAM using tmpfs and therefor not persistent. We make use of the haveged service to create enough random data to not slow down the boot process.

Sadly the problem does not seem to be a theoretical one and it seems people are not aware of the dangers a shared /etc/pacman.d/gnupg directory implies. This especially affects system images, virtual machines and installer images you can download.

For example the Arch Linux fork Manjaro ships a static private key on their installer image which also gets copied to the target system. The result is that all Manjaro users have the same "private" key and anybody can easily use this to sign any package that the package manager will accept. It might be even worse as users expect package verification is set up correctly and care less about the source they install packages from.

If you just realized you made a terrible mistake, you can reinitialize your keyring with a new random and secret key using these commands:

$ rm -rf /etc/pacman.d/gnupg
$ pacman-key --init
$ pacman-key --populate archlinux

Keep in mind that package verification was essentially unavailable to this point and it’s up to you what consequences you draw from this. They only real solution is to re-install everything from a trusted source.

tl;dr: Make sure pacman’s PGP keys in /etc/pacman.d/gnupg are random and remain secret.

How your parents installed Arch

For those who think the new Arch Install Scripts are no longer the Arch Way, here are the original install instructions of Arch Linux 0.1 published in March 2002.

                              ARCH LINUX #VERSION#
                     SHORT & SWEET INSTALL INSTRUCTIONS
                 ==========================================

1.  Make your swap/root partitions
      # fdisk /dev/discs/disc0/disc

2.  Make your target filesystems
      # mkswap /dev/discs/disc0/partX
      # mkreiserfs /dev/discs/disc0/partY

3.  Activate your swap partition
      # swapon /dev/discs/disc0/partX

4.  Mount your target root filesystem under /mnt
      # mount /dev/discs/disc0/partX /mnt

5.  Initialize the pacman database
      # mkdir -p /mnt/var/lib/pacman
      # touch /mnt/var/lib/pacman/pacman.db

6.  Add the filesystem package from /arch/pkg/filesystem-0.X-Y.tar.gz
      # cd /arch/pkg
      # pacman --add -r /mnt filesystem-0.X-Y.tar.gz

7.  Mount any other data partitions you may have created
      # mount /dev/discs/disc0/partX /mnt/home
      # mount /dev/discs/disc0/partY /mnt/usr

8.  Install some base packages
      # cd /arch
      # ./installworld /mnt

9.  Uncompress the linux source to /mnt/usr/src
      # cd /mnt/usr/src && tar zxvf /arch/linux-2.4.XX.tar.gz

10. Mount a dev and proc under your new system and chroot
      # mount -t devfs none /mnt/dev
      # mount -t proc none /mnt/proc
      # chroot /mnt /bin/bash

11. Build & install your new kernel from /usr/src/linux
      NOTE: make sure you ask for "/dev file system support" and
            "Automatically mount at boot", since we use DevFS.

12. Install a lilo bootloader
      # vi /etc/lilo.conf
      # lilo

13. Edit settings
      # vi /etc/rc.conf
      # vi /etc/resolv.conf
      # vi /etc/fstab
      # vi /etc/modules.conf (if needed)

14. Exit your chroot shell
      # exit

15. Reboot!

16. Verify success and install any additional packages you want.
      # pacman -A /mnt/cd/arch/pkg/whatever-1.1-1.pkg.tar.gz

PHP 5.4.1 in – Suhosin out

Today I finally moved PHP 5.4.1 into our [testing] repository. Our previous tests of 5.4.0 were quite successful. In short PHP 5.4 is ready for production now and will be in our [extra] repository in a couple of days.

When I first packaged PHP 5.4.0 one of the most concerning issues was the lack of a compatible Suhosin patch and extension. Unfortunately this situation has not changed within the last two months: there is still no information if or when Suhosin will be released for recent versions of PHP. Therefor I decided to remove the Suhosin patch and its extension from our repository. To not break compatibility on minor updates the current version in [extra], PHP 5.3.11, includes a patched version of Suhosin; the last official one was dedicated for 5.3.9.

Suhosin is a safe guard for scripts and PHP itself. It consists of  patch to PHP and an extension which can be used independently. The patch aims to protect the PHP core against buffer overflows and string format vulnerabilities. Scripts written in PHP are protected by the Suhosin extension. It can be configured to disallow or limit certain user input. There is also a feature to transparently encrypt cookie and session data. More of Suhosin’s features are described on its website.

Certainly some of Suhosin’s features are a good idea as they might protect you in case PHP itself or scripts fail. This is why it was added years ago when I took over maintainership of PHP. Fortunately some of its features got added to PHP itself over time. But today we are confronted with several problems:

  • There is no Suhosin for PHP 5.4. This means we would be stuck at 5.3 and cannot ship the most recent packages which is one of the goals of Arch Linux.
  • The upstream project has become less and less active in the past years. There are certain indicators which make Suhosin an unhealthy open source project: There is still only one main contributor, the development process is mostly closed, there is no public repository for the patch, no bug tracker, the forums are in maintenance mode for years and the latest news entry is from 2007. Of course things might be different in the background, but this is the situation which is publicly visible.
  • The delays between PHP releases and their Suhosin counterparts are increasing. I often have to patch the Suhosin patch itself to use it with latest minor updates of PHP. This leaves us with a PHP version that is neither supported by the PHP project nor the Suhosin authors. While these modifications are mostly trivial I cannot exclude the possibility that I break PHP in a dangerous way. The situation is even worse on major update like we have now. Adjusting Suhosin to PHP 5.4 is not trivial. In addition to this testing development releases or release candidates often means that you have to disable Suhosin.
  • There are no plans of the project to ever move any of Suhosin’s features into PHP itself. In fact the Suhosin author is strictly against such a move.

In general we have a “Do not patch” policy in Arch and try to keep our packages as close to upstream as possible. In its current state I no longer have any excuse to break this rule for Suhosin; in fact I now have a lot of arguments which support this policy. So even if a new version of Suhosin was released tomorrow, next week or in a few months, it wont be re-added to our PHP packages. The upstream projects would need some changes to its process to be reliable again.The relationship with the PHP project itself has to improve and it should at least be tried to apply certain features into the PHP core project.

Verify all the packages

Once you have verified the finger prints of the five master keys you might want to actually configure pacman to verify packages. While this was already possible in the past, it is now way easier to get started thanks to pacman 4.0.3 and the keyring package.

If you already have played around with signature verification you could start from scratch by (re)moving the /etc/pacman.d/gnupg directory. I also assume that you have disabled signature checking by setting SigLevel = Never in /etc/pacman.conf.

All you need to do now is to install the archlinux-keyring package, initialize your local pacman keyring, finally import all the packager keys and set the trust level for the master keys. This is achieved by the following commands:

# pacman -Syu archlinux-keyring
# pacman-key --init
# pacman-key --populate archlinux

The last command will ask you to confirm each of the master keys. Now you can set SigLevel to PackageRequired for all official repositories in /etc/pacman.conf. For example:

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

From now on pacman will check the signatures of all packages before installation and the keys are automatically managed by the keyring package.

Note: You can also set the SigLevel globally in the [options] section. The downside is that you wont be able to install your own packages using pacman -U anymore if they are not signed by a trusted key. Future versions of pacman will have a separate SigLevel option for packages that are installed from you local hard drive.