pkgstats version 3: lookup package statistics from your terminal

pkgstats is a tool that gathers and analyses installed packages of Arch Linux users. It started as a small shell script back in 2008 and helps us among other things to determine which packages are no longer used but also which packages from the AUR are popular candidates.

Previously I rewrote the server part and added historical statistics per package and the ability to compare the popularity of several packages over time. It also introduced a public API which is now used by the pkgstats client.

The new pkgstats client

Today I released the initial version of pkgstats v3, the client that can be installed via pacman. While in the past it was only used to send usage data to the server, the cli has now an interface to search for package statistics and compare different packages.

Search for packages

You can now search for packages by using pkgstats search <query>

~> pkgstats search firefox
firefox                             77.57
firefox-developer-edition           7.42
firefox-i18n-en-us                  3.76
firefox-i18n-de                     3.60
firefox-i18n-fr                     2.08
firefox-ublock-origin               1.93
firefox-i18n-en-gb                  1.90
firefox-nightly                     1.58
firefox-adblock-plus                1.51
firefox-extension-https-everywhere  1.32

10 of 288 results

See more results at https://pkgstats.archlinux.de/packages#query=firefox

Show and compare packages

You can lookup the popularity of a single package or compare a number of packags at once. Simply type pkgstats show <package> [<package>...]

~> pkgstats show nodejs go php python rust
python         99.73
nodejs         65.26
go             61.34
php            33.10
rust           20.49

See more results at https://pkgstats.archlinux.de/compare/packages#packages=go,nodejs,php,python,rust

Direct links to web graphs

As you saw in the examples above, pkgstats now also prints an URL representing your query. Following these link renders a visual representation of historical data.

Searching for packages

Comparing packages over time

Submit your package list

As with previous versions of pkgstats a list of your installed packages can be submitted to the server. This is done on a weekly basis triggered by a systemd timer. If you like to inspect what data would be transmitted you can print the JSON output via pkgstats submit --dump-json

~> pkgstats submit --dump-json 
{
  "version": "3",
  "system": {
    "architecture": "x86_64"
  },
  "os": {
    "architecture": "x86_64"
  },
  "pacman": {
    "mirror": "https://mirror.pkgbuild.com/",
    "packages": [
      "a52dec",
      "aalib",
      "accountsservice",
      "acl",
       ...
      "zsh",
      "zstd",
      "zvbi",
      "zxing-cpp"
    ]
  }
}

Integrated help and shell completion

The new client comes with an integrated help command for each sub command and flag. The package also ships with shell completions for Bash, Zsh and Fish.

Feedback welcome

I rewrote the initial shell script in Go and released it as version 2.5. Version 3 is a continuation of this effort and represents just the beginning. If you have any suggestions to improve pkgstats or ideas for interesting and useful features please let me know.

Further information

You’ll find more information on the pkgstats home page and GitHub.

PHP 7.1 beta packages for Arch Linux

The first beta version of PHP 7.1 has been released and its time to have a look at the next iteration of the PHP 7 series. You will find a set of packages in my repository:

[php]
Server = https://repo.pierre-schmitz.com/$repo/os/$arch

Insert these lines on top of the other repository definitions in your /etc/pacman.conf. A copy of the PKGBUILDs I used to create these packages are available in my git repository.

Packaging

I intend to update these with beta versions and release candidates till the final release of PHP 7.1.0 later this year. Even though I will try to provide a smooth update path, please be prepared to encounter problems.

Despite of a having a new module API, third party modules seem to work fine after a simple rebuild, in contrast to our first contact with PHP 7. All these modules are available in my repository as well.

New features

With the new minor release we will get more improvements of the scalar and return type declarations introduced in PHP 7.0. My favorite new features are:

  • Nullable Types
    Being able to declare a type to be either specific or null was a missing feature in version 7.0 which lead people to not declare any type at all.
  • Void Return Type
    You are now able to declare a function to never return anything; another missing piece of the new return type declarations.
  • Iterable type
    We are finally able to declare a type that matches an array but also classes that implement the Traversable interface. In short it is anything you can use with foreach(). This means we no longer need to put primitive arrays into traversable objects if we like to use type hinting.
  • Class constant visibility modifiers
    Using constants internally is less awkward as we are now able to declare them with private visibility. People no longer need to abuse private properties to document that certain constants should not be used from a foreign context.

A complete list of changes can be found in the PHP 7.1 NEWS file. Also see the continuously updated UPGRADING file.

Testing and benchmarking

While PHP 7.1 is still under development the packages I provide are configured with production settings. Optimizations are turned on, all debugging functions and information are disabled and stripped from the binaries. This means you may use these to test and benchmark your applications and server setups.

Let me know of any issues and share your experiences with the first minor update of PHP 7.

Using OPcache to speed up your CLI scripts

With PHP 7 it is now possible to use OPcache efficiently. It can be configured to store the compiled byte code on disk and use it on the next run of a script. Complex command line scripts like Composer and PHPUnit tests will benefit from such a cache as it reduces startup time dramatically.

In addition to benefiting cli scripts, this cache is also used as second level cache in general. This should increase performance if the SHM backed OPcache runs out of memory or the PHP process gets restarted.

To enable the second level cache simply ensure the following ini directives are set:

zend_extension=opcache.so
opcache.enable=1
opcache.enable_cli=1
opcache.file_cache=/tmp/php-opcache

In this case I am going to store the byte code in /tmp which means it wont survive a reboot and will be kept in RAM by default. We now need to ensure that the mentioned directory /tmp/php-opcache is created by systemd on boot. Place the following content in a file called /etc/tmpfiles.d/php-cli-opcache.conf:

d /tmp/php-opcache 1777 root root 1d

This will also prune all files that are older than one day. You might want to adjust these values to your needs or even use a persistent storage in /var/tmp instead of /tmp.
To initially create this folder simply run

# systemd-tmpfiles --create /etc/tmpfiles.d/php-cli-opcache.conf

You may now check your configuration by running some cli scripts:

$ composer -V

The compiled byte code should now appear in /tmp/php-opcache:

$ tree /tmp/php-opcache               
/tmp/php-opcache
├── 896f8ffaef254102552141eb4ab1b214
│   └── usr
│       └── bin
│           └── composer.bin
└── 896f8ffaef254102552141eb4ab1b214phar:
    └── usr
        └── bin
            └── composer
                ├── bin
                │   └── composer.bin
                ├── src
                │   ├── bootstrap.php.bin
                │   └── Composer
                │       ├── Command
                │       │   ├── AboutCommand.php.bin
                │       │   ├── ArchiveCommand.php.bin
                │       │   ├── ClearCacheCommand.php.bin
...

PHP 7 on Arch Linux

I have been working on PHP 7 packages for a while. They are available in my repository:

[php]
Server = https://repo.pierre-schmitz.com/$repo/os/$arch

Insert these lines on top of the other repository definitions in your /etc/pacman.conf
A copy of the PKGBUILDs I used to create the packages are available in my git repository.

Upstream changes

PHP 7 is possibly one of the most interesting releases since version 5 which was published over a decade ago. Besides up to twice the performance PHP now offers scalar type declarations and return types. The official migration guide lists all new features and incompatible changes.

Packaging changes

With PHP 7 I am trying to move closer to upstream defaults and be less opinionated. While this requires a more responsible configuration by the user it also eliminates some surprises. And after all this is the Arch way. As a bonus, scripts like Composer work fine with the default configuration now.

Core package changes

  • php-pear is no longer available
  • php-mssql was removed by upstream
  • php-ldap is now included in the core php package as its dependencies were already met
  • The mysql extension was dropped upstream and is no longer available

Configuration changes

  • openssl, phar and posix modules are now built in. Remove the corresponding directives from you php.ini, e.g. extension=openssl.so
  • The include_path is no longer set to /usr/share/pear by default
  • open_basedir is no longer defined. This also means that packaged scripts do not need to be put into an awkward /usr/share/webapps directory.
  • PHP-FPM does no longer provide a logrotate configuration. By default errors are logged to syslog (these will be picked up by journald).
  • FPM pool configuration is now stored in /etc/php/php-fpm.d by default.
  • The upstream FPM service file is now installed which does not include PrivateTmp=true.

Third party extensions

We offer a few PHP modules that are maintained by third parties and are not part of the PHP source distribution. So far only modules that are in active development were able to provide updated versions. This means we have to drop most of these PHP extensions, especially those that might be considered dead.

The following list provides an overview of modules I gathered so far. It is possible we could re-upload updated packages once upstream provides us with compatible versions.

Package Status Solution
graphviz incompatible remove PHP bindings
php-apcu compatible update to version 5
php-geoip incompatible remove package
php-memcache incompatible remove package, project seems dead
php-memcached incompatible remove package, move back when upstream version is available
php-mongo incompatible remove package, superseded by the mongodb driver
php-xcache incompatible remove package, project seems dead
uwsgi-plugin-php incompatible remove PHP support, upstream update seems possible
xdebug compatible update to latest release candidate of version 2.4

In addition to this I will introduce a new module called php-apc-bc which will provide the legacy apc_ function for use with the APCu extension. Before version 5 this feature was provided by APCu itself.

Testing in Production

While I have been working on PHP 7 for months now one really only sees how it performs when put into production. So a few days ago I started running all services at archlinux.de (this also includes this very blog) with that latest version of PHP.

So far its running fine. The update went smoother than expected and the needed configuration changes were close to nonexistent.

There are some regular but very few core dumps produced by php-fpm. These did not result in 5xx server responses though and I assume that they are related to shutting down fpm workers. It should be investigated but it does not seem critical.

Before updating I had to make two small source code changes:

  • Engine errors are now of the type \Error. \Error and \Exception implement the \Throwable interface. So I had to adjust a type hint, or in this case remove it so it would also work in PHP 5.6.
  • The return type of the \SessionHandlerInterface are now enforced. If you return anything other than a string in your custom session handler the operation will fail.

Your Feedback

Now it is your turn. The packages provided in my repository are ready and I will ensure an upgrade path once the updated packages are available in the [extra] repository. I do not plan to hold them back for very long but I’d like to get some feedback first. After all it is a major update, we drop a lot of extensions and compatibility with many scripts out there is unknown.

Try these packages, test your own code and send me your feedback and suggestions.

PHP 5.5 on Arch Linux

Packages of PHP 5.5 are available in my repository. Note that these are debug builds and might require the use of the [testing] repository.

[php]
Server = https://repo.pierre-schmitz.com/$repo/os/$arch

Only try this at home. APC gets replaced by APCu for user data caching. Use the new opcache module to cache the PHP scripts. XCache is also available as a development snapshot.

Let me know of any issues but also successes. More will follow up soon.