Cees-Jan

Test lowest, current, and highest possible on Travis

During DPC I've had a talk with Rafael about making sure you test all your possible versions, lowest, current, and highest. The talk was ignited by the infamous composer.lock file (whether to commit it or not).

Composer Travis Hybrid

The setup

Setting travis up to test against those 3 targets is rather easy. As usual you add each PHP version to the php array. The first twist is that for each version you add a highest and lowest dependencies environment var to the build matrix as such:

- php: 5.6
  env: dependencies=lowest
- php: 5.6
  env: dependencies=highest

We can use that variable (or it's absence) to determine what composer command to run. For the default job we check if $dependencies is not set and then run composer install:

if [ -z "$dependencies" ]; then composer install; fi;

For the lowest we check if $dependencies is set to lowest and then run composer update --prefer-lowest:

if [ "$dependencies" = "lowest" ]; then composer update --prefer-lowest -n; fi;

For the highest we check if $dependencies is set to highest and then run a regular composer update:

if [ "$dependencies" = "highest" ]; then composer update -n; fi;

The end result (I've left the actually running of test out to provide a minimal example):

language: php

php:
  - 5.6
  - 7

matrix:
  include:
    - php: 5.6
      env: dependencies=lowest
    - php: 7
      env: dependencies=lowest
    - php: 5.6
      env: dependencies=highest
    - php: 7
      env: dependencies=highest

before_script:
  - composer self-update -q
  - if [ -z "$dependencies" ]; then composer install; fi;
  - if [ "$dependencies" = "lowest" ]; then composer update --prefer-lowest -n; fi;
  - if [ "$dependencies" = "highest" ]; then composer update -n; fi;
The result on travis

After you updated your travis.yml you'll end up with a build matrix like this:

Travis build matrix

The after match

Now that we have the testing setup things are going to get interesting. In my case, most issues were from overly loose constraints in my computer.json. One example is a package targeted at ^1.0, that resulted in errors and the correct targeting is ^1.0.6 due to a bugfix in 1.0.6. You'll find similar issues and errors on your path but it will be worth it in the end as your package will work for your set constraints.

A side note

Jordi has a very valid point here. Travis has limited CPU time available and it shows best during the evening hours when both the EU and US are active. I'm not surprised anymore by builds waiting up to an hour to start. If you're a small package maintainer like me that doesn't makes a lot of commits (and thus builds) or doesn't support a lot of PHP versions limiting the build matrix might not have a big import. But that doesn't make the following suggestion less interesting. Test highest and lower only for hhvm, PHP7, and your lowest and highest PHP5.x version (5.3 and 5.6 for example). That way you ensure lowest and highest work most likely with all your supported PHP versions. (The change something breaking between those 5.x versions you didn't test is rather nihil.)


Categories: PHP - TravisCI - TravisCI SeriesTags: PHP - TravisCI