How Xdebug made HHVM look super awesome
Two weeks ago I tweeted about HHVM
beeing a lot faster then standard PHP on a certain set of Travis CI
builds. No matter how awesome that tweet looked, the timings it showed are influenced and accurate. Let me tell you why.
So @HipHopVM is said to be fast but this difference is just ridiculous. (Testing operations on images.) pic.twitter.com/3HO3dSB9IZ
— Cees-Jan Kiewiet (@WyriHaximus) February 18, 2014
What happened
In my effort to get a full year of daily contributions on Github I was going over my repo's nitpicking something simple to do that day. Found that the .travis.yml
of HotSpots could use some white space between sections. Used Github's fancy webbased editor since it was a quick fix. Committed and decided to check out the test results:
The difference between vanilla
PHP and HHVM
is huge. Looking at the tests them self it's 8 seconds on HHVM
versus 2 minutes on vanilla
. How awesome that result may look it's not realistic. So lets find out what happened.
How is is set up
First off I've setup the github the repo with a commit hook to travis and it includes an .travis.yml
file with the testing configuration for Travis. It includes the following PHP versions to test against:
- 5.3
- 5.4
- 5.5
- 5.6
- hhvm
The script
directive is the following:
php vendor/bin/phpunit --coverage-text --coverage-clover ./build/logs/clover.xml
And the after script, will be important later on, is this:
php vendor/bin/coveralls
Getting the hint
Two days after the tweet I get this response to it.
@WyriHaximus Noob question: why does job 45.5 ends with a InvalidConfigurationException "json_path is not writable" where others don't?
— DUVERGIER Claude (@C_Duv) February 20, 2014
Hinting there is something wrong. Checking build #45.5 line #97 and no surprise it's there.
What was causing it
With that hint I started looking around and realized it must have been because the clover.xml test
coverage file was missing. Then it starting to make sense, HHVM
doesn't support Xdebug
so there is no code coverage collected, just test results. Knowing that Xdebug
slows things down tremendously when doing code coverage, it could explain the difference. Normally it's barely noticeable but due to the huge amount of asserts (going over the resulting image pixel by pixel) it stands out. To be sure, and find out the real performance gap, a simple VM using vagrant was set up with HHVM
and vanilla
PHP install.
The Accurate results
The end result isn't as spectacular as my initial screenshot. But it does confirm my suspicion that Xdebug
slowed the tests down on Travis
. Both are run with repeat on 100 to get a decent timing difference. But it does show HHVM
still as a clear winner.
PHP
php vendor/bin/phpunit --repeat 100
Resulted in:
HHVM
hhvm vendor/bin/phpunit --repeat 100
Resulted in:
Conclusion
My initial tweet was sent out because I was surprised by the huge timing discrepancy. The community responded in was great way. And I hope no one has the wrong impression about HHVM
vs. PHP
performance due to that tweet. HHVM
is still the clear winner, completing the tests a good 10% faster. This oopsie
has been a valuable lesson for me to always properly check my facts. Would like to thank Claude for pointing me in the right direction.