A recent set of Linux distro benchmarks appears to show some surprising results. Yet it also shows the dangers of relying too heavily upon such benchmarks to make real-world technology decisions.
Phoronix is one of the few Web sites with a heavy focus on Linux hardware testing. Its editors have also developed a comprehensive, and highly regarded, testing and benchmarking suite for Linux distros. As a result, Phoronix is a notable source for comparative data on Linux distro performance, involving both multiple distros and different releases of the same distro.
Recently, Phoronix offered up comparative benchmarks of four leading Linux distros:
“With it being a while since we last compared many Linux distributions when it comes to their measurable desktop performance, we decided to run a new round of tests atop four of the most popular Linux distributions: OpenSuSE, Ubuntu, Fedora, and Mandriva. To see where these Linux distributions are at, we used their latest development releases and then performed all package updates as of 2009-07-15. Following that, we ran an arsenal of tests using the Phoronix Test Suite.”
Different distros typically run different versions of key software, and this case was typical. While three of the distros installed the EXT4 file system by default, for example, a fourth — Mandriva — uses EXT3. To some extent, these differences make exact comparisons very difficult, although benchmarking can still provide useful information about general performance trends.
Or can they? Perhaps that’s a debatable point.
On one hand, the Phoronix benchmarks indicate the performance benefits of the new EXT4 file system. The Dbench test, which simulates filesystem load, clearly showed Mandriva lagging behind the other distros. Even though Dbench, like most such tests, does a poor job of simulating real-world server loads, such results are still noteworthy when one takes them in context.
On the other hand, a few of the Phoronix Test Suite results showed problems with the Fedora development release. The Apache benchmark, for example, made Fedora look downright anemic compared to its competitors.
The Phoronix author took a tentative stab at explaining these results: He speculated that it could be due to “SELinux on Fedora, debugging options, or a combination of other factors.”
In fact, it turns out that Rawhide, the Fedora development code base, is compiled by default with its debugging options enabled. These options impose a significant performance handicap, since they require the operating system to perform a variety of running, often very resource-intensive, checks.
In defense of Phoronix, the benchmarks were apparently run with each distro’s default, “out of the box” installation options. Yet such options in a development release are predictably very different than they would be in a production release, as Fedora’s default use of a debugging mode indicates.
While this particular test appears to be seriously flawed, other Phoronix benchmarks are far more useful, in my opinion. Always evaluate such results, however, with a grain of salt: No matter how carefully it is done, benchmarking remains as much an art form as it is a science.
Madhav Kobal's Blog 