On benchmarks

Do you know this one?

Phoronix tested md5sums of ISO images of distributions. The winner was openSUSE, scoring e29311f6f1bf1af907f9ef9f44b8328b, which gave it a noticeable lead before second Slackware (b026324c6904b2a9cb4b88d6d61c81d1), which is quite closely followed by Fedora (9ffbf43126e33be52cd2bf7e01d627f9) and Debian (9ae0ea9e3c9c6e1b9b6252c8395efdc1). The difference between these two distributions, as you can see, is only very small. Ubuntu completely flopped in this test, achieving only 1dcca23355272056f04fe8bf20edfce0, which is surprising, especially considering that its previous release scored a very nice c30f7472766d25af1dc80b3ffc9a58c7. (source).

Ok, that's just a joke, but the sad part is, as somebody pointed out, that it wouldn't be really that surprising if Phoronix actually did something like that. And, probably even more sad, there would be people who'd really take it as if it meant something and started adding comments about how last openSUSE is pretty good, last Ubuntu is so disappointing, and Fedora and Debian are not really that different.

So take this from somebody who has already done a lot of performance work: Benchmarks, on their own, mean almost nothing if you don't understand them. Especially if they are seriously flawed (I mean, testing filesystem performance by doing CPU-intensive tasks? Hallo? Probably even FAT16 could provide the same results in those tests on an SSD.), but even if the results are useful numbers, it is still necessary to understand what the numbers actually say. I think I wouldn't even have a big problem forging a "benchmark" where KDE would get better (and correct) numbers than LXDE by finding a scenario that'd be twisted enough.

And even then, it is still necessary to keep in mind what is compared. Comparing the default setup of Fedora and openSUSE means also comparing GNOME and KDE, which you may or may not want, but if you compare the distributions this way, it is affected by differences between the desktops, and if you compare the desktops, it is affected by the differences between the distributions. And in either case, it may or may not apply to another distribution or another desktop.

Yet one more thing to understand is what is measured and how it affects performance as a whole. Ages ago there was a Dot article that also mentioned performance improvements to be brought by Qt4 in some specific areas, yet even now there are numbers of people seriously disappointed by KDE4's performance only because they thought that since Qt4 improves in some areas, KDE4 will get exactly the same improvement, regardless of how much these improvements matter for the whole of KDE4 or how much of KDE4 was rewritten when porting from KDE3. When I fixed Exmap to work again and did a little benchmark as a part of it, there wasn't really much more to it than to show that Exmap works again and that it is very easy to lose a lot of advantage by a simple mistake, yet people had no problems drawing all kinds of strange conclusions from that. Since making right conclusions is unexpectedly difficult for most people when it gets to benchmarks, I really need to remember not to just post numbers again without also saying what it means.

And, finally, there is still the question of other costs and whether it is worth it. Various KDE components often have resource demands, but then they are also often worth it. I mean, we all could still use TWM, or, heck, Windows 95, but seriously, how many of us really would? This, ultimately, is always about what works the best for you.


Most discussion wars stars that people are comparing oranges and they blame bananas from that.

As example. Ubuntu sucks because it is so brown and simple. But OpenSUSE wins because it is green and its desktop is very configurable. So Ubuntu is not so good OS as OpenSUSE is.

Just like you said, there are wrong conclusion what are used. Like you can change the theme/color theme easily. You can change the desktop environment from Gnome to KDE SC or vice versa. And you are running same Linux OS on both of them. Maybe other allows users to change OS from Linux to Hurd on them? (Does OpenSUSE or Ubuntu offer a Hurd OS or are they just using Linux OS?).

It is just so easy to start wars by showing two different desktops and say "this is bad" and "this is good". Even that they would be using same OS, same distribution of that OS and even same desktop environment, with different theme and settings and preinstalled applications. Some people does not even understand the terms OS, distribution, desktop environment, theme as example. And they do very harsh judgement by their knowledge what is based to marketing and second hand information.

Same thing with numbers and graphs. You nailed the point, you need to understand the technology how it works until you can say anything about comparison.

By fri13 at Wed, 03/10/2010 - 10:46

I think Phoronix is doing good to every aspect of Linux desktop, plus it's building the community and keeping the momentum over graphics development. Just take a look at how hackers are mentioning Phoronix in their reports. Their 'strange' benchmarks often spotted regressions in place you could never imagine!

I use KDE SC 4 and I'm really aware that in my machine it's using lots of ram, but I like it an I will keep it ;-)

By Enrico Ros at Wed, 03/10/2010 - 15:30

Same here. If I let my PC run overnight, Nepomuk fills up my 8 GB swap and 4 GB ram with one HUGE process (I believe it's soprano something) and I have to disable Nepomuk, wait 5 min for the thrashing to stop, and I can finally start doing something. But I'll just wait until it's fixed cuz I like pain... If I had to use that PC for work, though, I'd disable Nepomuk right away :D

By jospoortvliet at Wed, 03/10/2010 - 16:29