OCT
24
2006

Distro Quality Assurance

Submitted by Thomas Zander

In Linux land we have an awful lot of choice in distro's. Choice is good as it ensures you have a better chance of getting exactly what you want.
In something as big as a linux distro (more often then not it ships man CDs worth of data) I feel that the effects are not helping people out anymore. Choosing your distro is just based on too many things. How hard is it to upgrade packages? Can you upgrade in a year time? Is the directory structure easy to understand? How easy is it to maintain your hardware? Is all the hardware supported?
And on the other side of the spectrum you have security patches and package management philosophies.
Most people don't look at that when choosing a distro, but in my experience there is something wrong with each distro I saw. And I bet its because the different distros don't steal enough from each other. Stealing package tweaks from other distros because they give a better user experience is a good thing.

The latest differentiation I saw is how and when to ship a package. Debian is famous for shipping a package only when its found to be well tested. Their stable release is really rock solid, and a package will only make it into the 'testing' section if it does not have known issues. In other words; it takes a little while after release before you can get it, but you can virtually count on your latest update not breaking things.

In KOffice we just closed a bugreport on spell checking not working in KWord. The reason for that, it turns out, is that Gentoo moves towards the opposite side of quality control. In this case KDELibs 3.5.4 could not have been shipped as is because a dependency was not available yet. Instead of letting it wait, the packagers choose to ship it but exclude spell checking. Which made it build even though the dependency was not there. Leading to spell checking missing in KWord.
http://bugs.gentoo.org/show_bug.cgi?id=152417

Choosing the distro for your machine clearly means more then most people judge it on. The quality control to ensure a minimum of bugs users are exposted to is high in a distro like Debian. On the other end of the spectrum we have a distro that makes choices that I would equate to not having any QA process at all.

After seeing a handful of weird bugs every week on IRC from gentoo, I'm struggling with my conviction that people can run my (LGPLed) code any way they like. With that I mean I don't mind people doing with it whatever they do, maybe have a logo in our software "NOT designed for...". :)

What do you think? Can we help distros provide a better user experience, or, towards the ridiculous side, should we close bugs from some distro as is?

Comments

Most changes distributors apply to our code are not intentionally bad.

Most of the time the packagers just don't know the full list of consequences and, realistically, they can't unless they know every bit of the software stack above that change.

However they are usually very cooperative if we tell them about some of these overlooked side effects and will either remove the change or find a different way to do it.

I think reports for bugs caused by distribution specific changes should be forwarded to their BTS and out BTS would need a flags for "closed, reason forwarded"

By krake at Tue, 10/24/2006 - 09:40

I'm pretty certain no distro is intentionally shipping things that are bad.

Its the care distros give to make sure the changes are for the better of their users. Or just for those users that want that new release NOW no matter what. (instant gratification) But naturally will complain when its not perfect.
And I've seen quite some complains being directed towards #koffice.

We need projects and ideas. What about we combine this with a buildfarm (build.opensuse.org) and have unit tests stating that things fail or don't compile under a certain disto which are then reported to the project owners?
Anyone with better ideas?

>I think reports for bugs caused by distribution specific changes should be forwarded to their BTS
> and out BTS would need a flags for "closed, reason forwarded"

Thats putting out the fires, I'm trying to find out ways to make sure the fire never gets big enough for end users to be hit by them. And as a sideeffect make sure end users will see the actual KDE quality and not a diminished on.

By Thomas Zander at Tue, 10/24/2006 - 10:51

Some form of joined testing might help a lot, but I guess the distributors would have to maintain their build systems to make sure it is up-to-date from their point of view.

Thats putting out the fires

It's not just about putting out fires. There will always be misdirected reports, because the difference between the projects is not visible to the user. Ideally all reports would first go to the distributor who then delegates upstream problems to the correct source project.

But we would still need to handle reports that show up in our BTS first.

By krake at Tue, 10/24/2006 - 11:10

This exactly what I've been sensing too. As software developer you're at the mercy of packagers to ship your code correctly. Or in my case (KMess), whether they even package it at all.

I've been bughunting or 2 hours why a contributor of my project didn't get kdDebug() output in Fedora Core. It turns out they've patched kdDebug() to drop all output, unless it was enabled in $KDEDIR/share/config/kdebugrc I owe "Beineri" a big thank-you for the site http://ktown.kde.org/~binner/distributor-patches/ Without that site I would have a very unproductive developer.

When a distro patches something, it's likely the package lacks something they'd like to have something original author didn't think of (of offer as a config option). These kind of changes should be documented somewhere, or at least mention a wish to the upstream developer. That would solve a lot already!

By vdboor at Tue, 10/24/2006 - 09:43

>This exactly what I've been sensing too. As software developer you're at the mercy of packagers to ship your code correctly. Or in my case (KMess), whether they even package it at all.

And as a distributor you're at the mercy of $upstream

- to write proper build scripts (often indeterministic, missing/broken destdir support, unneeded hardcoded limitations to specifc versions of build tools or even completely broken)

- to release consistently versioned tarballs

- not to re-release a fixed tarball with the same signature

- not to use tarball urls with a query part

- to stay within the build dir and not to wildly write-access the system

- to be continued... :)

All this costs time when packaging, that could be spent better.

> When a distro patches something, it's likely the package lacks something they'd like to have something original author didn't think of (of offer as a config option). These kind of changes should be documented somewhere, or at least mention a wish to the upstream developer. That would solve a lot already!

Patches are indeed a problem. Both distributors not creating them in a protable way and not sending them upstream as well as upstream not reacting on patches. Part of the philosophy of Gentoo is that packages do not become a "distro-fork" and that patches reach $upstream.

By carlo at Thu, 10/26/2006 - 19:18

Actually, the problem is that the dependency check was broken from kdelibs itself, and the direct way to fix that was chosen, removing a feature because it was creating a build bug and (at the first glance) not breaking anything.

I can provide you with a few thousands cases where instead we got to fix the bugs caused downstream, if you're so interested in quality assurance, I think I fixed my share of bugs with configure.in.in in the past.

The bug is open now for what? 3 days, that's less than a working week. I've had my problems lately and I cannot work 24 hours a day on Gentoo alone, I need to pay my bills too, but I was working on the bug already without all this fuss. Now I had to ask nicely to get a pause from my work to fix this, as you seem to be perfect and ready to blame others.

I don't think anybody should talk about distributions' QA without first _working_ as a distributor, because there's a big difference between what you ship downstream and what the users expect to receive. This 3.5.5 release had so many patches that if users were expected to use the vanilla sources they'd be crying...

By Diego Elio Pettenò at Wed, 10/25/2006 - 20:04

Removing a library and shipping the product instead of waiting for the proper fix seemed fine to you. Thats ok for me!
You correctly state that it didn't break anything at first glance, and I accept that.

What would be very usefull is to find a way to make sure that any such mistakes are caught before the users see them.
Gentoo has a large userbase that will emerge new stuff as soon as it hits the mirrors, so it is in Gentoos as well as KDEs interrest to keep such breakages to a minimum.

So, how can we get (automated) quality assurance into Gentoo. Preferably in a way that also work in the various other distros.

How about that idea to use a buildfarm and let it fail if a component is missing. That would work for this case, but also for the gentoo KOffice1.6 update that made Krita disapear due to a too old lcms. Would that be an idea?

By Thomas Zander at Thu, 10/26/2006 - 08:36

Well, the ~arch tree is not ensured to work, that is known. If users mixes arch and ~arch (the only way to have a too old lcms) that is totally unsupported. In general, ~arch _is_ supposed to be for identifying problems before normal users get the software.

I cannot stop users from using the latest versions just committed, but you cannot pretend that we're always perfect from day one.. especially considering that at the moment we're a bit understaffed.

There might have been rough points for KOffice 1.6, but at least the last three releases of KDE itself were smooth, if we exclude some troubles with HAL versions needed by KDE and GNOME that conflict, and prevent us from marking 3.5.4 stable.

There is a tinderboxing project being worked on by official devs and non devs, but it's mostly for stable now.

One thing I can think of is limited to KDE 3.x or in general autotools based packages, and it is to make the configure _fail_ if needed software is not found.. unfortunately the KDE policy goes totally in the other direction, by disallowing failures when prerequisites are missing, even if they were requested explicitly.

By Diego Elio Pettenò at Thu, 10/26/2006 - 18:10

>What would be very usefull is to find a way to make sure that any such mistakes are caught before the users see them.
Gentoo has a large userbase that will emerge new stuff as soon as it hits the mirrors, so it is in Gentoos as well as KDEs interrest to keep such breakages to a minimum.

Well, as soon as the ebuilds hit the tree. ;) Personally I'm less active within Gentoo lately, partly because I can't stand the "wanna_have_0day_packages" requests anymore. The problem is not on the side of us devs though. Gentoo users are expected to file bugs at bugs.gentoo.org (unless they are sure it is a kde.org bug), because it's impossible to test all possible build combinations. And ebuilds don't go directly stable, they stay in testing for at least a month. And this month minimal testing time is part of our QA. So I'm sorry to tell you that you deal with a crowd of Gentoo users, who are using testing ebuilds, but don't have a clue what this implies on their side. Maybe it's worth writing an article in the Gentoo Weekly Newsletter about it, to refresh the information, but unfortunately such bug reports will always reach you, as there are always people who do do not hear. The worst I've seen happen is users filing identical bug reports in both bugzillas at the same time, hoping one team may be faster than the other... :(

>How about that idea to use a buildfarm and let it fail if a component is missing. That would work for this case, but also for the gentoo KOffice1.6 update that made Krita disapear due to a too old lcms. Would that be an idea?

Gentoo is a meta distribution with each user compiling/building his very own distribution. kdelibs alone offers ~20 use flags to en-/disable specific configure flags etc. and is keyworded for ten architectures, not to count the libraries and their use flags. I'm quite sure you understand that it is impossible to test all combinations.

Coming back to the actual issues: None of the relevant ebuilds were marked stable and the spell checker functionality regression I'd even see as a minor issue from a distribution point of view, if it happened in stable.

btw.: What's with the broken++ KOffice build tests? Doesn't look like you have a build farm testing all and everything. :)

By carlo at Thu, 10/26/2006 - 18:57