KDE Linux on KDE Blogs

I Just Wanted to Print a Sandwich

Silverhadch — Tue, 01 Sep 2026 00:00:01 +0000

Or: how I ended up developing KDE Linux.

Now

Then

Most of the people who work on KDE Linux arrived the same way. They were KDE developers first, and then either helped start the project or wandered over to it once it existed. I went the other direction entirely. I was a user. Then a user who filed bugs. Then a user who fixed them. Somewhere in there I became one of the more active contributors, and only after all of that did I become a KDE developer.

So my start in KDE was not really KDE at all. It was KDE Linux.

I was searching for a distro, a very particular one. I wanted the latest KDE, rolling enough, atomic, and flicker-free with systemd-boot. After a lot of searching I stumbled over KDE Linux and started using it.

Oh boy. Young pre-alpha KDE Linux was definitely something.

If I gave you a current KDE Linux image and one of the old ones, you would believe them to be completely separate projects. Stuff broke so comically often back then that I eventually hit the stage of grief where you stop filing bug reports and start fixing things yourself.

So I made myself an Invent account and just started fixing the issues I ran into. From systemd-boot spitting out error messages because the UKI had the wrong permissions, to roughly thirty commits to get network printers working properly, because I wanted to print a picture of a sandwich.

Don't ask why. That will remain a secret you'll have to ask me about in person if we ever meet at Akademy.

That printer saga taught me the first real lesson: unlike my personal projects, here you will get a lot of pushback if something is wrong, and you will go back and forth until it's good enough. If you want to collaborate in a team with actual people, and not just with your friend where the two of you scream at each other over regressions, you need to learn patience.

Patience is the key to everything. You'll see it come up here again. And again.

Pre-Alpha: mostly plumbing

After a while of fixing the distro I was daily driving, it stopped feeling like everything was constantly collapsing. That's when I got invested in seeing this project succeed and started making frequent fixes, to the point where I went from being a random non-KDE contributor to someone the rest of the community started trusting with real work.

A lot of my early work was stabilisation, because as I said, KDE Linux back then was not very stable. And one problem guided us then and still guides us today, hopefully not for much longer: the choice of base distro.

You see, KDE Linux is not a traditional mutable distro. It's an image-based system. Writes to /usr are taboo and there is no pre-installed package manager to install software with.

Now you might have spotted a small problem. Where the heck does /usr come from?

Well, that sentence isn't quite true. There is a package manager that gets used to populate /usr, and then afterwards we rip it out and close the door behind it. KDE Linux currently uses pacman and the Arch Linux repos to build its images. Which means any problem Arch has, KDE Linux inherits. And as far as most Linux users know… Arch is not famously known for being stable.

So KDE Linux had a small problem. The KDE stack we build ourselves in a separate packages pipeline relies on the ABI and versions of certain non-KDE libraries provided by pacman, and therefore by Arch. It is not uncommon for the Arch base to move too fast and brick the KDE stack. We were chasing a target we couldn't possibly catch up with often enough, and we got bricked images out of it.

So it was decided to rely on Arch's infrastructure (that later turned out to be a mistake) and pin to a snapshot. Working together with Harald, we got the packages pipeline to dedicate a repo snapshot for the main image pipeline, so our stack wouldn't die because of our base distro. And since KDE Linux was designed as an image-based system, swapping the base distro shouldn't be disruptive to users should the need ever arise. But let's not get ahead of ourselves.

Now, another thing Arch is known for is the AUR, a public index of build scripts anyone can submit to. That openness is the entire point and it is why the AUR is so useful, but it also means nobody reviews a package on your behalf before it runs on your machine.

For the past year or longer Arch Linux and the AUR have been the victim of repeated DDoS and malware attacks, and KDE Linux got caught in between, because we were using the AUR to a degree. Which meant we had to get the hell out of the crossfire and start removing our AUR dependency bit by bit.

Before KDE Linux I was an AUR packager, and I still package for a couple of distros, so I took over maintainership for some of it and then started helping remove our AUR dependency altogether. And since some of my own AUR packages were now being pulled into KDE Linux, meaning I was now part of a KDE project's supply chain, I started thinking that maybe I should just do it and become a KDE developer.

Throughout the pre-alpha I got some guidance from KDE devs like Nate and Harald, and the KDE community started to grow on me. So I asked Harald to sponsor my KDE developer application.

A short detour about growing up

Before I get to what happened next, I should stop and say something about the person who filed that application.

KDE helped me grow as a developer and as a person.

Because, as some of you might know, I was not that old. I was basically seventeen, almost eighteen, and somewhat immature, and therefore early on I was, let's put it gently, a bit overwhelmed and abrasive in my language towards others. I repeatedly got into conflict with people because of my poor choice of words and insulting behaviour, because I was severely lacking in the social skills department. I've got to admit that.

However, after a while, and after having gotten employ**ed, my social skills improved and I started being cringy instead of abrasive. Depending on who you ask, that might be a downgrade, not gonna lie.

I would say I've grown as a person through interacting with the rest of the KDE community. So it wouldn't be too far-fetched to say that KDE Linux had a positive effect on my life.

The alpha announcement, and the four-day pipeline curse

Eventually KDE Linux reached a certain level of maturity and stability. The pipelines, however, still weren't very happy.

We were approaching the end of the pre-alpha era and preparing a release and an announcement, but the pipelines had a tendency to fall in on themselves every four days. So for a while it wasn't uncommon for the majority of KDE Linux dev work to be plumbing.

Eventually the pipelines started to get along with us too, and the KDE Linux alpha release was announced at Akademy, where I looted a crap ton of stickers that my younger sister then proceeded to take from me because she liked Katie.

How I develop a distro I don't run

Not long after that, the way I worked on the project had to change completely.

Until then, my development process basically amounted to doing sysexts on my daily driver, which was KDE Linux. But getting employ**ed meant I couldn't just stay on an alpha distro, even though KDE Linux was nowhere near as unstable by then as it had been in its early life.

So I moved into the ublue custom images scene and started doing a "KDE Linux but at home" thing, which accidentally turned into a small competition between the two despite me developing both. Ironic. That project eventually died off because the COPR for KDE master builds went unmaintained, so I switched to NixOS, which has been my daily driver ever since.

Which raises a fair question: isn't the alpha stage the part where I made massive changes, features and improvements? How did I do that on NixOS, where building anything outside the Nix way means fighting the fact that nothing lives where a normal build expects it to?

Two things. CI-built images, and the built-in Docker script and QEMU.

(That was three things, but never mind, I'm too tired to move my mouse cursor back and delete it, because I've been writing this in a flow state for two hours.)

Our CI builds images of every MR and pushes them to Harald's server, so you can download an image and boot and test it in QEMU. That is how I develop KDE Linux to this day. There's also the Docker script which builds MR images locally, in case you have a faster machine than the CI, which most people probably do. I also have a second machine that I use to test some images on real hardware.

Alpha: the people who wear cool hats

With the tooling sorted, I could get back to what I actually wanted KDE Linux to be.

Do you know that distro where people wear cool hats? Yes, Fedora.

They're quite known for pushing the Linux desktop into the future and showing off the latest and greatest of Linux. And in case you know what KDE Linux's goal is… it's pretty much that, for KDE. Showing off the latest of KDE. But how can you do that if the latest and greatest of Linux is missing?

So I started joining Fedora Matrix rooms, doing some drive-by contributions and lurking. The change proposals especially caught my eye, along with the fact that both Fedora and KDE Linux want to push Btrfs forward. So I started implementing changes even before Fedora got a chance to do them, like KMSCON.

And one thing Btrfs is known for is snapshots. Small problem, though: Linux distros don't usually create good Btrfs layouts by default. We didn't either, so Harald moved our layout to RootFSv2.

The specific issue is having /home and all its users be one single subvolume. That means, firstly, that only root can take snapshots, because /home is owned by root. And secondly, that every user's data gets rolled back to a given point together, so users can "lose" their data so that another user can recover theirs.

That has some pretty nasty wheel war potential.

So we needed to separate user data from each other. But how? KDE doesn't do useradd and friends. That falls way, way, way, way lower in the shared stack between almost all Linux distros. That's shadow-utils.

Living in the shadows

The work here you won't find in the kde-linux repo, and for a time it looked like I was on hiatus. In reality I had been living in the shadows (pun intended), working behind the scenes to push the Btrfs experience of Linux distros forward, not just for KDE Linux or Fedora, who were also very interested in this approach.

So I worked on shadow-utils for months, from bugfixing the Btrfs backend to adding new features to revamping the whole CI. I've been kind of stuck there, to be honest. Though in a good way. I like the project, and C is my favourite language (shocker, treacherous!).

Eventually the work landed in KDE Linux as RootFSv3 plus snapper for kio-snapshot, which added everything needed for users to be in charge of their own home snapshots, instead of members of the wheel group getting into a wheel war over rolling back all of /home rather than just their own subvolume.

That was a hell of a lot of work, and it showed how patient one has to be. It took a dozen months of mostly waiting.

So always remember: Rome wasn't built in a day. Patience is the key to success.

Modularity, and not reinventing the wheel

You also need to learn modularity, because throwing everything into the main repo is a bad idea for maintainability.

In the early alpha stage I wrote a small helper that improves UX in an interactive zsh session when you try to run a file without the executable bit set. (I got tired of chmod +x-ing things constantly after adding Java support and running .jar files straight from the terminal.)

We also started slowly populating the kde-linux group with a lot of repos over time, since a lot of that work can actually be shared with other distros and eventually landed in the system group. Which also shows: please don't reinvent the wheel. We eventually realised that throwing distro engineering into the kde-linux namespace might not be that great an idea, and that collaboration might be the better thing.

Escaping Arch

Now, remember when I was ranting about Arch up top? It kind of got worse, with our needs slowly not being fulfilled. The attacks on the AUR got worse, stability deteriorated, and the archive infrastructure we depended on turned out to be fragile, which is fair enough, since keeping old snapshots around is not what a rolling release is built for.

So I started the not-so-uncontroversial discussion of finally biting the bullet and fleeing Arch. We and even non-KDE contributors started discussing what we should do.

Because of the shared history with Fedora mentioned above, I was of the opinion that rebasing on Fedora might not be a bad idea, since Fedora and KDE are actually good buddies and our goals align better than with Arch. However, some pointed out that in a way we'd just be exchanging Arch for Fedora, introducing other problems, and still be at the mercy of a base distro.

So a third, more radical approach was suggested by GNOME people, who are also making their own atomic distro, and by the ublue crew (remember the custom images stuff): BuildStream, and taking complete control of the whole stack, so we're the ones in total control rather than our base distro. Which carries the risk of what Uncle Ben warned Peter about. With great power comes great responsibility.

This was around the middle of the alpha. So I started working on a Fedora version of KDE Linux, which later became the foundation for culling our Arch repos. Because up to that point the base-distro agnosticism wasn't entirely true: our stack came from our own self-made Arch repos. So instead I decided to port it to kde-builder-based tarballs, since that only relies on the base distro in one place, defining build dependencies.

We later of course decided to go with the BuildStream approach, and my Fedora work became fedora-plasma-canary, which again comes full circle back to ublue. Lol.

Fun fact: I was actually re-introducing kde-builder-based building of our stack, because the very earliest builds of KDE Linux, before I even took part in it, used kde-builder and didn't build Arch packages. So we basically reverted back to kde-builder, our upstream tooling that we also use for regular KDE development, on and outside of KDE Linux.

Around this time we worked out how decisions actually get made in the project. The running joke is that we are a semi-constitutional monarchy, with Harald as supreme banana emperor and the more active contributors as the magic council of elders. In practice it is a handful of people who talk a lot, usually agree, and sit entirely within KDE's actual governance like everyone else. We had like only one vote so far.

Late alpha: ISO hell

Governance turned out to be the easy problem of that period. The hard one was a file format.

And there is one thing we had to hear an unbearable number of times.

"Where .iso???"

In case you don't know, KDE Linux was distributed as a raw GPT disk. Basically as if your SSD right now were a file. That could be burned onto a USB stick or imported as a hard disk in QEMU. The VM use case was slightly more complicated, because you then need to add another hard disk image to the VM to actually install onto. The old .raw file is a hard drive in file form that you insert into your "PC."

For real installs it didn't matter, because firmware can boot off a hard disk of any kind — which is presumably how the machine you're reading this on got started. So you burn the hard disk file onto a portable hard disk (a USB stick, or a floppy — though it won't fit on a floppy, believe me, I tried, it was 7 GB and 435 MB too large) or a real SSD, if for some reason you have two lying around and are allergic to USB sticks.

Now, GPT hard disks are very flexible. You can do a lot of software gore with them, like putting them on the 9.4 billion terabyte drives that get invented in 2413. And doing cursed things with partitions. Making an image of one is very easy and you don't need to do anything other than have a properly working hard drive for it to work. (Technically you can also image a broken hard drive, but the intention of the KDE Linux live media is to be able to install, and booting is kind of required for that.)

However, ISO 9660 (the filesystem) and El Torito (the requirements for making an ISO 9660 filesystem bootable) are incredibly awful. They are a nightmare to deal with and unbelievably outdated.

Unlike firmware, which will treat a file as a hard disk, please and thank you, these want you to throw everything into an outdated filesystem and insert certain metadata and files in certain places. And then things like systemd-gpt-auto-generator, which rely on GPT byte sectors and which we use to boot, have to be fixed up in the initrd to go from ISO 9660 byte sectors to GPT byte sectors and then boot them. Working around the thing in the first place.

Absolute nightmare. I spent weeks reading specs and source code and more source code to only slightly understand what the hell the .iso wanted from me to be El Torito compatible. After a while I learned systemd had taken a huge part of that work and integrated it. From this day on, I shall fight and die for you, systemd.

And do you know who thought up these horrible ideas, which were never intended to be used outside of optical discs? ISO 9660 came out of a 1985 industry meeting at a hotel in Lake Tahoe and became a standard in 1988. El Torito arrived a decade later, in 1995, as a joint proposal from Phoenix Technologies and IBM, named, so the story goes, after the restaurant in Irvine where it was designed. Its idea of booting is to make the disc pretend to be a floppy, which tells you roughly everything about the era it comes from. The same IBM that gave us the ThinkPad, which is the only reason I didn't throw mine out of the window over this.

Of course this is a gross oversimplification, and don't expect me to explain in detail what these specs said, because I'm trying my hardest to forget the experience. And once again patience saved the day — just the day, mind you, my sanity fell victim to specs older than I am — since I also had to wait until ISO 9660 and El Torito support landed in a systemd release.

Lasciate ogne speranza, voi ch'intrate.

And you know what for? Basically nothing.

It slightly improved the VM install experience, which could also have been solved on the VM software side. Because nobody burns a Linux ISO onto a CD or a DVD anymore in the big 26, and that was the only thing actually gained by complying with El Torito and ISO 9660.

So if you manage to cram the over-7-gigabyte ISO onto a floppy disk, please let me know.

Late alpha: an April Fools joke that got away from me

And since we were just talking about floppy emulation: do you know DOS?

It was quite popular back then, before Microsoft won the DOS wars. How fairly it won is a question people are still arguing about on the internet, and I am not opening that door. Either way, DOS was on its way out once NT got good enough for home use, and the good old days of talking to hardware directly in real mode with clever hyper-optimisations are long over.

However, you can still experience it yourself in KDE Linux. Because as an April Fools joke, I wanted to add DOS support to KDE Linux.

It seems like nobody realised it was a joke. All the other devs thought I was dead serious.

Too awkward to now reveal that it was a joke, I instead decided to extend the Package Compatibility Helper to install certain software on demand via a so-called integration tree. It hooks into mimetypes for opening things by right-clicking, into binfmt (asking the kernel to hand a file off to PCH when it's executed), and into stub binaries in /usr/bin.

For DOS and Windows, PCH first checks against clear mimetypes, and then, if it's still unclear whether you need Wine or DOSBox, we inspect the binary. Because Microsoft was obsessed with throwing stuff into the DOS header. You can tell something is a Win16 program because the DOS header is a subtle ad telling you to go buy Microsoft Windows, or Win32 because the DOS header reminds you that, in case you try to run the Windows version of Kate under DOS, this program cannot be run in DOS mode.

Quite a fun rabbit hole to go down.

Where we are now

If I had to define my work and everyone else's in the late alpha stage, it's finishing the stabilisation, starting the polish, and fulfilling long-standing project goals.

For being an alpha, and soon a beta, this distro has become unrecognisable compared to its pre-alpha state. It's more polished and stable than most released distros I've tried, and I do mean that. Of course I might just be biased, but I can't wait to see where this journey goes, and whether this project will finally give KDE its deserved world domination.

Closing

So, I've been rambling about the important steps and my experiences developing KDE Linux and watching the project grow alongside myself. But all blogs have to end somewhere, and my hands are starting to hurt.

If I have any tips for new contributors who are interested in helping this ship sail, and it's the thing that has been guiding me through this whole journey: cooperation and patience. Not just between KDE devs, but with the entire Linux stack.

The day you plant the seed is not the day you eat the fruit.

This Month in KDE Linux: August 2026

Nate Graham and John Veness — Mon, 31 Aug 2026 21:15:01 +0000

Welcome to this edition of This Month in KDE Linux, dedicated to KDE’s next-generation operating system.

Here’s how KDE Linux evolved in August:

Data safety

A major data safety project reached a state of usability!

First, Hadi Chokr transformed users’ home folders into Btrfs sub-volumes and turned on automatic Btrfs snapshots for all user files.

Then, Bharadwaj Raju integrated the new kio-snapshot system he’s been working on and added support in Dolphin.

Now you can easily view and roll back to older versions of your files, even without an off-device backup:

This doesn’t replace a backup, of course! But it does amount to a sort of homedir-level undo feature that can keep your data safe from accidental deletions, or other destructive changes that are less catastrophic than a disk failure or having the device lost or stolen. And these snapshots will be useful to use as backup sources, so you’re not trying to back up a moving target.

Another few layers in the data safety onion, shall we say!

CJKV input support by default

Nate Graham finished the project to make Chinese, Japanese, Korean, and Vietnamese text input work out of the box. Now everything needed to type in these languages is pre-installed, and all you have to do is turn it on.

QA & testing

Thomas Duckworth and Bhushan Shah continued to develop, refine, and maintain KDE Linux’s automatic QA system, which continued to pay dividends this month. The system caught multiple complex integration issues introduced accidentally in commits made to KDE software — all of which are now fixed, so users of other operating systems will never have to experience them.

Security

Yago Raña Gayoso removed an unnecessary override that gave everyone in the wheel group default access to the Docker daemon’s socket.

Nate Graham fixed an oversight that let blacklisted kernel modules into the unified kernel images.

Bugs fixed

Harald Sitter fixed an issue that could make updates fail right after the update was published, or when run on machines with weak CPUs or slow network connections.

Hadi Chokr fixed an issue with user-visible version numbers of KDE Linux builds not including the intended level of precision.

Documentation

Nate Graham added quick links to KDE Linux’s official documentation right there in the launcher menu’s “Help” section.

Nate Graham also documented how to free up disk space, added a dedicated “Help & Support” page to the website, and added a link to the project’s news feed.

Philip Grant refined and documented a way to integrate KeePassXC with the Flatpak packaging of Firefox and Chromium (and their derivative browsers).

Julius Künzel fixed the styling of the standard Hugo alert bubbles, and Nate Graham ported the docs to use them.

Grab bag

Paul Brown added a neat feature to the set-up-systemd-extension tool that lets you create new extensions at different locations and with different names.

Hadi made the installer report its progress in a more granular way, so there’s less waiting around and wondering if the installation process got stuck.

Bhushan Shah removed the “UEFI shell” entry from the bootloader menu, since it was useless for practically everyone, and now the menu is less cluttered.

How you can help

KDE Linux is making steady progress towards its Beta milestone, and is now 85% of the way there.

There’s lots to do! If you’re a fan of the project, please help out; there are many ways:

User support: help support people on discuss.kde.org using KDE Linux.
Issue reporting: install KDE Linux and report issues.
Documentation: improve the docs by submitting merge requests.
Flatpak: fix packaging or code issues in Flatpak-packaged apps.
OS development: help build KDE Linux! There’s plenty to do.

This Month in KDE Linux: July 2026

Nate Graham and John Veness — Mon, 03 Aug 2026 00:00:01 +0000

Welcome to the inaugural edition of This Month in KDE Linux hosted on blogs.kde.org!

Previously, these posts lived on Nate Graham’s personal blog. But KDE Linux is growing up and becoming a more important part of KDE, larger than its individual contributors! So it’s time to move the public communication to KDE infrastructure, and that’s where you’ll be able to read these posts going forward.

Without further ado, here’s how KDE Linux evolved in July:

QA & testing

This month, Thomas Duckworth and Bhushan Shah integrated the next-gen QA system they’ve been building!

It’s now testing tons of conditions to make sure KDE Linux continues to work as expected, and it also acts as a full-stack integration test suite. To illustrate the utility of such a thing, it’s already identified and protected users from two significant issues:

A login-breaking KWin crash caused by a regression in gcc from Arch Linux (already fixed).
A regression in automatically unlocking your wallet at login (also already fixed).

These tricky problems could only have been caught by the kind of full-stack integration test suite that KDE Linux has now.

If you’re interested in the details, Thomas Duckworth’s blog post on how the system works is worth a read. Thanks very much to Thomas and Bhushan for building this lovely system!

Security

Hadi Chokr implemented some kernel-hardening tweaks we learned about from the SecureBlue project that were relevant to KDE Linux and wouldn’t noticeably burden users.

Efficiency & performance

Nate Graham removed the kernel modules for various hardware watchdogs that aren’t relevant for desktop use, which slightly reduces CPU usage and improves boot times.

Thomas Duckworth switched the CUPS printing system to use socket activation, which allows its services to only run when needed.

Software support package downloader + support for Java and DOS apps

Hadi Chokr implemented a major improvement to the “Package Compatibility Helper” tool that now allows it to — with the user’s consent — download software from Flathub to handle files that the user tries to open, but that nothing on the system can currently open.

Hadi then made this work for Java apps; the first time you try to launch one, the tool will ask you if you want to download a Java Runtime Environment (JRE). If you approve, it will do so and then launch the app using the JRE:

After that, Hadi implemented support for DOS apps; trying to launch one will offer up DOSBox. Now you can have that authentic 1993 Doom experience:

This system can be extended to support even more things in the future! And it’s been written in a generic enough way that it would be useful for any other image-based OS using Flatpak, as well.

Bugs fixed

Philip Grant fixed the mechanism that ensures that existing KDE Linux systems have the same Flatpaks installed as are pre-loaded on new systems, as it had various bugs that prevented it from working properly.

David Fundament fixed the script that disables USB auto-suspend for input devices so that it works with devices lacking a product name.

WenChao Zhang restored the esp4 and esp6 kernel modules that had been removed to mitigate the “DirtyFrag” vulnerability. The vulnerability has been fixed now, and missing those modules had broken IPsec VPNs, so they’re back now.

Hadi Chokr made WWAN modems work out of the box.

Documentation

Thomas Duckworth documented how to use and extend the new QA system.

Nate Graham documented how to make the Fcitx 5 input method (for CJKV input) work with apps still using the XWayland compatibility layer.

Hadi Chokr documented how to set up bootloader menus so you can choose between booting KDE Linux or another OS on the same computer. However, setting up such a system is left as a manual exercise for the reader, as currently the KDE Linux installer does not yet formally support complex disk setups. This will come later.

Grab bag

Prajna Sariputra pre-installed vulkan-virtio, improving performance in VMs whose graphics drivers support Vulkan.

Thomas Duckworth made the set-up-systemd-extension and toggle-developer-mode tools only prompt for authentication once.

Nate Graham removed the “Push Services” library and System Settings page, because it’s not mature enough for shipping yet.

How you can help

KDE Linux is making steady progress towards its Beta milestone, and is now 83% of the way there.

There’s lots to do! If you’re a fan of the project, please help out; there are many ways:

User support: help support people on discuss.kde.org using KDE Linux.
Issue reporting: install KDE Linux and report issues.
Documentation: improve the docs by submitting merge requests.
Flatpak: fix packaging or code issues in Flatpak-packaged apps.
OS development: help build KDE Linux! There’s plenty to do.

openQA Testing in KDE Linux

Thomas Duckworth — Mon, 13 Jul 2026 00:00:00 +0000

The openQA-based testing system has recently been integrated into KDE Linux (hooray!), and I thought it was about time I did a little write-up.

The nature of KDE Linux, in which the whole system ships as a single signed image rather than a pile of packages, is (in theory) wonderful for reliability. However, this raises an uncomfortable question: how do we make sure that image actually works before we ship it to people? OpenQA is the answer!

TL;DR: we boot each build in a virtual machine, run tests that interact with it to ensure the system installs and upgrades properly and that desktop functionality works. Once the tests pass, the user gets an end-to-end tested image. This replaces the rather rudimentary basic-test machinery, which simply booted up the live image and checked if the boot was blessed and if any units failed.

The test flow

A single build goes through three stages.

install-system takes the live ISO, boots it in a VM, and runs a real installation onto an empty virtual disk, just like a real user would. sanity-test then boots that freshly installed disk and verifies the system actually comes up and behaves. In between, while we're testing the upgrade path in parallel, an upgrade-system stage installs the previous release and upgrades it to the current build to check whether the previous release can actually be upgraded to the new build. Each stage hands its disk to the next.

They're wired together as a dependency chain, so in the openQA web interface the whole run shows up as a single connected graph. If installation fails, the later stages don't bother running, as there's nothing to test.

Our CI pipelines now approximately look like this:

Interesting architectural tidbits

We do a few things differently compared to your stock-standard openSUSE or Fedora openQA instances.

Selenium testing instead of needle testing

Normal openQA tests operate through "needles". These aren't sewing needles; rather, they're screenshots of the virtual machine in some desired state with some JSON metadata attached. This metadata defines certain areas to match or ignore, and the test code can click matched areas. While needles are certainly effective at interacting with the system exactly how a user would, they have drawbacks. It's quite annoying to make and constantly update needles, as well as keep them from breaking every time there are slight changes in user interfaces.

Luckily for us, we already have a battle-tested way of interacting with user interfaces for testing: selenium-webdriver-at-spi . It's already widely used across unit tests in KDE projects, hence our decision to use it affords us a lot more flexibility, maintainability, and consistency. It also enables app developers to run their own tests on KDE Linux with openQA down the track.

Essentially, we have a Python unittest script on the system that we're testing (see the sysext section below for details), which attaches itself to an application. It then interacts with the app by leveraging the AT-SPI2 accessibility API to send clicks and read the screen, in a similar vein to screen-reading software such as Orca.

Ephemeral workers in CI jobs

openQA instances usually have long-running workers that are hosted on servers. It's a bit of a painful ordeal to get all that infrastructure up and running. On top of that, hosted workers need to do an upload-download rigmarole involving large assets from the server, such as the .iso files and the generated hard disk. This makes things very slow for no good reason.

We already have CI runners that work perfectly well for this and can be spun up when needed, giving us effortlessly simple scaling. So, we spin up an openQA worker container in a CI runner, which submits jobs to the openQA server. It has its own UUID, which is shared with the job, so the worker running in CI is always assigned the right job.

This saves us from the bandwidth rigmarole because all the assets are generated and consumed within the one container, so we can simply keep all the assets on the worker and never upload them to the server. As a result, we save a lot of storage space on the openQA server, so we can run it with fairly minimal hosting requirements.

The use of systemd system extensions to inject tests

How do we actually get our Selenium tests on the system, you may ask? Enter the humble system extension , or sysext, for short.

We include a few things in our sysext:

The Python unittests themselves.
A bootstrap script with some system configuration, so we have an appropriate environment set up for testing.
A venv, so we can make use of the Python ecosystem. This is created in the bootstrap script.

All of this is packaged up into an EROFS .img file, which we mount to the worker's VM. This is then automounted by an associated udev rule in upstream KDE Linux, with the bootstrapping script being triggered by an associated service shortly afterward.

Since we have the capability to inject tests and configuration into the system, we're able to test things that would otherwise be impossible to test with needles. For example, we test if essential desktop processes have ever crashed, if any systemd services failed, if networking works, and if commands we ship with KDE Linux work properly. All of these tests leverage direct access to the innards of the system.

Interaction with the system through SSH

To actually poke at the system and have the worker run these tests sequentially, we need some way of interacting with the system. openQA provides some facilities to interact with a serial terminal, but this proved to be very fragile and unreliable, with buffering issues everywhere.

Instead, we set up SSH with our sysext and use the facilities provided by the Python library Fabric to run all our tests in a robust manner.

Each test runs in a transient systemd service created by systemd-run. This runs the test as the intended user, groups its processes in a cgroup, gives it an isolated journal stream for output, and returns its service exit status synchronously. The harness can then collect the unit's journal even when the test fails, and we keep everything neat and tidy.

Staging images before we publish, and how we test updates

To prevent users from downloading an image that still needs to be tested, we create a staging directory on storage.kde.org, scoped to the imaging stage's job ID, that stores the built artifacts in a directory tree. It has a layout that mirrors the public-facing tree, so we can simply merge it in once tests pass.

However, this throws a spanner in the works when we try to test system upgrades because we obviously can't upgrade to an image that hasn't been published yet!

To fix this, the solution is simple. In the sysext, we simply point systemd-sysupdate to the staging directory we've already created. This has some drawbacks, though. For the moment, we can't test delta updates through kde-linux-sysupdated. That shouldn't be too difficult to fix in the near future, but we're waiting on KDE Linux to be entirely hosted on storage.kde.org before we jump on that. The bigger issue here is that we really don't have a good way of testing updates from the chunk store. A better story for this still needs to be worked out, but for the time being the upgrade test is good enough.

What's to come

We have a few things we're aiming towards:

As mentioned above, testing delta/chunked upgrades.
Leveraging openQA to allow app developers to test their own apps atop KDE Linux.
Generalizing all our openQA glue so other projects can use the architecture we've built.
By extension, porting the aforementioned glue from admittedly fragile bash scripts to Python or some other more appropriate language.
Testing installs using manual partitioning and Full Disk Encryption.

…and probably many more things that we haven't thought of yet. Exciting times!