CI/CD on KDE Blogs

New Craft cache 25.01 published

Julius Künzel, and Hannah von Reth — Sun, 02 Feb 2025 00:00:00 +0000

A new Craft cache has just been published. The update is already available for KDE's CD, CI (Windows/Android) will follow in the next days.

Please note that this only applies to the Qt6 cache. The Qt5 cache is in LTS mode since April 2024 and does not recieve major updates anymore. We highly recommend to port your Qt5 app packaged by Craft to Qt6 as soon as possible!

Changes (highlights)

General

We added CI for flake8, isort and black with the help of tox (which makes it easy to run them locally too) to all Craft repositories. To be able to do so we did a lot of best pratice cleanup beforehand like eg. removing star imports.

Craft Core

Drop support for MSVC 2017
Introduced a CraftBool helper. This allows handy things like self.subinfo.options.dynamic.withMyLib.asOnOff instead of 'ON' if self.subinfo.options.dynamic.withMyLib else 'OFF'
Fix: let the Meson build system respect the buildStatic option
Handle --enable-static --enable-shared in AutoToolsPackageBase instead of in every single blueprint
Python packages (Linux and Windows MSVC; macOS is work in progress):
- Build them ourself instead of using the pre-build binaries from pypi.org
- Use proper staging
- Allow to deploy/package them
Properly set Craft env when branch is switched (eg. with Craft Master in CI)

Blueprints

libjpeg-turbo 3.0.3
Multiple fixes for build of shared vs. static libs
libvpx 1.15.0
Add minGW 14.2 (not the default yet!)
7z 24.09
KShimgen 0.6.1
linuxdeploy-plugin-qt 2.0.0-alpha-1-20250119
qtkeychain 0.15.0

About KDE Craft

KDE Craft is an open source meta-build system and package manager. It manages dependencies and builds libraries and applications from source on Windows, macOS, Linux, FreeBSD and Android.

Learn more on https://community.kde.org/Craft or join the Matrix room #kde-craft:kde.org

New Craft cache 24.10 published

Julius Künzel, and Hannah von Reth — Wed, 16 Oct 2024 00:00:00 +0000

A new Craft cache has just been published. The update is already available for KDE's CD, CI (Windows/Android) will follow in the next days.

Please note that this only applies to the Qt6 cache. The Qt5 cache is in LTS mode since April 2024 and does not recieve major updates anymore.

Changes (highlights)

Craft Core

Drop Python2 support
Require at least Python 3.9

Blueprints

Qt 6.8.0
FFmpeg 7.1
Kirigami Addons 1.5.0
KDE Frameworks 6.7.0
KDE Plasma 6.2.0
Removed snoregrowl
Removed ctemplate

About KDE Craft

KDE Craft is an open source meta-build system and package manager. It manages dependencies and builds libraries and applications from source on Windows, macOS, Linux, FreeBSD and Android.

Learn more on https://community.kde.org/Craft or join the Matrix room #kde-craft:kde.org

KDE Goals - Our Cumulative Culture

Farid Abdelnour — Mon, 26 Aug 2024 00:00:00 +0000

Every two years, the KDE community selects three goals that serve as focal points for the entire community's efforts in the coming years. This cyclical process of goal-setting and community-wide focus is a great example of KDE's Cumulative Culture in action.

This concept, typically observed in human societies, refers to the ability to build upon previous knowledge and innovations to create increasingly complex and effective solutions. In KDE's case, each cycle of goals represents a new layer of accumulated wisdom, i.e. new features and more stability.

The First Cycle (2018-2020)

The first cycle of goals laid the groundwork with its focus on community growth, privacy, and usability.

Streamlined Onboarding: Focused on attracting and retaining new contributors by making the onboarding process smoother and more engaging.
Privacy Software: Prioritized user privacy and security, ensuring KDE software respects user data and complies with security standards.
Usability & Productivity: Aimed to enhance the usability and productivity of KDE software, making it powerful yet easy to use.

The Second Cycle (2020-2022)

The second cycle tackled more complex challenges. Goals like Wayland implementation improvements (which layed the foundation for the Plasma 6 release), improving the app ecosystem, and ensuring consistency in design and functionality.

Wayland: This task aimed at stabilizing Wayland support accross KDE apps.
All About the Apps: Improved KDE's app infrastructure, enabling more efficient app delivery and better support services.
Improve Consistency across the Board: Ensured uniformity in design and functionality across KDE software, improving usability and reducing redundancy.

The Third Cycle (2022-2024)

The third cycle, which is currently coming to an end, was about progress and adaptation. A focus to include environmental responsibility, operational efficiency, and inclusive design.

Sustainable Software: Focused on making KDE software more energy-efficient and environmentally friendly by implementing practices that reduce resource consumption and ensure long-term sustainability.
Automate and Systematize Internal Processes: Aimed to streamline KDE’s internal workflows by automating repetitive tasks, adding code tests across projects and creating a Quality Assurance team to name a few.
KDE For All: Seeked to make KDE software accessible and inclusive for all users.

A New Cycle A Comin' (2024-2026)

Now, as we enter the fourth cycle of the KDE Goals, we see the full power of this cumulative process. Each goal, whether fully achieved or not, contributes to the collective knowledge and capability of the KDE community. Ideas and partial solutions from past cycles become a solid foundation of knowledge and experience that support future efforts.

The commmunity is currently voting on the following proposals for the next KDE Goals cycle that will guide our efforts and shape our focus for the coming years:

KDE Goals at Akademy

The three most voted goals will be announced at Akademy, where there will also be a wrap-up talk about the achievements of the current goals. Also, there will be Birds-of-a-feather (BoF) sessions with the new goal champions.

Join the Matrix room and keep an eye on the website for the latest KDE Goals updates.

New Craft cache 24.08 published

Julius Künzel — Sun, 18 Aug 2024 00:00:00 +0000

A new Craft cache has just been published. The update is already available for KDE's CD, CI will follow in the next hours or days.

Please note that this only applies to the Qt6 cache. The Qt5 cache is in LTS mode since April 2024 and does not recieve major updates anymore.

Changes (highlights)

Qt 6.7.2
FFmpeg 7.0.1
llvm 18.1.8
boost 1.86.0
OpenSSL 3.3.1 (for Android too, which was still on 1.1.1v until recently)
CMake 3.30.0
Ninja 1.12.1
Removed qt-installer-framework (Windows)

About KDE Craft

KDE Craft is an open source meta-build system and package manager. It manages dependencies and builds libraries and applications from source on Windows, macOS, Linux, FreeBSD and Android.

Learn more on https://community.kde.org/Craft or join the Matrix room #kde-craft:kde.org

Farewell, Binary Factory! Add Craft Jobs for Your Apps to KDE's GitLab Now

Ingo Klöcker — Tue, 30 Jan 2024 00:00:00 +0000

This is the final update on the migration of the Craft jobs from Binary Factory to KDE's GitLab. Since the last blog the last missing pieces have been put in place.

We now build a KF6 runtime which is used for the nightly flatpaks of many of the apps that will be part of KDE's Megarelease 6.

Moreover, additionally to signing the sideload APPX packages (see previous blog) the Windows Craft jobs now also sign the NSIS (.exe) installers and all binaries included in the installers. This completes the port of the Windows Craft jobs from Binary Factory to KDE's GitLab.

Now is the time to add GitLab jobs to your project for builds previously run on Binary Factory. The 24.02 release branch has been cleared for using our signing and publishing services, so that you can prepare builds of AppImages, Flatpaks, Android packages, macOS installers, and Windows installers for the 24.02 release of your project, or any other release if you release independent of KDE Gear. To enable those builds add one or more of the following GitLab templates to your project's .gitlab-ci.yml.

craft-appimage.yml (Qt 5), craft-appimage-qt6.yml (Qt 6)
flatpak.yml
craft-android-apks.yml (Qt 5), craft-android-qt6-apks.yml (Qt 6)
craft-macos-arm64.yml (Qt 5), craft-macos-arm64-qt6.yml (Qt 6)
craft-macos-x86-64.yml (Qt 5), craft-macos-x86-64-qt6.yml (Qt 6)
craft-windows-x86-64.yml (Qt 5), craft-windows-x86-64-qt6.yml (Qt 6)
craft-windows-mingw64.yml (Qt 5), craft-windows-mingw64-qt6.yml (Qt 6)

All jobs except for the Flatpak job use Craft for building and packaging your app. You may have to add a .craft.ini file to your project's root folder for overriding the defaults of Craft and the Craft blueprints of your project or your project's dependencies.

What's Next

Next I'll work on making it possible to create and publish Android Application Bundles (AAB) additionally to APKs for your Android apps. Application Bundles contain the binaries for all supported architectures in a single package (instead of multiple different APKs for each architecture). This packaging format is required for new applications published on Google Play.

From GitLab to Microsoft Store

Ingo Klöcker — Wed, 20 Dec 2023 00:00:00 +0000

This is an update on the ongoing migration of jobs from Binary Factory to KDE's GitLab. Since the last blog a lot has happened.

A first update of Itinerary was submitted to Google Play directly from our GitLab.

Ben Cooksley has added a service for publishing our websites. Most websites are now built and published on our GitLab with only 5 websites remaining on Binary Factory.

Julius Künzel has added a service for signing macOS apps and DMGs. This allows us to build signed installers for macOS on GitLab.

The service for signing and publishing Flatpaks has gone live. Nightly Flatpaks built on our GitLab are now available at https://cdn.kde.org/flatpak/. For easy installation builds created since yesterday include .flatpakref files and .flatpakrepo files.

Last, but not least, similar to the full CI/CD pipeline for Android we now also have a full CI/CD pipeline for Windows. For Qt 5 builds this pipeline consists of the following GitLab jobs:

windows_qt515 - Builds the project with MSVC and runs the automatic tests.
craft_windows_qt515_x86_64 - Builds the project with MSVC and creates various installation packages including (if enabled for the project) a *-sideload.appx file and a *.appxupload file.
sign_appx_qt515 - Signs the *-sideload.appx file with KDE's signing certificate. The signed app package can be downloaded and installed without using the Microsoft store.
microsoftstore_qt515 - Submits the *.appxupload package to the Microsoft store for subsequent publication. This job doesn't run automatically.

Notes:

The craft_windows_qt515_x86_64 job also creates .exe installers. Those installers are not yet signed on GitLab, i.e. Windows should warn you when you try to install them. For the time being, you can download signed .exe installers from Binary Factory.
There are also jobs for building with MinGW, but MinGW builds cannot be used for creating app packages for the Microsoft Store. (It's still possible to publish apps with MinGW installers in the Microsoft Store, but that's a different story.)

The workflow for publishing an update of an app in the Microsoft Store as I envision it is as follows:

You download the signed sideload app package, install it on a Windows (virtual) machine (after uninstalling a previously installed version) and perform a quick test to ensure that the app isn't completely broken.
Then you trigger the microsoftstore_qt515 job to submit the app to the Microsoft Store. This creates a new draft submission in the Microsoft Partner Center. The app is not published automatically. To actually publish the submission you have to log into the Microsoft Partner Center and commit the submission.

Enabling the Windows CD Pipeline for Your Project

If you want to start building Windows app packages (APPX) for your project then add the craft-windows-x86-64.yml template for Qt 5 or the craft-windows-x86-64-qt6.yml template for Qt 6 to the .gitlab-ci.yml of your project. Additionally, you have to add a .craft.ini file with the following content to the root of your project to enable the creation of the Windows app packages.

[BlueprintSettings]
kde/applications/myapp.packageAppx = True

kde/applications/myapp must match the path of your project's Craft blueprint.

When you have successfully built the first Windows app packages then add the craft-windows-appx-qt5.yml or the craft-windows-appx-qt6.yml template to your .gitlab-ci.yml to get the sign_appx_qt* job and the microsoftstore_qt* job.

To enable signing your project (more precisely, a branch of your project) needs to be cleared for using the signing service. This is done by adding your project to the project settings of the appxsigner. Similarly, to enable submission to the Microsoft Store your project needs to be cleared by adding it to the project settings of the microsoftstorepublisher. If you have carefully curated metadata set in the store entry of you app that shouldn't be overwritten by data from your app's AppStream data then have a look at the keep setting for your project. I recommend to use keep sparingly if at all because at least for text content you will deprive people using the store of all the translations added by our great translation teams to your app's AppStream data.

Note that the first submission to the Microsoft Store has to be done manually.

APKs now built on invent

Ingo Klöcker — Mon, 13 Nov 2023 00:00:00 +0000

The migration of jobs from Binary Factory to KDE's GitLab continues. Last week almost all jobs that built APKs were migrated to invent.kde.org. The only remaining Android job on Binary Factory doesn't use Craft.

The Android APK jobs running on invent are the first jobs that make use of our CI Notary Services delegating tasks that require sensitive information (e.g. the keys to sign the APKs or the credentials to upload the APKs to our F-Droid repositories) to services running outside of GitLab. Currently, the apksigner service and the fdroidpublisher service are live. The configuration of the services can be found in the sysadmin/ci-utilities repository.

As before, the APKs are published in our F-Droid repositories:

KDE Android Release Builds provides APKs built from the 23.08 branches.
KDE Android Nightly Builds provides APKs built from the master branches.

Many apps have already switched from Qt 5 to Qt 6, but the Qt 6 APKs are not yet ready for public consumption. Not even as nightly builds. Therefore many "nightly" builds are many weeks old.

The migration to invent has a few implications. On Binary Factory all APKs (release and nightly) were rebuilt once a day even if nothing changed. On invent the APKs are rebuilt whenever a change is pushed to the release/23.08 branch or the master branch of a project. Another change is that on invent any KDE developer can manually trigger a new pipeline to build the APKs (e.g. if a bug in a dependency was fixed). On Binary Factory you often needed to ask someone to trigger a build for you.

Building APKs If you want to start building APKs for your project then add the craft-android-apks.yml template for Qt 5 or the craft-android-qt6-apks.yml template for Qt 6 to the .gitlab-ci-yml of your project. Note that you must use the include:project format (example).

The jobs will build unsigned APKs. To enable signing your project (more precisely, a branch of your project) needs to be cleared for using the signing service. This is done by adding your project to the project settings of the apksigner. The master branch is cleared by default for all applications listed in the project settings, so that you only need to add two lines: The repo path on invent and the ID of your application. The key used to sign your application will best be created by us directly on the machine that does the signing.

Once you think the APKs are ready for publication you can enable publishing in our F-Droid repositories by adding your project to the project settings of the fdroidpublisher. Because some Qt 6 APKs are not yet ready for publication although their build succeeds, by default the master branch is not cleared for publishing. This means you will have to add five lines to the project settings of fdroidpublisher.

Outlook To complete the Android jobs/services a job/service for submitting APKs to Google Play will go live soon. It's currently only used by Itinerary but, in view of our Make a Living initiative, we hope to publish more apps on Google Play to create some revenue to fund our work. Creating and publishing Android Application Bundles (AAB), the new packaging format required for new applications published on Google Play, will follow soon after.

Then we'll add signing of Windows artifacts and installers and submission to the Microsoft Store, signing and publishing of Flatpaks, and signing and publishing of builds for macOS.

NeoChat Published in the Microsoft Store

Ingo Klöcker — Mon, 16 Jan 2023 00:00:00 +0000

Carl Schwan already announced it on discuss.kde.org and at @neochat@fosstodon.org.

In this blog we’ll see how it was done and how you can publish your KDE app in the Microsoft Store.

Reserving a Name and Age Rating Your App

The first step requires some manual work. In Microsoft Partner Center you need to create a new app by reserving a name and complete a first submission. How to do this has been described by Christoph Cullmann in the Windows Store Submission Guide. Don’t hesitate to reserve the name of your app even if you are not yet ready for the first submission to the Microsoft Store. Once a name is reserved nobody else can publish an app with this name.

The first submission needs to be done manually because you will have to answer the age ratings questionnaire. NeoChat was rated 18+ because it allows you to publish all kinds of offensive content on public Matrix rooms. Filling out the questionnaire was quite amusing because I did it together with the NeoChat crowd at #neochat:kde.org.

On the first submission of NeoChat I chose to restrict the visibility to Private audience until it was ready for public consumption. I created a new customer group NeoChat Beta Testers with the email address of my regular Microsoft Store account in Microsoft Partner Center and then selected this group under Private audience. This way I could test installing NeoChat with the Microsoft Store app before anybody else could see it.

Don’t spend too much time filling out things like Description, Screenshots, etc. under Store Listings because some of this information will be added automatically from the AppStream data of your app for all available translations.

Semi-automatic App Submissions

The next submissions of NeoChat were done semi-automatically via the Microsoft Submission API with the submit-to-microsoft-store.py script while writing this Python script and the underlying general Microsoft Store API Python module microstore. The script is based on a Ruby prototype (windows.rb) written by Harald Sitter.

The idea is that the script is run by a (manual) CI job that the app’s release manager can trigger if they want to publish a new version on the Microsoft Store.

To run the script locally you need the credentials for an Azure AD application associated with KDE’s Partner Center account. Anything else you need to know is documented in the script’s README.md.

Making NeoChat Publically Available

The last step of the process to get NeoChat published in the Microsoft Store was another manual submission which just changed the visibility to Private audience. This could also have been done via the Microsoft Submission API (but not with the current version of the script), but I think it’s good to have a last look at the information about the app before it is published. In particular, you may have to fill out the Notes for certification, e.g. if your app cannot be tested without a service or social media account. For NeoChat we had to provide a test account for Matrix.

Moreover, you may want to fill out some details that are currently not available in the AppStream data, e.g. a list of Product features, the Copyright and trademark info, or special screenshots of the Windows version of your app.

What’s Next

On our GitLab instance, we want to provide a full CI/CD pipeline for building and publishing our KDE apps on the Microsoft Store (and many other app stores). A few important things that require special credentials or signing certificates are still missing to complete this pipeline.

And we want to get more KDE apps into the Microsoft Store.

If you need help with getting your KDE app into the Microsoft Store, then come find me in the #kde-windows room.

Updates after publication:

2023-01-31: Updated link to script after merge of the MR