JUN
27
2011

Randa Review: the buildsystem

I'm back now (since two weeks already) from the KDE Platform sprint in Randa and I have to say it was great.

Randa is located very nice in the Swiss mountains, next to Matterhorn, and very well suited not only for KDE sprints, but also for cycling (I had my bike with me there) :-)

I'm everytime impressed by the exceptional passion and extraordinary talent of the people in our community.
Probably this is because our community is "self-filtered": you don't get in for the money,
but because of personal motivation. ... and so only people who are highly motivated and
"obsessed" by software find their way to us.
David, Stephen, Milian, Sune, Aaron, Volker, Kevin (to name just a few) -- what an amazing list of people.

So, what was in it from the buildsystem POV ?

A lot.
What I realized is that we are now in a different situation than early 2006 when
I introduced CMake into KDE. Back then there were not many free software projects
(outside KDE) which used CMake. So it was ok to put our own cmake modules, macros
and scripts into kdelibs, and so they were available to "everybody" in our
community. Now we have 2011 and now there are a lot of projects outside KDE
which use CMake, among them a set of libraries from KDE developers, but which don't
depend on kdelibs.
For all these projects our CMake extensions are not available, except if they make copies
of them in their own projects.

Making our CMake stuff available outside KDE
So, one major topic regarding the buildsystem was can we make our extensions available also for non-KDE projects ?

Yes, we can !

The following list of actions all work towards this goal:

  • get automoc into cmake, so every cmake user can use it. Gregory ported it in Randa
    away from Qt to STL and the crossplatform library cmsys from cmake. Now we have to figure out
    with the cmake developers in which way this can be integrated nicely into cmake.
  • a small set of modules should qualify for directly going into cmake, e.g. CheckStructHasPointer
  • we have more than 100 Find-modules alone in kdelibs. Getting them into cmake is straightforward.
    You adopt one of the modules, and post to the cmake mailing list "Here is a new module, I want to
    be the maintainer." Then you'll get cmake git access and you'll have to maintain it then
    in cmake, i.e. fix bugs cmake users report, make sure source compatibility is kept, etc. So
    while straightfoward, it requires work and dedication. Once a Find-module made it into
    cmake, we cannot rely on it immediately. We'll have to wait until the next version of cmake
    is released, and if possible until this version has made it at least in the newest versions
    of the major distributions. So all in all, this doesn't sound too promising for us.
  • Instead, we'll create a separate package, and move the Find-modules into that package.
    Then everybody who needs a Find-module can depend on this package. It will be in kdesupport,
    and it will be named "extra-cmake-modules".
    And it will have relatively short release cycles, so new stuff can become available quickly.
    Raphael and me will get this going.
    This will be straightforward, since all the modules in kdelibs have absolutely
    no dependencies on KDE (of course except the obviously KDE-related ones)

Modularization of KDE
Another big topic at the sprint was the repositors split with git.
With these many small git repositories it becomes much harder for the developers to build
KDE, compared to how it was with our big KDE modules.
To help with that, we have kdesrc-build, buildtool, and we found out that Gnome
also has something similar called jhbuild.

I had a look at this issue and noticed that CMake basically has built-in support
for such things since 2.8.0 with its ExternalProjects support.
I worked on this and we now have "Superbuild" CMakeLists.txt for kdesupport and kdegraphics
(Superbuild because it consists only of subprojects, so it is the superproject of these
projects).
Why do we need yet another solution ?

  • it is indeed actually not an additional solution, since it is simply CMake, not a extra new tool
  • it has the additional feature that it can also generate source-packages in flexible
  • granularity, e.g. it can generate a kdegraphics tarball as we had with the big kdegraphics module

  • it should be possible to write an all-in-one CMakeLists.txt, which covers all of KDE SC.
    Since it also handles the dependencies, it is possible to enable only the project
    you are interested in, and it tells you which other projects it depends on, so
    you might enable them to. This way e.g. a source tarball containing everything required
    to build e.g. kruler can be created.

Superbuild can be found already at http://projects.kde.org/projects/kde/superbuild

Testing
We also figured out that we should really try to set up continuous builds for the KDE frameworks,
so we can better ensure quality. More on that later.

How to install properly
With the more fine grained git repositories as we have them already e.g. in kdesupport and
kdegraphics, a lot more packages which contain libraries have to be installed now.
Some libraries we have simply install their headers and library files, some also install
pkg-config files, some install Config.cmake files for cmake. Even kdelibs doesn't do it
in the best possible way, among others because this was new for me back then, and now it has
to stay to keep source compatiblity. E.g. kdepimlibs and kdevplatform do it quite good.
A cmake-based library should install a FooConfig.cmake file along with a FooConfigVersion.cmake
file and export its targets using install(TARGETS ... EXPORT ...).
This file can contain all necessary information about the installed package, including version
numbers, enabled or disabled features, and, especially important for our Windows crew,
the different libraries for debug- and release-versions.

So, what do we do to make more packages install a nice FooConfig.cmake file ?

  • in kdeexamples/ there is a new subfolder buildsystem, which contains now one example,
    "HowToInstallALibrary", which shows how to "perfectly" install a library from a cmake-based
    project
  • It will hopefully not be necessary anymore to additionally install a pkg-config file. cmake will
    support a new command line mode, in which it can be used like pkg-config in purely
    Makefile-based builds or autotools builds (cmake --find-package -DNAME=LibXml2 -DMODE=COMPILE -DCOMPILER_ID=GNU -DLANGUAGE=C)
    With the help from David and Ryan (the dconf guy) I was even able to write a working
    cmake.m4, which autotools-based projects can use to find a library which has been built itself
    with cmake

All in all, we got a lot done, but the work just started.
A first success is that basically our macro_optional_find_package() was accepted into cmake (in a somewhat different way, it will be possible to disable each non-required find_package() via a CMAKE_DISABLE_FIND_PACKAGE_PackageName switch), but it has the same effect. So, one thing which can then also be used outside of KDE.

Next in list, the pkg-config-compatible mode.
Let's see.

Comments

Did some German slip into your English text? ;)

"Superbuild because it besteht only of subprojects"


By krake at Tue, 06/28/2011 - 07:38

Ooops, sorry.
Fixed.

Alex


By Alexander Neundorf at Tue, 06/28/2011 - 19:33

Having a buildsystem dir in kdeexamples/ is a great idea, will look at it for future work!

About "macro_optional_find_package()" getting accepted into cmake, did it got renamed when it went in? I like what the function does, but the naming is terrible. It should be something like "find_optional_package()" IMO.


By aurélien gâteau at Tue, 06/28/2011 - 23:11

The macro was not simply accepted.

Instead, probably with cmake 2.8.6, every normal call to find_package() can be disabled via a CMAKE_DISABLE_FIND_PACKAGE_ variable.
(if you try to disabled a REQUIRED one you get an error).

So it works out of the box for all non-REQUIRED find_package() calls, no extra work required anymore.

Regarding the prefix "macro_": I decided to add some prefix so it is obvious this is not a built-in thing coming with cmake. We have now a git repository "extra_cmake_modules", so I think all macros we will have there will have the prefix "ecm_".

Alex


By Alexander Neundorf at Wed, 06/29/2011 - 17:22

Sounds good, thanks for the clarification.


By aurélien gâteau at Thu, 06/30/2011 - 11:03