Meta-Programming Is Fun

Meta-Programming is becoming an increasingly important part of my life as a software developer. It's a fascinating way to take programming to a new and higher level. Ian has blogged some thoughts about Meta-Programming. I would like to respond to them:

First of all you might ask: What the hell is Meta-Programming? I tried to give some answers in my presentation about Meta-Programming in KDE at aKademy. A simple answer is: Let the computer write your programs. Stop doing the boring work yourself and focus on the interesting bits. I have written many many thousands lines of code in my life. Solving a problem for the first time is interesting, solving it a second time starts to get boring, solving it again is just plain annoying. The same patterns occur many times, the code to be written is obvious from the beginning, it becomes grunt work to do what is already clear in your mind. The more experience you get the more boring it becomes to write standard code. But there is a way out: Put the boring repetitive schematic work into code, write a program that does this job for you and stick to the interesting and challenging parts of programming. That's Meta-Programming.

KConfig XT was one of my first bigger attempts at Meta-Programming. I was tired of writing lots of stupid code to access configuration data again and again. Creating a generic solution to this problem in plain C++ is hard and doesn't give a nice interface to the application developer. Creating specialized solutions for one specific application however is simple and results in a very nice interfaces, but it's tedious work. The consequence of this for me was to write a program which generates the simple, but tedious code automatically, so that the application developer can focus on more important tasks. The result is quite satisfying. More and more applications get ported to KConfig XT, more and more code stupid becomes obsolete. As I consider deleting code as one of the finest tasks of a software developer I take this as success.

My next Meta-Programming project was to solve the annoying problem of handling XML data. How much XML parsing code have you written? Probably too much. Is it fun? No, definitely not. SAX and DOM are better than directly parsing angle brackets, but if you just want to transfer the data of an XML file into a C++ object representation all this parsing stuff is just a waste of time. Instead of writing all this DOM node iterating or SAX callback code you would simply want to tell your program: Just do it. And that's exactly what my kxml_compiler is supposed to do. Tell it how the XML looks like and all the necessary code is automatically generated. It's there and it works reasonably well for a project in this early stage.

Ian notes that maintaining code generators is "nightmarishly complex" and concludes that it is not fun. He is right about the complexity but I have to disagree with the fun part. Meta-Programming gives you an incredibly amount of power. When you do Meta-Progrmming you get into the habit of solving problems "once and for all" which is made possible by the higher level of abstraction and indirection. This is a really interesting challenge. Writing code is nice, writing code that writes code is fun (well, and writing code that write code which writes code, like for example an XML writer adds the insane part of the fun).

The key point is to use tools which help with writing meta code. kconfig_compiler is a big pile of spaghetti code, that's hard to maintain. For kxml_compiler I thus introduced libkode, a library which helps with creating code. With libkode instead of writing C+ statements to a file you create file, class, function, variable and all the other objects which constitute a program and the library does all the file writing stuff, including nice formatting , keeping track of declarations, includes and other tedious stuff. It also helps with translating working code into a code generator by the "kode --codify" tool which converts a program into another program which is able to create the original program. You can then take parts out of it, replace special names, variables, constructs by generic code and the code generator is done. For me this has proven to be a very useful tool which speeds up development and helps to not get lost in the madness of quoting and double quoting.

Ian seems to go into the direction of template frameworks. This can be a very efficient and convenient way to generate code. I have done some work using the Velocity template engine from the apache project with Java and this turned out to be a very pleasent way to generate HTML code, because it gives a good separation of presentation on one hand and logic and data on the other hand, combined with a very simple but still flexible and powerful way to make use of templates. For complex problems it develops a tendency to become messy, though, because the complexity then appears on multiple levels in several different ways.

That's the reason why I prefer to create C++ code with C++. I want to create C++ code and writing C++ is what I know best, so it seems very logical to me to use plain C++ for the code generator instead of a template engine which adds an additional language or format, or another programming language like Perl or Python. Another reason is that code generators quickly become large and complex projects. To have the power of a real programming language which is suitable for these kind of projects helps a lot.

To explore all these aspects of Meta-Programming is one of the most interesting, challenging and exciting things I have done in the area of programming in the last time. I see it as a kind of natural evolution of software development, like going from assembler to a high-level programming language. It will make our programming life easier and will allow us to write better code in less time. Yes, and it definitely is fun.