Implementing C++ implicit type conversions on method arguments in Smoke based language bindings

I'm sorry about the unwieldy title to this blog - I couldn't think of a shorter snappier way of putting it, but I'll try explain the tricky problem with 'C++ implicit type conversions' that I've managed to solve.

Although I've been living in Gran Canaria for over four years, until recently most of my books were still laying around my flat in cardboard boxes. A few months ago I went to the IKEA sale and picked up two of their 'Linnarp' bookshelves for a good price. I was excited about my 'Linnarps' until I read that building them involved hammering nails, and my DIY phobia cut in. I looked at the size of the nails and thought my tiny hammer from my fret saw set was going to be too small, and I didn't have anything bigger. So I tried to buy a hammer in the Spanish shops and found that they don't seem to do small to medium sized hammers, only what I would regard as quite big hammers. Maybe it is to do with the latin 'macho' mind set. Personally, I would rather not use a hammer at all, than use one that I was unhappy with. So my 'Linnarps' just sat there in their boxes for a few more months, until I went back to the UK for Christmas and was able to buy a nice small Stanley hammer.

After waiting patiently to pick up the right hammer, I returned to Gran Canaria and set to with working on the 'Linnarps' and got them built. I could finally put all my treasured books in bookcases where I could actually find them. My computer books filled a whole 'Linnarp', and one of the ones I like most is 'Compilers - Principles, Techniques and Tools' by Aho, Sethi and Ullman or the Dragon Book. I have the first edition with a red dragon on it (the dragon subsequently went from red to green to purple). The dragon on the cover is a metaphor for conquering complexity in compiler design - a mean looking beast, not like our friendly Kongi. If you want to be a knight who slays dragons and don't want to end up as toast, you had better be pretty well prepared and study things like LALR parsers and so on.

If you love solving very complex compiler implementation problems, then C++ is the language for you! It is arguably the most complex language of all with respect to implementing a compiler, and implementing a language binding for a dynamic language to the C++ api involves dealing with some of those same complexities. One of the big 'dragons' that has been bugging me since I first started working on QtRuby over six years ago, is how to deal with the way you can pass an instance of a certain type as an argument to a C++ method, and as long as there is a constructor which takes the argument and can construct an instance of the same type as the target argument then it will work. So if you pass an 'int' to a method expecting a QVariant, then because there is a QVariant constructor that takes an 'int', the C++ compiler will construct a QVariant from the 'int' for you, and pass that to the method.

Here is an example in JSmoke JavaScript of using the Qt 4.6 animation framework:

button = new QPushButton("Animated Button");

animation = new QPropertyAnimation(button, "geometry");
animation.setStartValue(new QRect(0, 0, 100, 30));
animation.setEndValue(new QRect(250, 250, 100, 30));


The code is pretty much the same as it would be in C++, and without implicit type conversions it would look like this, which is just that bit clunkier:

button = new QPushButton("Animated Button");

animation = new QPropertyAnimation(button, new QByteArray("geometry"));
animation.setStartValue(new QVariant(new QRect(0, 0, 100, 30)));
animation.setEndValue(new QVariant(new QRect(250, 250, 100, 30)));


There are several possible ways I had thought of to solve the problem:

  • One solution would be to just special case all the instances in the api where this can occur, but there are quite a lot of places and for QtRuby only some of the more important uses have been special cased.
  • Another way would be to generate extra methods in the Smoke libraries for all the combinations of extra arguments types that are possible, but it would probably mean the libs would be significantly larger and the api would be more cluttered when you looked at it with introspection tools.
  • Instead of adding the extra methods at library code generation time, another approach could be to generate all the extra possible methods at startup after loading the Smoke libs, and add the extra methods to the statically generated lookup tables.

The problem seemed pretty intractable, and rather than solving it badly I preferred not to solve it at all, just like I had waited to get hold of exactly the right hammer before assembling my bookshelves. In the end I came up with the idea of looking up and matching possible constructors for argument type conversions from the existing introspection data, which is pretty elegant. As the complete api is described in the Smoke lookup tables, it is possible to find all the info with no changes needed to the Smoke libs. The method matching code first tries to match without looking for implicit type constructors, and only if that fails does it go into a second more expensive phase. That way, the exacting matching for the majority of methods that don't have any implicit conversions should run at much the same speed as before.

In JSmoke you can set a 'Qt.Debug.trace' global variable to add various sorts of debug tracing. One option is 'Qt.Debug.MethodMatches' which show how methods were matched against their argument types. If we look at the output from matching the QByteArray argument in the QPropertyAnimation constructor, it looks like this:

Qt.Debug.trace = Qt.Debug.MethodMatches;
animation = new QPropertyAnimation(button, "geometry");
Qt.Debug.trace = Qt.Debug.None;

    Argument type conversion [email protected]:-1 for 
            QByteArray::QByteArray(const char*) 
            module: qtcore index: 530 matchDistance: 1
Method [email protected]:-1 for 
    QPropertyAnimation.QPropertyAnimation([object QPushButton:0x09307a78], geometry):
        QPropertyAnimation::QPropertyAnimation(QObject*, const QByteArray&) 
        module: qtcore index: 3922 matchDistance: 4

For each C++ type in the target method, the corresponding JavaScript argument is matched and given a score as to how close the match was. In the case above, an argument type of 'const char *' in the QByteArray constructor was given a value of 1, whereas a QString argument type would have been prefered and given a value of 0. The first argument to the QPropertyAnimation constructor is a 'QObject*' and we provided a QPushButton which is three levels down in the inheritance heirarchy, and so is given a 'matchDistance' of 3, giving the total of the match scores or 'matchDistance' for the QPropertyAnimation constructor of 4. You can see the class heirarchy by using the handy smokeapi command line tool:

$ smokeapi -r qtgui -p -c QPushButton

An extra wrinkle in the C++ complexity of implicit argument type conversions is that as well as matching constructors, the compiler will also look for type conversion operators such as 'QImage::operator QVariant()'. So if you pass a QImage to a method expecting an argument type of 'QVariant' it will use the operator method to construct the QVariant. The JSmoke overloaded method resolution code also handles this case as it can dynamically look up and call those operator methods to do the conversions.

The next stage in implementing the JSmoke runtime will be to add a 'method selector cache' which will allow methods that have already been resolved against a certain set of JavaScript argument types to be retrieved given the JavaScript type signature. The approach I have taken to the implicit type conversions will work well with a cache, as the result of the lookup is encoded into a QVector with one entry for the main method, with optional extra entries for any type conversion methods that are needed. So a language binding based on a language independent introspection library doesn't have to be slower than a conventional static binding as long as the caching scheme works well.


Great read, thanks!

By eike hein at Mon, 02/01/2010 - 04:40

One thing I liked about the kjsembed bindings was that I was able to include implicit conversions from something like a string '#aacccc' to a QColor. In kjsembed, there was quite some overhead, but since jsmoke has a cache of this, it might be possible to provide js-friendly conversions like this with a low average cost (despite the initial hit).

By Richard Moore at Mon, 02/01/2010 - 22:30

Yes, you can already pass a javascript string everywhere a QColor is expected. But that may well clash with other instances that can be created from strings. For example, in this code:

    Qt.Debug.trace = Qt.Debug.MethodMatches;
    painter.setBrush(new QBrush("#aacccc"));
    Qt.Debug.trace = Qt.Debug.None;

    Argument type conversion [email protected]_test.js:-1 for QPixmap.QPixmap(#aacccc):
        QPixmap::QPixmap(const QString&) module: qtgui index: 9342 matchDistance: 0
    Argument type conversion [email protected]_test.js:-1 for QImage.QImage(#aacccc):
        QImage::QImage(const QString&) module: qtgui index: 6501 matchDistance: 0
        QImage::QImage(const char*) module: qtgui index: 6502 matchDistance: 1
    Argument type conversion [email protected]_test.js:-1 for QBrush.QBrush(#aacccc):
        QBrush::QBrush(Qt::GlobalColor) module: qtgui index: 1425 matchDistance: 100
        QBrush::QBrush(Qt::BrushStyle) module: qtgui index: 1394 matchDistance: 100
    Argument type conversion [email protected]_test.js:-1 for QGradient.QGradient(#aacccc):
    Argument type conversion [email protected]_test.js:-1 for QColor.QColor(#aacccc):
        QColor::QColor(const QString&) module: qtgui index: 1617 matchDistance: 0
        QColor::QColor(unsigned int) module: qtgui index: 1616 matchDistance: 100
        QColor::QColor(Qt::GlobalColor) module: qtgui index: 1614 matchDistance: 100
        QColor::QColor(const char*) module: qtgui index: 1618 matchDistance: 1
        QColor::QColor(QColor::Spec) module: qtgui index: 1620 matchDistance: 100
Method [email protected]_test.js:-1 for QBrush.QBrush(#aacccc):
    QBrush::QBrush(const QColor&) module: qtgui index: 1424 matchDistance: 0
    QBrush::QBrush(const QImage&) module: qtgui index: 1400 matchDistance: 0
    QBrush::QBrush(const QPixmap&) module: qtgui index: 1399 matchDistance: 0
    QBrush::QBrush(Qt::GlobalColor) module: qtgui index: 1425 matchDistance: 100
    QBrush::QBrush(Qt::BrushStyle) module: qtgui index: 1394 matchDistance: 100
    QBrush::QBrush(const QBrush&) module: qtgui index: 1401 matchDistance: 100
    QBrush::QBrush(const QGradient&) module: qtgui index: 1402 matchDistance: 100

There are matches on QBrush constructors for QColor, QImage and QPixmap which can all take a string as an argument to their constructors.

You can pass javascript regular expressions or dates everywhere a QDate/QTime/QDateTime or QRegExp is expected in the C++ api. One improvement that would be good perhaps would be to never return QVariants, but always convert them to the values they wrap. The JSmoke bindings really need people trying them out and suggesting api conveniences like that.

By Richard Dale at Tue, 02/02/2010 - 11:37

Very interesting. The same problem occurs at other places as well, e.g. when generating remote stubs for some kind of remote method invocation. Microsoft implements such a thing for their dynamic features embedded in static languages (C# etc.) so they can call COM-objects etc. like they would be normal/native objects. Does D-BUS support method overloading? If yes, then your method dispatching algorithm might be of use there as well. Then in a dynamic language (Python, Ruby, JavaScript) remote stubs can be generated on the fly. So maybe you can write it in a manner so it can be reused at such places?

By Mathias Panzenböck at Tue, 02/02/2010 - 14:34

I based my argument matching on the code in the Qt qscript runtime that dynamically resolves overloaded slots, and calls the one that gives the best match. That's where I got the term 'match distance' from, for how closely a JavaScript argument matches the C++ target type. The resolving slots problem is very similar to resolving overloaded methods in the Smoke lib. However, that code didn't have support for implicit type conversions on the arguments, which is really the main subject of this blog.

Does D-BUS support method overloading?

No and so it is quite simple to call the right DBus method from dynamic languages. The Qt QtDBus api makes a lot of use of QVariants though, and so this particular improvement to JSmoke will make using that api a lot tidier.

By Richard Dale at Tue, 02/02/2010 - 15:00