New Smoke proposal? Bindings broken by design...

I've been following Ashley and Richard's conversations about the new Smoke proposal for KDE4 and I have to say it all reminds me exactly why I have grown to loathe bindings. First, let me say that I have loads of respect for both Ashley and Richard as they are extremely talented developers. Both of them helped me enormously with the now discontinued Qt# bindings. With that said, I think KDE bindings developers and enthusiasts are completely missing the mark for a possible future KDE/Qt binding roadmap.

It seems there is a growing number of KDE'ers with the will to change the KDE binding's situation. If we continue down the path we're now taking for KDE4 we're going to get the current binding's situation perhaps with more visibility in the community. Perhaps the kdebindings module can be cleaned up to the point where it builds reliably well. Perhaps we'll see the adoption of the some of the better bindings (shout out to Ruby, Python and KJSEmbed) in stand alone applications or scripts in Plasma. If these are your only goals, then please, just ignore this post. If however, you want more for KDE bindings then read on...

Let's take the new smoke proposal for instance. Here we already have an enormous library that is about to get even bigger by creating a QObject derived proxy for every non-QObject class in KDE/Qt libraries. Think about that for a second. That means every non-polymorphic class in the current libraries will now have a virtual table. Forget that the hot code paths in KDE/Qt will now have this extra overhead and just think what this will do to the size of this already enormous lib. Further, we'll have to continue overriding each and every existing virtual function and then on top of that we'll need to make it a slot.

What Ashley, Richard and all other bindings developers are doing is actually quite a remarkable achievement, but it is also the source of the problem. I believe the entire concept of bindings as we know it is fundamentally broken. Think about it for a second... what is a binding, really, after all?

Creating a traditional language binding is nothing short of trying to export an entire language's features by writing a shared library. IMHO, the mother of all fundamentally broken designs. Well, actually, there is one that is worse, but you have to read on for that one...

Actually, it is worse than that, because most binding authors see the above as not that challenging enough... no, no, we have to make it harder by writing a program that automatically writes the shared library exporting an entire language's features. LOL.

And what do we get for this back breaking work?

1. A platform that will never match the speed or performance of the original library by design!
2. A platform that will never have the same feature set as the original library by design!
3. The inability to ever write a shared library that can be consumed by the original language by design!
4. The inability to ever write a shared library that can be consumed by other languages bindings "
5. Awkward and arresting incontinuities between the original language's types and the target language "
6. Awkward and arresting incontinuities between the original language's features and the target language "
7. All of this has to be repeated for every new library that we add and every change we make to the current libraries "
8. Third party library makers that are in some way affiliated with KDE/Qt will have the same issues "

In short, a second class development platform by design!

Now, it seems there is a growing number of developers with the will to change this situation. If we continue down the path we're now taking for KDE4 we're going to get the above, but perhaps with more visibility in the community. Perhaps the kdebindings module can be cleaned up to the point where it builds reliably well. Perhaps we'll see the adoption of the some of the better bindings (shout out to Ruby, Python and KJSEmbed) in stand alone applications or scripts in Plasma. If that is the goal, then as I said at the top ;), just ignore this post.

The only real way around the problems I've highlighted above is to, as I've been saying to anyone who will listen, build the binding into the compiler. Actually, it is more like building the binding into the _toolchain_ just like Microsoft has done with the Common Language Runtime. Although, Microsoft is not the first to come up with this idea, they've done the most thorough job of it. One way of solving our problems, of course, is by building a C++ compiler for Mono/PNET based upon the ECMA spec. The advantages here would be an already developed CLR implementation for Windows/Linux/MacOSX. The disadvantages would be the legal uncertainty surrounding this whole path. I'm not in favor of this, myself.

As an aside, I must say that I find it absolutely hilarious the position that the CLR/Mono folks find themselves in. They have this new platform that is supposed to be a binding killer, but they still have to write a binding for their old technologies because they can not be naturally consumed by the new platform. They have to graft their object model that was hacked into the library onto the object model of their new platform. Thinking about how fundamentally broken that is... and I have to say that this is truly the mother of all fundamentally broken designs. Thus, even though they have this new platform, they also are crippled by the problems with traditional bindings. LOL.

The other option would be to adapt LLVM to support a higher level bytecode presentation. I've talked about this in numerous locations and I'm sure some are probably tired of hearing me say it again and again. If you haven't heard about LLVM you can read a little about it here.

To all of you who are tired of hearing me go on and on about this... It is an enormous undertaking, but I can't help but believe that with the renewed interest among KDE developers towards the bindings, and with very talented hackers such as Ashley, Richard, Alexander, Ian, and the rest that we could get much farther and much faster with this approach than with devoting so much time into what I feel is a withering end.


I came here through your semi-troll on the KDE bindings mailing list. ;)

I'm sorry, but you spend a lot of breath attacking the current solution and practically none at all on a possible viable replacement or strategy. You say Mono. Then you say no Mono. You say LLVM. LLVM what? Come on...

The *last* thing I would want to to see is people like Richard be discouraged. Korundum is one of the most exciting things to happen to KDE, IMHO. Just because Qt# failed doesn't mean Korundum doesn't have a future.

By Navindra Umanee at Sat, 06/11/2005 - 21:18

First, it wasn't a troll. I'm also not trying to discourage anyone. Well, only if the truth itself is discouraging. I also made clear at the top that if your goals are just to continue with the current situation, with mild improvements that you could skip it.

BTW, I've talked with Richard before about this. I don't think you have to worry that this will send him into a depression. He well knows the current limitations. I'm just trying to point towards a possible way out of those limitations.

BTW, pointing out that Korundrum has some limitations does not equal saying because Qt# failed that Korundrum has no future. In fact, I said the opposite. It's just what *kind* of future it has. Read a little harder, please.

And if you've been following the lists and this blog you'll find that I _have_ talked about a possible replacement strategy quite a bit. You're also greatly simplifying what I _do_ say in this blog. Mono is one particular implementation of the Common Language Runtime. The CLR comes from the specs that Microsoft submitted to ECMA. One of those included specs, but one which Mono has not implemented, details a C++ compiler for the CLR. That is one avenue, but no it isn't the one I personally want to see KDE take.

As I said in the post, you can find out more about LLVM here. Just click through and keep on clicking through if you are interested. If not, then quit being offended and use your Korundrum bindings, just as I said :)

By Adam Treat at Sat, 06/11/2005 - 21:29

Please understand that I'm not saying that the current bindings are completely no good or that everyone should just stop using them. This post isn't directed at bindings users as much. It is more for the bindings developers.

Also, remember, as I said at the top, if the current bindings situation meets your goals and/or you're happy with the current roadmap in spite of the problems I detailed, then that is absolutely fine. For instance, KJSEmbed really don't have anything to worry about here, because the goal of the binding isn't to provide a complete replacement for KDE/Qt C++. That's not what Ian and the KJSEmbed team are trying to do and as such they aren't effected by the problems I pointed out.

By Adam Treat at Sat, 06/11/2005 - 21:34

I just think you should spend more time explaining your replacement strategy, because honestly I do not get it. If you want KDE to follow your strategy, I think you owe it to explain it better.

I read your long rant, and everyone knows these problems. Richard went into KDE bindings knowing about them. Then Ashley came up with a general solution to it all -- Smoke to bind all bindings. As you say, it was amazing and impressive, and yes I for one have been very happy with what I've seen of QtRuby.

So Smoke is fat, everyone knows that. How is Mono better? Mono is fat too and like you say GTK# still has to be ported. So, what was your point? I'm sorry if I didn't get it, but your post on the kde-bindings mailing list pointed here and here I just read a rant about issues everyone knows about.

Then you say LLVM. What about LLVM? I assume LLVM is not fat and this is why you are so excited. However, does this thing even work? What Unix distribution is deploying it? What about Windows or MacOS?

By Navindra Umanee at Sat, 06/11/2005 - 21:52

"If you want KDE to follow your strategy, I think you owe it to explain it better."

Gotcha. Expect a detailed post about this soon.

"How is Mono better?"

Well, it is better because it solves every problem I pointed out above. With one caveat being GTK#.

"What about LLVM?"

Well, you'll have to read more about it, but real quick... LLVM is a tool chain that includes a ported g++ frontend that can target a JIT or native code. It is actually almost a complete replacement for the g++ toolchain. I've done a fair amount of work so that Qt can compile and link with it.

"I assume LLVM is not fat and this is why you are so excited."

The question doesn't make sense. LLVM is not a library. It is a development toolchain including a C++ compiler and a JIT. Apples and oranges. To answer why I'm excited about think you have to understand what the CLR does and understand that LLVM has a ported g++ that can conceivably compile KDE/Qt C++ into a bytecode representation. This bytecode representation can then be consumed by other language compilers to allow native binding from language to language. Imagine Ruby would be able to directly use KDE/Qt C++ headers and link directly with KDE/Qt C++ shared libraries all in Ruby like syntax.

"However, does this thing even work?"

Yes, it works now. No, we'd have to add a non trivial amount of work to make it into a CLR-like platform.

"What Unix distribution is deploying it?"

Debian. Apple has also hired the lead author to work on it.

"What about Windows or MacOS?"

Yes, it works on both.

More later.

By Adam Treat at Sat, 06/11/2005 - 22:05

Now this is making more sense to me. Okay Mono... You've listed many of the problems already. Such as the fact that Managed C++ isn't really C++. A C++ compiler for Mono doesn't mean you can compile Qt for Mono does it? What about all the C and C++ libraries Qt links to? My head a splode.

Then there's the fact Redhat won't even *ship* Mono. I imagine there is a damn good reason for that.

Thanks for the details on LLVM. You've done some work on porting Qt on LLVM. In that case, *wow*. For some reason this never registered with me after reading your initial post. This is possibly something worth talking about. Heck, I'd go as far as saying this is a very exciting development... possibly the dot audience would like to hear more about it.

What I'd like to hear is how LLVM is anything better than Mono, its relevance to KDE, and how it helps the KDE/Qt bindings issue. Looking forward to that! :)

By Navindra Umanee at Sat, 06/11/2005 - 22:13

A more complete name for Managed C++ is "Managed Extensions to C++." MC++ permits precisely the type of scenario that you describe: mixing managed and unmanaged code. A unique feature among the .NET languages is that MC++ permits writing code that is a fixture of managed and unmanaged code. It is able to work with both kinds of objects using IJW ("it works works") marshalling. MC++ is also able to consume both traditional native code libraries and managed code libraries.

The "normal", unmanaged C++ sections of code in MC++ are not restricted, but of course when writing the managed sections, certain restrictions do allow so that the code is compatible with a managed environment.

My understanding is that Trolltech has built Qt on Windows using MC++.

By Marcus at Sat, 06/11/2005 - 23:31

AFAIK, the legal problems are synonymous with the Redhat problem.

Now, onto whether Managed C++ is a real C++ ... well, you have to differentiate between Microsoft's early C++.NET and the new spec that I linked to below. The updated spec is completely different from MS early attempt at integrating C++ into .NET. And I must say, it does a very good job.

Now, onto LLVM... Yes, I submitted bugs and patches and had Qt4 building and linking with LLVM a few months ago. There was still a couple small problems, but I believe they've been fixed... however I haven't tested LLVM against Qt since that time.

Regardless, I will write up more about what I would like to see a little later.

By Adam Treat at Sun, 06/12/2005 - 00:29

Redhat's problem with Mono is with the Microsoft-specific class libraries that the mono project is reimplementing, the VM itself has never been raised as problematic.

By Waldo Bastian at Sun, 06/12/2005 - 08:57

The VM is problematic in that it is still effectively Microsoft's IP. They do claim the ideas that they've put in around the VM, specific to .Net, as their own. The only catch is at the moment Microsoft has to license all of those claims (including any patents etc.) royalty free and reasonably. All that could change should the ECMA stuff be revoked as standards (Microsoft can quite easily make that untenable).

The VM, and just about everything associated with Mono and .Net, is most definitely a problem for Red Hat. Quite wisely, Red Hat would rather be safe and not play the game of mindshare with Microsoft or have to explain away threats from Microsoft to their customers. It just simply is not worth the trouble it causes. The fact that Red Hat are now heavily involved in the Java application server business is also another major factor.

By segedunum at Sun, 06/12/2005 - 18:51