Skip to content

Multimedia Frameworks Yesterday, Today, Tomorrow

Thursday, 11 May 2006  |  scott wheeler

OfB: Scott, perhaps summarizing what you've said before, can you give us a roadmap for KDE multimedia? Where is KDE Multimedia heading? Where is the place of KDE Multimedia in the future of KDE?

Scott: Well, I suppose roadmap is probably a pretty good analogy -- since there are a lot of ways to get there. Right now those of us in that are more active in KDE multimedia have tried to lay out a set of requirements for multimedia frameworks; they need to support audio and video decoding, routing that to the appropriate places (sound card, sound server, etc.) and probably some basic recording features. Then we've tried to evaluate what's available to us and how we can make use of the things that are available to best support KDE as a desktop. We learned a lot from aRts, so we're trying to make sure that the options that we're considering will be viable long term; but also we don't want to be explicitly tied to a single framework -- at least not yet. So what some of us started on at aKademy was a framework for abstracting away the basic features -- playing stuff, mostly -- of the media frameworks and making that pluggable. We're pretty sure that we'll have a few implementations around and we'll be able to make a decision on what should be the default at a later time.

The general consensus at aKademy between the multimedia developers is that GStreamer will probably be the most suitable default. But that also means that packagers, administrators and in some cases users will be able to choose what best fits their needs. Naturally we would hope for a default solution that "just works", will be maintained forever, never change its interfaces in a binary incompatible way and solve world hunger, but I suppose we've grown a bit more cynical and cautious. Right now it looks like the best way forward is to pick the most sane default that we can, and hope that users won't ever have a reason to change it, but also to insulate ourselves from being locked into a single framework for the entire KDE 4 lifespan.


That was 2004. There haven't been any major breakthroughs in the last couple of years that invalidate the sentiments there.

The problem isn't as Christian suggests that "no framework 'does it all' so having this flexibility is a good thing", but rather that at present, no framework meets the basic requirements -- as was indicated in the above interview in 2004.

Now, I've been a GStreamer advocate inside of KDE for years. I like the GStreamer project. I like its general architecture, I like its community and I like the general direction that it's headed. But when I put on my KDE hat, I still can't recommend using it directly in kdelibs. (From what I understand of the organization of the GNOME project, this is more or less equivalent to the decision there for GStreamer to remain a "desktop" rather than a "platform" component.)

All other arguments aside, GStreamer doesn't offer a stable API. I can understand why that's the case, but as such, because of the (sane) library policies within the KDE project on binary compatibility we cannot simply use a GStreamer binding as our multimedia solution. Period. I was a little surprised by Christian's posting because we've talked about this multiple times.

So, how do you solve something like that? Well, you could look at the core functionality from GStreamer and how that would be used in the applications in KDE's SVN repository and produce a KDE-ified, task-oriented binding for those core elements that would insulate us against GStreamer version changes. That would probably look a lot like a playbin binding. It would also look a lot like Phonon.

Phonon isn't an attempt to wrap a complete media framework. And it's not just about being indecisive as Jono suggested. That's not to say that there isn't an element of indecisiveness that's played into things, but I think to understand that, especially for a multimedia framework, you have to look at the last few years of history of KDE's multimedia.

We were decisive for KDE 2. And it came back to haunt us. At that time, aRts was a pretty sane choice for a multimedia framework. In 2006 it's not. What we saw was that application developers that either weren't satisfied with the capabilities or performance of our default tended to use other systems. Originally I got a lot of flack for checking in the GStreamer backend to JuK several years ago from the KDE community. But naturally the GStreamer folks were quite happy to see that and today we see a number of KDE applications that support GStreamer and other media systems. What we learned is that a lot can change in the better part of a decade and that application developers are going to pick the things that fit their needs, regardless of what is our default. (How many high end multimedia apps for KDE use aRts?)

But most applications aren't pro-audio apps and they're not non-linear editors. A rich multimedia experience on the desktop is mostly about making it easy for application developers to do the simple stuff (embed video or audio in an application, maybe do a little recording here and there, mix stuff). And that's really what Phonon is for. For the other stuff, which is no doubt cool, I don't think that the lack of those features in Phonon is going to stop anyone from developing the application they want to. It might mean that they spend a couple of hours framework shopping at the get-go.

Now, if I turn around and look at things from a broader perspective, yes, it would be ideal to have a framework that the full weight of the community could be put behind. Having a de facto standard would most likely speed the maturation of that framework and give ISVs something to target. I know as a library developer that one of the best ways to achieve such a point of becoming a de facto standard is to have major projects relying on your library.

But unfortunately, making the call right now, as much as I hope to see GStreamer do well in the future, I wouldn't say that it's a foregone conclusion that GStreamer will remain in its currently dominant position in, say, 2012, which is probably about the time that KDE 4 will be on the way out. If it does, then certainly revisiting this debate when designing KDE 5 will be worthwhile.

Update: After reading some of the comments floating around in blogosphere, there are a couple of points that should be clarified: Phonon does exist and is already semi-functional. Not counting the examples and UI related code, it's under 5k lines of code. GStreamer, for instance, is around 180k lines of code and aRts is around 90k. Maintaining Phonon, which is basically just a collection of interfaces and a little bit of logic is not on the same order as maintaining a multimedia framework.