Wanted: Brave Developers

Introducing: KExtProcess
Those who have tracked my KDE commits this year will have noticed that most of them went to KExtProcess, my pet Swiss Army Knife for anyone who writes code that calls external applications.

Due to several reasons I haven't been able to put a whole lot of time in it, so it took me almost 5 months to get from version 0.3 to 0.4, while the amount of man-hours actually put into it probably doesn't exceed 80 hours. But at long last it's there now, KExtProcess 0.4!

If you've never heard of KExtProcess before, I won't be surprised. The three previous versions were basically just my own internal milestones towards this version. KExtProcess 0.4 has been the target version where I wanted to give it more exposure, so if you are a developer and use any of the following technologies in your code, read on:

  • KProcess
  • kdesu
  • ssh
  • KIO-Fish
  • su

KExtProcess is a library for application developers. Although I do have two demo applications, KExtProcess is not an application; it's purely a library.

KExtProcess is named and modelled after KProcess, the KDE class for calling external applications.

Crossing the Border: Beyond KProcess
The really cool thing about KExtProcess is that it is called an Extended KProcess for a reason: it is not limited to the computer and the user account you're running your application with. That's right, it allows you to run processes as another user with almost the same API as KProcess!

With just a few lines of code you can easily run an application over SSH on your server, write to its input and read its output. Writing GUI applications for system administration is a lot easier. And it doesn't even require the user to have KDE installed on his or her server, even X is not required!

And since nobody is going to read on without a teaser image (and I need you, so you'll have to read on :) ), here's a screenshot of the demo application showing KExtProcess in action:

Crossing More Borders: Multiple Hops
Many administrators don't allow logging in as root over SSH directly, and many remote systems in larger companies require 'club-hopping' over multiple servers instead of allowing direct connections from workstation to end-node.

As I have been working on exactly such a setup on my previous job I know the related problems all too well: KIO-SFTP and KIO-Fish cannot be used for most remote administration if you can't login as root remotely. KExtProcess is explicitly designed to overcome this problem by allowing the application to chain multiple steps together and treat them as a single logical process.

Crossing the Ultimate Border: Empowering the User
Compared to using KExtProcess for a direct SSH login to [email protected] it doesn't take a lot more code to login as user@server instead and have KExtProcess then use a second KExtProcess::SuProcess step to 'su' to root.

But, that wouldn't be very flexible, no?

Nope, and that's why KExtProcess sports Profiles, which hold all the steps needed before starting the process in a single entity. On top of that there's the profile editor that can be plugged from a GUI easily to have the user manage the profiles him- or herself.

Profiles are shared across KDE applications, so a profile that works for one application will work with all of them, maximizing the ease of use for the user.

GUI-wise the profile editor is not really finished yet. At aKademy I plan to sit down with Aaron and the rest of the usability gang to replace the dialog with something prettier. From a developer's point of view that doesn't make much of a difference as the API will not have to change much, if at all. In case you are interested in the current profile editor, here's a screenshot:

So Why do I Need You?
That's actually pretty simple: KExtProcess isn't of much value if nobody uses it. So I need developers who want to port their applications to it. I want feedback on the API, bug reports, applications using KExtProcess and ideally people who want to help out making KExtProcess kick even more ass in its next versions.

Be aware though, KExtProcess is not production-ready yet! It's ready for early-adopting developers now, but certainly not for end users on mission critical systems. The two most important known issues:

  • If you enter passwords in a profile they will be stored in plaintext instead of inside KWallet. The obvious workaround is to not store passwords in the profile for now. Just leave the field empty and KExtProcess will prompt you for the password instead.
  • Newline handling is flawed at best. SuProcess and SSHProcess need PTY support even if you don't enable it for the 'real' process on the final hop. Data that comes back through a PTY, however, uses CR/LF rather than just LF, and currently KExtProcess 'solves' this by treating the two as equivalent. Needless to say that this basically prohibits the use of KExtProcess on binary data for now.

Final Words
This blog entry is getting way too long already, so I'll end it here. The upcoming days I'll post additional blog entries with the rest of what I wanted to tell and some more screenshots. For now, if you're the application developer of, say, KSystemLog, KPackage or anything else that uses su, SSH or could otherwise benefit from distributed process execution, take a look at the source code of the KExtProcess Tail Demo to see the ease of use. I'll post a better example later this week, so stay tuned for more cool KExtProcess news!


... you only need to integrate this into that part of KDE that is responsible for launching applications from the kmenu or the desktop will enhance that quite considerably???

That would be a real nice thing. Being able to spawn processes on another computer from a simple mouseclick.

So: feature request: can you add stuff to make KExtProcess aware of the current SCPM profile? Just a thought.

By Ruurd Pels at Sun, 08/14/2005 - 21:54

Wow, you found another possible application of KExtProcess :)

A related one that came up on IRC last week would be to integrate into Konqueror-the-filemanager and when you are browsing fish: or sftp: offer the ability in the context menu to run remotely.

There are dozens of other applications, like remote debugging from KDevelop, embedding KExtProcess into fish: to overcome the limitations when you need root, integrate into Konsole for distant remote shells and GUI support, and whatnot. I planned to mention them in my blog, but it became too long, so that will be a separate post later this week.

Besides, it's bed time here :)

By Martijn Klingens at Sun, 08/14/2005 - 22:16

Wow! This sounds very interesting.

Maybe I can use this stuff to support remote working copies in Cervisia. I will definitely take a look at it.

Bye, Christian

By Christian Loose at Mon, 08/15/2005 - 08:24

If you could look at it for Cervisia that would be awesome, since remote working copies are simple line-based text protocols and as such the only thing that I have really tested so far.

I don't rule out bugs; in fact I expect tons of them, but it's a nice use case that fits extremely well within my goals for 0.5.

Be sure to report back any feedback!

(Which reminds me I need a Bugzilla component... *clicketyclick* Done!)

By Martijn Klingens at Mon, 08/15/2005 - 15:10

That sounds great. I had a request if Kalva could run the commands it builds on a remote host. Imagine a programmable videorecorder that delegates the work to a dedicated tv host :-)

Though I must say I run completely out of time. This feature would be great but before I can dig into it I have still so much work in front to make Kalva even locally a complete app...

By Andreas Silberstorff at Mon, 08/15/2005 - 10:09

Congrats on your release; I know you have been working a long time on this.
Also; your 80 hours are continuous hours. Never underestimate the time you spent on things while you are "logged-out" ;)

By Thomas Zander at Mon, 08/15/2005 - 11:36

That's true, the amount of hours spent on thinking (even if just subconsciously) is hard to count. It's the way I probably designed most of the APIs, as well as the general concept behind the chaining, although the latter was already in KExtProcess 0.3 and thus isn't included in the 80 hours I mentioned.

By Martijn Klingens at Mon, 08/15/2005 - 14:38

This sounds like a really cool utility class. One wacky idea it gives me is that it opens the possibility of running java applets on a different machine to the browser. Quite why you would want to is another question!

By Richard Moore at Mon, 08/15/2005 - 12:46

One possible use for KExtProcess that I found during development is to run management tools remotely. YaST and KControl modules (helpdesk personal that does troubleshooting for a user!) being the main examples, but I can imagine remote Java apps as well.

Together with KParts and QXEmbed it should be fairly trivial to write a top-notch management console that just embeds existing system instead of having to start from scratch.

Note though that the current KExtProcess focuses on console forwarding; X11 support is there only by virtue of SSH's implicit support for it, but not an explicit feature. I hope to add better X11 support in 0.5, but it's tricky for non-SSH, especially when 'su' is needed, since 'sux' is not always available and the custom C++ code that kdesu uses won't work remotely.

The best thing I can think of is probably to have a small perl helper that works like kdesu that can be uploaded in a fish-like fashion. Better ideas are welcome, so be sure to look me up at aKademy if you know something.

By Martijn Klingens at Mon, 08/15/2005 - 14:44

I can see this library becoming an awesome addition to KDE.

It may be outside of the scope of your project, but it would be great to use this to replace the KDE-su dialog that pops up within certain areas of KDE, such as KControl. While it's probably one of the most mundane uses for such a system, it would be best to keep all of that functionality in 1 place. :)

If it were to be used for such things, might I reccomend adding sudo support? I don't use Kubuntu, but I know it would be quite a boon to them, in addition to other distros that might want to take the same path.

By you at Mon, 08/15/2005 - 14:04