Rich on KDE Blogs

A trip down memory lane

Rich — Sat, 17 Nov 2012 00:00:00 +0000

I was digging through some old backups on Friday looking to see if I had any old versions of the Qt source code lying around after Eirik mentioned during his devdays talk that the release tar balls for lots of the early releases including Qt 1.0 had got lost... I didn't find those, but I found some gems I didn't know I had.

First, there's a presentation on the KDE internals I don't know where I gave, I suspect it was at MANLUG (the Manchester Linux User Group).

Next, there's a presentation The design of the K Desktop Environment I gave at Simplinux in Faro, Portugal back in 1999. This one has a bit more depth than the first It's from back before we released KDE 2.0. This one talks about us using CORBA. The biggest thing in retrospect that happened at this conference is I met Nuno Pinheiro KDE's artist in chief. The colour scheme of the slides might make your eyes hurt (that's why Nuno's so important), I recommend the text version myself!

Finally there's a presentation The Technology Behind KDE 2.0 that was given after the release of KDE 2.0 but before the release of KDE 2.1. I've no idea where I gave this one. It mentions the removal of CORBA, and brings in important tools like KSycoca that we still use today. It also talks about ~~DBUS~~DCOP which was the conceptual parent of the DBUS API that is used all over linux today.

A nice little trip down memory lane I hope you'll agree.

Edit: Changed to a pdf link for the last one since google docs required a sign in.

An Unexpected Journey

Rich — Sat, 19 May 2012 00:00:00 +0000

Recently, when building qt5, I'd started noticing some very strange errors from the configure script. The errors seemed to indicate that an awk script was being used as a filename - very strange. Even stranger was that other people weren't hitting this issue - just me. Never a good sign. Today, I finally got around to debugging it and the issue was rather weird.

My initial thought was that my version of bash was incompatible with the script in some way, so I copied the code for the function that was erroring and made a standalone version - it worked fine.

I then tried to run the configure script using sh -x to watch it was doing, but unfortunately that seemed to confuse the script. Finally, I started to read the code. The relevant function reads:


$awk ' BEGIN {

lots of awk

} '

Finally, I spotted that $awk was not being set. If you look at the script then you can see that the qt5 configure looks for gawk, nawk then finally awk. So, the obvious step was to see if I had a version of awk installed. Using 'which' told me I didn't have one, but using rpm -q told me I did - curiouser and curiouser.

I checked what the gawk package installed and I could see that at least part of it was there, but running it gave an IO error - very weird. In fact, I could see that I was running a symlink that pointed to a file that wasn't there. At this point, I assumed a bad update of some kind, so I ran:

zypper install gawk

It said my gawk was the lastest. I then tried

zypper install -f gawk

to force the install and it gave an IO error. Spot the pattern? Every time I try to do stuff to the gawk binary I get an IO error. At this point, I looked at /var/log/messages and saw a lot of messages like this:

May 19 18:51:06 linux-h33o kernel: [ 58.719815] EXT4-fs error (device
sda6): ext4_ext_check_inode:403: inode #393297: comm rpm: bad header/extent:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0)

Not good.

This seemed like some kind of file system corruption, so I backed up my files immediately before I carried on investigating. Running fsck from a rescue system showed a few errors, but nothing major, and all were fixable. After allowing the repair I rebooted, sure in the knowledge that the problem was solved.

All I needed to do was reinstall the corrupted gawk package:

linux-h33o:/home/rich/src # zypper install -f gawk Loading repository
data...
Reading installed packages...
Forcing installation of 'gawk-4.0.0-3.1.2.x86_64' from repository 'openSUSE-12.1-Oss'.
Resolving package dependencies...

The following package is going to be reinstalled: gawk

1 package to reinstall.
Overall download size: 820.0 KiB.
No additional space will be used or freed after the operation. Continue? [y/n/?] (y):
Installing:
gawk-4.0.0-3.1.2 [error] Installation of gawk-4.0.0-3.1.2 failed: (with
--nodeps --force)
Error: Subprocess failed. Error: RPM failed: error:
unpacking of archive failed on file /bin/gawk: cpio: rename failed -
Input/output error error:
gawk-4.0.0-3.1.2.x86_64: install failed

Oh dear. It's not good when fsck says the file system is ok, but the driver disagrees.

At this point, I started doing some serious googling to figure out wtf was going on. Happily, I came across the following bug report that let me resolve the issue https://bugzilla.kernel.org/show_bug.cgi?id=32182 . By following the debugfs steps described, I was able to kill the bad inode. Thank fully this fixed my file system.

So what can we learn from this? Obviously, we can learn that the configure script wasn't the source of the problem, but simply the way it manifested. It also shows that the configure script has a bug in that it doesn't report when awk is missing. We can also learn that a bit of googling can solve a lot of problems.

A final point to note if anyone is considering doing evil things with debugfs like this is that as soon as I figured out I had a corrupt file system I made a backup. This gives you a nice warm glow inside as you know you can attempt things that would otherwise be insanely risky.

What's New in Qt 5 for SSL?

Rich — Sat, 14 Apr 2012 00:00:00 +0000

With the availability of the Qt 5 alpha, I thought I'd try to summarise what's been done in the SSL stack. Most of the changes in Qt 5 for SSL are incremental improvements, or things that will form the basis of future changes. In this post I'll try to highlight the main changes:

QSslCertificate::subjectInfo() and QSslCertificate::issuerInfo() now return a QStringList instead of a QString

It's pretty common for a certificate to contain more than entry of a specific type, but in Qt 4 the API only let you access the first one.

QSslCertificate::isValid() has been deprecated.

Originally this method was just used to check the validity of the dates for which a certificate was valid, when the certificate blacklist was introduced to deal with issues such as the Commodo compromise, checking for blacklisted certificates was added too. Unfortunately the name of the method gave the misleading impression that simply calling it was enough check the validity of a certificate - it isn't. There is now a new isBlacklisted() method which can be used to check if a certificate is blacklisted (and checking the dates is trivial anyway).

QSslCertificate::alternateSubjectNames() is deprecated and replaced by QSslCertificate::subjectAlternativeNames().

The new name reflects the actual name of this field in the RFCs etc.

The QSsl::TlsV1 enum value was renamed to QSsl::TlsV1_0

Since there are now multiple versions of TLS this is a pretty obvious change.

QSslCertificate::serialNumber() now always returns the serial number in hexadecimal format.

The old code returned the serial number as a hex string if it was long, but as an integer if it was short. For consistency we now always use the same format.

The openssl network backend now reads the ssl configuration file allowing the use of openssl engines.

This change means that people wanting to use openssl engines such as hardware accelerators can do so using their openssl config file and Qt will respect the setting. This was possible before, but required setting a flag at compile time.

QSSlCertificate::toText()

There's now a method to convert a certificate to human readable text (for certain values of human).

QsslCertificate::verify()

There is now a method to verify a certificate chains validity. This means that you can check a certificate chain against the root store directly.

QSslCertificate::extensions()

X.509 certificates can contain extensions (and almost all do). Qt previously only supported the subject alternative name extension. In Qt 5 this new method returns a list of all extensions, and does its best to convert them into a sensible structure. Some extensions such as basicConstraints and subjectKeyIdentifier (for example) are supported to the extent that the structure is defined. Other extensions can also be accessed, though the structure of the information returned may change between versions.

This change isn't massively useful in itself, but it provides a foundation for future improvements.

QSslSocket can now pause on ssl errors

Currently, a nested event loop is required in order to process the sslErrors signal from QSslSocket. In Qt 5, you can use the setPauseMode() method of QAbstractSocket to tell the socket to pause when the error signal is emitted. This means that no data will be transmitted until you tell the socket to continue, allowing the nested event loop to be avoided.

It is intended to extend this facility to cover authentication requests too. In future, you will probably also be able to request that the socket be paused at the end of the handshake even if there were no errors so you can perform future checks. Unfortunately time constraints meant that only the first step described above was completed (and that QNetworkAccessManager does not make use of this facility).

QSslConfiguration::setSslOption()

In Qt5 you can enable and disable various bug workarounds etc. using this method. This change was backported to 4.8.

New QT_NO_SSL define

In older versions of Qt you could make code conditional on SSL support being available using #ifdef QT_NO_OPENSSL, but this is tied to the openssl backend. In order to allow for additional backends in the future such as one using GnuTLS there is now a QT_NO_SSL define too. This means that unless you actually depend on the openssl backend (eg. because you are using the native handles to perform additional openssl calls yourself) you should use QT_NO_SSL.

Support for opaque QSslKeys

Qt 5 adds support for a new type of QSslKey that is 'opaque' this new type can be used to build things like PKCS#11 support into code using Qt. An example of this is at http://git.iksaif.net/?p=qsslkey-p11.git;a=tree

In addition to these new features, there have also been lots of bug fixes etc. too, but you can see those in the bug tracker.

Using GnuTLS with QTcpSocket

Rich — Sat, 14 Jan 2012 00:00:00 +0000

It's been quite a while since I last wrote a blog post, but it's not because I haven't been coding, in fact quite the opposite. The Qt opengov project is finally underway and I've been doing quite a lot of work on the various SSL classes. I'm now an official Qt approver, so as you can see the process of getting non-nokia developers the ability to commit to Qt is working.

In Qt 4.x and currently in Qt5 even though the various QSsl classes provide an abstraction from the underlying SSL implementation, there is only one backend and it uses openssl. I recently made a change that means we can add new backends during the Qt5 life time by separating the concepts of SSL support from the availability of SSL - of course this doesn't change anything if openssl is the only game in town.

Over the last couple of weekends, I've been investigating GnuTLS as a potential implementation that could form the basis of a second backend. I started with the easy part - handling X.509 certificates, and last weekend managed to implement some code that showed that we could implement this part of the Qt API using GnuTLS. I did hit some issues of course, but most were minor. The GnuTLS maintainer Nikos Mavrogiannopoulos has already fixed the minor documentation issues I spotted, and has even implemented a couple of features that I found were missing - definitely a sign that this library is being actively maintained I think you'll agree.

Today I attempted the more challenging task of trying to make GnuTLS work through a QTcpSocket. This integration is essential for using the library in Qt since without it features such as proxy support etc. would not be available - it really is a 'must have'. As you might guess from the fact I'm bothering to blog about it the results were positive, so let's take a look.

What I've done for this prototype is implement a QObject that provides the same basic outline as a QIODevice subclass. In order to keep things simple, I haven't yet tried to make something that is tied to that specific API at this point. I've also not tried to integrate this into QSslSocket, instead this code aims to demonstrate that doing so will be feasible in future.

The API I've implemented is reasonably close to what QSslSocket offers, the header file should make it clear that most of what you'd expect is there:

class SslSocket : public QObject
{
 Q_OBJECT

public:
 SslSocket(QObject *parent=0);
 ~SslSocket();

 QByteArray read(qint64 maxsize);
 
public slots:
 void connectToHost(const QString &hostname, int port);
 void startHandshake();

 qint64 write(const QByteArray &data);

signals:
 void handshakeComplete();
 void readyRead();
 void error();

private slots:
 void dataReceived();

private:
 void setupSession();
 void handshakeInternal();

 ssize_t readInternal(void *buffer, size_t length);
 ssize_t writeInternal(const void *buffer, size_t length);
 ssize_t writeVectorInternal(const giovec_t *iov, int iovcnt);

 static ssize_t read_callback(gnutls_transport_ptr_t transport, void *buffer, size_t length);
 static ssize_t write_callback(gnutls_transport_ptr_t transport, const void *buffer, size_t length);
 static ssize_t write_vector_callback(gnutls_transport_ptr_t transport, const giovec_t *iov, int iovcnt);

 SslSocketPrivate *d;
};

Now you've seen the header, let's take a look at the implementation. The first method we'll look at is the one that performs the initial setup of our SSL session. This happens before we even attempt to send any data to the server since once of the things we're going to do is change the way data is sent to run it via QTcpSocket:

void SslSocket::setupSession()
{
 qDebug() << "Initialise client session";

 // Setup the trust store
 gnutls_certificate_allocate_credentials(&d->x509cred);
 gnutls_certificate_set_x509_trust_file(d->x509cred, "/etc/ssl/ca-bundle.pem", GNUTLS_X509_FMT_PEM);

 // Configure the session
 gnutls_init(&d->session, GNUTLS_CLIENT);
 gnutls_credentials_set(d->session, GNUTLS_CRD_CERTIFICATE, d->x509cred);

 const char *err;
 gnutls_priority_init(&d->priority_cache, "NORMAL", &err);
 gnutls_priority_set(d->session, d->priority_cache);

 // Setup the transport functions to use QTcpSocket
 gnutls_transport_set_ptr(d->session, this);
 gnutls_transport_set_pull_function(d->session, read_callback);
 gnutls_transport_set_push_function(d->session, write_callback);
}

The first thing this method does is setup the trust store, and point it to the location of the CA bundle on my opensuse 12.1 system. In fact, this is a total waste of time, since verifying the certificate isn't something I've implemented yet! The next part is rather more useful however, we initialise a session and tell GnuTLS that we're acting as an SSL client. After this, we set the priority of the various ciphers etc. that are available. This step seemed unimportant, but I discovered that if you fail to do it then rather than using a sane default, GnuTLS dies with an internal error. The last setup step we do is to tell the library that we want to use our own functions to send and receive data rather than using the built-in ones.

For each of the read and write functions, we provide a callback. This is a static method (since a pointer-to-member-function is a no-no for reasons I won't go into). Our static method then calls the member function that implements our callback:

ssize_t SslSocket::read_callback(gnutls_transport_ptr_t transport, void *buffer, size_t length)
{
 SslSocket *self = static_cast<SslSocket *>(transport);
 return self->readInternal(buffer, length);
}

ssize_t SslSocket::readInternal(void *buffer, size_t length)
{
 qDebug() << "readInternal, length" << length << ", available" << d->socket->bytesAvailable();

 if (d->socket->bytesAvailable() < qint64(length)) {
 gnutls_transport_set_errno(d->session, EAGAIN);
 return -1;
 }

 return d->socket->read(static_cast<char *>(buffer), length);
}

As you can see the static function simply casts the user data pointer (which we provided using gnutls_transport_set_ptr earlier) to our class, then calls the appropriate method. The read implementation simply sees if we have enough bytes in our socket's buffer to satisfy the request, and if not tells the library to try again. If we do, then naturally we read the data. The data we need can only become available when the eventloop is running since that's when QTcpSocket performs its data transfers. If we simply retried immediately then we'd see no more data than we did during our first attempt.

The write function is similar, but since QTcpSocket has buffering of its own we can simply perform the write as requested. My real implementation supports a slightly more complex form of the write function that can process requests to write several blocks of data at once, but the basic concept is the same.

In order to allow the retry behaviour I mention above, there's a simple state machine. The SslSocket has the following basic states:

enum State
{
 Disconnected,
 Connecting,
 Handshaking,
 Encrypted
};

If new data is received then we look the state and retry the relevant operation as follows:

void SslSocket::dataReceived()
{
 qDebug() << "dataReceived() state is " << d->state;

 if (d->state == Handshaking) {
 handshakeInternal();
 }
 else if (d->state == Encrypted) {
 emit readyRead();
 }
}

The handshakeInternal() method is the one that actually performs the SSL handshake, and as you can see if we're in the appropriate state then we retry each time more data is received. The actual implementation is pretty simple:

void SslSocket::handshakeInternal()
{
 qDebug() << "start handshake, state is" << d->state;
 int result = gnutls_handshake(d->session);
 qDebug() << "shake result" << gnutls_strerror(result) << result;

 if (result == 0) {
 qDebug() << "handshake completed";
 d->state = Encrypted;
 emit handshakeComplete();
 }
 else if (gnutls_error_is_fatal(result)) {
 qDebug() << "fatal error in handshake";
 emit error();
 d->socket->disconnect();
 d->state = Disconnected;
 }
}

As you can see, once we get a real result from the handshake (ie. one that can't be solved with more data) then we emit either the handshakeComplete() signals or the error() signal and update the state of the SslSocket. If the handshake is completed successfully then we can now send and receive encrypted data.

The read and write methods are similar. In both cases, we need to consider the possibility that we need to retry (indicated by either an interrupted, or again error). The implementations are as follows:

QByteArray SslSocket::read(qint64 maxsize)
{
 QByteArray buffer;

 if (d->state != Encrypted) {
 qFatal("Write before encrypted is not supported yet");
 return buffer;
 }

 buffer.resize(maxsize); // ### This could get pretty big!
 ssize_t result;
 do {
 result = gnutls_record_recv(d->session, buffer.data(), maxsize);
 } while( result == GNUTLS_E_INTERRUPTED || result == GNUTLS_E_AGAIN );

 buffer.truncate(result);

 return buffer;
}

qint64 SslSocket::write(const QByteArray &data)
{
 qDebug() << "write";

 if (d->state != Encrypted) {
 qFatal("Write before encrypted is not supported yet");
 return -1;
 }

 ssize_t result;
 do {
 // ### It's possible that this could fail due to the need to read some data
 // for example if a renegotiation is underway. Not sure.
 result = gnutls_record_send(d->session, data.constData(), data.size());
 qDebug() << "write result" << result;
 } while( result == GNUTLS_E_INTERRUPTED || result == GNUTLS_E_AGAIN );

 return result;
}

In order to test this, I wrote a very simple class that will perform an HTTP GET and read the response. The result is that we can can connect to an HTTPS server and see the root page - not very exciting I guess but a sign that everything is working!

GetSlash::GetSlash(SslSocket *sock)
 : QObject()
{
 this->sock = sock;
 connect(sock, SIGNAL(handshakeComplete()), this, SLOT(start()));
 connect(sock, SIGNAL(readyRead()), this, SLOT(gotData()));
}

void GetSlash::start()
{
 sock->write(QByteArray("GET / HTTP/1.0\n\n"));
}

void GetSlash::gotData()
{
 QByteArray result = sock->read(2000);
 qDebug() << result;
}

Finally there's a basic main() function that ties all this together, and a little bit of code to log what's going on. The end result is that I now feel confident that we could implement a working backend for QSslSocket etc. using GnuTLS without too much trouble.

I'll add the code to my normal qt-examples repository later, but if anyone needs it (or earlier experiment experimenting with the certificate APIs) then just let me know.

EDIT: The code is now available here https://gitorious.org/qt-examples/qt-examples/commit/40e57647242079745c46d84e15d92f2093af300f

Qt SSL Update

Rich — Sun, 06 Nov 2011 00:00:00 +0000

It's been a while since I blogged, so I thought I'd post an update on what I've been up to.

Along with nearly a thousand other Qt developers, I headed over to Munich for the Qt Dev Days. It was my first time at the event, and I was surprised just how big it was. There were a lot of interesting talks, with highlights including Jens' talks on Qt Quick components for the desktop and using QML for rapid prototyping and Peter's talk on Secure Networking.

Before all the talks however, we had a day full of Qt contributor sessions. This included a lot of discussions about how the open governance model should work, and also technical sessions on individual areas. I've posted my notes from a couple of sessions to the new Development mailing list, including my notes from the various discussions on SSL.

I'd like to thank the KDE eV for sponsoring my travel and hotel, Nokia for my Dev Days pass (and a shiny new N9), and my own company, Westpoint, for letting me attend.

In terms of coding, I've been fairly busy too. I've recently added support for disabling fancy new features of SSL/TLS like session tickets, compression and server name indication that since they're so shiny and new (ahem, most of them are ten years old at least) seem to be incompatible with various expensive SSL accelerators. This change has been backported to Qt 4.8 so we should be able to use it in KDE 4.x too.

I've also made a few changes that are only in the Qt 5.0 branch since they're really too big a change to slip into the 4.8 tree at this stage. The major one is the ability to verify a certificate chain without needing to connect to a server.

Most recently, I've written a pretty large improvement to the handling of certificates to add support for certificate extensions. This change has been handled under the open governance system using gerrit, which has resulted in a much faster turn-around than the previous gitorious based approach of merge requests. The new code makes it possible to access all the certificate extensions, with a design that makes it easy to extend as new ones are required. The change is currently in the process of getting merged, and assuming I don't get bitten by an old openssl version on one of the test platforms should be in the main tree tonight.

I have a bunch of further improvements to the SSL support in progress, with my work on adding OCSP support being one of my main targets. There are several other low-hanging fruit like the bug work arounds I mentioned too though, so I'll probably try to address some of those as well.

If you're working on a KDE application and Qt's SSL support doesn't offer what you need, then please let me know so that I can make sure that what you're missing gets added to the todo list.

SSL at the Qt Contributor Summit

Rich — Sun, 03 Jul 2011 00:00:00 +0000

I haven't written any posts for a few weeks, so I figure it's time for an update. I, along with a bunch of other people who hack on Qt, attended the Qt Contributor Summit in Berlin. This was a very useful event, since it brought together a good mixture of both Nokia developers and external developers - as I would have hoped, KDE was one of the most common afiliations. One of the main things I wanted to do during the event was get together with the people responsible for the Qt SSL code and ensure that the plans for Qt5 will meet the needs of KDE (and others who want non-trivial SSL usage).

The discussions with Peter and Markus (the guys responsible for this code) were very productive, to the point that the only reason we didn't get a couple of the smaller items merged then and there were that they didn't have access to the Nokia internal network to do it. Peter, Markus and I have been tracking the stuff we'd like to see sorted in Qt's SSL for quite some time in a wiki page at http://developer.qt.nokia.com/wiki/Improving_Qts_SSL_Support and after the discussions, I forsee few problems in getting any of the improvements on the list incorporated into Qt.

One complex area we touched on was the possibility of changing the Qt APIs so that the various authentication required, and ssl certificate errors signals can be handled without a nested event loop. We even managed to figure out a way of doing this that would remain backwards compatible with code using the current API, so I have some hope that this can be achieved.

Following the conference I've already had a couple of merge requests approved - printing certs as text https://qt.gitorious.org/qt/qtbase/merge_requests/2 and support for duplicated elements in certificate names https://qt.gitorious.org/qt/qtbase/merge_requests/5 (source incompatible with Qt 4). Currently I'm finishing off the code for validation of certificate chains (when not connecting to a server) which should be done in the next week or two, the code is at https://qt.gitorious.org/~rich/qt/richs-qtbase/commits/verify-certificate if you want to look.

Once I've completed the changes above, I currently trying to decide between access to certificate extensions, EV certificate support and finishing integrating my OCSP code.

Improving SSL Security with Certificate Change Notification

Rich — Sun, 22 May 2011 00:00:00 +0000

Improving the security of SSL is a hot topic these days, and trust in certifcate authorities is rightly at an all time low. One way of improving the situation that doesn't rely on believing that a 3rd party will actually do their job properly is to notify users when the certificate for a site changes. There are already extensions for some browsers that offer this facility, so I sat down today to write a proof of concept that looked at how this implemented in Qt.

What I've implemented is a class that can simply be plugged into QSslSocket based code to give the user a warning if the certificate for a site has changed since their last visit. The monitor class can keep track of many sockets and the cache is persistent. The idea is that the application tells the certificate monitor about the SSL socket, so that it becomes monitored by calling the addSocket() method, this method simply adds the socket to a QSignalMapper:

void SslCertificateMonitor::addSocket(QSslSocket *socket)
{
 d->mapper->setMapping(socket, socket);
 connect(socket, SIGNAL(encrypted()), d->mapper, SLOT(map()));
}

As you can see, we're connecting to the encrypted() signal of the socket in order to ensure that we get informed as soon as the SSL handshake is complete, and before any sensitive data (like cookies) are transmitted to the client. Unfortunately, this is also the cause of one of the problems of the implementation with the current Qt APIs as I'll describe later.

The core of the implementation is the socketReady() method which is called whenever the encrypted() signal for a socket we're monitoring is emitted. We'll take a look at the method a chunk at a time:

void SslCertificateMonitor::socketReady(QObject *sockobj)
{
 QSslSocket *sock = qobject_cast<QSslSocket *>(sockobj);
 if (!sock)
 return;

 QString peerName = sock->peerName();
 if (peerName == d->lastPeerAccepted)
 return; // Fast path for most recently accepted certificate

Since we're using QSignalMapper to let us know which socket we're dealing with in any given call, we first cast the QObject pointer we're given down to a QSslSocket. This means we can now call all the SSL functions. First we extract the name of the 'peer' (this is a fancy name for the site we're connecting to). The next two lines are actually part of an optimisation - since we'll often get repeated calls for the same site, I've added a simple single-entry cache that lets us quickly handle two requests that follow one after the other. A more complete implementation of this concept would probably use a slightly larger cache. If we've just accepted the site, then we know that there's no need to go through the full set of checks again, so we simply approve it immediately.

The next step is to check that the cache itself exists, and if not then create it. In this example, the cache is simply a directory that contains a file named after the host being visited. In the file, we store a cryptographic hash of the certificate that was used.

 if (!hasCertificateCache()) {
 bool ok = createCertificateCache();
 if (!ok)
 return;
 }

Finally, we get to the real meat of the class - the algorithm that determines if we should warn the user. This checks if we have a cached value, and if so checks for differences. If the certificate has changed then the signal certificateWarning() is emitted.

 QSslCertificate certificate = sock->peerCertificate();

 // Have we been here before?
 if (hasCachedCertificate(peerName)) {
 if (isMatchingCertificate(peerName, certificate)) {
 d->lastPeerAccepted = peerName;
 return; // All is well
 }

 // Cert has changed
 d->acceptCurrent = false;
 QString message = tr("The certificate for %1 has changed since you previously visited, " \
 "it could be that someone is trying to intercept your communication.");
 message = message.arg(peerName);
 emit certificateWarning(peerName, message);
 }
 else {
 // The certificate is new. We don't show anything to user because then
 // we're simply training them to click through our warning message without
 // thinking.
 d->acceptCurrent = true;
 }

People using the class need to connect to the certificateWarning() signal, and if they wish the connection to be approved should call the acceptCertificate() method. This operates in a similar way to the QSslCertificate ignoreSslErrors() method, ie. expecting the client to use a nested event loop (such as a modal dialog) to interact with the user. If the acceptCertificate() method is called then the value of acceptCurrent will be set true. The value of this variable governs what we do next:

 // If the user has chosen to accept the certificate or the certficate is new
 // then we store the updated entry.
 if (d->acceptCurrent) {
 d->lastPeerAccepted = peerName;
 addCertificate(peerName, certificate);
 }
 else {
 // Certficate has been considered dangerous by the user
 sock->abort();
 }

The code above is the core of the entire implementation, but we should take a look at the internals of two of the functions it uses: the method for writing an entry into the cache, and the method for checking for changes. Both of the methods are very simple, adding a certificate to the cache is simply a matter of writing the certificate digest to a file:

void SslCertificateMonitor::addCertificate(const QString &peerName, const QSslCertificate &cert)
{
 QString cacheEntry = d->cacheDir + QLatin1Char('/') + peerName;
 QFile f( cacheEntry );
 if (!f.open(QIODevice::WriteOnly))
 return;

 f.write(cert.digest());
 f.close();
}

Checking if the certificate has changed is just as easy - we just compute the digest of the certificate the site has provided and compare it with the value we recorded previously:

bool SslCertificateMonitor::isMatchingCertificate(const QString &peerName, const QSslCertificate &cert)
{
 QString cacheEntry = d->cacheDir + QLatin1Char('/') + peerName;
 QFile f( cacheEntry );
 if (!f.open(QIODevice::ReadOnly))
 return false;

 QByteArray oldDigest = f.readAll();
 f.close();

 if (oldDigest != cert.digest())
 return false;

 return true;
}

In order to test the code worked, I wrote a simple app that will connect via SSL and dump the response. The code that integrates the monitor is simple:

MonitorTest::MonitorTest()
 : QWidget()
{
 monitor = new SslCertificateMonitor(this);
 connect(monitor, SIGNAL(certificateWarning(const QString &, const QString &)),
 SLOT(certificateWarning(const QString &, const QString &)));

This creates the monitor and connects the warning signal to the slot that will inform the user about the issue. When we actually create a socket, we have to call:

 monitor->addSocket(sock);

This tells the monitor to watch the certificates for our socket. Finally, we have to let the user choose what to do when a changed certificate is spotted. In the example, this is as simple as displaying a dialog:

void MonitorTest::certificateWarning(const QString &host, const QString &message)
{
 QMessageBox::StandardButton result =
 QMessageBox::warning( this,
 tr("Certificate for %1 has changed").arg(host),
 message + QLatin1String("\n\nAre you sure you wish to continue?"),
 QMessageBox::Yes | QMessageBox::No );

 if (result == QMessageBox::Yes)
 monitor->acceptCertificate();
}

This means that when a certificate is found to have changed, the user sees a message like the one in the screenshot below:

As you can see, this proves that the basic concept works ok. Unfortunately there are some major API limitations that mean this code isn't as useful in practice as it could be:

It cannot be extended to QNetworkAccessManager right now as that neither gives us access to the underlying QSslSocket nor provides a function for encrypted() that is analogous to the sslError() forwarding function.
Since we need to tap the encrypted() signal, it's not safe for the application to send data in response to this signal (which is the way it is normally used). We really need to have a signal that works immediately prior to the one the application uses, or get the application to use a signal sent by this class.

In addition there are some easily fixable issues, for example it doesn't track the date or anything else from the certificate. This could easily be added though eg. by using the modification time on the cache entry. This is a good time however to be finding that there are API issues since there are plans for a reworking of the QSslSocket APIs for Qt5. Hopefully experiments like this will let us significantly improve things for the future.

The code as usual is in my qt-examples gitorious repository.

QPcap - A Qt-Style Wrapper Around libpcap

Rich — Sat, 19 Mar 2011 00:00:00 +0000

I've been working on a new library for the last couple of weeks that provides a Qt style API for libpcap. For those who aren't familiar with it, libpcap is the library used for capturing raw packets used by tools like tcpdump and wireshark. The pcap library has been around for years, and is built in C rather than C++. The API is pretty simple, but it's not one that can be trivially integrated into a gui application. The aim of the library I've been working on is to make using packet capture in Qt programs a simple matter of using a QObject and connecting to its slots.

Developing a wrapper for a C api is a fairly common activity for C++ developers, though not one that is done every day. My aim here is to provide an API that feel familiar to developers using Qt, so I've tried to provide something that uses the Qt style rather than simply wrapping the pcap methods into a C++ class. A side effect of this is that it will probably seem a little bit unfamiliar to people who know libpcap but, I feel that the benefit of making something that is easy to use by Qt developers outweighs this.

I'm only going to cover two aspects of the pcap wrapper in my post today: how the packet processing is triggered, and how the callback used by pcap is converted into a Qt-style signal. I'll do another post later about how the wrappers around the packet data work, but for now that's still in flux so writing about it seems a bit pointless.

A pcap capture sits there grabbing data from the wire, but doesn't do anything until you call pcap_dispatch. One approach we could take would be to repeatedly call this function, but this would lead to high cpu load as we'd often find there was no work to do. Fortunately on unix platforms pcap provides a function that gives us a file descriptor we can use for the select() system call - pcap_get_selectable_fd(). Qt, like other modern gui toolkits provides a way for us to integrate file descriptors into the select call of their event loop. In Qt, this comes in the form of the QSocketNotifier class.

In qpcap, I've used QSocketNotifier to know when I should call pcap_dispatch. This means we can process the packets and won't be burning CPU time pointlessly. The code that sets this up is in QPcap::start() as follows:

 int fd = pcap_get_selectable_fd(d->handle);
 d->notifier = new QSocketNotifier( fd, QSocketNotifier::Read, this );
 connect( d->notifier, SIGNAL(activated(int)), this, SLOT(dataAvailable()) );
 d->notifier->setEnabled(true);

The dataAvailable() slot this code calls is at the heart of QPcap, but is trivial. It just calls the pcap_dispatch method we referred to earlier. All the work is done by the callback function passed to it which is called once for each packet received.

void QPcap::dataAvailable()
{
 pcap_dispatch( d->handle, -1 /* all packets*/, (pcap_handler)&QPcap::packet_callback, (uchar *)this );
}

The callback function used by pcap_dispatch is a static method of QPcap. This is required to avoid issues with the difference between the calling conventions of C++ methods (which have the implicit 'this' argument) and C functions. We use the user data argument of the pcap_dispatch function to let us know which instance of QPcap should be notified. After setting the header and packet members of our data object, we trigger the packetReady() signals so that objects connected to our class will be informed.

void QPcap::packet_callback( uchar *self, const pcap_pkthdr *header, const uchar *packet )
{
 QPcap *qpcap = reinterpret_cast<QPcap *>(self);
 qpcap->d->header.header = header;
 qpcap->d->packet = packet;

 qpcap->packetReady();
 qpcap->packetReady( qpcap->d->packet );
}

This post only covers a couple of functions of qpcap, if you want to look at it in more detail, then the code is available on gitorious at http://gitorious.org/qpcap/qpcap/. I'll be writing at least one more post covering the wrapper classes offered by the library for accessing packets, but I'll leave that until the classes are a bit more complete.

Loading and Scaling Images in a Thread

Rich — Sat, 22 Jan 2011 00:00:00 +0000

In a previous blog post, I showed a simple example of using threads to perform complex calculations (SHA hashes) in a worker thread. I used them there because generating the hash of a DVD ISO can take a while, and GUIs that block make everyone cry. In this post, I'll use the same technique to load and scale an image whilst still letting my GUI startup instantly.

To begin with, lets look at the declaration of my Resizer class - this is the class that does most of the work. As you can see it's just a normal QObject, and provides slots to let us specify the input, target size etc. It also has a couple of signals one to provide the result of the work, and another to indicate that something went wrong. One thing to note is that the API is based around QImage not QPixmap, that's because it will be running outside the main thread, meaning GUI classes like QPixmap cannot be used safely.

class Resizer : public QObject
{
 Q_OBJECT
public:
 Resizer( QObject *parent=0 );
 ~Resizer();
public slots:
void setSize( const QSize &size );
void setAspectRatioMode( const Qt::AspectRatioMode mode );
void setInput( const QImage &amp;input );
void setInput( const QString &amp;filename );

void start();

signals:
void error();
void finished( const QImage &output );
private:
struct ResizerPrivate *d;
};

The implementation is pretty simple, I'll skip over the setters etc., since they simple do what you'd expect. In fact the only interesting method is the start() slot since that's where all the work is done.

struct ResizerPrivate
{
 QSize size;
 Qt::AspectRatioMode aspectMode;
 QImage input;
 QString inputFilename;
};

void Resizer::start()
{
 if ( !d->inputFilename.isEmpty() ) {
 d->input.load( d->inputFilename );
 }

 if ( d->input.isNull() ) {
 emit error();
 return; 
 }

 QImage output = d->input.scaled( d->size, d->aspectMode, Qt::SmoothTransformation );
 emit finished( output );
}

The start() method first looks to see if a filename has been provided, if it has then it tries to load it. If the load failed (or the user has provided no input at all) then the error signal is emitted and we return. Finally, if we have an image to work with, we perform a smooth scaling operation then emit the finished() slot with the result. As you can see, we haven't had to do anything special to deal with threads, since we're using signals to return our scaled image.

The main function of this example is where all the threading is dealt with. The Viewer class is simply a UI file that wraps a QLabel and provides a slot that will set the label's pixmap based on a QImage.

int main( int argc, char **argv )
{
 QApplication app( argc, argv );
if ( argc != 2 ) {
 printf( "Usage: %s <filename>\n", argv[0] );
 return 1;
}

Viewer *view = new Viewer();

QThread *thread = new QThread();
Resizer *resizer = new Resizer();
resizer->moveToThread( thread );

QObject::connect( thread, SIGNAL(started()), resizer, SLOT(start()) );
QObject::connect( resizer, SIGNAL(finished(const QImage &amp;)), view, SLOT(setImage(const QImage &amp;)) );

view->show();

resizer->setInput( QString::fromLocal8Bit(argv[1]) );
resizer->setSize( QSize(400,400) );

thread->start();

return app.exec();

}

The first interesting thing we do is create the QThread, note that we are using it directly here rather than subclassing it. We then create our Resizer object. When we've done that, we move it into our QThread, this will ensure that it runs there. Next, we connect the thread's started() signal to our start() method so that we'll begin loading and scaling as soon as the thread is run. We also connect our finished() signal (which will be in our thread) to the view (which will be in the main thread). Qt will automatically ensure that the image is passed safely from one thread to the other.

Since we haven't started our thread yet, we can safely set the input and size directly rather than calling these methods as slots. If we wanted to change these while the thread was running (which our resizer doesn't really support in this case) then we'd want to use signals or QMetaObject's invokeMethod() in order to do so safely.

Finally, we start() our thread then start the event loop. When the thread starts, the started() signal is emitted, which will trigger our resizer to begin processing. Because this is happenning asynchronously, our GUI will display immediately even if we're loading and scaling very large images. The same technique can be used in many scenarios, for example if you have expensive startup operations that need to be performed by your application, then you might want to consider using this technique.

As usual, the code is in my gitorious repository.

Getting the details of an SSL connection

Rich — Sun, 09 Jan 2011 00:00:00 +0000

SSL is a pretty complex topic, and whilst Qt makes the APIs pretty easy to use, there's a lot going on underneath and it can be hard to debug. In order to make life a bit easier, I've written a simple tool that will connect to an SSL service then dump every piece of information Qt makes available concerning it. Of course the aim here is both that the tool be useful in itself, and also that it provide a good illustration of how to use the APIs concerned.

The first step, naturally enough, is to connect to the service using SSL - as you might expect, this uses the QSslSocket class. In order to be informed if we manage to connect successfully, or if we hit a problem we connect to some useful signals/

encrypted() - This is emitted if we manage to successfuly connect to the service.
sslErrors(const QList<QSslError> &) - This is emitted if there are problems establishing a secure connection.
error(QAbstractSocket::SocketError) - This is emitted if the connection fails. It is declared in QAbstractSocket. Note that ssl errors can get reported here too.

The connectToHost() method is the one that triggers the actual connection, and by default performs the SSL handshake etc. for us:

void Connector::connectToHost()
{
 qDebug() << "Connecting...";
d->sock = new QSslSocket(this);
connect( d->sock, SIGNAL(encrypted()), this, SLOT(ready()) );
connect( d->sock, SIGNAL(error(QAbstractSocket::SocketError)),
 this, SLOT(socketError(QAbstractSocket::SocketError)) );
connect( d->sock, SIGNAL(sslErrors(const QList&lt;QSslError> &amp;)),
 this, SLOT(sslError(const QList&lt;QSslError> &amp;)) );
d->sock->connectToHostEncrypted( d->host, d->port );

}

If we get an error during the connection, then one of the two error handling slots we've connected gets called. If it's a socket error then we have a problem such as a refused connection we can't do much, however if it's an error caused by one of the SSL checks then we handle it differently. Normally, we should act as Qt does by default and treat SSL errors as hard errors - if we don't then SSL is unable to protect us against attacks. For the purposes of this example however, we want to be more lenient since we're specifically writing a way to figure out what's going on. As a result, our SSL error handling looks like this:

void Connector::sslError( const QList<QSslError> &errors )
{
 foreach( const QSslError &error, errors ) {
 qDebug() << "SSL Error: " << error.errorString();
 }

 // This is only used because we are interested in dumping all the info
 // do NOT use if you want real security (it introduces man-in-the-middle
 // attacks).
 d->sock->ignoreSslErrors();

If we manage to establish the connection ok, then we'll call a couple of functions that tell us the details. The first gives us the SSL certificate used by the remote server we've connected to, the second tells us the cipher we're using. We'll pass these to a couple of functions that print out the details, then terminate the application:

void Connector::ready()
{
 qDebug() << " === Peer Certificate ===";

 QSslCertificate cert = d->sock->peerCertificate();
 dumpCertificate( cert );

 QSslCipher cipher = d->sock->sessionCipher();
 dumpCipher( cipher );

 qDebug() << "Done";

 qApp->quit();
}

First, we'll look at how we dump the information about the cipher we're using. QSslCipher provides accessors that let us see the details - it's worth remembering that users will not be able to understand this information, so any interface for general use should be designed with rather more care.

void Connector::dumpCipher( const QSslCipher &cipher )
{
 qDebug() << "\n== Cipher ==";

 qDebug() << "Authentication:\t\t" << cipher.authenticationMethod();
 qDebug() << "Encryption:\t\t" << cipher.encryptionMethod();
 qDebug() << "Key Exchange:\t\t" << cipher.keyExchangeMethod();
 qDebug() << "Cipher Name:\t\t" << cipher.name();
 qDebug() << "Protocol:\t\t" << cipher.protocolString();
 qDebug() << "Supported Bits:\t\t" << cipher.supportedBits();
 qDebug() << "Used Bits:\t\t" << cipher.usedBits();
}

Now let's take a look at how we dump the certificate information. A certificate contains a lot of information (not all of which is accessible using Qt's API). In order to make the output managable, I've grouped it into three sections - information about the subject, information about the issuer, and finally information about the certificate itself.

The first thing we do is dump the certificate itself in a format known as PEM which a base64 string. Then we move onto the subject. We have quite a bit of information about the subject of a certificate, but two parts are particularly important for SSL - the common name and the subject alternative names. These two fields are the ones that are matched against the name of the host we're connecting to in order to determine if the certificate is valid for this host. After the information about the subject, we dump similar information about the issuer. Finally we display the information about the certificate itself which here is primarily just the dates for which it is valid.

	
void Connector::dumpCertificate( const QSslCertificate &cert )
{
 qDebug() << cert.toPem();
qDebug() &lt;&lt; "== Subject Info ==\b";
qDebug() &lt;&lt; "CommonName:\t\t" &lt;&lt; cert.subjectInfo( QSslCertificate::CommonName );
qDebug() &lt;&lt; "Organization:\t\t" &lt;&lt; cert.subjectInfo( QSslCertificate::Organization );
qDebug() &lt;&lt; "LocalityName:\t\t" &lt;&lt; cert.subjectInfo( QSslCertificate::LocalityName );
qDebug() &lt;&lt; "OrganizationalUnitName:\t" &lt;&lt; cert.subjectInfo( QSslCertificate::OrganizationalUnitName );
qDebug() &lt;&lt; "StateOrProvinceName:\t" &lt;&lt; cert.subjectInfo( QSslCertificate::StateOrProvinceName );

QMultiMap&lt;QSsl::AlternateNameEntryType, QString> altNames = cert.alternateSubjectNames();
if ( !altNames.isEmpty() ) {
 qDebug() &lt;&lt; "Alternate Subject Names (DNS):";
 foreach (const QString &altName, altNames.values(QSsl::DnsEntry)) {
 qDebug() &lt;&lt; altName;
 }

 qDebug() &lt;&lt; "Alternate Subject Names (Email):";
 foreach (const QString &altName, altNames.values(QSsl::EmailEntry)) {
 qDebug() &lt;&lt; altName;
 }
}

qDebug() &lt;&lt; "\n== Issuer Info ==";
qDebug() &lt;&lt; "CommonName:\t\t" &lt;&lt; cert.issuerInfo( QSslCertificate::CommonName );
qDebug() &lt;&lt; "Organization:\t\t" &lt;&lt; cert.issuerInfo( QSslCertificate::Organization );
qDebug() &lt;&lt; "LocalityName:\t\t" &lt;&lt; cert.issuerInfo( QSslCertificate::LocalityName );
qDebug() &lt;&lt; "OrganizationalUnitName:\t" &lt;&lt; cert.issuerInfo( QSslCertificate::OrganizationalUnitName );
qDebug() &lt;&lt; "StateOrProvinceName:\t" &lt;&lt; cert.issuerInfo( QSslCertificate::StateOrProvinceName );

qDebug() &lt;&lt; "\n== Certificate ==";
//qDebug() &lt;&lt; "Serial Number:\t\t" &lt;&lt; cert.serialNumber(); // This seems buggy
qDebug() &lt;&lt; "Effective Date:\t\t" &lt;&lt; cert.effectiveDate().toString();
qDebug() &lt;&lt; "Expiry Date:\t\t" &lt;&lt; cert.expiryDate().toString();
qDebug() &lt;&lt; "Valid:\t\t\t" &lt;&lt; (cert.isValid() ? "Yes" : "No");

}

The end result of all this is a small command line tool that will connect to the host specified and dump all this information. It should be useful for debugging any issues that may be encountered when connecting to an SSL site using Qt. There are a number of possibile enhancements that could be made (eg. dumping the whole certificate chain) but even now, the output of this tool should make tracking down SSL issues in Qt applications a lot simpler. The code, as ever, is in my qt-examples gitorious repository. An example of the output generated is here.

Rendering UI files to PDF

Rich — Mon, 03 Jan 2011 00:00:00 +0000

As a followup to my previous blog post about rendering widgets to SVG, lets take a look at rendering them to PDF. I won't go into as much detail as the previous blog post since the code is mostly the same. This time, instead of a renderToSvg() method, we have a renderToPdf() method. All the rest of the code is basically the same as the previous example.

The renderToPdf() method makes use of the ability of QPrinter to generate PDF files (incidentally, it can also generate postscript). The important part of the code is as follows:

 QPrinter pdf;
 pdf.setOutputFormat( QPrinter::PdfFormat );
 pdf.setOutputFileName( pdfFile );
 QPainter p;
 p.begin(&pdf);
 target->render(&p);
 p.end();

We create a printer (using the default printing mode etc.), then tell it to use PDF output. Next we set the output filename. Finally, we render the widget as before. Like QSvgGenerator, QPrinter is a QPaintDevice so the API is just the same. The output file can be seen at http://xmelegance.org/devel/networkrequests.pdf, and the code is in gitorious qt-examples repository as before.

I mentioned in my previous post that QImage is also a QPaintDevice, so an exercise I'm leaving to you is to make 3rd version of the code that will render widgets to an image.

Rendering UI files to SVG

Rich — Mon, 03 Jan 2011 00:00:00 +0000

In response to a question on IRC yesterday, here's a quick example for the New Year - rendering designer's UI files to SVG using QSvgGenerator. As usual, the code is simple since Qt gives us everything we need.

The main program loads the UI file shows it, it also sets up our SVG capture. Loading the UI file is done using QUiLoader allowing us to load any file specified on the command line. We then figure out what to call the SVG file we're going to generate. Next we setup our Renderer object by telling it what to capture and the name of the file to write it to. In order to ensure the UI is ready, we use a single shot timer to trigger the capture one second after the application starts:

int main( int argc, char **argv )
{
 QApplication app( argc, argv );
QString filename = QString::fromLocal8Bit( argv[1] );
QUiLoader loader;
QFile f( filename );
if ( !f.open(QIODevice::ReadOnly) )
return 1;

QWidget *ui = loader.load( &amp;f );
if ( !ui )
return 1;

QString svgFile = filename.remove(".ui");
svgFile = svgFile.append(".svg");

Renderer render;
render.setTarget(ui);
render.setSvgFile(svgFile);
QTimer::singleShot( 1000, &amp;render, SLOT(start()));

ui->show();
app.exec();

}

The implementation of the actual capture is simple too, and is all handled by a single method 'renderToSvg()'. This method creates the generator and tells it the filename to use. Then it creates a QPainter and sets up drawing onto the QSvgGenerator - this is possible because QSvgGenerator is a QPaintDevice (like QWidget, QImage etc.). Finally we call the render() method of the widget we're capturing in order to perform the actual drawing.

bool Renderer::renderToSvg()
{
 if (svgFile.isEmpty()) {
 qDebug() << "No svg file set";
 return false;
 }

 QSvgGenerator svg;
 svg.setFileName(svgFile);
 QPainter p;
 p.begin(&svg);
 target->render(&p);
 p.end();
 
 return true;
}

The result of running this on a simple UI file can be seen at http://xmelegance.org/devel/networkrequests.svg. The code itself is available from my gitorious qt-examples repository as normal.

When I first wrote the above code, I tried to get the widget to render directly to the svg file, but this turned out to hit QTBUG-2278. The code that triggers the bug was:

 QSvgGenerator svg;
 svg.setFileName(svgFile);
 target->render(&svg);

So if I were you, I'd create the painter myself!

Implementing a Reusable Custom QNetworkReply

Rich — Sat, 28 Aug 2010 00:00:00 +0000

In my last blog post, I showed how to use a proxy QNetworkAccessManager to restrict network accesses to sites included in a whitelist. One limitation the previous version had is that it stopped the disallowed requests by rewriting the request itself to be to an invalid url. This then caused the default implementation of QNetworkAccessManager to generate an error reply for us. This post will look at how to create a custom reply directly, to allow us to display messages to the user etc. or even provide 'virtual' content.

The approach I've taken is to try to write a reusable utility class for sending basic network replies. I've called the class QCustomNetworkReply since if there's a reasonable level of support then I'll try to work it up into a merge request for Qt. To begin with, lets see how the whitelisting proxy looks now that it uses the new class:

QNetworkReply *WhiteListNetworkAccessManager::createRequest( Operation op,
 const QNetworkRequest &req,
 QIODevice *outgoingData )
{
 // If host is not whitelisted then kill it
 if ( !isAllowed( req.url().host() ) ) {
 QCustomNetworkReply *reply = new QCustomNetworkReply();
 reply->setHttpStatusCode( 403, "Forbidden" );
 reply->setContentType("text/html");
 reply->setContent( QString("<html><body><h1>That url is not in the whitelist</h1></body></html>") );

 return reply;
 }

 QNetworkReply *reply = QNetworkAccessManager::createRequest( op, myReq, outgoingData );
 return reply;
}

The new code operates if the request is to be disallowed. First it creates our custom reply, it then specifies the HTTP response code to send. Finally it sets up the content type and the content itself. Finally, we return our custom reply. As you can see, the API of the QCustomNetworkReply class is fairly simple to use (in fact the only required part is setting the content).

Now we've seen how it's used, lets take a look at how the custom network reply works. The class declaration is fairly simple:

class QCustomNetworkReply : public QNetworkReply
{
 Q_OBJECT

public:
 QCustomNetworkReply( QObject *parent=0 );
 ~QCustomNetworkReply();

 void setHttpStatusCode( int code, const QByteArray &statusText = QByteArray() );
 void setHeader( QNetworkRequest::KnownHeaders header, const QVariant &value );
 void setContentType( const QByteArray &contentType );

 void setContent( const QString &content );
 void setContent( const QByteArray &content );

 void abort();
 qint64 bytesAvailable() const;
 bool isSequential() const;

protected:
 qint64 readData(char *data, qint64 maxSize);

private:
 struct QCustomNetworkReplyPrivate *d;
};

The first group of methods are those for setting the various headers on our response. They are really just convenience wrappers around existing methods (for example making some protected functionality public). I won't go into any more detail about them since the implementations are obvious looking at the source. The two setContent methods are where things start to get interesting:

void QCustomNetworkReply::setContent( const QString &content )
{
 setContent(content.toUtf8());
}

void QCustomNetworkReply::setContent( const QByteArray &content )
{
 d->content = content;
 d->offset = 0;

 open(ReadOnly | Unbuffered);
 setHeader(QNetworkRequest::ContentLengthHeader, QVariant(content.size()));

 QTimer::singleShot( 0, this, SIGNAL(readyRead()) );
 QTimer::singleShot( 0, this, SIGNAL(finished()) );
}

The first method is simple a convenience and allows us to use a QString rather than a bytearray for the content, yhe second is where the real action is. First, we store the content. Next, we zero the offet that stores how much data has been read from our reply (remember a QNetworkReply is a QIODevice) and open our io device. Now that we have the content, we also set the response header that specifies the amount of data we have. Finally, we use two single shot timers to cause the readyRead() and finished() signal to be emitted when the event loop is reentered. We can't simply emit these signals immediately since the reply has not yet been returned by our QNetworkAccessManager, so nothing is listening for them.

The final part of the code is to provide a basic implementation of a QIODevice to allow the stored content to be read back out:

void QCustomNetworkReply::abort()
{
 // NOOP
}


qint64 QCustomNetworkReply::bytesAvailable() const
{
 return d->content.size() - d->offset;
}

bool QCustomNetworkReply::isSequential() const
{
 return true;
}


qint64 QCustomNetworkReply::readData(char *data, qint64 maxSize)
{
 if (d->offset >= d->content.size())
 return -1;

 qint64 number = qMin(maxSize, d->content.size() - d->offset);
 memcpy(data, d->content.constData() + d->offset, number);
 d->offset += number;

 return number;
}

That's all there is to it - see all fairly straight forward. The nice part is that now we have the QCustomNetworkReply, we don't need to do any of that work again and can simply reuse this class whenever we want to send data directly to clients of QNetworkAccessManager. As usual, the code is available from my qt-examples git repository at http://gitorious.org/qt-examples/.

A White-listing QNetworkAccessManager

Rich — Sat, 05 Jun 2010 00:00:00 +0000

My last blog post showed how a proxy class can be used to monitor the requests being made by a QNetworkAccessManager, and illustrated it using an example that displayed those requests graphically. This post will cover another use of proxy QNetworkAccessManagers, specifically modifying requests and responses. The example we'll use is a very simple one, we'll look at a class that will restrict a QNetworkAccessManager so that it can only access domains listed in a white-list.

Before we start, lets consider the reasons why restricting the domains a QNetworkAccessManager can access might be useful. One major use is when developing hybrid applications using QtWebkit - restricting the domains that can be accessed allows you to expose internal facilities to Javascript without allowing data to be transmited to arbitrary sites. Another use is in system's like Plasma where it would allow applets to be restricted to certain domains. The example we'll use is a mini-browser that can only access www.kde.org.

The class definition for our white-listing manager is very simple. It has a method to add domains to the whitelist, and one to test if a domain is white-listed. The only other method is the reimplementation of createRequest().

class WhiteListNetworkAccessManager : public QNetworkAccessManager
{
 Q_OBJECT

public:
 explicit WhiteListNetworkAccessManager( QObject *parent = 0 );

 bool isAllowed( const QString &domain );

public slots:
 void addDomain( const QString &domain );

protected:
 QNetworkReply *createRequest( Operation op,
 const QNetworkRequest &req,
 QIODevice *outgoingData );

private:
 QMap<QString, int> allowedDomains;
}

Looking at the two methods to manage the white-list first, we can see that they're both trivial. The first simply adds a domain to the map of allowed domains, the second tests if the map contains the specified domain as a key:

void WhiteListNetworkAccessManager::addDomain( const QString &domain )
{
 allowedDomains.insert( domain, 1 );
}

bool WhiteListNetworkAccessManager::isAllowed( const QString &domain )
{
 return (allowedDomains[domain] == 1); // Rely on Qt's default of 0 for non-existent ints
}

The only non-trivial part of this is that we rely on knowing that QMap will return a default value of 0 for maps with int based values if the key is not present.

The final method of our class is almost as trivial as these, it checks if the domain a request is being made to is on the white-list and if it is hands the request to the base-class. If however the request is to a non-allowed domain, it modifies the request to use an invalid url (one with an unsupported protocol). This causes the base-class implementation of the method to fail the request. It would be possible for us to create a response object of our own instead, but this would require quite a lot more code as we'd need to implement the full API of QNetworkReply. This might be the topic of a further example, but it's not required here.

QNetworkReply *WhiteListNetworkAccessManager::createRequest( Operation op,
 const QNetworkRequest &req,
 QIODevice *outgoingData )
{
 QNetworkRequest myReq( req );

 // If host is not whitelisted then kill it
 if ( !isAllowed( req.url().host() ) ) {
 myReq.setUrl( QUrl( QString("forbidden://localhost/") ) );
 }

 QNetworkReply *reply = QNetworkAccessManager::createRequest( op, myReq, outgoingData );

 return reply;
}

We've now covered the full implemtation of our class - that really is all there is to it. Using it is simply a matter of telling a QWebView to use it, and adding some domains to the white-list. An example usage is shown below:

 QWebView view;
 WhiteListNetworkAccessManager proxy;

 proxy.addDomain( QString("www.kde.org") );
 view.page()->setNetworkAccessManager( &proxy );

As usual, the full source code for this example can be found in my gitorious repository a http://gitorious.org/qt-examples/qt-examples/trees/master.

QNetworkAccessManager Proxy Classes

Rich — Sat, 01 May 2010 00:00:00 +0000

This blog post will demonstrate how you can tap into QNetworkAccessManager (QNAM), the class is used by QtWebkit for all its networking. QNAM is also the recommended API for high-level networking protocols such as HTTP in Qt. The class is very simple, but provides some powerful features like disk caching, cookie handling, and support for Socks and HTTP proxies. There are many cases when it is useful to be able to watch the requests being made through QNAM, and this can be acheived using a proxy QNAM class.

To illustrate the technique I'll use a simple proxy class to capture the requests as they are made, and display the information from the request and the response. The code was originally written foor the Arora web browser, but I've separated it out into a reusable stand-alone example. The result will be a display like this:

I won't describe the implementation of the viewer itself, but instead will focus on the proxy class, which is incredibly simple. The core method that QNAM uses is createRequest(). This method is used to implement various convenience functions like get() and post() and is where we need to take control. Our reimplementation is shown below in full:

QNetworkReply *NetworkAccessManagerProxy::createRequest( Operation op,
 const QNetworkRequest &req,
 QIODevice * outgoingData )
{
 QNetworkReply *reply = QNetworkAccessManager::createRequest( op, req, outgoingData );
 viewer->addRequest( op, req, outgoingData, reply );
 return reply;
}

All we're doing is calling the baseclass implementation and passing the information about the request to our viewer. It's really that simple. If we wanted to though, we could return a custom QNetworkReply which would let us directly control what happens to the request. This technique can be used to do things like restrict access of a browser to certain white-listed addresses, or to modify the content returned.

Making QtWebkit use our proxy class is simply a matter of calling setNetworkAccessManager, which we do in our main() function. The entire main function is shown below:

int main( int argc, char **argv )
{
 QApplication app( argc, argv );
 QWebView view;

 NetworkAccessManagerProxy proxy;
 NetworkAccessViewer viewer;
 proxy.setNetworkAccessViewer( &viewer );

 view.page()->setNetworkAccessManager( &proxy );

 view.setUrl( QUrl("http://www.kde.org/") );
 view.show();

 return app.exec();
}

Hopefully this is enough information to show just how easy using this powerful technique can be. In future I hope to provide a few further examples that show more complex proxy classes. As usual I've put all the code in my qt-example git repository on gitorious at http://gitorious.org/qt-examples/, enjoy.

Tokamak4 - A New Hope

Rich — Wed, 24 Feb 2010 00:00:00 +0000

Today I'm leaving Tokamak 4, so I thought I'd write a post about what I've been up to while I've been here.

The first day or so I spent getting my machine sorted out as trunk was causing some issues with my graphics driver leading to a hang in the DRI layer of the kernel. I also had a chance to triage some ksnapshot bug reports and look over some patches.

The next day was largely taken up with talks about the current status of the various parts of plasma, kwin and related technologies. We got a little hacking done but not much as it was more important to make sure we're all coordinating our development.

The remaining days became a bit of a blur, so in no particular order:

I refactored the plasma javascript scriptengine to make it easier to use it with QScriptEngines that are created externally. This is required before we can integrate things like QML.
Worked out what is required for QML integration to be implemented in a way that is compatible with our existing javascript bindings etc. This will require some small changes to the API offered by Qt for QML, which may be a problem. If we don't get those changes then QML is likely to be trapped outside the main engine which will reduce what it can do unless we write another set of bindings.
Made some fixes to the webslice applet to improve error handling.
Started work on an idea I've had in the back of my head for a while, I won't say too much about it here, but I'll include a pretty picture to whet your appetite:

In addition to this I've taken part in lots of discussions on topics ranging from activities/contexts, scripting of animations, silk and more. I'd like to thank Will and the rest of the openSuSE team for making this such a pleasant and productive sprint.

A Simple Threading Example

Rich — Thu, 14 Jan 2010 00:00:00 +0000

A topic that I've not mentioned in any of my blog posts is threading, not because I have anything against it, simply because a simple use-case hadn't come up. Today I was coding something easy to describe, where using threads was a good solution, so let's take a look at it.

The problem I needed to solve was to calculate cryptographic hashes of files that could be very large - for example 4 gigabyte DVD isos. As with any graphical application it is very important to ensure that the GUI remains responsive while this work is done. Since the problem involves very little communication between the part of the application calculating the hash, and the rest of the code a worker thread is an ideal solution.

The code that actually does the work is very simple, it uses the handy QCryptographicHash class to do the work (stored in the variable hasher):

HasherThread::HasherThread(QObject *parent) :
 QThread(parent)
{
 hasher = new QCryptographicHash( QCryptographicHash::Sha1 );
}

void HasherThread::run()
{
 QFile f( filename );
 if ( !f.open(QIODevice::ReadOnly) ) {
 emit error( QString("Unable to open file %1").arg(filename) );
 return;
 }

 hasher->reset();

 char buffer[16*1024];
 qint64 count;
 do {
 count = f.read( buffer, sizeof(buffer) );
 if ( count == -1 ) {
 emit error( QString("Read error") );
 break;
 }
 hasher->addData( buffer, count );
 } while( !f.atEnd() );

 emit completed( hasher->result().toHex() );
}

The code opens the file to be hashed, then reads through it in 16K chunks which it passes to the hasher. If this code were executed in the application's gui thread then this would cause the interface to freeze until the hash was ready, but the code lives in a class that inherits QThread.

One important feature to note about the code above is that the communication with the rest of the application is via signals (the error and completed signals to be precise). This is very important as it means that when we use the class, we can simply use it like this:

 hasherThread = new HasherThread( this );
 connect( hasherThread, SIGNAL(error(const QString &)),
 ui->resultEdit, SLOT(setText(const QString &)) );
 connect( hasherThread, SIGNAL(completed(const QString &)),
 ui->resultEdit, SLOT(setText(const QString &)) );

The special thing in the code above, is that we didn't have to do anything special - Qt just solved our inter-thread communication issues for us. What I mean, it that if our code had simply tried to output the result by calling ui->resultEdit->setText() directly then we would be attempting to access the GUI from outside the GUI thread which would most likely cause our application to crash. Instead, since we're using the default type of connections 'AutoConnection', what this means is that Qt will spot if the thread in which a signal is emitted is different from the one in which it is received and handle the necessary synchronisation for us - nice!

Cutting the Web Down to Size

Rich — Tue, 20 Oct 2009 00:00:00 +0000

A feature that has been floating around in a few places but hasn't been a significant feature in KDE is web slicing. This is the ability to take a piece of a web page (commonly a div) and render it as a standalone object. It's useful for stuff like putting a weather forcast on your desktop, watching new comments on a forum etc. This weekend I made a start on implementing it as tool for Project Silk.

The first version I wrote was pretty simple - it lets you specify the element you want rendered, then presents it to the client application as a QPixmap. This was fairly easy thanks to the new QWebElement API, the core of the code is as simple as this:

QPixmap Slicer::renderElement( const QString &selector )
{
 QWebFrame *frame = d->page->mainFrame();
 d->page->setViewportSize( frame->contentsSize() );

 QWebElement element = frame->findFirstElement( selector );
 if ( element.isNull() )
	return QPixmap();

 QRect rect = element.geometry();
 QPixmap result = QPixmap( rect.size() );
 result.fill( Qt::white );

 QPainter painter( &result );
 painter.translate( -rect.x(), -rect.y() );
 frame->render( &painter, QRegion(rect) );

 return result;
}

This is enough to let you specify a CSS selector and render it to a pixmap thanks to the nice API of QWebElement. Doing the same using the DOM API would be simple enough, but a lot more verbose (and probably less flexible). The code above was enough for me to make it trivial to write tools that present a slice of a web page as part of their UI.

As ever in life, some people are never satisfied so a common response was to ask if I could make it so the slice is interactive. This is a bit of a can of worms as everyone has their own definition - eg. simply responding to requests from the page to repaint could be trivially added to the code above and would offer the same functionality as that I'm told MacOS offers. Rather than try the half-way solution, I wrote a prototype that actually gives you a fully interactive slice - eg. you can type into the form elements. It's not production quality, but it proves the point. You can see the result below:

Around the point I got this working, Kenneth let me know that a commit had just landed in QtWebkit that would make the earlier version even simpler - there's now a render() method on QWebElement that does exactly the same thing my original code did. This is great news, as that solution is the correct one for a lot of use-cases.

Of course, by this time, I was quite tired but Sebas was still feeling lively. The result is that 'live slice' code gained support for being embedded in QGraphicsView overnight using the new QGraphicsWebView. There's a support class that makes it available as a widget like before, but the implementation is using this (new in 4.6) class. All in all this means we can nicely integrate this functionality into apps using QGraphicsView like plasma or amarok, and into QWidget based apps too.

All in all, a successful couple of days in the stealthy advance of project silk.

Adding custom objects to Qt Webkit

Rich — Sun, 05 Jul 2009 00:00:00 +0000

One question I've seen come up several times on #qt and qt-interest is how to add custom (application specific) APIs to those available by default in QtWebKit. This is actually pretty easy (once you know how) as I'll show below. This post will show a simple example of how to make an object available from javascript, including calling methods on the object and returning values from C++.

There are two things you really need to know in order to perform this integration, the first is the addToJavaScriptWindowObject() method of QWebFrame, this allows will make the specified QObject visible from javascript. The second thing you need to know is that objects published in this way will vanish each time the javascript window object is cleared - ie. every time the user navigates to a new page. To prevent this from causing problems, QWebFrame provides a signal that tells you whenever the object is cleared allowing you to re-add your custom API. This is actually much simpler than it sounds!

The core of this is really implemented in two methods in the example, they're shown below:

void MyApi::setWebView( QWebView *view )
{
 QWebPage *page = view->page();
 frame = page->mainFrame();

 attachObject();
 connect( frame, SIGNAL(javaScriptWindowObjectCleared()), this, SLOT(attachObject()) );
}

void MyApi::attachObject()
{
 frame->addToJavaScriptWindowObject( QString("MyApi"), this );
}

This code is all that you need in order to make all of the public slots of the MyApi object visible to javascript. The MyApi class provides two public slots:

public slots:
 void doSomething( const QString ¶m );
 int doSums( int a, int b );

The first slot simply logs a message to the debug output, the second returns the sum of its two arguments (yes, slots can return things!). They're called from javascript like this:

 MyApi.doSomething( 'Hello from JS page 2!!!!' );

 sum = MyApi.doSums( 2, 3 );
 alert( 'C++ says the sum is ' + sum );

And that's all there is to it! You can download the code from http://xmelegance.org/devel/qtwebkitextension.tar.gz.

First Hack with ItemsViewsNG

Rich — Fri, 10 Apr 2009 00:00:00 +0000

People reading planet will have seen Thomas Zander's post about the new ItemViews framework the Trolls^H^H^HQt Software guys have been working on. It's very experimental right now, but I thought I'd have a quick look. I decided that a fun hack to write would be to take a standard listview that displays a list of URLs (boring!) and write a custom view that instead displays the rendered web page. I only spent a couple of hours on it, so the code is a hack (eg. you need to resize it to get the pages to display once they've loaded) but the results look ok. The whole code for this example comes in under 120 lines.

So, that's my hack in response to Thomas's challenge, who's next?

Here's an updated version that correctly updates the items when the page is loaded without requiring a resize.

And now version 3 that disables the scrollbars in the previews making things look a lot nicer.