Recently, when building qt5, I'd started noticing some very strange errors from the configure script. The errors seemed to indicate that an awk script was being used as a filename - very strange. Even stranger was that other people weren't hitting this issue - just me. Never a good sign. Today, I finally got around to debugging it and the issue was rather weird.

My initial thought was that my version of bash was incompatible with the script in some way, so I copied the code for the function that was erroring and made a standalone version - it worked fine. I then tried to run the configure script using sh -x to watch it was doing, but unfortunately that seemed to confuse the script. Finally, I started to read the code. The relevant function reads:

$awk ' BEGIN {

lots of awk

} '

Finally, I spotted that $awk was not being set. If you look at the script then you can see that the qt5 configure looks for gawk, nawk then finally awk. So, the obvious step was to see if I had a version of awk installed. Using 'which' told me I didn't have one, but using rpm -q told me I did - curiouser and curiouser.

I checked what the gawk package installed and I could see that at least part of it was there, but running it gave an IO error - very weird. In fact, I could see that I was running a symlink that pointed to a file that wasn't there. At this point, I assumed a bad update of some kind, so I ran:

zypper install gawk

It said my gawk was the lastest. I then tried

zypper install -f gawk

to force the install and it gave an IO error. Spot the pattern? Every
time I try to do stuff to the gawk binary I get an IO error. At this point, I looked at /var/log/messages and saw a lot of messages like this:

May 19 18:51:06 linux-h33o kernel: [ 58.719815] EXT4-fs error (device
sda6): ext4_ext_check_inode:403: inode #393297: comm rpm: bad header/extent:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0)

Not good.

This seemed like some kind of file system corruption, so I backed up my files immediately before I carried on investigating. Running fsck from a rescue system showed a few errors, but nothing major, and all were fixable. After allowing the repair I rebooted, sure in the knowledge that the problem was solved.

All I needed to do was reinstall the corrupted gawk package:

linux-h33o:/home/rich/src # zypper install -f gawk Loading repository
data...
Reading installed packages...
Forcing installation of 'gawk-4.0.0-3.1.2.x86_64' from repository 'openSUSE-12.1-Oss'.
Resolving package dependencies...

The following package is going to be reinstalled: gawk

1 package to reinstall.
Overall download size: 820.0 KiB.
No additional space will be used or freed after the operation.  Continue? [y/n/?] (y):
Installing:
gawk-4.0.0-3.1.2 [error] Installation of gawk-4.0.0-3.1.2 failed: (with
--nodeps --force)
Error: Subprocess failed. Error: RPM failed: error:
unpacking of archive failed on file /bin/gawk: cpio: rename failed -
Input/output error error:
gawk-4.0.0-3.1.2.x86_64: install failed

Oh dear. It's not good when fsck says the file system is ok, but the driver disagrees.

At this point, I started doing some serious googling to figure out wtf was going on. Happily, I came across the following bug report that let me resolve the issue https://bugzilla.kernel.org/show_bug.cgi?id=32182 . By following the debugfs steps described, I was able to kill the bad inode. Thank fully this fixed my file system.

So what can we learn from this? Obviously, we can learn that the configure script wasn't the source of the problem, but simply the way it manifested. It also shows that the configure script has a bug in that it doesn't report when awk is missing. We can also learn that a bit of googling can solve a lot of problems.

A final point to note if anyone is considering doing evil things with debugfs like this is that as soon as I figured out I had a corrupt file system I made a backup. This gives you a nice warm glow inside as you know you can attempt things that would otherwise be insanely risky.

With the availability of the Qt 5 alpha, I thought I'd try to summarise what's been done in the SSL stack. Most of the changes in Qt 5 for SSL are incremental improvements, or things that will form the basis of future changes. In this post I'll try to highlight the main changes:

There are job offers floating sporadically on planetkde so I guess this one would fit too especially that there are many related technologies involved.

You may remember my story about some crazy Qt device. Now there is apparent expansion both vertically and to other types of high-end devices, so:

This offer is for permanent jobs in Warsaw office, rather for Polish speakers. More on my Polish blog.

It's been quite a while since I last wrote a blog post, but it's not because I haven't been coding, in fact quite the opposite. The Qt opengov project is finally underway and I've been doing quite a lot of work on the various SSL classes. I'm now an official Qt approver, so as you can see the process of getting non-nokia developers the ability to commit to Qt is working.

In Qt 4.x and currently in Qt5 even though the various QSsl classes provide an abstraction from the underlying SSL implementation, there is only one backend and it uses openssl. I recently made a change that means we can add new backends during the Qt5 life time by separating the concepts of SSL support from the availability of SSL - of course this doesn't change anything if openssl is the only game in town.

Over the last couple of weekends, I've been investigating GnuTLS as a
potential implementation that could form the basis of a second backend. I started with the easy part - handling X.509 certificates, and last weekend managed to implement some code that showed that we could implement this part of the Qt API using GnuTLS. I did hit some issues of course, but most were minor. The GnuTLS maintainer Nikos Mavrogiannopoulos has already fixed the minor documentation issues I spotted, and has even implemented a couple of features that I found were missing - definitely a sign that this library is being actively maintained I think you'll agree.

Today I attempted the more challenging task of trying to make GnuTLS work through a QTcpSocket. This integration is essential for using the library in Qt since without it features such as proxy support etc. would not be available - it really is a 'must have'. As you might guess from the fact I'm bothering to blog about it the results were positive, so let's take a look.

What I've done for this prototype is implement a QObject that provides the same basic outline as a QIODevice subclass. In order to keep things simple, I haven't yet tried to make something that is tied to that specific API at this point. I've also not tried to integrate this into QSslSocket, instead this code aims to demonstrate that doing so will be feasible in future.

The API I've implemented is reasonably close to what QSslSocket offers, the header file should make it clear that most of what you'd expect is there:

class SslSocket : public QObject
{
    Q_OBJECT

public:
    SslSocket(QObject *parent=0);
    ~SslSocket();

    QByteArray read(qint64 maxsize);
    
public slots:
    void connectToHost(const QString &hostname, int port);
    void startHandshake();

    qint64 write(const QByteArray &data);

signals:
    void handshakeComplete();
    void readyRead();
    void error();

private slots:
    void dataReceived();

private:
    void setupSession();
    void handshakeInternal();

    ssize_t readInternal(void *buffer, size_t length);
    ssize_t writeInternal(const void *buffer, size_t length);
    ssize_t writeVectorInternal(const giovec_t *iov, int iovcnt);

    static ssize_t read_callback(gnutls_transport_ptr_t transport, void *buffer, size_t length);
    static ssize_t write_callback(gnutls_transport_ptr_t transport, const void *buffer, size_t length);
    static ssize_t write_vector_callback(gnutls_transport_ptr_t transport, const giovec_t *iov, int iovcnt);

    SslSocketPrivate *d;
};

Now you've seen the header, let's take a look at the implementation. The first method we'll look at is the one that performs the initial setup of our SSL session. This happens before we even attempt to send any data to the server since once of the things we're going to do is change the way data is sent to run it via QTcpSocket:

void SslSocket::setupSession()
{
    qDebug() << "Initialise client session";

    // Setup the trust store
    gnutls_certificate_allocate_credentials(&d->x509cred);
    gnutls_certificate_set_x509_trust_file(d->x509cred, "/etc/ssl/ca-bundle.pem", GNUTLS_X509_FMT_PEM);

    // Configure the session
    gnutls_init(&d->session, GNUTLS_CLIENT);
    gnutls_credentials_set(d->session, GNUTLS_CRD_CERTIFICATE, d->x509cred);

    const char *err;
    gnutls_priority_init(&d->priority_cache, "NORMAL", &err);
    gnutls_priority_set(d->session, d->priority_cache);

    // Setup the transport functions to use QTcpSocket
    gnutls_transport_set_ptr(d->session, this);
    gnutls_transport_set_pull_function(d->session, read_callback);
    gnutls_transport_set_push_function(d->session, write_callback);
}

The first thing this method does is setup the trust store, and point it to the location of the CA bundle on my opensuse 12.1 system. In fact, this is a total waste of time, since verifying the certificate isn't something I've implemented yet! The next part is rather more useful however, we initialise a session and tell GnuTLS that we're acting as an SSL client. After this, we set the priority of the various ciphers etc. that are available. This step seemed unimportant, but I discovered that if you fail to do it then rather than using a sane default, GnuTLS dies with an internal error. The last setup step we do is to tell the library that we want to use our own functions to send and receive data rather than using the built-in ones.

For each of the read and write functions, we provide a callback. This is a static method (since a pointer-to-member-function is a no-no for reasons I won't go into). Our static method then calls the member function that implements our callback:

ssize_t SslSocket::read_callback(gnutls_transport_ptr_t transport, void *buffer, size_t length)
{
    SslSocket *self = static_cast<SslSocket *>(transport);
    return self->readInternal(buffer, length);
}

ssize_t SslSocket::readInternal(void *buffer, size_t length)
{
    qDebug() << "readInternal, length" << length << ", available" << d->socket->bytesAvailable();

    if (d->socket->bytesAvailable() < qint64(length)) {
        gnutls_transport_set_errno(d->session, EAGAIN);
        return -1;
    }

    return d->socket->read(static_cast<char *>(buffer), length);
}

As you can see the static function simply casts the user data pointer (which we provided using gnutls_transport_set_ptr earlier) to our class, then calls the appropriate method. The read implementation simply sees if we have enough bytes in our socket's buffer to satisfy the request, and if not tells the library to try again. If we do, then naturally we read the data. The data we need can only become available when the eventloop is running since that's when QTcpSocket performs its data transfers. If we simply retried immediately then
we'd see no more data than we did during our first attempt.

The write function is similar, but since QTcpSocket has buffering of its own we can simply perform the write as requested. My real implementation supports a slightly more complex form of the write function that can process requests to write several blocks of data at once, but the basic concept is the same.

In order to allow the retry behaviour I mention above, there's a simple state machine. The SslSocket has the following basic states:

enum State
{
    Disconnected,
    Connecting,
    Handshaking,
    Encrypted
};

If new data is received then we look the state and retry the relevant
operation as follows:

void SslSocket::dataReceived()
{
    qDebug() << "dataReceived() state is " << d->state;

    if (d->state == Handshaking) {
        handshakeInternal();
    }
    else if (d->state == Encrypted) {
        emit readyRead();
    }
}

The handshakeInternal() method is the one that actually performs the SSL handshake, and as you can see if we're in the appropriate state then we retry each time more data is received. The actual implementation is pretty simple:

void SslSocket::handshakeInternal()
{
    qDebug() << "start handshake, state is" << d->state;
    int result = gnutls_handshake(d->session);
    qDebug() << "shake result" << gnutls_strerror(result) << result;

    if (result == 0) {
        qDebug() << "handshake completed";
        d->state = Encrypted;
        emit handshakeComplete();
    }
    else if (gnutls_error_is_fatal(result)) {
        qDebug() << "fatal error in handshake";
        emit error();
        d->socket->disconnect();
        d->state = Disconnected;
    }
}

As you can see, once we get a real result from the handshake (ie. one that can't be solved with more data) then we emit either the handshakeComplete() signals or the error() signal and update the state of the SslSocket. If the handshake is completed successfully then we can now send and receive encrypted data.

The read and write methods are similar. In both cases, we need to consider the possibility that we need to retry (indicated by either an interrupted, or again error). The implementations are as follows:

QByteArray SslSocket::read(qint64 maxsize)
{
    QByteArray buffer;

    if (d->state != Encrypted) {
        qFatal("Write before encrypted is not supported yet");
        return buffer;
    }

    buffer.resize(maxsize); // ### This could get pretty big!
    ssize_t result;
    do {
        result = gnutls_record_recv(d->session, buffer.data(), maxsize);
    } while( result == GNUTLS_E_INTERRUPTED || result == GNUTLS_E_AGAIN );

    buffer.truncate(result);

    return buffer;
}

qint64 SslSocket::write(const QByteArray &data)
{
    qDebug() << "write";

    if (d->state != Encrypted) {
        qFatal("Write before encrypted is not supported yet");
        return -1;
    }

    ssize_t result;
    do {
        // ### It's possible that this could fail due to the need to read some data
        // for example if a renegotiation is underway. Not sure.
        result = gnutls_record_send(d->session, data.constData(), data.size());
        qDebug() << "write result" << result;
    } while( result == GNUTLS_E_INTERRUPTED || result == GNUTLS_E_AGAIN );

    return result;
}

In order to test this, I wrote a very simple class that will perform an HTTP GET and read the response. The result is that we can can connect to an HTTPS server and see the root page - not very exciting I guess but a sign that everything is working!

GetSlash::GetSlash(SslSocket *sock)
    : QObject()
{
    this->sock = sock;
    connect(sock, SIGNAL(handshakeComplete()), this, SLOT(start()));
    connect(sock, SIGNAL(readyRead()), this, SLOT(gotData()));
}

void GetSlash::start()
{
    sock->write(QByteArray("GET / HTTP/1.0\n\n"));
}

void GetSlash::gotData()
{
    QByteArray result = sock->read(2000);
    qDebug() << result;
}

Finally there's a basic main() function that ties all this together, and a little bit of code to log what's going on. The end result is that I now feel confident that we could implement a working backend for QSslSocket etc. using GnuTLS without too much trouble.

I'll add the code to my normal qt-examples repository later, but if anyone needs it (or earlier experiment experimenting with the certificate APIs) then just let me know.

EDIT: The code is now available here https://gitorious.org/qt-examples/qt-examples/commit/40e57647242079745c46d84e15d92f2093af300f

First, I've changed the branches that are being built on build.kde.org to reflect the branching of 4.8.

Second I've incorporated a build of Qt 4.8 branch that is triggered once a week to keep up with fixes there.

Third I've tried to get a build of Qt 5 up and running also, however this is where I've hit some bumps. I've been unable to get Qt 5 to build and as such not all other jobs are working :(

It's been a while since I blogged, so I thought I'd post an update on what I've been up to.

Along with nearly a thousand other Qt developers, I headed over to Munich for the Qt Dev Days. It was my first time at the event, and I was surprised just how big it was. There were a lot of interesting talks, with highlights including Jens' talks on Qt Quick components for the desktop and using QML for rapid prototyping and Peter's talk on Secure Networking.

Before all the talks however, we had a day full of Qt contributor sessions. This included a lot of discussions about how the open governance model should work, and also technical sessions on individual areas. I've posted my notes from a couple of sessions to the new Development mailing list, including my notes from the various discussions on SSL.

I'd like to thank the KDE eV for sponsoring my travel and hotel, Nokia for my Dev Days pass (and a shiny new N9), and my own company, Westpoint, for letting me attend.

In terms of coding, I've been fairly busy too. I've recently added support for disabling fancy new features of SSL/TLS like session tickets, compression and server name indication that since they're so shiny and new (ahem, most of them are ten years old at least) seem to be incompatible with various expensive SSL accelerators. This change has been backported to Qt 4.8 so we should be able to use it in KDE 4.x too.

I've also made a few changes that are only in the Qt 5.0 branch since they're really too big a change to slip into the 4.8 tree at this stage. The major one is the ability to verify a certificate chain without needing to connect to a server.

Most recently, I've written a pretty large improvement to the handling of certificates to add support for certificate extensions. This change has been handled under the open governance system using gerrit, which has resulted in a much faster turn-around than the previous gitorious based approach of merge requests. The new code makes it possible to access all the certificate extensions, with a design that makes it easy to extend as new ones are required. The change is currently in the process of getting merged, and assuming I don't get bitten by an old openssl version on one of the test platforms should be in the main tree tonight.

I have a bunch of further improvements to the SSL support in progress, with my work on adding OCSP support being one of my main targets. There are several other low-hanging fruit like the bug work arounds I mentioned too though, so I'll probably try to address some of those as well.

If you're working on a KDE application and Qt's SSL support doesn't offer what you need, then please let me know so that I can make sure that what you're missing gets added to the todo list.

Please note: This blog is personal and opinions are of my own and do not represent RIM

I've been quiet for a while now. Lots of things going on in my life. I have not forgotten KDE, but I'm still unable to work on anything at this time (more on that in future post).

But I'm glad to say that having Qt as part of PlayBook is awesome!
I look forward to seeing what people will do.

We've ported a number of projects to QNX for the PlayBook and can be found here: BlackBerry GitHub

"Don't browse, search" is already well known demand and it changes the way people expect to use your app.

Recently I needed to finalize first incarnation of my Global Search feature in Kexi. Being late by two days after Calligra beta 3 tagging, I made it fully working only for RC1 now. But it's here.

Since so far I missed any blog on our beloved planet pointing to this interesting resource I do it myself now;

Porting Windows Phone to Qt

A rather cute way to escape your Windows Phone lock in :-)

Short personal notice: I am currently in Cologne for a business trip lasting two weeks so I am staying over the weekend. If any KDE people around Cologne want to go for a beer until next Thursday, let me know :)

So, back to the subject. This blog entry is about a rather weird behavior of QBuffer I've debugged recently.

Some friends of mine were seeing a weird problem with some of their code using Qt4 that had previously worked in Qt3.
They broke it down to this minimal test case:

QByteArray data( 20, '\0' );

QDataStream writeStream( &data, QIODevice::WriteOnly );
QDataStream readStream( &data, QIODevice::ReadOnly );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // this fails, d == 0

Looking at the content of the byte array "data" confirmed that the write operation had been successful, i.e. "data" looks like this (hex encoded)

0000000500000002000000000000000000000000

So why did the second read operation return 0?

We have already determined that the write operations worked as expected, i.e. "data" contains the 4 byte representations for 5 and 2.

After studying the QDataStream code we concluded that it would not cause the observed effect since it basically calls the QIODevice's read method and then casts the result into the given result type.

So clearly the data QDataStream was seeing in the device was not the 0x00000002. However, a QIODevice::peek() refuted that quite annoyingly

qDebug() << readStream.device()->peek( 4 ).toHex();

Results in "00000002"

Damn!

At this point we were mostly out of ideas so we tried to manually set the read index to specific values:

const qint64 pos = readStream.device()->pos();
readStream.device()->seek( pos );

qint32 d;
readStream >> d; // still no luck, d == 0

const qint64 pos = readStream.device()->pos();
readStream.device()->seek( 0 );
readStream.device()->seek( pos );

qint32 d;
readStream >> d; // HAH! that worked!

Clearly something is going on behind the scenes that is undone or fixed when seeking away from the current position and repositioning again.

Lets expand the code a bit more:

QByteArray data( 20, '\0' );

QBuffer writeBuffer( &data );
writeBuffer.open( QIODevice::WriteOnly );

QBuffer readBuffer( &data );
readBuffer.open( QIODevice::ReadOnly );

QDataStream writeStream( &writeBuffer );
QDataStream readStream( &readBuffer );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // this fails, d == 0

This is equivalent to the first code snippet, we just explicitly create the QBuffer objects that handle reading/writing to the QByteArray.

Following the trail we finally discovered that QIODevice, the base class of QBuffer, is buffering reads in some sort of internal buffer.

Meaning our "readStream" was seeing a situation that was out-of-date, i.e. seeing the state of the memory buffer at the time of its first read:
"0000000500000000" instead of the correct "0000000500000002".

Why QIODevice::peek() was clearly bypassing that internal buffer is up to speculation. It was probably easier to implement than to return what the device would actually be using at the next read operation.

Conclusion: when using a QBuffer (directly or indirectly) for reading, remember to always also specify QIODevice::Unbuffered for open flags, otherwise it will waste memory on buffering already in-memory data and messing up read/write behavior.

The correct code for reading and writing a shared memory buffer with two QDataStreams therefore looks like this:

QByteArray data( 20, '\0' );

QDataStream writeStream( &data, QIODevice::WriteOnly );
QDataStream readStream( &data, QIODevice::ReadOnly | QIODevice::Unbuffered );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // Finally!