Buffered Buffer

Short personal notice: I am currently in Cologne for a business trip lasting two weeks so I am staying over the weekend. If any KDE people around Cologne want to go for a beer until next Thursday, let me know :)

So, back to the subject. This blog entry is about a rather weird behavior of QBuffer I've debugged recently.

Some friends of mine were seeing a weird problem with some of their code using Qt4 that had previously worked in Qt3. They broke it down to this minimal test case:

QByteArray data( 20, '\0' );

QDataStream writeStream( &data, QIODevice::WriteOnly );
QDataStream readStream( &data, QIODevice::ReadOnly );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // this fails, d == 0

Looking at the content of the byte array "data" confirmed that the write operation had been successful, i.e. "data" looks like this (hex encoded)

0000000500000002000000000000000000000000

So why did the second read operation return 0?

We have already determined that the write operations worked as expected, i.e. "data" contains the 4 byte representations for 5 and 2.

After studying the QDataStream code we concluded that it would not cause the observed effect since it basically calls the QIODevice's read method and then casts the result into the given result type.

So clearly the data QDataStream was seeing in the device was not the 0x00000002. However, a QIODevice::peek() refuted that quite annoyingly

qDebug() << readStream.device()->peek( 4 ).toHex();

Results in "00000002"

Damn!

At this point we were mostly out of ideas so we tried to manually set the read index to specific values:

const qint64 pos = readStream.device()->pos();
readStream.device()->seek( pos );

qint32 d;
readStream >> d; // still no luck, d == 0

const qint64 pos = readStream.device()->pos();
readStream.device()->seek( 0 );
readStream.device()->seek( pos );

qint32 d;
readStream >> d; // HAH! that worked!

Clearly something is going on behind the scenes that is undone or fixed when seeking away from the current position and repositioning again.

Lets expand the code a bit more:

QByteArray data( 20, '\0' );

QBuffer writeBuffer( &data );
writeBuffer.open( QIODevice::WriteOnly );

QBuffer readBuffer( &data );
readBuffer.open( QIODevice::ReadOnly );

QDataStream writeStream( &writeBuffer );
QDataStream readStream( &readBuffer );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // this fails, d == 0

This is equivalent to the first code snippet, we just explicitly create the QBuffer objects that handle reading/writing to the QByteArray.

Following the trail we finally discovered that QIODevice, the base class of QBuffer, is buffering reads in some sort of internal buffer.

Meaning our "readStream" was seeing a situation that was out-of-date, i.e. seeing the state of the memory buffer at the time of its first read: "0000000500000000" instead of the correct "0000000500000002".

Why QIODevice::peek() was clearly bypassing that internal buffer is up to speculation. It was probably easier to implement than to return what the device would actually be using at the next read operation.

Conclusion: when using a QBuffer (directly or indirectly) for reading, remember to always also specify QIODevice::Unbuffered for open flags, otherwise it will waste memory on buffering already in-memory data and messing up read/write behavior.

The correct code for reading and writing a shared memory buffer with two QDataStreams therefore looks like this:

QByteArray data( 20, '\0' );

QDataStream writeStream( &data, QIODevice::WriteOnly );
QDataStream readStream( &data, QIODevice::ReadOnly | QIODevice::Unbuffered );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // Finally!