Short personal notice: I am currently in Cologne for a business trip lasting two weeks so I am staying over the weekend. If any KDE people around Cologne want to go for a beer until next Thursday, let me know :)

So, back to the subject. This blog entry is about a rather weird behavior of QBuffer I've debugged recently.

Some friends of mine were seeing a weird problem with some of their code using Qt4 that had previously worked in Qt3.
They broke it down to this minimal test case:

QByteArray data( 20, '\0' );

QDataStream writeStream( &data, QIODevice::WriteOnly );
QDataStream readStream( &data, QIODevice::ReadOnly );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // this fails, d == 0

Looking at the content of the byte array "data" confirmed that the write operation had been successful, i.e. "data" looks like this (hex encoded)

0000000500000002000000000000000000000000

So why did the second read operation return 0?

We have already determined that the write operations worked as expected, i.e. "data" contains the 4 byte representations for 5 and 2.

After studying the QDataStream code we concluded that it would not cause the observed effect since it basically calls the QIODevice's read method and then casts the result into the given result type.

So clearly the data QDataStream was seeing in the device was not the 0x00000002. However, a QIODevice::peek() refuted that quite annoyingly

qDebug() << readStream.device()->peek( 4 ).toHex();

Results in "00000002"

Damn!

At this point we were mostly out of ideas so we tried to manually set the read index to specific values:

const qint64 pos = readStream.device()->pos();
readStream.device()->seek( pos );

qint32 d;
readStream >> d; // still no luck, d == 0

const qint64 pos = readStream.device()->pos();
readStream.device()->seek( 0 );
readStream.device()->seek( pos );

qint32 d;
readStream >> d; // HAH! that worked!

Clearly something is going on behind the scenes that is undone or fixed when seeking away from the current position and repositioning again.

Lets expand the code a bit more:

QByteArray data( 20, '\0' );

QBuffer writeBuffer( &data );
writeBuffer.open( QIODevice::WriteOnly );

QBuffer readBuffer( &data );
readBuffer.open( QIODevice::ReadOnly );

QDataStream writeStream( &writeBuffer );
QDataStream readStream( &readBuffer );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // this fails, d == 0

This is equivalent to the first code snippet, we just explicitly create the QBuffer objects that handle reading/writing to the QByteArray.

Following the trail we finally discovered that QIODevice, the base class of QBuffer, is buffering reads in some sort of internal buffer.

Meaning our "readStream" was seeing a situation that was out-of-date, i.e. seeing the state of the memory buffer at the time of its first read:
"0000000500000000" instead of the correct "0000000500000002".

Why QIODevice::peek() was clearly bypassing that internal buffer is up to speculation. It was probably easier to implement than to return what the device would actually be using at the next read operation.

Conclusion: when using a QBuffer (directly or indirectly) for reading, remember to always also specify QIODevice::Unbuffered for open flags, otherwise it will waste memory on buffering already in-memory data and messing up read/write behavior.

The correct code for reading and writing a shared memory buffer with two QDataStreams therefore looks like this:

QByteArray data( 20, '\0' );

QDataStream writeStream( &data, QIODevice::WriteOnly );
QDataStream readStream( &data, QIODevice::ReadOnly | QIODevice::Unbuffered );

qint32 a = 5;
writeStream << a;

qint32 b;
readStream >> b;

Q_ASSERT( b == 5 );

qint32 c = 2;
writeStream << c;

qint32 d;
readStream >> d;

Q_ASSERT( d == 2 ); // Finally!

A slight delay of my flight from Düsseldorf to Graz gives me time to recap the awesome time at the Desktop Summit 2011 in Berlin.

Having been part of the programme committee I was looking forward to see at least a small subset of the talks live, though I hope I will have the opportunity to see many more once the videos have been processed and are available online.

Selecting talks hadn't been easy, both because of the huge amount of proposals we've received and because deciding on suitable topics for such a wide range of attendees is no piece of cake either.

I think the session I liked most was the lightning talk style collection of presentations of our communities' GSOC students and GNOME's women outreach program interns. Not only did they often take on difficult challenges, they all showed amazingly sophisticated solutions.

I also found it extremely encouraging that these programs seem to strengthen and even expand the diversity of our communities. I mean, sure, GNOME, KDE and other Free Software projects are international by design, with contributors from many countries. But due to various reasons the bulk of people are usual from North America or Europe.
So seeing so many students/interns from Asian, Latin American and Eastern European countries was one of my personal highlights of these summit. I hope that the incredibly high percentage of women in this group of presenters indicates that we are on the right path to improving our rather embarrassing track record in making our female peers feel welcome in our midst.

Companies often store their customer information in databases managed by customer relation management tools.

SugarCRM is one such popular (and open source) system, built on the extremly wide spread AMP stack (Apache HTTP, MySQL, PHP). Therefore its main user interface is web based, i.e. accessible through standard web browsers and thus also relatively platform independent.

Additional to that SugarCRM is also capable of being a web service, allowing interaction with programs of any sort through passing requests and responses formatted as XML over HTTP.
The technology deployed for this is called SOAP.

Building a SOAP client with just XML parsing libraries would be a daunting task, so there are many SOAP frameworks which help with that. Now, as KDE developers we would want to use such a framework which is as Qt-like as possible, ideally one being built with Qt.

Under these requirments our champion is KD SOAP, which is not only a Qt based library capable of talking SOAP, it also includes a code generator tools which will take a web service's API description file and generate C++/Qt proxy code for working with the web service as if it would be a local object in your application.
For the developers among this blog's readers, you can think of this as an equivalent to using qdbusxml2cpp for D-Bus services.

When we talk about contacts in the context of KDE we of course imply Akonadi, or more specifically having an Akonadi resource for accessing contact data on a certain backend.

KD SOAP enabled me to quickly develop such a resource for accessing SugarCRM addressbook, which I have now put into KDE's git.
It is still lacking polish such as KWallet integration, but is can already be used to view, edit, add and delete SugarCRM contacts through KDE addressbook interfaces.

Like all KDAB's add-on products to Qt and like Qt itself, KD SOAP is available both under Free Software (GPL) license as well as under a commercial license.
The SugarCRM resource is Free Software and licenced under GPL2 and GPL3.

The following are two screenshots showing sample contact data in both SugarCRM's native web interface as well as in KDE's native addressbook:

I am not talking about His Steveness or this kind of jobs (congratulations to both involved parties!), I am talking about these.

Traditionally we have been writing in a very start-to-end fashion, where execution starts at an entry point and ends when the task is done.
At first there wasn't any reason not to do this as there was no user interaction during a programs execution point or only at controlled points, e.g. a console program asking for a Y/N descision.

Then, with the switch to event based main execution, that changed to having multiple entry points but each usually still starting an execution chain that finished a complete task in one go.

One of the reasons for this "complete task in one go" style is that it is a lot easier to code. All necessary data only has to live during this uninterrupted execution, nothing else can interfere and maybe make some of the data invalid, etc.

This is of course no problem for anything that can be done fast, but long duration processing like the introduction of network I/O with its latency and overall time requirements made it necessary to come up with new styles.

One of them is staying with the start-to-end fashion but move the execution to a parallel processing unit. e.g. multi threading.

Another one is processing the task in smaller steps, using the a similar technique already used for separating the tasks, e.g. event based.

Given that Qt already has event driven processing, doing it for things like networking was a natural choice.
However, mainly because Qt's I/O classes are relatively low level and at that time weren't as unified as they are nowadays, KDE developers came up with an even nicer to use idea: jobs.

A job is basically a context for processing a single task (though such a task could of course be an aggregation of sub tasks). It is started and ends at some point, it might report progress inbetween, might even be able to be suspended and resumed.

Implementing a task as a job is usually more challenging than the traditional uninterrupted execution path, but it is really not much more difficult to use: you create the job object, connect to its result signal and start it.

Sure, compared to a normal method call you don't get the result at that part of your code where you start the thread, instead you need an addition method (the slot the result signal is connected to).
However, you also have advantages, like having all data necessary for the task stored in the job object and not having to keep it around elsewhere (but still having access to it from whereever you interact with the job).

Unfortunately they are not yet widely used outside of I/O bound tasks, quite some other long duration processing is implemented in keeping the event loop running but not actually returning from the current code context, e.g. "blocking" user input by showing some kind of process dialog, returning either when finished or when the user explicitly cancelled.

At first this looks just like another good option for doing long duration task processing, however this can lead to multiple execution contexts within applications that are not prepared for that (because they know they are not using threads so they don't expect things to be called twice).

This possibilty of unexpected re-entrancy is often overlooked because the magnitude of operations is caused by user input and thu,s by taking that out of the equation, makes the approach "safe" again.

I put "safe" in quotes because it isn't true. Events can not only caused by the application's user, it could be caused by a timer reaching its timeout, or a socket becoming available for reading.

One could work around some of these by means of flags or similar status variables but even for application code this becomes really messy at some point.
It is almost an invitation for disaster for any form of library code.

In KDE PIM we've got bitten by that a lot during the last couple of releases due to libraries and applications assuming things like instant access to data. We had to effectively split a lot of our code into smaller parts that could deal with having some data arrive in chunks or not at all.

That was a lot of work and mostly only possible because none of these libraries were public API, thus allowing us to change them without caring about even source compatibilty.

So my advice to anyone adding new libraries that involve long duration processing, don't assume that all applications using the library will be fine with the library blocking the user input as its sol method of avoiding re-entrancy.

Think about providing job based API and let application developers hook it up to a progress dialog if that's how they want it processed.
Heck, if it is really so difficult to connect the job's result signal to an appropriate dialog slot, provide a convenience function that does that and extracts and returns the result from the job.

Harald has asked me to organize a KDE release party here in Graz.

However, I am too lazy (wait! busy! sorry for my English, busy is the word I was looking for) to do that there won't be a dedicated event to celebrate KDE's newest achievement.

Fortunately the release date, August 4th, is a Wednesday or more precisely the first Wednesday of the month.
And on such rare occasions (happens only twelve times a year) Free Software users from Graz and areas close by gather at a nice Italian restaurant to have Pizza, Beer (some even have Cappuchino and Tiramisù) and talk about stuff that comes to their minds.

For the German native speakers among you, we call this "LUGG Stammtisch" :)
(as in Linux User Group Graz, Stammtisch should be self-explanatory)

So if you think you can stand yet another pizza or fancy having some beer with awesome people, including at least one well know KDE developer ;), join us at Cosa Nostra, Hans-Sachs-Gasse 10.

Beginning at 20:00, until we are thrown out (gah! until we are politely asked to leave) :D

See you there!

With Akonadi most operations are running behind the scenes, carried out by background helper processes called Akonadi Agents.

While we do have respective progress monitoring in KMail2, users will eventually take advantage of fact that they are no longer tied to specific applications. At which point they might want to be able to check on the status of these background processes without launching some front end applications.

Back in April, during one of our development sprints, I've created a Plasma DataEngine which provides information about running Akonadi agents.

I was kind of hoping that somebody with actual widget skills would be curious enough to try some Plasma widgets on top of it, alas this didn't happen.

Therefore I sat down today and wrote one myself, using the opportuntiy to also have a first real attempt in doing some KDE<->JavaScript coding.

The code for it more or less looks like this:

layout = new LinearLayout( plasmoid );
layout.orientation = QtVertical;

engine = dataEngine( "akonadiagents" );
agents = engine.sources;

resources = new Array;

addAgent = function( name ) {
  label = new IconWidget();
  label.orientation = QtHorizontal;
  layout.addItem( label );
  resources[ name ] = label;

  engine.connectSource( name, plasmoid );
}

plasmoid.dataUpdated = function( name, data ) {
  label = resources[ name ];

  label.text = data.name;
  label.icon = data.typeIcon;

  label.infoText = data.statusMessage;
  if ( !data.online ) {
    label.enabled = false;
  } else {
    label.enabled = true;
    if ( data.status == 1 ) { // running
      label.infoText = data.statusMessage + " (" + data.progress + "%)";
    }
  }
}

while ( agents.length > 0 ) {
  agent = agents.pop();

  // ideally this would be using the agent type's capabilities, but the DataEngine::Data as returned by
  // DataEngine::query() is not accessible from within JavaScript (or at least nobody on #plasma knew)
  if ( agent.indexOf( "resource" ) != -1 ) {
    addAgent( agent );
  }
}

I am sure this is not very pretty from the point of view of skilled JavaScript users, you are welcome to beat me on that :D

Anyway, screen cast of the Plasmoid in action, as usual on blip.tv

It shows the widget running in plasmoidviewer side by side with Akonadiconsole to demonstrate that the data engine really exposes the same data.
I start with toggling a local VCard file resource between Offline and Online state, which the widget visualizes by disabling/enabling the respective item.

Finally I synchronize all collections of the IMAP resource, showing status and progres reporting.

After all the blogging about our (as in KDE PIM developers) Akonadi porting, I thought I'll address a couple of things other application developer might consider for the next release cycle.

The most prevalent use of PIM API seems to be KABC::StddAddressBook.

Find By Uid

A common use case is to look for a contact object by its identifier, acquired by user selection somewhen in the past.
The code to do that usually looks like this:

KABC::AddressBook *addressBook = KABC::StdAddressBook::self();

KABC::Addressee contact = addressBook->findByUid( uidString );

This is quite easy but it always has a couple of issues, mainly the call to

KABC::StdAddressBook::self()

because its first invocation will load the whole address book and it will do so synchronously.
Potentially blocking the application for the whole duration or, sometimes even worse, introducing unexpected reentrancy in single threaded applications due to use of nested event loops.

Note: while the same sequence is possible with Akonadi, due to Akonadi jobs being based on KJob and it having the capability of "blocking" by use of a nested event loop, I really recommend that you use the porting effort as an opportunity to get rid of such emulated synchronousity whenever possible.

The equivalent using Akonadi API looks like this

Akonadi::ContactSearchJob *job = new Akonadi::ContactSearchJob( this );
job->setQuery( Akonadi::ContactSearchJob::ContactUid, uidString );

connect( job, SIGNAL( result( KJob* ) ), SLOT( contactSearchResult( KJob* ) ) );

and

void contactSearchResult( KJob *job )
{
  if ( job->error() != 0 ) {
    // error handling, see job->errorString()
    return;
  }

  Akonadi::ContactSearchJob *searchJob = qobject_cast( job );

  const KABC::Addressee::List contacts = searchJob->contacts();
}

Also see API docs for ContactSearchJob

Find by Email

Another use case I came across is searching by Email. The KABC code usually looks pretty identical:

KABC::AddressBook *addressBook = KABC::StdAddressBook::self();

KABC::Addressee contact = addressBook->findByEmail( emailString );

Unsurprisingly, the Akonadi code looks also quite similar to the previous use case:

Akonadi::ContactSearchJob *job = new Akonadi::ContactSearchJob( this );
job->setQuery( Akonadi::ContactSearchJob::Email, emailString );

connect( job, SIGNAL( result( KJob* ) ), SLOT( contactSearchResult( KJob* ) ) );

The code for the result slot is the same.

Who Am I

Another use case for StdAddressBook is not directly available in Akonadi:

KABC::AddressBook *addressBook = KABC::StdAddressBook::self();

KABC::Addressee contact = addressBook->whoAmI();

Even this had its shortcomings, because this depends on the users adding themselves to the address book and marking the respective entry as "this is me".

The concrete use cases I've seen this deployed for are getting the user's full name and/or their email address.

It might be better to use KPIMIdentities::IdentityManager instead.
It contains the identity users can configure in the Systemsettings module for personal information, so if there is data yet, you could embed the respective KCM in a dialog and let the user configure this centrally for all of KDE.

Modifying a contact

Using KABC::StdAddressBook modifying a contact (for example one retrieved by findByUid()) worked like this:

addressBook->insertAddressee( contact );

KABC::Ticket *ticket = addressBook->requestSaveTicket();
if ( !ticket ) {
  // error
} else if ( !addressBook->save( ticket ) ) {
    // error
    addressBook->releaseSaveTicket( ticket );
  }
}

With Akonadi, you will need the Akonadi::Item or at least the item's identifier.
Assuming you retrieved the contact using ContactSearchJob, consider keeping the item or Item::id() around, see

ContactSearchJob::items()

(inherited).

Lets say you have just the item's identifier:

Akonadi::Item item( itemId );
item.setPayload( contact );
item.setMimeType( KABC::Addressee::mimeType() );

Akonadi::ItemModifyJob *job = new Akonadi::ItemModifyJob( item );

connect( job, SIGNAL( result( KJob* ) ), SLOT( contactModifyResult( KJob* ) ) );

Slot code similar to the one for search.

Buildsystem adjustments
Since you are already linking against KDE PIM Libs classes, you already have the necessary

find_package( KdepimLibs 4.5 REQUIRED )

However, your target_link_libraries need to be extended to contain

${KDEPIMLIBS_AKONADI_LIBS}

for base things like Akonadi::Item and

${KDEPIMLIBS_AKONADI_CONTACT_LIBS}

for things like

Akonadi::ContactSearchJob

.

Includes for Akonadi core classes are like

#include <Akonadi/Item>

for contact related specializations they are like

#include <Akonadi/Contact/ContactSearchJob>

In both cases also available in lowercase variants with ".h" suffix.

Other useful classes

Our Akonadi porting efforts have resulted in a couple of useful classes for various aspects of dealing with PIM data.
For contacts such classes would be: ContactEditor (also available as a dialog), ContactViewer (again also available as a dialog), ContactsTreeModel and others.

As usual: in case of questions, our IRC channel is #akonadi, freenode network, or send an email to our mailinglist (kde-pim@kde.org).

Also available on TechBase

After my last blog I was asked whether I feel that the migrator is now ready for testing.
I think it is.

If one wants to repeat the same test scenario (or variations of it), there are a couple of tricks to do that with as less effort as possible.

As an initial step create a new user account. This ensures that the data and config on your main account is safe even in cases of anything going wrong.

Then, before coping all necessary data, shutdown KMail/Kontact if running. This ensures that all config and data files have been written to disk.
Actually, if you are using Disconnected IMAP, I recommend you synchronize all of them before doing that. While the migrator is supposed to handle unsynchronized caches as well, it is more work to make this work repeatedly and you will probably only do that if you really want to test this corner cases.

Copying data to a new user account is in my experience best done using "tar" because creation and extraction can be done with the respective user account, i.e. do not require any elevated priviledges and adjusts file ownership and rights accordingly.

Initial copying of data and config

Run the tar command in the original user's home directory, so that all paths in the archive are relative to home and work the same way on the new user account.

% tar cvf /tmp/migratortest.tar \
    .kde/share/config/kmailrc .kde/share/config/mailtransports .kde/share/config/emailidentities .kde/share/config/kcmkmailsummaryrc \
    .kde/share/apps/kmail .kde/share/config/kwalletrc .kde/share/apps/kwallet

If you have your local mail in any other directory than the default, e.g. $HOME/Mail, you need to add this as well.

On the test user account you can now run the extraction, again in the home directory

% tar xvf /tmp/migratortest

Deactivating "surprises"

Copying the config "as-is" can lead to unpleasent surprises, e.g. the mirgratd Akonadi setup starting to download mails from a POP3 server, making them unavailable for the main user account.

If you don't have any interval checking or "Check on startup" settings, you can skip this, but it doesn't hurt to make sure.

As the test user, open the copies KMail config, e.g.

% kate .kde/share/config/kmailrc

Look for the section labelled "[General]" and make sure it has this entry

checkmail-startup=false

Then check all section beginning with "[Account" and make sure they have this entry

check-interval=0

At this point you can now safely start KMail to do stuff with the copied data, e.g. setting flags, tagging mails, reducing the amount of data, whatever.

Creating the test environment

All the things above can be done with whatever version of KDE SC you are normally using, so you need an environment to run the new software in.

You can of course run the whole test user account with that version, but I personally prefer to have all usual tools available in their stable version and not having to build/install all of them again.
The following is therefore a description of how I prefer to do it.

First, I need an environment setup "script"

KDEDIR=/dvl/kde/trunk/install
KDEDIRS=$KDEDIR
AKTESTHOME=$(pwd)
KDEHOME=$AKTESTHOME/.kde
KDETMP=$AKTESTHOME/kdetmp
KDEVARTMP=$AKTESTHOME/kdevartmp
PATH=$KDEDIR/bin:$PATH
LD_LIBRARY_PATH=$KDEDIR/lib:$LD_LIBRARY_PATH
export KDEDIRS PATH LD_LIBRARY_PATH AKTESTHOME KDEHOME KDETMP KDEVARTMP

XDG_DATA_DIRS=$KDEDIR/share:/usr/local/share:/usr/share
export XDG_DATA_DIRS

XDG_DATA_HOME=$AKTESTHOME/.local/share
XDG_CONFIG_HOME=$AKTESTHOME/.config
export XDG_DATA_HOME XDG_CONFIG_HOME

cd $AKTESTHOME

$KDEDIR is where the new software version is installed to, here that is "/dvl/kde/trunk/install".

Creating a new test scenario works like this:

- Open a Konsole window with several tabs, I recommend at least four

- Create a test base directory, e.g.

mkdir /tmp/akonaditest

- Copy the environment file

cp testenv /tmp/akonaditest

- Switch the Konsole window into input forwarding mode, e.g.

CTRL + SHIFT + ,

- Change (all tabs simultaniously due to the previous step) into the the test directory

cd /tmp/akonaditest

- Source the environment

source testenv

- Now, in one the other tabs (so that it is the only one doing it)

dbus-launch > dbusenv

- Then again in the master tab (the one which forwards its input)

source dbusenv

- Switch of input forwarding, e.g.

CTRL + SHIFT + -

- Copy data and config (if you have KMail running in the test account, shut it down before doing that )

cp -ra $HOME/.kde .

Starting the runtime processes

Tab 1: run kdeinit4 and wait until it has stop writing insane amounts to log output :)
Tab 2: run nepomukserver
Tab 3: start Akonadi, run

akonadictl start

Running the migrator

Tab 4:

kmail-migrator --interactive

If you are asking yourself why I am using all these different tabs, I am doing that to have the log output of these processes separated from each other.
E.g. Tab 3 will contain only output from Akonadi server and Akonadi Resources, so I can look there if the migrator reported an Akonadi error.

Re-running a test

If you have Disconnnected IMAP accounts, you can rerun the test in the same environment if you choose to "Keep Local Copies" when the migrator asks how to deal with the cached emails after import.

If you chose the delete option, you'll have to copy the cache again:

cd .kde/share/apps/kmail
rm -r dimap
cp -ra $HOME/.kde/share/apps/kmail/dimap .

To rerun the full test, remove all newly created resource through "akonadiconsole", as well as the following config files: kmail2rc, kmail-migratorrc

To rerun only part of the test, remove the respective resources, kmail2rc and edit kmail-migratorrc, removing the respective Resource section.

Since you are working with a copy, you can modify kmailrc and remove all accounts you don't want to be part of the test run.

Disconnected IMAP import options

Import of Disconnected IMAP cache is following three options that can be set in kmail-migratorrc.
Part of the installation is a base config containing this:

[Disconnected IMAP]
ImportNewMessages=true
ImportCachedMessages=true
RemoveDeletedMessages=false

Without any change to this file ($KDEDIR/share/config/kmail-migratorrc) or an override in the local file, any Disconnected IMAP account will get all new (locally added but not synchronized yet) and normal cache messages imported.

If you enable the third one and have unsynchronized local deletes, the import process will attempt to delete these messages from the IMAP server. This list of locally deleted message identifiers is stored in "kmailrc", in the Account section of the respective IMAP account.
This can obviously only be done once, the messages are then gone from the server.

If you disable all three, you will get almost the same setup as for normal IMAP accounts, i.e. migrator just listing the folders, but it will also set cache policies on each folder to make them keep all emails that they download.

Misc

You can also run KMail (the new one) instead of the migrator, it will run the migrator for you and then continue its own startup.

You can check the finished setup either with KMail or just Akonadiconsole (second tab has a folder tree).
I personally always keep Akonadiconsole running in an additional tab.

You can use the current KMail to add/remove accounts, locally add/delete messages from Disconnected IMAP folders (if you want to test that), etc.

In case of questions, I usually hang out on our IRC channel #akonadi, freenode network, or send an email to me or our mailinglist (kde-pim@kde.org).

After blogging about our progress on KMail's data and config migration to Akonadi for a couple of times, I felt that it was time for a screencast showing the migrator in action.

The KMail test setup migrated here has most of the common account types: POP3, IMAP, Disconnected IMAP, local MBox file and, of course, KMail's local folders.

After starting up, the migration tool checks whether it will be dealing with one or more Disconnected IMAP accounts. If there is at least on unmigrated account of that type, just like in this test scenario, it asks the user how to handle the old cache after import.

Bascially, a Disconnected IMAP account in KMail has local copies of the mails on the IMAP server, so it can access them even when offline (hence "Disconnected IMAP").
The migrator imports these copies into Akonadi's cache so you don't have to download them again after the switch to KMail2.

Note:normally you don't need the old copies anymore, but for example if you want to test migration several times, you will want to opt for keep the copies for the next round.

In the screencast I am option to delete all successfully imported copies (the migrator will always keep those which it failed to import for whatever reason) in order not to duplicate disk space usage.

The migrator then continues to work through the configured accounts, starting with a POP3 one.
Retrieving the authentication data for the server requires access to the user's KDE wallet, which prompts for the password protecting it.

Next account is an IMAP server without local message copies (normal IMAP).
While this does not require import of already downloaded messages, it might still have only locally available information that is worth migrating, e.g. tags not supported by IMAP.

After finishing this rather fast check for metadata, the migrator continues with the test setup's local MBox account.

At this stage we get to the interesting part: the Disconnected IMAP account.
A couple of its folders have locally cached messages, two of them quite a lot (5000 each).

Finally, after having processed all configured accounts, the migrator's last task is to make KMail's local folders available to the Akonadi setup.
It therefore creates a special Akonadi resource which can deal with the mixture of Maildir and MBox folders and then checks whether any of these folders should be used for special roles such as "inbox", "outbox", "sent mail" and so on.

In the scenario of the screencast it can even make them the default for their respective role, just like they were in a traditional KMail setup.

Video available in OGG Theora and Flash on Blip.tv

Those who have read my previous blog entry will remember that I have been working on a way to directly operate on a mixed tree storage layout.

"Mixed tree" means that the mail folder tree (hence "tree") consists of Maildir and MBox folders nested in each other in any combination, e.g. Maildir inside Maildir, MBox inside Maildir, Maildir inside MBox.

Since this is highly KMail specific, it should probably be called "KMail tree" or "KMail mail directory", but there is always room for improvement, especially with naming.

Anyways, the Akonadi resource I've blogged about last time is actually built around a nicely self contained mail access system based on what I called "Akonadi FileStore".

It does not only provide the usual operations like loading, saving and deleting mails, it also provides (read-only) access to KMail generated meta data about e-mails, such as flags and tagging.

This meta data access capability is what makes this "Mixed Maildir Store" actually powerful enough to help with more than just local mail directory migration, so special thanks goes to Casey Link for extraction of the reader code from KMail sources and putting it into a library I just had to make use of.

Up to this point the KMail migrator, i.e. the utility which makes an existing KMail setup usable for the upcoming KMail2, could only create server access facilities and restore local mail access.
Due to the "Mixed Maildir Store" it can now also make use of the cache "Disconnected IMAP" accounts have created, mails which have been downloaded from an IMAP server and stored locally for faster and offline access.

So instead of just creating an IMAP handler for each Disconnected IMAP account, KMail migrator now also retrieved the cached e-mails and stores them into Akonadi's cache, including again additional meta data such as flags or tags.

Ideally any Disconnected IMAP account is full synchronized at the time of migration, i.e. all local changes have been uploaded to the server already.
However, the migrator is also be capable of dealing with unsynchronized changes, though deletion of mail on the server that have been deleted locally is currently disabled by default, better have some mails twice than losing any mail in case the implementation of that feature is not 100% correct.

One part I am not yet quite satisfied with is that the migrator tool adjust, as in changes, the KMail configuration file to point KMail2 to the new locations of things.
At no point does it change the actual e-mail data, so maybe it should also keep the config untouched, e.g. for re-running the migration process in case something didn't work right away.