When I upgraded my KDE-setup to 4.7.0 I run into the problem that when starting the brand new KMail based upon Akonadi that the Nepomuk-feeder started to index my imap-folders.

After my last blog I was asked whether I feel that the migrator is now ready for testing.
I think it is.

If one wants to repeat the same test scenario (or variations of it), there are a couple of tricks to do that with as less effort as possible.

As an initial step create a new user account. This ensures that the data and config on your main account is safe even in cases of anything going wrong.

Then, before coping all necessary data, shutdown KMail/Kontact if running. This ensures that all config and data files have been written to disk.
Actually, if you are using Disconnected IMAP, I recommend you synchronize all of them before doing that. While the migrator is supposed to handle unsynchronized caches as well, it is more work to make this work repeatedly and you will probably only do that if you really want to test this corner cases.

Copying data to a new user account is in my experience best done using "tar" because creation and extraction can be done with the respective user account, i.e. do not require any elevated priviledges and adjusts file ownership and rights accordingly.

Initial copying of data and config

Run the tar command in the original user's home directory, so that all paths in the archive are relative to home and work the same way on the new user account.

% tar cvf /tmp/migratortest.tar \
    .kde/share/config/kmailrc .kde/share/config/mailtransports .kde/share/config/emailidentities .kde/share/config/kcmkmailsummaryrc \
    .kde/share/apps/kmail .kde/share/config/kwalletrc .kde/share/apps/kwallet

If you have your local mail in any other directory than the default, e.g. $HOME/Mail, you need to add this as well.

On the test user account you can now run the extraction, again in the home directory

% tar xvf /tmp/migratortest

Deactivating "surprises"

Copying the config "as-is" can lead to unpleasent surprises, e.g. the mirgratd Akonadi setup starting to download mails from a POP3 server, making them unavailable for the main user account.

If you don't have any interval checking or "Check on startup" settings, you can skip this, but it doesn't hurt to make sure.

As the test user, open the copies KMail config, e.g.

% kate .kde/share/config/kmailrc

Look for the section labelled "[General]" and make sure it has this entry

checkmail-startup=false

Then check all section beginning with "[Account" and make sure they have this entry

check-interval=0

At this point you can now safely start KMail to do stuff with the copied data, e.g. setting flags, tagging mails, reducing the amount of data, whatever.

Creating the test environment

All the things above can be done with whatever version of KDE SC you are normally using, so you need an environment to run the new software in.

You can of course run the whole test user account with that version, but I personally prefer to have all usual tools available in their stable version and not having to build/install all of them again.
The following is therefore a description of how I prefer to do it.

First, I need an environment setup "script"

KDEDIR=/dvl/kde/trunk/install
KDEDIRS=$KDEDIR
AKTESTHOME=$(pwd)
KDEHOME=$AKTESTHOME/.kde
KDETMP=$AKTESTHOME/kdetmp
KDEVARTMP=$AKTESTHOME/kdevartmp
PATH=$KDEDIR/bin:$PATH
LD_LIBRARY_PATH=$KDEDIR/lib:$LD_LIBRARY_PATH
export KDEDIRS PATH LD_LIBRARY_PATH AKTESTHOME KDEHOME KDETMP KDEVARTMP

XDG_DATA_DIRS=$KDEDIR/share:/usr/local/share:/usr/share
export XDG_DATA_DIRS

XDG_DATA_HOME=$AKTESTHOME/.local/share
XDG_CONFIG_HOME=$AKTESTHOME/.config
export XDG_DATA_HOME XDG_CONFIG_HOME

cd $AKTESTHOME

$KDEDIR is where the new software version is installed to, here that is "/dvl/kde/trunk/install".

Creating a new test scenario works like this:

- Open a Konsole window with several tabs, I recommend at least four

- Create a test base directory, e.g.

mkdir /tmp/akonaditest

- Copy the environment file

cp testenv /tmp/akonaditest

- Switch the Konsole window into input forwarding mode, e.g.

CTRL + SHIFT + ,

- Change (all tabs simultaniously due to the previous step) into the the test directory

cd /tmp/akonaditest

- Source the environment

source testenv

- Now, in one the other tabs (so that it is the only one doing it)

dbus-launch > dbusenv

- Then again in the master tab (the one which forwards its input)

source dbusenv

- Switch of input forwarding, e.g.

CTRL + SHIFT + -

- Copy data and config (if you have KMail running in the test account, shut it down before doing that )

cp -ra $HOME/.kde .

Starting the runtime processes

Tab 1: run kdeinit4 and wait until it has stop writing insane amounts to log output :)
Tab 2: run nepomukserver
Tab 3: start Akonadi, run

akonadictl start

Running the migrator

Tab 4:

kmail-migrator --interactive

If you are asking yourself why I am using all these different tabs, I am doing that to have the log output of these processes separated from each other.
E.g. Tab 3 will contain only output from Akonadi server and Akonadi Resources, so I can look there if the migrator reported an Akonadi error.

Re-running a test

If you have Disconnnected IMAP accounts, you can rerun the test in the same environment if you choose to "Keep Local Copies" when the migrator asks how to deal with the cached emails after import.

If you chose the delete option, you'll have to copy the cache again:

cd .kde/share/apps/kmail
rm -r dimap
cp -ra $HOME/.kde/share/apps/kmail/dimap .

To rerun the full test, remove all newly created resource through "akonadiconsole", as well as the following config files: kmail2rc, kmail-migratorrc

To rerun only part of the test, remove the respective resources, kmail2rc and edit kmail-migratorrc, removing the respective Resource section.

Since you are working with a copy, you can modify kmailrc and remove all accounts you don't want to be part of the test run.

Disconnected IMAP import options

Import of Disconnected IMAP cache is following three options that can be set in kmail-migratorrc.
Part of the installation is a base config containing this:

[Disconnected IMAP]
ImportNewMessages=true
ImportCachedMessages=true
RemoveDeletedMessages=false

Without any change to this file ($KDEDIR/share/config/kmail-migratorrc) or an override in the local file, any Disconnected IMAP account will get all new (locally added but not synchronized yet) and normal cache messages imported.

If you enable the third one and have unsynchronized local deletes, the import process will attempt to delete these messages from the IMAP server. This list of locally deleted message identifiers is stored in "kmailrc", in the Account section of the respective IMAP account.
This can obviously only be done once, the messages are then gone from the server.

If you disable all three, you will get almost the same setup as for normal IMAP accounts, i.e. migrator just listing the folders, but it will also set cache policies on each folder to make them keep all emails that they download.

Misc

You can also run KMail (the new one) instead of the migrator, it will run the migrator for you and then continue its own startup.

You can check the finished setup either with KMail or just Akonadiconsole (second tab has a folder tree).
I personally always keep Akonadiconsole running in an additional tab.

You can use the current KMail to add/remove accounts, locally add/delete messages from Disconnected IMAP folders (if you want to test that), etc.

In case of questions, I usually hang out on our IRC channel #akonadi, freenode network, or send an email to me or our mailinglist (kde-pim@kde.org).

After blogging about our progress on KMail's data and config migration to Akonadi for a couple of times, I felt that it was time for a screencast showing the migrator in action.

The KMail test setup migrated here has most of the common account types: POP3, IMAP, Disconnected IMAP, local MBox file and, of course, KMail's local folders.

After starting up, the migration tool checks whether it will be dealing with one or more Disconnected IMAP accounts. If there is at least on unmigrated account of that type, just like in this test scenario, it asks the user how to handle the old cache after import.

Bascially, a Disconnected IMAP account in KMail has local copies of the mails on the IMAP server, so it can access them even when offline (hence "Disconnected IMAP").
The migrator imports these copies into Akonadi's cache so you don't have to download them again after the switch to KMail2.

Note:normally you don't need the old copies anymore, but for example if you want to test migration several times, you will want to opt for keep the copies for the next round.

In the screencast I am option to delete all successfully imported copies (the migrator will always keep those which it failed to import for whatever reason) in order not to duplicate disk space usage.

The migrator then continues to work through the configured accounts, starting with a POP3 one.
Retrieving the authentication data for the server requires access to the user's KDE wallet, which prompts for the password protecting it.

Next account is an IMAP server without local message copies (normal IMAP).
While this does not require import of already downloaded messages, it might still have only locally available information that is worth migrating, e.g. tags not supported by IMAP.

After finishing this rather fast check for metadata, the migrator continues with the test setup's local MBox account.

At this stage we get to the interesting part: the Disconnected IMAP account.
A couple of its folders have locally cached messages, two of them quite a lot (5000 each).

Finally, after having processed all configured accounts, the migrator's last task is to make KMail's local folders available to the Akonadi setup.
It therefore creates a special Akonadi resource which can deal with the mixture of Maildir and MBox folders and then checks whether any of these folders should be used for special roles such as "inbox", "outbox", "sent mail" and so on.

In the scenario of the screencast it can even make them the default for their respective role, just like they were in a traditional KMail setup.

Video available in OGG Theora and Flash on Blip.tv

Those who have read my previous blog entry will remember that I have been working on a way to directly operate on a mixed tree storage layout.

"Mixed tree" means that the mail folder tree (hence "tree") consists of Maildir and MBox folders nested in each other in any combination, e.g. Maildir inside Maildir, MBox inside Maildir, Maildir inside MBox.

Since this is highly KMail specific, it should probably be called "KMail tree" or "KMail mail directory", but there is always room for improvement, especially with naming.

Anyways, the Akonadi resource I've blogged about last time is actually built around a nicely self contained mail access system based on what I called "Akonadi FileStore".

It does not only provide the usual operations like loading, saving and deleting mails, it also provides (read-only) access to KMail generated meta data about e-mails, such as flags and tagging.

This meta data access capability is what makes this "Mixed Maildir Store" actually powerful enough to help with more than just local mail directory migration, so special thanks goes to Casey Link for extraction of the reader code from KMail sources and putting it into a library I just had to make use of.

Up to this point the KMail migrator, i.e. the utility which makes an existing KMail setup usable for the upcoming KMail2, could only create server access facilities and restore local mail access.
Due to the "Mixed Maildir Store" it can now also make use of the cache "Disconnected IMAP" accounts have created, mails which have been downloaded from an IMAP server and stored locally for faster and offline access.

So instead of just creating an IMAP handler for each Disconnected IMAP account, KMail migrator now also retrieved the cached e-mails and stores them into Akonadi's cache, including again additional meta data such as flags or tags.

Ideally any Disconnected IMAP account is full synchronized at the time of migration, i.e. all local changes have been uploaded to the server already.
However, the migrator is also be capable of dealing with unsynchronized changes, though deletion of mail on the server that have been deleted locally is currently disabled by default, better have some mails twice than losing any mail in case the implementation of that feature is not 100% correct.

One part I am not yet quite satisfied with is that the migrator tool adjust, as in changes, the KMail configuration file to point KMail2 to the new locations of things.
At no point does it change the actual e-mail data, so maybe it should also keep the config untouched, e.g. for re-running the migration process in case something didn't work right away.

In an attempt to follow up on my blog about Akonadi porting xplained I am going to write about Akonadi migration.

It is basically the data storage related cousin of porting:
Porting is, as we learned, about adapting applications to a new way of handling data.
Migration is about adapting data to new ways of being accessed.

The last couple of months I unfortunately had too little time for development on either KDE or Akonadi so I spent the available time on thinking about mail migration.

You see, mail is a special case when we talk about migration because it has a couple of key differences when compared to other data types:

- amount
- location
- state/properties

The difference in amount of data is the most obvious one. Most users will have more messages in mail fodlers than contacts in their address book, messages will on average be a lot bigger than contacts, e.g. due to containing attachments.
It is actually very likely that the amount of mail data is several orders of magnitudes greater than that of contact data.
For example I have probably around 100 contacts in my address book and probably around 100 000 messages (if not more), consuming several Gigabytes of disk space.

The different in location of data is referring to the likeliness of messages being stored on a server vs. other data types like contacts.
Recent developments like Google Contacts have shifted that somewhat but it is still more likely to encounter a setup with contacts being locally and mail being remotely stored.
Due to this almost inevitable remote storage any migration process will have to deal with local caching of some sort, e.g. in the case of KMail the maildir used to cache mails of KMail's "Disconnected IMAP" account type.

The difference in data state or properties is a lot less obvious than the other two.
Even in the most basic usage scenario we have the message state changing from unread to read but it is highly likely that messages have additional properties attached such as "important" or "you have replied to this one".
This alone wouldn't be a problem if it would be part of the message or at least part of the file the message is stored in. However not all on-disk formats support that so applications like KMail had to find alternative ways of storing this additional information, e.g. "index" files.

Lets have a look at a couple of scenarios to see how these influence the miration process.

For comparison we'll take migration of a local address book:
- get location of address book file (e.g. KDE's std.vcf)
- create Akonadi storage handler for a VCard file
- point it to the location of the file
- done

A similar approach will work for a local maildir directory:
- get location of the maildir root directory (e.g. $HOME/Mail, $KDEHOME/share/apps/kmail/mail)
- create Akonadi storage handler for MailDir
- point it to the location of the file
- done

The messages stay right where they are, so amount of data is irrelevant. MailDir can encode most of the state data into the file name, so not an immediate problem either. Storage is local, no server involved, no caching to deal with.

Now have a look at another form of local mail storage: mbox
- get location of the mbox file (again rather trivial)
- create Akonadi storage handler for MBox
- point it to the location fo the file
- done

Done? Not quite. Doing it that way we'll lose state data in a way that's probably not acceptable to our users.

So what could we do?
One option would be to make the Akonadi storage handler for MBox understand, e.g. KMail's index files, but that is quite ugly, involved maintaining old code (at least the reading part) and is KMail specific or requires the Akonadi MBox resource to understand all kinds of such additional files.

I'll get back to this later but lets have a look at another example first: mbox file within a maildir tree
- get location of the mbox file (again rather trivial)
- create Akonadi storage handler for MBox
- point it to the location fo the file
- done

Since this shares the same problem as stand-alone mbox, I'll skip the related problems.
However, we have some additional issues here, one being that the Akonadi MBox resource will create a top level folder in Akonadi, thus "moving" the mail box folder out of the tree while keeping the file in it.
One possible solution would to have a resource which can handle mboxes inside maildir trees, but since mbox folders and maildir folders behave differently (e.g. mbox folders need to be "compacted" to really delete mails) we don't consider this a proper solution unless we run out of alternatives.

To not forget about the server location problem, lets finally also look at disconnected IMAP:
- get server connection values and login credentials
- create Akonadi storage handler for IMAP
- tell it about server and user
- done

As an attention paying reader you already know that we are not quite done yet :)
So what is it this time?
Can't be state of message, everything is on the server. Can't be the resulting folder locations, IMAP servers have always been treated separately.

Obviously it has something to do with the "disconnected" part, so lets have a closer look at that.
It means that KMail has a maildir tree somewhere that is more or less a copy of what's on the IMAP server.
More or less because it is like synchronizing between a remote and a local directory, i.e. changes on either side are not immediately visible on the other side, they are applied at resychronization times.

This leads to two complications for the migration process:
- the users will be very angry if we have them download all those message again
- some messages might have been added or deleted locally and the respective changes have not been synchronized yet

Again a possible solution would be to make the Akonadi IMAP resource understand this local cache and transaction logs, but again this is not a very clean solution.

I am sorry that this is already quite a long blog but maybe you are still interested in some of my thoughts on how to make this work nevertheless.

If we treat the process more like a form of importing instead of simply reattaching different storage handlers, we gain the possibility to change format and locations in a way that allows us to inform the users about these changes and most likely also allow for an advanced mode for people with really specific needs.
So instead of silently doing things in the background, KMail2 could bring up a GUI saying that it has detected a KMail1 setup and lets you choose between ignoring that, importing that the way it sees fit or letting you switch to an import GUI for customization.

Our scenarios above can then be handled like this:

MailDir: no difference there but potentially allowing a customized import routine to move the messages to a new base directory

MBox: top level mbox files can be handled by the Akonadi MBox resource, the importer can be KMail specific and understand the index files, attaching the additional information to the resulting Akonadi message items.
MBox files in side the maildir tree can be read by the importer and added as a new folder to the top level folder managed by the Akonadi MailDir resource, potentially allowing a customized import routine to move the mbox file and treat it like a top level folder instead.

Disconnected IMAP: similar to the "in-tree mbox" case, the importer can be made to understand KMail's form of caching and transaction state handling. However, differently to the "mbox -> maildir folder" conversion, we do not want the Akonadi IMAP resource to add the messages on the server, most of them will already be there.

Again my main idea is to let the importer understand what it is actually importing, in this case how the message is addressed on the server, and attach this information to the message when adding it to a folder managed by the Akonadi IMAP resource.
The IMAP resource will therefore only have to be extended to understand this additionally attached information, not how that used to be stored on disk.

I hope to have some time during the Chrismas holidays to experiement with some of the ideas.

For quite some time almost every blog by a KDE PIM developer is about Akonadi in one for or the other, often about "Akonadi porting" or "porting to Akonadi".

Akonadi itself can already be difficult to explain, combined with "porting" it probably has only meaning left if you are a developer.

The thing anyone else will be able to extract are delays.
Delays in the sense that we, the developers working on KDE PIM, are enthusiastic about our plans on what we want to achieve for a certain KDE release, only to be disappointed that we didn't get as far as we originally hoped for.

For example an initial guess was to have KAddressBook and KOrganizer ported in time for the 4.3 release, but lack of resources meant we didn't even get to start working on KOrganizer in any significant form and the new KAddressBook would lack some features.
So KAddressBook is held back and finished for 4.4, KOrganizer is still being worked on.

So what is all this porting business actually about and why does it take so long?

Imagine a situation where you want to cook something (well, you can imagine anything els of course, but it will greatly improve the understanding of the following example if you restrict your fantasy to cooking for now).

After you've decided what you want to cook, you'll have to get your ingredients, which usually means going out for shopping. When you get home, probably after quite some time and travelled distance, you'll have to prepare these ingredients (wash, peel, chop, etc.) and then start the actual cooking.

That works pretty well, millions of people do between that once in a while and every day.
So you meet other cooks and discover that while you have individual styles for preparing ingredients and the cooking process, you all despise the shopping chore, especially when shops are crowed, roads are jammed, weather is nasty, and so on.

On one of those lazy sunday afternoons you read about outsourcing and immediately convince your fellow cooks to outsource shopping to a shopping specialist.
You snicker at the thought of this poor fellow having to brave the nasty weather, travel the jammed roads, wait in line at stores, while you comfortably wait for the devlivery.

However, you discover that you have to adjust your approach to cooking to accomodate for this for of ingredient acquisition.
With your initial approach you knew when you would be doing what for how long (not counting unavoidable delays during shopping).
With your new approach you don't anything between placing your order and the arrival of the goods. Sure, you could be waiting at the door step, but that gets boring after a while. You can spend some time preparing everything you already have at home, but usually that won't keep you busy long enough. So you do something else instead, maybe reading Planet KDE and learning that those lazy developers have balantly copied your outsourcing idea :)

Switching to a different reality there is this cook called KMail (in this reality cooks are, for an unknown reason, actually called "mail user agents").
An absolut expert in preparing an ingredient called "message" (weird reality, I know), peeling of envelopes, chopping into pieces (jokingly called "multi parts"), etc. You know, the usual cooking stuff, just done expertly.

But again the shopping for "messages" at shops called "servers", with owners of sometimes questionable character, low quality offerings and so on, is making KMail's life unpleasent.
So of course outsourcing that to a shopping specialist (in this reality, weird as it is, called "Akonadi") is the way to go.

Unfortunately our cook KMail discovers, while adjusting to the different cooking approach, that all these years of shopping had resulted in acquiring certain habits (in another reality, the author of this blog has the habit of not using shopping lists because eventually he'll recognize things to buy at the shop anyway).

Habits that need to be unlearned or replaced by habits which fit the new situation better. Habits our cook might not have been aware of or vowed never to think about again.
Fortunately there is a group of motivational trainers (having the surprisingly sensible name "KDE PIM Developers") who will help to overcome those.

It takes time, but it is totally worth it.

Good evening from Osnabrück; Tom shares some facts from the day 1 with you, so all I have now (before putting my hand on svn commit) is some graphics. Recently (except for updating opensuse) I have switched from winnt5 to winnt6 on my notebook (thanks Adriaan!).

I need to admit I am regular user of Akregator under Vista now (which is -- repeat-after-me -- fully functional -- aKregator I mean, not Vista ;)). I am also slowly moving to KMail as well on the OS. The latter KDE app runs with POP3/SMTP/IMAP/SASL support offered out of the box.

Read on for the gfx...

The first KMail hacking day is coming to a close. It has been a fun time -- working with many oldtime PIMsters and several new contributors as well. I noticed that KOrganizer was getting a little attention too.

This weekend we PIMsters are having a virtual meeting on #kontact; hopefully to give a lot of love to KMail for a potential KDE 4.0 release. Most of the KDE PIM applications are in dire need of attention and, in their current state, have almost no hope of being part of a KDE 4.x release.

The geeko is hungry, so for the past few days Coolo and I have been feeding it the carcasses of many of the more serious bugs in KMail. Our focus has been on online IMAP, as that has had the most egregious bugs in our opinion, but we're also after low-hanging fruit anywhere else in KMail. We're basically doing it because We Care A Lot, but we also want some tangible improvements in KDE 3.5 in openSUSE 10.3, besides all the KDE 4 work we're doing at the moment. The fixes are all going into 3.5 branch, of course, and are being ported to the enterprise branch and 3.5.5+, so rest-of-world benefits too.

So if you are using 3.5 branch svn, please hammer on KMail, especially online IMAP, and report any crashes you see. I'll try and post a list of the bugs we've scalped so far later, but for now I need to take a break and clear my head.