MAY
26
2010

Status of automated KMail migration

Those who have read my previous blog entry will remember that I have been working on a way to directly operate on a mixed tree storage layout.

"Mixed tree" means that the mail folder tree (hence "tree") consists of Maildir and MBox folders nested in each other in any combination, e.g. Maildir inside Maildir, MBox inside Maildir, Maildir inside MBox.

Since this is highly KMail specific, it should probably be called "KMail tree" or "KMail mail directory", but there is always room for improvement, especially with naming.

Anyways, the Akonadi resource I've blogged about last time is actually built around a nicely self contained mail access system based on what I called "Akonadi FileStore".

It does not only provide the usual operations like loading, saving and deleting mails, it also provides (read-only) access to KMail generated meta data about e-mails, such as flags and tagging.

This meta data access capability is what makes this "Mixed Maildir Store" actually powerful enough to help with more than just local mail directory migration, so special thanks goes to Casey Link for extraction of the reader code from KMail sources and putting it into a library I just had to make use of.

Up to this point the KMail migrator, i.e. the utility which makes an existing KMail setup usable for the upcoming KMail2, could only create server access facilities and restore local mail access.
Due to the "Mixed Maildir Store" it can now also make use of the cache "Disconnected IMAP" accounts have created, mails which have been downloaded from an IMAP server and stored locally for faster and offline access.

So instead of just creating an IMAP handler for each Disconnected IMAP account, KMail migrator now also retrieved the cached e-mails and stores them into Akonadi's cache, including again additional meta data such as flags or tags.

Ideally any Disconnected IMAP account is full synchronized at the time of migration, i.e. all local changes have been uploaded to the server already.
However, the migrator is also be capable of dealing with unsynchronized changes, though deletion of mail on the server that have been deleted locally is currently disabled by default, better have some mails twice than losing any mail in case the implementation of that feature is not 100% correct.

One part I am not yet quite satisfied with is that the migrator tool adjust, as in changes, the KMail configuration file to point KMail2 to the new locations of things.
At no point does it change the actual e-mail data, so maybe it should also keep the config untouched, e.g. for re-running the migration process in case something didn't work right away.

Comments

Have you thought about robustness features in case the kmail reader library imports old kmail bugs that could crash the mixed tree resource? Something like a log feature so that the message/folder index currently being processed is tracked, so that if the resource crashes while parsing the old metadata, it can easily be identified by a developer, or worked around by blacklisting the old metadata import for that message/folder index that caused a crash on a previous run?


By Will Stephenson at Thu, 05/27/2010 - 09:17

I didn't know that there were problem with the index reading which could even result in a crash.
Maybe we can guard our index reading code against these, we are only using a portion of the stored data.

I haven't looked through the list yet, but some of these might be caused by KMail using the values read from the index, not the index reading itself.
Is any of them specifically hinting at a crash when reading the index?


By krake at Thu, 05/27/2010 - 11:53