After my last blog I was asked whether I feel that the migrator is now ready for testing.
I think it is.

If one wants to repeat the same test scenario (or variations of it), there are a couple of tricks to do that with as less effort as possible.

As an initial step create a new user account. This ensures that the data and config on your main account is safe even in cases of anything going wrong.

Then, before coping all necessary data, shutdown KMail/Kontact if running. This ensures that all config and data files have been written to disk.
Actually, if you are using Disconnected IMAP, I recommend you synchronize all of them before doing that. While the migrator is supposed to handle unsynchronized caches as well, it is more work to make this work repeatedly and you will probably only do that if you really want to test this corner cases.

Copying data to a new user account is in my experience best done using "tar" because creation and extraction can be done with the respective user account, i.e. do not require any elevated priviledges and adjusts file ownership and rights accordingly.

Initial copying of data and config

Run the tar command in the original user's home directory, so that all paths in the archive are relative to home and work the same way on the new user account.

% tar cvf /tmp/migratortest.tar \
    .kde/share/config/kmailrc .kde/share/config/mailtransports .kde/share/config/emailidentities .kde/share/config/kcmkmailsummaryrc \
    .kde/share/apps/kmail .kde/share/config/kwalletrc .kde/share/apps/kwallet

If you have your local mail in any other directory than the default, e.g. $HOME/Mail, you need to add this as well.

On the test user account you can now run the extraction, again in the home directory

% tar xvf /tmp/migratortest

Deactivating "surprises"

Copying the config "as-is" can lead to unpleasent surprises, e.g. the mirgratd Akonadi setup starting to download mails from a POP3 server, making them unavailable for the main user account.

If you don't have any interval checking or "Check on startup" settings, you can skip this, but it doesn't hurt to make sure.

As the test user, open the copies KMail config, e.g.

% kate .kde/share/config/kmailrc

Look for the section labelled "[General]" and make sure it has this entry

checkmail-startup=false

Then check all section beginning with "[Account" and make sure they have this entry

check-interval=0

At this point you can now safely start KMail to do stuff with the copied data, e.g. setting flags, tagging mails, reducing the amount of data, whatever.

Creating the test environment

All the things above can be done with whatever version of KDE SC you are normally using, so you need an environment to run the new software in.

You can of course run the whole test user account with that version, but I personally prefer to have all usual tools available in their stable version and not having to build/install all of them again.
The following is therefore a description of how I prefer to do it.

First, I need an environment setup "script"

KDEDIR=/dvl/kde/trunk/install
KDEDIRS=$KDEDIR
AKTESTHOME=$(pwd)
KDEHOME=$AKTESTHOME/.kde
KDETMP=$AKTESTHOME/kdetmp
KDEVARTMP=$AKTESTHOME/kdevartmp
PATH=$KDEDIR/bin:$PATH
LD_LIBRARY_PATH=$KDEDIR/lib:$LD_LIBRARY_PATH
export KDEDIRS PATH LD_LIBRARY_PATH AKTESTHOME KDEHOME KDETMP KDEVARTMP

XDG_DATA_DIRS=$KDEDIR/share:/usr/local/share:/usr/share
export XDG_DATA_DIRS

XDG_DATA_HOME=$AKTESTHOME/.local/share
XDG_CONFIG_HOME=$AKTESTHOME/.config
export XDG_DATA_HOME XDG_CONFIG_HOME

cd $AKTESTHOME

$KDEDIR is where the new software version is installed to, here that is "/dvl/kde/trunk/install".

Creating a new test scenario works like this:

- Open a Konsole window with several tabs, I recommend at least four

- Create a test base directory, e.g.

mkdir /tmp/akonaditest

- Copy the environment file

cp testenv /tmp/akonaditest

- Switch the Konsole window into input forwarding mode, e.g.

CTRL + SHIFT + ,

- Change (all tabs simultaniously due to the previous step) into the the test directory

cd /tmp/akonaditest

- Source the environment

source testenv

- Now, in one the other tabs (so that it is the only one doing it)

dbus-launch > dbusenv

- Then again in the master tab (the one which forwards its input)

source dbusenv

- Switch of input forwarding, e.g.

CTRL + SHIFT + -

- Copy data and config (if you have KMail running in the test account, shut it down before doing that )

cp -ra $HOME/.kde .

Starting the runtime processes

Tab 1: run kdeinit4 and wait until it has stop writing insane amounts to log output :)
Tab 2: run nepomukserver
Tab 3: start Akonadi, run

akonadictl start

Running the migrator

Tab 4:

kmail-migrator --interactive

If you are asking yourself why I am using all these different tabs, I am doing that to have the log output of these processes separated from each other.
E.g. Tab 3 will contain only output from Akonadi server and Akonadi Resources, so I can look there if the migrator reported an Akonadi error.

Re-running a test

If you have Disconnnected IMAP accounts, you can rerun the test in the same environment if you choose to "Keep Local Copies" when the migrator asks how to deal with the cached emails after import.

If you chose the delete option, you'll have to copy the cache again:

cd .kde/share/apps/kmail
rm -r dimap
cp -ra $HOME/.kde/share/apps/kmail/dimap .

To rerun the full test, remove all newly created resource through "akonadiconsole", as well as the following config files: kmail2rc, kmail-migratorrc

To rerun only part of the test, remove the respective resources, kmail2rc and edit kmail-migratorrc, removing the respective Resource section.

Since you are working with a copy, you can modify kmailrc and remove all accounts you don't want to be part of the test run.

Disconnected IMAP import options

Import of Disconnected IMAP cache is following three options that can be set in kmail-migratorrc.
Part of the installation is a base config containing this:

[Disconnected IMAP]
ImportNewMessages=true
ImportCachedMessages=true
RemoveDeletedMessages=false

Without any change to this file ($KDEDIR/share/config/kmail-migratorrc) or an override in the local file, any Disconnected IMAP account will get all new (locally added but not synchronized yet) and normal cache messages imported.

If you enable the third one and have unsynchronized local deletes, the import process will attempt to delete these messages from the IMAP server. This list of locally deleted message identifiers is stored in "kmailrc", in the Account section of the respective IMAP account.
This can obviously only be done once, the messages are then gone from the server.

If you disable all three, you will get almost the same setup as for normal IMAP accounts, i.e. migrator just listing the folders, but it will also set cache policies on each folder to make them keep all emails that they download.

Misc

You can also run KMail (the new one) instead of the migrator, it will run the migrator for you and then continue its own startup.

You can check the finished setup either with KMail or just Akonadiconsole (second tab has a folder tree).
I personally always keep Akonadiconsole running in an additional tab.

You can use the current KMail to add/remove accounts, locally add/delete messages from Disconnected IMAP folders (if you want to test that), etc.

In case of questions, I usually hang out on our IRC channel #akonadi, freenode network, or send an email to me or our mailinglist (kde-pim@kde.org).