NOV
13
2011

Akonadi misconception #1: where is my data?

I regularly see the same misconception and fear popping up on the mailing lists, bug reports and IRC: if the Akonadi database gets corrupted, I will lose my data.

To make it clear from the beginning: the Akonadi database is NOT your MAIN data storage.
Even if it gets destroyed, removed, your data is still safe.

So what is the database?

1) It is an interface: the Akonadi server and the underlying database is a common interface to your (PIM-alike) data for different applications, so those applications do not have to deal with the data files directly.

2) But I see my email headers and even the body in the database and in $HOME/.local/share/akonadi/file_db_data. Why? Because the database is also a cache towards your data. Common, frequently accessed parts (like e-mail headers) are stored in the database. These are usually stored permanently and kept in sync with your original data source (IMAP folders, mails on the local disc).
Parts requested infrequently are either stored in the database or in the above folder. The actual place is decided upon the size of the data. These parts are from time to time cleaned up from the cache, if they were not used for a certain period of time. Technically it is possible to configure when the cache is cleaned or if it is cleaned at all, but the regular user should not have to deal with it.

3) Is there anything I might lose by deleting the database? Yes, there is, and that is the metadata added to your data. That can be anything extra information that cannot be stored in the format of your data, like Nepomuk tags or other custom information. In case of emails, you might think that read/forwarded/etc. can be lost. Luckily this is not the case (since KDE 4.7.2), as the email storage formats can store these informations natively.

The above explains why you will not lose any critical information by losing your akonadi database.

Good, but where is my data if not in the database? This depends on what kind of data we are talking about.

1) E-mail: in case of IMAP (online or disconnected) your data is on the IMAP server. With disconnected IMAP there are windows when your local cache contains data that is not yet syncronized to the server, deleting the local cache in this case indeed will make you lose the unsynchronized files. This is not Akonadi specific though, this is true for any disconnected IMAP client.
In case of POP3, the mails is stored immediately after download in a local maildir folder. The actual place of the folder depends on your configuration, it can be just as $HOME/Mail, as $HOME/kde./share/apps/kmail/ or $HOME/.local/share/.local-mail (for new installations).

2) Calendars and contact information: they can be either on a server (Kolab server, LDAP server) and only cached in Akonadi as explained, or they can be in local vcard or .ics file. The actual location of these files again depends on your setup. The new standard place for them is $HOME/.local/share/contacts.

Still there were reports of data losing, why? Unfortunately programmers are not perfect and introduce bugs in the codebase. One of the most severe bugs caused real data losing when copying mails from one folder to another. This is fixed with KDE 4.7.2+ and any recent Akonadi server. There are other bugs around, but none will cause losing your original data files.

Finally, what will happen if the database gets corrupted? Of course, it needs to be recreated. You can try by shutting down akonadi server (akonadictl stop), removing the $HOME/.local/share/akonadi and syncronize all your resources again (this will take some time). If the database is not recreated, you need to do a full cleanup by removing also the configuration files under $HOME/.config/akonadi.
Then you need to add back your resources (IMAP/POP3/contact files/etc) and syncrhonize them. In case of emails, you need to check your filters, they most probably try to move the mails into a wrong folder.
Yes, this is a lot of work and should be avoided as much as possible, but it should be done only in the rare case of real database corruption.

I hope this will clear some confusion about the data storage inside Akonadi.

And a few word about the database itself.
We use MySql. I don't know the original reason why it was picked up (ask Volker about that ;) ), but I know some people don't like it for whatever reason.
Most of them try to use SqLite. A friendly warning: don't. This is explained also in the akonadi wiki. All what is written there is true, and I experienced myself as well.
Unfortunately recently I learned that MySQL has some severe issues in certain cases: NFS mounted home directories and failed suspend/resume. In these cases the database gets corrupted (this is a MySQL bug!), and cannot be easily restored. I did not experience this corruption myself, but was reported from time to time.
What remains is trying another database server, namely PostgreSQL. Akonadi supports it, it is possible to change the database backend in the akonaditray or in the akonadiconsole application. Changing backends means importing all your data again to the database, like you'd start from scratch. PostgreSQL support is not that well tested, so expect some rough edges, but we would like to see people try it out and reporting their experience with it.
It is my personal opinion only, but if PostgreSQL proves to work fine, we might switch to that as the default backend, given the problems with MySQL.

What about other database backends? There were plans to use Virtuoso, to reduce the number of database severs needed for a KDE desktop, but the work was not completed. See the techbase aritcle.

UPDATE: Christophe Giboudeaux raised a point about PostgreSQL, that their database format changes between major releases in an incompatible way and there is no automated, easy way for upgrade (you need to dump it with the old version and import with the new one). Sad, that there is no perfect solution.

Comments

I for one would love to see virtuoso used as a backend. If more apps manage to move to stringi/nepomuk for indexing then we could at least theoretically have just one database for "everything".


By teho at Sun, 11/13/2011 - 17:51

Postgres has an in-place upgrade tool since a couple of versions ago. It is not automatic (thankfully: it's offline, one-way, potentially ressource-intensive, probably not what you want yet, etc) but could be automated by akonadi itself (since akonadi knows about the usage pattern, it can make those upgrade decisions on behalf of the user). Taking a db dump (as part of the regular backup process) wouldn't hurt anyway.

I'll try akonadi with postgres again. I had tried a long time ago, but the many pim bugs at the time prompted me to stick with the recomended option. I trust pg much more than mysql and have it runing on my machines for many projects, whereas mysql is only there for akonadi. So switching to postgres would improve my resource usage, maintenance burden, and peace of mind. YMMV.


By moltonel at Mon, 11/14/2011 - 13:27

Thanks for the explanation.

Regarding the email storage, I am still puzzled, though... I would also expect to find my emails somewhere on the local drive but I can only find them in the database.

I fetch my emails using POP and already try to search in the usual locations ($HOME/Mail, $HOME/.local/share/.local-mail, etc...). Plus there are a few Gigabyes so it should not be that easy for them to hide :)

Do you have an idea? Seems to be a configuration issue but I have not found any solution so far.


By Benoit Walter at Mon, 11/14/2011 - 16:18

Find the folder you are looking for in KMail. The toplevel entry of the folder tree should be something like Local Folders (unless you renamed it). Find the Local Folders account in the KMail account configuration. Click on Modify and see where it points to.
In case you have multiple Local Folders or similar accounts, look at the POP3 accounts configuration, see in the Advanced tab where does it deliver the mail and find the account that corressponds to that folder.


By amantia at Mon, 11/14/2011 - 19:07

Thanks a lot for your answer!

My Emails are in "KMail Folders" (which exists besides "Local Folders") and I cannot access to its configuration (I did the migration several months ago).
I can maybe try to copy all my emails from "KMail Folders" into "Local Folders" and try somehow to remove "KMail Folders".

Do you have any idea?


By Benoit Walter at Mon, 11/14/2011 - 20:49

Strange you couldn't access the configuration. Double click on Modify should get you there. KMail Folders means it is a migrated account, so the mails should be just where your mails were prior to migration (migration doesn't copy the mails themselves). Of course you can remove the account and add a new one back, but for that you'd need to know where the mails are. Try a global search in your $HOME (including the hidden subdirs).
You can also post on kdepim-users@kde.org, probably easier to discuss there then through a blog.


By amantia at Tue, 11/15/2011 - 07:11