Akonadi misconception #1: where is my data?

Sunday, 13 November 2011  |  Amantia

I regularly see the same misconception and fear popping up on the mailing lists, bug reports and IRC: if the Akonadi database gets corrupted, I will lose my data.

To make it clear from the beginning: the Akonadi database is NOT your MAIN data storage. Even if it gets destroyed, removed, your data is still safe.

So what is the database?

  1. It is an interface: the Akonadi server and the underlying database is a common interface to your (PIM-alike) data for different applications, so those applications do not have to deal with the data files directly.

  2. But I see my email headers and even the body in the database and in $HOME/.local/share/akonadi/file_db_data. Why? Because the database is also a cache towards your data. Common, frequently accessed parts (like e-mail headers) are stored in the database. These are usually stored permanently and kept in sync with your original data source (IMAP folders, mails on the local disc). Parts requested infrequently are either stored in the database or in the above folder. The actual place is decided upon the size of the data. These parts are from time to time cleaned up from the cache, if they were not used for a certain period of time. Technically it is possible to configure when the cache is cleaned or if it is cleaned at all, but the regular user should not have to deal with it.

  3. Is there anything I might lose by deleting the database? Yes, there is, and that is the metadata added to your data. That can be anything extra information that cannot be stored in the format of your data, like Nepomuk tags or other custom information. In case of emails, you might think that read/forwarded/etc. can be lost. Luckily this is not the case (since KDE 4.7.2), as the email storage formats can store these informations natively.

The above explains why you will not lose any critical information by losing your akonadi database.

Good, but where is my data if not in the database? This depends on what kind of data we are talking about.

  1. E-mail: in case of IMAP (online or disconnected) your data is on the IMAP server. With disconnected IMAP there are windows when your local cache contains data that is not yet syncronized to the server, deleting the local cache in this case indeed will make you lose the unsynchronized files. This is not Akonadi specific though, this is true for any disconnected IMAP client. In case of POP3, the mails is stored immediately after download in a local maildir folder. The actual place of the folder depends on your configuration, it can be just as $HOME/Mail, as $HOME/kde./share/apps/kmail/ or $HOME/.local/share/.local-mail (for new installations).

  2. Calendars and contact information: they can be either on a server (Kolab server, LDAP server) and only cached in Akonadi as explained, or they can be in local vcard or .ics file. The actual location of these files again depends on your setup. The new standard place for them is $HOME/.local/share/contacts.

Still there were reports of data losing, why? Unfortunately programmers are not perfect and introduce bugs in the codebase. One of the most severe bugs caused real data losing when copying mails from one folder to another. This is fixed with KDE 4.7.2+ and any recent Akonadi server. There are other bugs around, but none will cause losing your original data files.

Finally, what will happen if the database gets corrupted? Of course, it needs to be recreated. You can try by shutting down akonadi server (akonadictl stop), removing the $HOME/.local/share/akonadi and syncronize all your resources again (this will take some time). If the database is not recreated, you need to do a full cleanup by removing also the configuration files under $HOME/.config/akonadi. Then you need to add back your resources (IMAP/POP3/contact files/etc) and syncrhonize them. In case of emails, you need to check your filters, they most probably try to move the mails into a wrong folder. Yes, this is a lot of work and should be avoided as much as possible, but it should be done only in the rare case of real database corruption.

I hope this will clear some confusion about the data storage inside Akonadi.

And a few word about the database itself. We use MySql. I don't know the original reason why it was picked up (ask Volker about that ;) ), but I know some people don't like it for whatever reason. Most of them try to use SqLite. A friendly warning: don't. This is explained also in the akonadi wiki. All what is written there is true, and I experienced myself as well. Unfortunately recently I learned that MySQL has some severe issues in certain cases: NFS mounted home directories and failed suspend/resume. In these cases the database gets corrupted (this is a MySQL bug!), and cannot be easily restored. I did not experience this corruption myself, but was reported from time to time. What remains is trying another database server, namely PostgreSQL. Akonadi supports it, it is possible to change the database backend in the akonaditray or in the akonadiconsole application. Changing backends means importing all your data again to the database, like you'd start from scratch. PostgreSQL support is not that well tested, so expect some rough edges, but we would like to see people try it out and reporting their experience with it. It is my personal opinion only, but if PostgreSQL proves to work fine, we might switch to that as the default backend, given the problems with MySQL.

What about other database backends? There were plans to use Virtuoso, to reduce the number of database severs needed for a KDE desktop, but the work was not completed. See the techbase aritcle.

UPDATE: Christophe Giboudeaux raised a point about PostgreSQL, that their database format changes between major releases in an incompatible way and there is no automated, easy way for upgrade (you need to dump it with the old version and import with the new one). Sad, that there is no perfect solution.