Akonadi-like access to data in files

Some of Akonadi's resource agents (usually just called resources) work on local files, some on files containing more than one data object, some on directories containing one data object per file.

For example the "VCard Resource" has one vcf file to work with which in turn contains any number of vcards, i.e. contacts.

Those single file storage containers have a couple of things in common so of course we want to share as much of code between their respective resources as possible.

Unfortunately the data inside the different files is formatted quite differently, so the parsing and creation of C++ objects is rather type specific. Another difference which can make common code difficult is the expected size of the files and each of its entries, i.e. a VCard file will most likely just contain a couple of dozen contacts, maybe in the lower hundret range while an MBox file can easily reach several thousands of entries and each of its entries (emails) is usually larger than one of the contacts.

So code that works well for contacts or calendars does not necessarily work well for messages and vice versa.

While thinking about possible ways to improve our situation I had the idea of using the same level of abstraction we are already using in Akonadi, i.e. generic "Items" which hold the type specific data as their "Payload".

In other words, if we had an "Akonadi Item File" we would get Akonadi items out if it and not have to care about whether those items transport contacts or emails.

This would still leave us with the problem of different file and item sizes. Again I decided to use concepts already proven useful in Akonadi: payload parts and gettings things on demand.

Payload parts refer to a concept where allow for a payload (remember that could be a contact or an email, etc) be split into parts that make sense for the respective data type, e.g. splitting an email into "Headers", "Body" and so on.

Getting things on demand refers to get whatever parts of an item you are interested in at any time, e.g. only getting headers when listing a mail folder and getting the rest when displaying a selected one.

In Akonadi we do this through jobs and telling those jobs what we expect them to return to us.

The respective Akonadi code would look similar to this:

const Collection collection = someModel->selectedCollection();

ItemFetchJob *job = new ItemFetchJob();
job->fetchScope().fetchPayloadPart( MessagePart::Header );

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( collectionListed( KJob* ) );

with collectionListed() doing something like that

ItemFetchJob *job = dynamic_cast( job );

const Item::List items = job->items();
// fetch the first item as a whole

job = new ItemFetchJob( items[ 0 ] );
job->fetchScope().fetchFullPayload();

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( itemFetched( KJob* ) );

In order to do something similar with files I came up with a concept I called the Akonadi Filestore. The main interface looks like this (omtting some of the methods not important for our example above)

class StoreInterface
{
  public:
    virtual Collection topLevelCollection() const = 0;

    virtual ItemFetchJob *fetchItems( const Collection &collection, const ItemFetchScope *fetchScope = 0 ) const = 0;

    virtual ItemFetchJob *fetchItem( const Item &item, const ItemFetchScope *fetchScope = 0 ) const = 0;
};

Assuming we have an implementation that operates on a file which contains messages, e.g. an MBox store, we can implement the example above quite similar to the respective Akonadi code. (note: all jobs are named like the ones from Akonadi but live in their own namespace. Collections, items, fetchscope are directly the classes from Akonadi)

// lets assume mStore is of type StoreInterface* and has been
// initialized properly with an MBox store implementation

// lets list the mails in the stores top level collection
ItemFetchJob *job = mStore->fetchItems( mStore->topLevelCollection() );
job->fetchScope()->fetchPayloadPart( Message::Header );

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( collectionListed( KJob* ) );

with collectionListed() doing again something like that

ItemFetchJob *job = dynamic_cast( job );

const Item::List items = job->items();
// fetch the first item as a whole

job = mStore->fetchItem( items[ 0 ] );
job->fetchScope().fetchFullPayload();

connect( job, SIGNAL( result( KJob* ) ), this, SLOT( itemFetched( KJob* ) );

As you can see we get on demand, payload part fetching with the only type specific thing being the name of the header payload part. In Akonadi resource this difference would removed because Akonadi forwards the payload part from the client requesting it, so the resource does not have to know any of those identifiers itself.

This is already quite nice but it gets better :)

• We no longer require the items to be of the same type, say a ZIP file containing contacts, calendars and emails (or like an Outlook PST file)

• We can include metadata that is not part of the common payload formats, e.g. flags of email messages if the store's format supports that (or if it is actually working with more than one file, e.g. KMail is saving this things into index files it keeps alongside the actual mail files).

Neither design nor implementation are fully production ready, I consider it a testbed for the concepts mentioned above. You can find it in PIM Playground (to get correct paths either checkout playground/pim/akonadi or create a directory "akonadi" into which you checkout "filestore")