Three days in Berlin (or improving the pim user experience)

Last weekend I've met with some of our old timer KDEPIM developers and some of the newer ones who are interested in KDEPIM or related technologies in the KDAB offices in Berlin. Being a KDAB employee for a few years already (wow, time is passing quickly), the place was familiar, the people looked familiar. The foosball table was slightly familiar, although some players changed their dresses in order to spread confusion inside the visitor's heads. That is the only reason we've lost against locals with 5-0, 5-0. Suprisingly enough, the table wasn't used that much. Why? Because people were busy either working and talking. What they talked about I don't know too much, as I focused on some issues I planned to fix, but others can testify that Volker did not sit too much in front of his computers, but was dragged from one place to another for various discussions. Most of us started on Friday around noon. I won't tell too much about what others did, that is mostly their problem. Especially if they were on the other side of the room and I didn't saw their screens, I can't tell if they worked at all or not. They looked like they did though. ;) I can share some things about what some sitting nearby did. See later. As recently I became the de-facto maintainer and bugfixer for mail filters, I worked mostly on them. Unfortunately the first attempt to fix a bug failed: a bug that bothers me, although it is not the most reported one. It is about mails not being filtered for a while after a resume from hibernation. It is hard to reproduce, and although I run into it, I couldn't reproduce reliably enough and in a way I can debug it. After fighting with it and realising it won't work, I gave up. See you next time. Meantime we had some excellent food in an Indian restaurant, then when finally everybody arrived (almost everybody to be honest, plus even some more from KDAB who didn't sign up, but show up there), we were ready to start with presenting the kdepim and Akonadi architecture. Old time pimsters Volker Krause, Kevin Krammer and Tobias Koenig helped me with it, and sometimes saved me, as my knowledge in some area proved to be superficial. We ended up with a pretty impressive drawing on the whiteboard: Hopefully for those being on site it was understandable.

Time passed, so we went out for dinner to a Greek restaurant at the corner. Nice place, third time being there, good food (seems Berlin has only good eating places). We even saw how Germany won over Ireland. Too bad Steveire (Stephen Kelly) was not there... We continued with some more talks and hacking, then everybody went to their sleeping places. Some left earlier, some later waiting for remote developers to show up and discuss the patches (he didn't show up).

Day 2 Next day started around 9 o'clock for some. Later for others. I continued with the filter debugging and fixing. I looked at the bugreports and tried to come up with some bugs that are both would help the users and myself. I'm selfish, I want to fix the bugs I run into. ;) One problem that bothered me for a long time is that email content from online IMAP is downloaded even if no filter for IMAP requires it (but a filter for e.g POP3 needs the full content). Now this is fixed and those who use online imap could notice a great speedup in mail syncing (depending on their filter setup). Another often reported issue was duplicating the mails, especially after spam filtering. You've ended up with a mail in the spam folder and a mail in the original place. This could be combined with a conflict dialog shown up as well for that mail. This problem actually revealed a not-well handled case down inside the Akonadi stack that was created by a reordering of filter commands in the filtering code. For those who want technical details: if there was a move action (like "move mail to spam folder") followed by a modify action (like "mark as read"), the filter reordered this to "mark as read" followed by a move. Unfortunately this exposed two problems, (mostly, but not only) IMAP specific: on IMAP you can't modify a mail, instead you create a new one and delete the old. What happened here is that a filter changed the mail (run through bogofilter/spamassasin), then the "mark as read" action was executed, which uploaded the changes, including the changed mail to the IMAP server. This caused a new ID for the item on the IMAP server. The mail with the old ID was deleted. Then the filter agent performed a move, but he had only the old ID, before the modifications. The move - if performed between two different resources, like IMAP and local maildir - is technically a copy followed by a delete. The copy worked fine, but the delete didn't. It tried to delete the old mail again (which was gone), not the modified version. That's how you ended up with a duplicate. The current solution is to reorder the filtering pipeline, so the move is performed before the flag modifications. The extra benefit is that the filtered message is not uploaded again just to be deleted the next second. Speedup, less network traffic, better user experience. This doesn't fix the main problem with Akonadi, but it is an acceptable workaround. The reordering is so far only in the master branch (KDE 4.10), but the relevant code will be backported to the 4.9 branch, so it will be in 4.9.3. And speaking of conflicts, that was the next hot topic, and one of the often reported bugs. It was mostly caused by conflicting flag changes (read/spam/important/etc), either caused by KMail itself plus the filtering or just KMail. After some discussion we agreed, that reporting conflicts on flag changes doesn't make sense, so we should not bother the user. It is not data loss, and in worst case some flag is reset. In normal case nothing wrong happens, as Akonadi is able to merge two changes in the flags (or so claims Volker). I have to admit that this fix was not completed at the sprint, I finished it today, as I noticed more code in KDEPIM that didn't disable conflict check on flag changes. While doing the changes, I did quite some refactoring, cleaning up the code as much as I could do for the parts I touched. The code was originally deep inside KMail (as most of other code), got extracted from it for 4.8.0, and now we are at the stage that we can make the extracted code more cleaner, we can remove some things that doesn't make sense anymore now that the filtering is not inside KMail. While reading the bug reports, I also run into one indicating that mails arriving to an MBOX account (like /var/spool/mail/username) are not filtered automatically. Sounds like an easy bug to fix and so was it. Now they are filtered. Finally another annoying issue was fixed: there was no indication that filtering is ongoing. Now when this happens, you can see in the KMail's progress bar.
The net result of the above: faster filtering, less annoying and useless error dialogs for the users. And according to bugzilla, 31 bugs less (some were duplicates though).

Nepomuk...one of the things that, well, so far I was always recommending for users to turn off. Slight detour here: what is Nepomuk and its relation to KMail? Nepomuk helps you to find data. It indexes all kind of data and with some queries you can find e.g every file where my name is mentioned, all email addresses from any file, etc. There is a process that goes through the akonadi data (emails, calendars, etc) and "feeds" to it for Nepomuk to be indexed. Then there are queries and code in KMail accessing Nepomuk: getting email addresses for composer's autocompletion, searching inside mail bodies, tagging your emails with custom tags, etc.
Unfortunately there are problems around it. One is that some queries ae processed quite slowly, that in turn slows down KMail, leading to poor user experience. E.g switching between mails, sending mails, etc. is slow. Or startup is slow or even blocked. I tried to fix the last part as I just run into it, but as this was my very first Nepomuk related patch, I'm not sure 100% succeeded. We found a problem with the Nepomuk API itself, and I informed Vishesh, the Nepomuk developer about it (even if I used non-blocking calls against Nepomuk, one call is blocking without question). He suggested something I might try later, although I'd be more happier if somebody with real Nepomuk knowledge could give a review of the pim Nepomuk usage.
Then there is the Akonadi feeder, that gives the data to Nepomuk. Something is not perfect there and the indexing causes a serious slowdown, where either itself or Nepomuk (actually its storage backend, virtuoso) starts to use the CPU *a lot*. This is the main reason I recommend to users to disable Nepomuk <em>so far</em>. We had Christian Mollekopf the author of the feeder in the sprint, and he worked on some optimizations. Hopefully this will improve the situation. Meantime we (and I) tried to convince Vishesh to use KMail, so he can see himself the problems our users face. 

As a break we had another presentation, this time about KMail itself, what components make up KMail, how they are distributed. As far as I saw, this was less interesting to the audience, they rather looked at their computers and hacked on something. Luckily our KMail maintainer, Laurent Montel, is super active, but I wouldn't mind more contributors. Too bad he wasn't at the sprint.

We had a lunch at a nearby place, nothing extra, but the food was (again) good. 

What did other do? Let's see what I can remember... Sune dreamed about crypto stuff and composite jobs. He worked on making some cryptographic code asynchronous and started to get faimiliar with kdepim code. I'd not say he picked up the easiest job.
Volker run around all the time, discussed various things like "spanish sync" with Alex Fiestas (<a href="http://community.kde.org/KDE_PIM/Meetings/PIM_October_2012_meeting#IndexingHeavy_Task_Scheduling_.28aka_.22Spanish_Scheduling.22.29">see here</a>), database backends with Martin, change recorder with David Faure (who remotely joined the meeting and got lost inside the change recorder code ... he has the solution now in his head, so be patient, we will end up with a better implementation for it that again speeds up Akonadi), job pipelines with me, and who knows what with others, as for a long time he just disappeared with a bunch of developers. They actually ended up in the lobby discussing "stuff". Milian Wolf, who is not (or not yet?) a KDEPIM developer, but mainly a KDevelop one, joined us and used massif to track down some ugly memory usage in KMail. And he did a good job in it. Previously KMail used more and more memory as you navigated between large folders (Alex mentioned some 2GB for him), while now it levels up at one point and doesn't increase. He might blog himself about, as he has also some nice graphs.
Then there were two guys from KDAB, who are old time pim developers (quiz: find their names in this blog), but they cooked something else, not related to KDEPIM, not related concretely to KDE, but to a lower level: to Qt. It is an amazingly cool stuff, but I don't want to give more details. Expect it to be presented a the <a href="http://qtconference.kdab.com/">Qt Developer Days</a>, either in a talk or at the KDAB booth. Don't worry, I'm pretty sure it will be freely available what they did and KDE can enjoy it in the future.
 I'm sorry that I don't remember what the rest did. In general I  know that Martin Klapetek worked on the social network resources, Mark Gaiser, who recently started to work on KDEPIM stuff eagerly listened to our presentations and worked on a QML calendar application, Alex kept reporting bugs and discussed improvements with Volker, while John Layt, the "timezone KDE guy" worked on plasma calendar related issues.
 As a KDE person Chani also joined us for a while, we quickly nominated her as the QML and Plasma person, so all questions related to them were immediately redirected to Chani. Jos Poortvilet was also supposed to join for some talks, but he could show up only on Sunday for personal reasons.
 At the end of the day everybody was so busy, so instead of going out for eating, we just ordered some pizza. And most of us stood in the office well past midnight.
 
<b>Day 3</b>
 Well, the above partially happened on Sunday. It was a shorter day for me, due to the late night standup before, and that I had to go to the airport after lunch. A lunch that wasn't exactly as planned. We went out for a Doner Kebab place that is supposed to be the best one in Berlin. It is just a small kiosk on the sidewalk in a street, but man, there was a big queue for it. On a Sunday! Locals says it is worse on weekdays. Even after almost an hour, we still had 10 persons in front of us, my departure time was approaching, so I gave up and instead bought some (quite good) chinese fast-food from a nearby place, then rushed to the airport.
 A long journey awaited me with a 3 hour stopover in Munich, but luckily I had a power supply there and even some network (they offer 30 minutes/day/phone number), so I could continue on the work I started at the sprint. After flying and driving another 2.5 hours, I arrived late (or rather early) morning next day back home, and after sleept until around 10. Then I started to work again for KDAB, a work that is just as enjoyable as working for KDE. After all, the two communities has a serious overlap. ;)
That's from the sprint. If you'll be at the Qt Developer Days Europe, we might meet there. I'm looking forward for a good conference.