Mimetypes are out of ksycoca

During the Akademy kde-on-mobile discussion, Kévin Ottens suggested that mimetypes be splitted out from the main sycoca database, in order to reduce the overall time kbuildsycoca takes every time it runs, and to make things more modular. Last monday I had a look and decided to go one step further and not use ksycoca at all for mimetypes. This makes the KMimeType subsystem really independent (usable without having run kbuildsycoca4 before, so no dependency on kdeinit+kded etc). Took me the whole week, but there we are.

Given that shared-mime-info delivers pre-processed output already, we "just" have to load what we need, on demand, in-process. I was afraid this would be slower and take more memory as a result, but see results below.

The KMimeType subsystem now loads directly from disk the information it needs. The first time you use findByPath("foo.jpeg") it loads the "globs" file generated by shared-mime-info once and for all into memory, and then uses that for fast matching. Same thing for the list of aliases (one big file, loaded once), and for "subclasses". The more expensive information is the XML file for each mimetype, but this is only needed when asking for the icon, the comment, or the list of extensions for a given mimetype. This used to be all parsed and cached into ksycoca, now it's done on demand by KMimeType.

The only mimetype-related thing that is still in ksycoca is one tiny entry per mimetype, with a pointer to the list of services (applications) which support this mimetype, for KMimeTypeTrader.

Here are the performance results:

• "kbuildsycoca4 --noincremental" (with warm caches) went from 2.8s to 2.1s
• Mimetype inheritance tests (KMimeType::is() on a list of 40 mimetypes, like PreviewJob does), went from 791,404 instr. loads to 568,345 instr. loads (QBENCHMARK and "./mytest -callgrind" rock). So, loading the subclasses file directly from disk is actually faster than loading a preprocessed form from ksycoca? Well, it's also that constructing a KMimeType is way faster now, since the mimetype details are not loaded in this case.
• Mimetype glob matching (reconizing that "foo.jpg" is a jpeg file) is also much faster: from 100k-120k instructions to 75k instructions. (This doesn't include the one-time-per-process parsing of the globs files).
• Something had to give, though. The worst case for loading mimetype details is a tree that shows all mimetypes, such as the KMimeTypeChooser widget. Instead of loading the icon and comments from ksycoca it now loads them from individual xml files, so even though I ported that code from QDomDocument to QXmlStreamReader (which made it 29% faster), the overall time for loading all mimetype details is much longer: 1182k instead of 113k instructions. However from the user's point of view, this means 460ms instead of 60ms when opening that tree of mimetypes, which actually takes quite longer because of the icon loading anyway. I think this is acceptable. The very common use case (mimetype lookup and matching) was optimized at the expense of the rare case (showing all mimetypes).

I find it interesting how not using a cache that was invented to improve performance, actually improves performance :-)

Interestingly, I thought it would be worse on memory usage too, but it's not: ./kmimetypechoosertest_gui in kdelibs-4.5: VmSize: 282060 kB, VmData: 14524 kB ./kmimetypechoosertest_gui in kdelibs-trunk: VmSize: 215976 kB, VmData: 11976 kB I'm not entirely sure why, in fact, apart from the fact that ksycoca (kmimetypefactory, more precisely) was loading some dicts unconditionally, like the alias list, which isn't loaded anymore in this particular use case. It can't be the size of KMimeType objects, they are not kept in memory in either case. So I'm still a bit surprised, but well, the good way, so not really worth spending time with massif ;)

The "visual" result of all this: kdelibs-4.5$ kmimetypefinder foo.cpp KSycocaPrivate::openDatabase: Trying to open ksycoca from "/var/tmp/dfaure-kde4/kdecache-dfaure/ksycoca4" text/x-c++src

kdelibs-trunk$ kmimetypefinder foo.cpp text/x-c++src

(This particular use case of starting one process for one mimetype lookup has of course worse performance now, but it's time to stop looking at performance when it's about 0.01s vs 0.03s for a one-time operation...)

Anyway the main point of these 2 commands was: no ksycoca needed. Now I'm not trying to send the message that ksycoca is bad. We definitely need it for things like "find the apps that support image/jpeg" or "find the apps that contain FooBar in their name or keywords". The number of app desktop files is so big, that we definitely need some kind of database index for these, we don't want to load all of them into memory in each process. But for mimetypes, thanks to shared-mime-info producing usable output, not using ksycoca actually makes sense, the information is already in a usable form even for fast lookups (mimetype by name, mimetype for a certain file, by name and/or content).

Writing all this makes me realize that the caching of kioslave .protocolinfo files into ksycoca is more like mimetypes than applications and even simpler: fewer items, and always used for looking up the details for a single protocol by name. So, logically I should extract it from ksycoca too. Well, after the vacations :)