JUL
31
2010

10-100 times faster thumbnail loading

Fast - tell me how do you implement a thumbnail loader - well the loading
and storing of the thumbnails that is?

Classically you would likely say something that included individual JPG
files for each thumbnails, and perhaps also something that included
multithreading.

Well that is at least how many application does it, and Linux even have a
standard for it, so applications can share the thumbnails.

I've implemented a much faster way to do it. This is something like 10-100
times faster, and I have code to prove it :-)

The Short Story
Store all the thumbnails in one big file, which is way
faster to navigate for the hard disk, than seeking around just to load a
few kilo bytes here and there.

The slightly longer story

First of all, I didn't come up with this idea, so kudos to whoever
introduced it to me many years ago - maybe someone planted the idea in my mind
while I was sleeping :-)

I'm no where close to being a hardware kind of guy, so I wont bother you
with the details or statistics, but the fact is that the hard disk waste
valuable milliseconds when it has to seek to a new location to load
data. Just try to create an application which loads dummy data from a bunch
of different files, and measure how much your application loads a
second. Now compare that to an application that loads dummy data from one
big file instead.

When you want to see the thumbnails in say a photo application like
KPhotoAlbum, then you are very likely to load them in a specific sequence,
so if you could get them located on disk in that sequence then the disk
would not need to skip around when loading.

I, at least haven't found a way to tell my Linux how to store things in a
specific order on disk, so the alternative is to store all the thumbnails
in one big file. That at least from my test seems to be way less
fragmented.

Try it out
OK, don't just trust what I say, try it. Download the demo from
http://www.kphotoalbum.org/data/ThumbnailLoader.tgz

compile it using:
qmake; make

run it like this:
ThumbnailLoader new /path/to/the/rootdirectory/of/all/you/images

you only need to specify the path the first time you run the demo.

To compare this to the old way of loading thumbnails, you may run the
demo like:
ThumbnailLoader old.

To avoid being cheated by your disk cache, make sure you load in a lot of
images, and that you scroll a lot around.

Cheers
Jesper.

Comments

Hey Jesper,

On Linux, you can emulate the situation with cold disk caches by running the following command: echo 1 > /proc/sys/vm/drop_caches Michael Pyne has recently worked on a cache implementation that does exactly what you need in a more generic way, the KSharedDataCache. He has blogged about his experience implementing it:

http://www.purinchu.net/wp/2010/04/25/implementing-a-shared-cache-part-1/ http://www.purinchu.net/wp/2010/04/30/implementing-a-shared-cache-part-2/ http://www.purinchu.net/wp/2010/05/08/implementing-a-shared-cache-part-3/ http://www.purinchu.net/wp/2010/05/20/implementing-a-shared-cache-part-4/

It's part of the 4.5 Platform.

Cheers, -- sebas

By sebas at Sat, 07/31/2010 - 13:30