10-100 times faster thumbnail loading

    blackie's picture
    2010
    31
    Jul

    Fast - tell me how do you implement a thumbnail loader - well the loading
    and storing of the thumbnails that is?

    Classically you would likely say something that included individual JPG
    files for each thumbnails, and perhaps also something that included
    multithreading.

    Well that is at least how many application does it, and Linux even have a
    standard for it, so applications can share the thumbnails.

    I've implemented a much faster way to do it. This is something like 10-100
    times faster, and I have code to prove it :-)

    The Short Story
    Store all the thumbnails in one big file, which is way
    faster to navigate for the hard disk, than seeking around just to load a
    few kilo bytes here and there.

    The slightly longer story

    First of all, I didn't come up with this idea, so kudos to whoever
    introduced it to me many years ago - maybe someone planted the idea in my mind
    while I was sleeping :-)

    I'm no where close to being a hardware kind of guy, so I wont bother you
    with the details or statistics, but the fact is that the hard disk waste
    valuable milliseconds when it has to seek to a new location to load
    data. Just try to create an application which loads dummy data from a bunch
    of different files, and measure how much your application loads a
    second. Now compare that to an application that loads dummy data from one
    big file instead.

    When you want to see the thumbnails in say a photo application like
    KPhotoAlbum, then you are very likely to load them in a specific sequence,
    so if you could get them located on disk in that sequence then the disk
    would not need to skip around when loading.

    I, at least haven't found a way to tell my Linux how to store things in a
    specific order on disk, so the alternative is to store all the thumbnails
    in one big file. That at least from my test seems to be way less
    fragmented.

    Try it out
    OK, don't just trust what I say, try it. Download the demo from
    http://www.kphotoalbum.org/data/ThumbnailLoader.tgz

    compile it using:
    qmake; make

    run it like this:
    ThumbnailLoader new /path/to/the/rootdirectory/of/all/you/images

    you only need to specify the path the first time you run the demo.

    To compare this to the old way of loading thumbnails, you may run the
    demo like:
    ThumbnailLoader old.

    To avoid being cheated by your disk cache, make sure you load in a lot of
    images, and that you scroll a lot around.

    Cheers
    Jesper.

    Comments

    Comment viewing options

    Select your preferred way to display the comments and click "Save settings" to activate your changes.
    sebas's picture

    KSharedDataCache and dropping disk caches

    Hey Jesper,

    On Linux, you can emulate the situation with cold disk caches by running the following command: echo 1 > /proc/sys/vm/drop_caches Michael Pyne has recently worked on a cache implementation that does exactly what you need in a more generic way, the KSharedDataCache. He has blogged about his experience implementing it:

    http://www.purinchu.net/wp/2010/04/25/implementing-a-shared-cache-part-1/ http://www.purinchu.net/wp/2010/04/30/implementing-a-shared-cache-part-2/ http://www.purinchu.net/wp/2010/05/08/implementing-a-shared-cache-part-3/ http://www.purinchu.net/wp/2010/05/20/implementing-a-shared-cache-part-4/

    It's part of the 4.5 Platform.

    Cheers, -- sebas

    Comment viewing options

    Select your preferred way to display the comments and click "Save settings" to activate your changes.