Marble's Secrets Part III: The Earth in a Download

Today we'll finish our first trilogy about Marble Desktop Globe:

• Part I was a Do-It-Yourself course about creating maps for Marble. We've seen how Marble manages to even display features such as aerial photos or OpenStreetMap.
• Part II showed how Marble paints the different map layers. We've also seen why Marble only needs very little disk space and memory for its default map - making it an ideal choice for solutions that involve little hardware resources (like the Asus EeePC and the OLPC).

In Part III we'll look beyond Marble's offline mode: We'll get to know how Marble fetches its data from the internet.

Caption: Left: Creating "KWorldClock 2" in 5 minutes using Marble's Qt Designer Widget Plugin. Note that this is just a proof that this would work today! - Actually Henry de Valence is working on a KWorldClock plasmoid already. Right: Marble showing off a real-time cloud map and Sun Shading with the "Earth At Night" map blended in.

By data we mostly refer to texture data. As of version 0.5.x (KDE 4.0) we only support a single texture layer. However this is about to change with the next version of Marble: David Roberts has added support for Sun Shading in Marble. If this wasn't exciting enough he also added a way to have the "Earth at Night" map blended onto the dark side of the earth. This includes a pretty nice Terminator line. On top of that he already added internal support for having real-time clouds as another layer! See the screenshot based on a current build of the Qt-version of Marble pre-0.6-SVN.

A really big mosaic of tiles ...

Actually I've met a lot of people recently who weren't aware of the fact that Marble downloads further texture data from the internet while browsing the map. On one hand the reason for this misconception is maybe the fact that we only provide very little additional online data right now (about 400 MB). On the other hand the map download is done automatically and very seamlessly in the background, so people will hardly notice. Andrew Manson created a nice YouTube video while he finished his GSoC Project "GPS support for Marble". During the first 1.5 minutes Andrew demonstrates what could be a summary of the "Marble Secrets Part II" hands-on paragraph: the video shows how you can hide and show layers using the legend tab. Watch closely at 1:35 where Andrew zooms into the "Earth at Night" Map: you can see how the map will suddenly show a lot more detail as new data has arrived on the computer in the background. Keep watching and you'll see the same happening with the "Satellite View" map.

From Part I we know that internally the current map gets provided at different sizes - each size at different resolutions. These different map "levels" are split into 675x675 pixel tiles each. Depending on the zoom level only those tiles that are actually visible will get loaded into the physical memory ("RAM") of our computer. This mechanism keeps physical memory consumption low. For each tile Marble will check first whether it's available in the physical memory's Tile Cache (yes, you were about to suggest this feature, but we got it implemented already!). If it's not available in the "Tile Cache" Marble will try to load it from the local hard disc. In case it's not there Marble will schedule the tile for download in Marble's very own HttpDownloadManager. But wait - what should we display in the meantime while there is no matching tile available? After all due to the way Marble works we can't simply "omit" the missing tile!

As a cheap but smart workaround Marble just scales up the tile at the biggest zoom level that is locally available for this particular place. It will then just cut out the area that is needed and will use this handcrafted DIY-tile as a temporary replacement until the "real" data arrives.

In the meantime the HttpDownloadManager will contact the server and will look for the tile in question (e.g. it might look for this one ) . It's only convenient that on the server all the tiles get stored in the very same directory hierarchy as on the local hard disc: {marble-data-base-url}/maps/earth/{theme}/{level}/{row}/{row}_{column}.jpg

So Marble will fetch the tile from the server and will store it locally on the hard disc - again sticking to the given tile directory hierarchy. Once it got stored locally Marble will update the temporary data of the texture tile to match the "real" data.

This mechanism works pretty well already. The HttpDownloadManager is also used to download the Wikipedia webpages that get displayed if you left-click on city labels or symbols.

If you're a real GIS person you might wonder whether it's possible to just replace this HttpDownloadManager with something "more sane" - like an enterprise database that has this data stored already. Yes, a suitable solution might be available in marble/src/lib/MarbleWidget.{cpp,h} where you'll find a method called: void MarbleWidget::setDownloadManager( HttpDownloadManager *downloadManager ); (soon to be changed to "AbstractTileProvider"). Just replace the HttpDownloadManager pointer with a pointer to your very own custom TileProvider class.

Get Hot New Maps ...

But what if you don't just want to zoom into maps that exist on your hard disc already. What if you want new maps on your computer? DXS to rescue! DXS is the successor of "Get Hot New Stuff (GHNS)". It's a web service that allows applications to download and install data from the Internet in one single click. So if you have the KDE 4 version of Marble installed then you can simply choose the "File" menu and pick "Get New Data ..." there. A dialog will appear that will allow you to download further completely new maps. Usually the maps that get downloaded via DXS only consist of a single tarball. The tarball has a size of a few Megabytes, contains a .dgml-file and might either have an installmap or pre-tiled data.

The beauty is that once the .dgml file and a few base tiles are in place Marble can start again to look for further tiles on the server.

But why doesn't Marble just download all the tiles from the server in a single download? Well, Blue Marble Next Generation is the most comprehensive free satellite map that covers the whole earth at a resolution of 500 meters per pixel. And this map takes about 350 Megabytes on the server already. With increasing resolution the size occupied on the server will grow exponentially. Even if you cover just the continents at a resolution of up to 30 meters per pixel ( Like NASA's NLT LandSat map does ) then your satellite map will occupy almost 200 Gigabytes on the server! So it's pretty obvious that for an application like Marble it's the best solution to just download the pieces that are actually needed! We plan to provide higher resolution maps on the server in the future. Given that we can't just change the maps easily once we start to provide them, this requires careful planning, so please be patient.

If you're using the Qt-only version of Marble then you're out of luck as it doesn't have the DXS feature built-in (It does have the automatic HttpDownloadManager though). You're then on your own and you need to find and install the maps manually.

Marble Junior Jobs

Today we have several exciting opportunities to let your name enter the CREDITS section in Marble. The "EASY Category" might only take you as little as an hour to complete!

To make sure that you won't waste time on a Junior Job ("JJ") while somebody else might already be about to submit a patch you can claim a JJ. Just tell us in the comments section that you have just started to work on it. Please be aware that once you have started time is running up! For reasons of fairness you should send us a preliminary patch within the following 8 hours. This will make your claim permanent for the next few days.

Of course the preliminary patch doesn't need to fully solve the problem but it should at least tackle a small part of it. If you don't send us the patch within 8 hours then the JJ will be up for grabs again. I hope this takes away the "uncertainty" whether it's worth to start or not ;-) First however you need to spend 10 minutes to get Marble from the sources and to compile it. You need to check out Marble "trunk". Once that is done you can take the plunge:

• Category EASY: The Sun Shading feature is only available via a single menu entry in the Qt-Only version of Marble. Please add such a menu entry to the KDE 4 version of Marble: just compile Marble for KDE 4. You need to apply a few changes to marble/src/marble_part.cpp and marble/src/marbleui.rc. Look at the existing menu entries in those files to see how things get implemented. As a reference for the menu name and its position have a look at marble/src/QtMainWindow.cpp.
• Category EASY: Marble doesn't save or restore the Sun Shading settings. Create a patch that solves this problem. Have a look marble_part.cpp (if you want to implement it for the KDE version of Marble) or have a look at QtMainWindow.cpp (if you want to implement it for the Qt-Only version) to see how it works.
• Category MEDIUM:: Currently Marble only tries to fetch tiles once. If the server doesn't reply Marble won't retry until Marble gets restarted. Find a solution that will reschedule the tile up to 3x in case of failure. Between the attempts Marble should wait 1 min. before the job gets rescheduled. Relevant classes for this job are marble/src/lib/HttpDownloadManager.{cpp;h} and marble/src/lib/HttpFetchFile.{cpp;h}
• Category MEDIUM:: This job will take a bit longer: Create a working dialog that will show the Tile Download Status in Marble's queue. The dialog should only be available by starting Marble with a special command line option or through a key stroke.
• Category HARD:: Marble downloads the tiles and saves them into the user's .marble directory. Until now Marble will never start to delete any local tiles. So the contents in .marble will grow until all tiles got fetched or until the HDD is full. Come up with a solution that will fix this problem. The solution should still preserve the tile directory structure and should work similar to QPixmapCache. Just that in this case the configurable cache is the hard disc.

For help or questions you can join us on IRC ( irc.kde.org, #kde-edu ) or send a mail to our mailing list.