Sponsored Links

Add your own satellite images to Virtual Earth with MapCruncher and Amazon S3 PDF Print E-mail
Written by Richard Marsden   
Monday, 06 April 2009 14:21

Raster images (eg. aerial photos) can be easily added to Microsoft's Virtual Earth control using the  MapCruncher utility. This utility calculates the required transformations and creates the necessary quadtree tile hierarchy.

This article shows you how to use MapCruncher to add a satellite photograph  to Virtual Earth, and to host the tiles on Amazon's S3 and CloudFront services.

I recently started to revamp the interactive maps on the EcoMapCostaRica.com website. Due to the lack of high resolution imagery of rural Costa Rica in both Google Maps and Microsoft Virtual Earth, the original implementation used a mixture of MapServer, OpenLayers, and KML. MapServer served WMS tiles derived from ESRI Shape files, and an ASTER satellite image (sourced from the Smithsonian's Global Volcanism Program).  A second version replaced the vector base layer with Virtual Earth satellite imagery, but unfortunately had limited zoom capabilities.

MapCruncher is a utility from Microsoft Research that has now been released as beta software for Virtual Earth. A detailed description can be found in the PDF research paper: MapCruncher: Integrating the World's Geographic Information, Jeremy Elson, Jon Howell, John R. Douceur;  Microsoft Research Redmond. MapCruncher can be downloaded from:  http://dev.live.com/virtualearth/mapcruncher/ .

Most of MapCruncher is intuitive to use, with a number of informative text displays explaining what to do next. After loading your source raster map (the ASTER satellite tile in my case), you match known points on the raster with points on Virtual Earth's imagery. These are known as correspondences. Here is a screenshot of MapCruncher after matching a correspondence:

Screenshot of MapCruncher

MapCruncher is capable of two kinds of transformations: affine (ie. linear), and quadratic. The affine transformation only requires three correspondences, whilst the more accurate quadratic requires six. Both transformations will benefit from more than the minimum number of points.

Set any other parameters that you require. For example, specific input colors can be set to be transparent using the Transparency tab. An important setting is the Maximum (Closest) Zoom on the Source Info tab. Use this to set the maximum zoom level that the tiles will be viewed at. Tiles are created as a hierarchy, setting this level to be too high will result in large numbers of tiles and require large amounts of disk space. I chose the high value of 17, but I needed to view the entire study area which is only about 500m across.

After matching the points, press the Render button. MapCruncher will calculate the new transformation and creates the tiles. This will take a few minutes. After completion, the Correspondences tab will include each correspondence's location error. All details of the MapCruncher run are written to the MapCruncherMetaData.xml file. This includes the transform coefficients, the bounding box, and individual tile information. We shall use the bounding box information later.

MapCruncher will also create the SamplePage.html file. This is an example page that uses the MapCruncher JavaScript control to display the newly-created tiles on Virtual Earth. This is useful as a simple test, but we shall be using Virtual Earth's native tile layer support instead.

Check your MapCruncher results. There may be two common problems that need fixing. The first is the sheer size of the results. This may happen because the maximum zoom is set too high for your requirements. The EcoMapCostaRica imagery needs to be used at a high resolution, so the maximum zoom was set to 17. With the entire ASTER satellite tile, this results in over 8GB of tiles. This is an impractical size for most online applications of this size. The solution is to cut the source file down. The source file can be clipped before it is loaded into MapCruncher. In addition, it is possible to set MapCruncher's clipping area. After rendering, MapCruncher will show a blue outline around your tile image. This has blue control nodes. Move these to change the clipping shape. Additional control nodes can be inserted. This proved useful to remove unimportant corners from the original tile that were poor quality imagery (eg. due to bad cloud cover). By combining these two clipping methods, the file size dropped from over 8GB to just over 300MB.

The second common problem involves the lateral accuracy of the transformation. This can often be seen visually, but the error column in the Correspondences tab gives a quantitiative feedback of the errors. Initial runs with the ASTER data had errors in the range of 20-500m. These large errors were partly due to the choice of correspondence locations, and partly due to the relatively poor resolution of the Virtual Earth imagery. It was difficult to find good locations to match in some areas due to cloud cover and/or the vegetation type. The relatively low resolution of the Virtual Earth imagery meant that the points could not be reliably located more accurately than about 100m.

The solution to these accuracy problems were two fold. First, I had some good GPS location points which could be correlated with locations in the study area. The GPS coordinates had a well-defined accuracy of 6 metres derived from a Trimble field computer during last year's field season. The ASTER imagery has a resolution of about 15m/pixel. Therefore I added three points located in the study area at known GPS coordinates. These were considerably more accurate that points matched on the Virtual Earth imagery. Secondly, I removed the correspondences with the worst errors. After re-rendering, the errors dropped from 20-500m to 0.81-42m. Importantly, the error for the three study area points was 4.3, 7.2, and 17m - much better than the 30-40m error that was previously seen in this critical area. The worst case for the entire ASTER image (42m) is still better than what is visible in the local Virtual Earth imagery.

Publishing the Image Tiles

The tiles have been created and are now ready for publishing. Although MapCruncher has its own JavaScript control that can be used with Virtual Earth, we chose to use the native Virtual Earth tile layer. This is because it is better documented, and is not classed as beta - ie. has a more stable API.

The tiles totalled over 300MB. This is a lot, and could pose a significant demand on a traditional webserver. Therefore I chose to publish to the Amazon Simple Storage Service (S3). This is a simple 'cloud' web service for bulk file storage. Files can also be made available as simple URLs - ideal for large web files such as images, movies, and map tiles. In addition, I added Amazon's CloudFront service as a front end. CloudFront acts as an automatic mirror for S3, mirroring the S3 contents to servers around the world. The end result is that client requests are served from a server close to the client - greatly reducing network latency.

Information and sign-up links for S3 and CloudFront can be found at Amazon Web Service. I also chose to use Bucket Explorer to manage my S3 account. Although the next version of Bucket Explorer will support CloudFront, the current version does not. Therefore I also used Amazon's free Manager for Amazon CloudFront. Bucket Explorer's promised support for CloudFront should make the process easier, and allow for advanced features such as meta data updates.

S3 accounts consist of "buckets". Files are located in buckets using a flat file system. However, '/' characters are allowed in files, allowing for pseudo-directories. After publishing the tiles to the pseudo-directory of ASTER in the sample bucket 'sample-bucket', they can be referenced with the URL: http://sample-bucket.amazonaws.com/ASTER/*.png 

S3 will let you create bucket names with uppercase characters. Bucket names can be used as sub-domain names, so it is strongly recommended that your bucket names are completely lowercase. Also, remember to make sure that both your bucket and your uploaded files are marked as 'public read'.

CloudFront works in terms of 'distributions'. These distributions map to individual Amazon S3 buckets. Therefore you will need to create a distribution to match the S3 bucket that you have created. Unlike the S3 buckets, you have no control over the name that CloudFront will choose for your distribution. It will be an unintelligible string of alphanumeric characters. Replace the name 'your_distrib' in the following code with this distribution name. The final URL for the above ASTER example will be: http://your_distrib.cloudfront.net/ASTER/*.png

After you have created your bucket, uploaded your tiles, and created the matching CloudFront distribution, you can test the above URL. Remember to replace the '*' with a tile name. Also, you will may have to wait up to 15 minutes for CloudFront to copy a newly created distribution to all of its servers.

Using the Tiles in Virtual Earth

We have published the tiles to the 'cloud'. Now we are finally ready to use them in a Virtual Earth application.

We use the VEMap object's AddTileLayer method to add a new tile layer containing the tiles. The tile specification is specified using a VETileSourceSpecification object. This requires the layer name (ASTER), the URL to the tiles, the geographic bounds, and the zoom range. The geographic bounds are taken from the MapRectangle entity in MapCruncher's MapCruncherMetaData.xml file (see above). Here is an example:

  1 
  2 
  3 
  4 
  5 
  6 
  7 
  8 
  9 
 10 
 11 
 12 
 13 
 14 
 15 
 16 
 17 
 18 
 19 
 20 
 21 
 22 
 23 
 24 
 25 
 26 
 27 
 28 
 29 
 30 
 31 
 32 
    // Create the VEMap object and display it
    map = new VEMap('Map');
    map.LoadMap();
    map.SetMapStyle('h');

    // These are the bounds of the tile data
    var bounds = [
           new VELatLongRectangle(new VELatLong(10.556914110, -84.757503122),
                                  new VELatLong(10.423837436, -84.620150214))];

    // This is the tile source
    // Replace your_distrib with the name of your CloudFront distribution
    // Also change the /ASTER/ path as required
    // VE will replace %4 with the correct tile name for the required 
    // position in the tile quadtree
    var tileSource = "http://your_distrib.cloudfront.net/ASTER/%4.png";

    // Create the tile source specification for a layer to be called "ASTER"
    // One distribution server
    // Tiles support zoom range of 12-17
    // Display it as complete opaque and a Z index of 100
    var tileSourceSpec = new VETileSourceSpecification("ASTER", tileSource);
    tileSourceSpec.NumServers = 1;
    tileSourceSpec.Bounds     = bounds;
    tileSourceSpec.MinZoom    = 12;
    tileSourceSpec.MaxZoom    = 17;
    tileSourceSpec.Opacity    = 1;
    tileSourceSpec.ZIndex     = 100;

    // Add the tile layer!
    map.AddTileLayer(tileSourceSpec, true);

 

Typically, the VEMap object and the various layers are created on a "onLoad" callback after the web page loads. Here is a complete web page that includes this callback and two size-setting methods:

   1 

   2 
   3 
   4 
   5 

   6 
   7 
   8 
   9 
  10 
  11 
  12 
  13 
  14 
  15 
  16 
  17 
  18 
  19 
  20 
  21 
  22 
  23 
  24 
  25 


  26 
  27 
  28 
  29 

  30 
  31 
  32 
  33 
  34 
  35 
  36 
  37 
  38 
  39 
  40 
  41 
  42 
  43 
  44 
  45 
  46 
  47 
  48 
  49 
  50 
  51 
  52 
  53 
  54 
  55 
  56 
  57 
  58 
  59 
  60 
  61 
  62 
  63 
  64 
  65 
  66 
  67 
  68 
  69 
  70 
  71 
  72 
  73 
  74 
  75 
  76 
  77 
  78 
  79 
  80 
  81 
  82 
  83 
  84 
  85 
  86 
  87 
  88 
  89 
  90 
  91 
  92 
  93 
  94 
  95 
  96 
  97 
  98 
  99 
 100 
 101 
 102 
 103 
 104 
 105 
 106 
 107 
 108 
 109 
 110 
 111 
 112 
 113 
 114 
 115 
 116 
 117 
 118 
 119 

 120 
 121 
 122 
 123 
 124 
 125 
 126 
 127 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<head>
<title>MapCruncher Sample Web Page</title>

<script src="http://dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=6.2" 
  type="text/javascript"></script>

<script type="text/javascript">

var div = null;
var map = null;



// The following two functions work together, to set the map object size 
// to match the window

function GetSize()
{
    var myWidth = 0, myHeight = 0;

        if( typeof( window.innerWidth ) == 'number' ) {
              //Non-IE
              myWidth = window.innerWidth;
              myHeight = window.innerHeight;
        } else if( document.documentElement && ( 
          document.documentElement.clientWidth || 
          document.documentElement.clientHeight ) ) {
              //IE 6+ in 'standards compliant mode'
              myWidth = document.documentElement.clientWidth;
              myHeight = document.documentElement.clientHeight;
        } else if( document.body && ( document.body.clientWidth || 
          document.body.clientHeight ) ) {
              //IE 4 compatible
              myWidth = document.body.clientWidth;
              myHeight = document.body.clientHeight;
        }

        if (myWidth > 0)
        {
              var result = new Array();
              result[0] = myWidth;
              result[1] = myHeight;
              return result;
        }
        else
        {
              return null;
        }
}

function SetMapSize()
{
    var browserSize = GetSize();
    var div = document.getElementById("Map");

    if (browserSize != null)
    {
        div.style.width = (browserSize[0] - 25) + "px";
        div.style.height = (browserSize[1] - 25) + "px";
    }
    else
    {
        div.style.width = "800px";
        div.style.height = "600px";
    }
    div.style.overflow = "hidden";
    div.style.position = "relative";

}



// This is the main map creation method
// It creates the VEMap map object and adds our tile data as a layer
// It is called when the page loads


function LoadPage()
{
    // Set the map div size - and tie to the resize events (see methods above)
    SetMapSize();
    window.onresize = SetMapSize;

    // Create the VEMap object and display it
    map = new VEMap('Map');
    map.LoadMap();
    map.SetMapStyle('h');

    // These are the bounds of the tile data
    var bounds = [
           new VELatLongRectangle(new VELatLong(10.556914110, -84.757503122),
                                  new VELatLong(10.423837436, -84.620150214))];

    // This is the tile source
    // Replace your_distrib with the name of your CloudFront distribution
    // Also change the /ASTER/ path as required
    var tileSource = "http://your_distrib.cloudfront.net/ASTER/%4.png";

    // Create the tile source specification for a layer to be called "ASTER"
    // One distribution server
    // Tiles support zoom range of 12-17
    // Display it as complete opaque and a Z index of 100
    var tileSourceSpec = new VETileSourceSpecification("ASTER", tileSource);
    tileSourceSpec.NumServers = 1;
    tileSourceSpec.Bounds     = bounds;
    tileSourceSpec.MinZoom    = 12;
    tileSourceSpec.MaxZoom    = 17;
    tileSourceSpec.Opacity    = 1;
    tileSourceSpec.ZIndex     = 100;

    // Add the tile layer!
    map.AddTileLayer(tileSourceSpec, true);

    // Position the map to be centred on the tile layer data
    map.SetCenterAndZoom( new VELatLong(10.48, -84.69), 13);
}


</script>
</head>

<!-- HTML BODY. Defines the LoadPage callback and the div that stores the map 
  -->
<body onload="LoadPage()">

<div id="Map"></div>


</body>
</html>

 

And here are the results of the above page:

MapCruncher results in Virtual Earth

We have our own high resolution satellite imagery positioned over the lower resolution satellite imagery of Virtual Earth. The entire image is located within the resolution of Virtual Earth, and our study area is positioned within the accuracy of our high resolution image. The image is fully tiled and can be zoomed to a useful level for the study area. Although the tiles consume over 300MB, they are located on servers distributed around the world using the cost effective Amazon S3 / CloudFront solution.


Comments (4)Add Comment
62
EcoMapCostaRica VE map
written by Richard Marsden, April 07, 2009
The current iteration of the EcoMapCostaRica can be found here:

http://www.ecomapcostarica.com/map/index_ve.shtml

Note that this is likely to change over the coming months, and will probably switch from being an alternative map to being the main map.
62
Amazon CloudFront Addenda
written by Richard Marsden, April 09, 2009
Note that when you create your CloudFront distribution, you can create a CNAME so that it is has a subdomain alias. This might be useful if the URL is publically visible, but is of less interest for an 'internal' tile server like above. See the CloudFront documentation for details.

The latest version of Bucket Explorer was released this week - it now supports CloudFront distributions. I've only created one distribution with it so far, but it is simpler than using the separate Amazon tool. It also allows you to set the metadata for S3 file objects. This is important because you can set the expiry information. Set Cache-Control to "max-age=2419200" to change the CloudFront fetch interval from the 24 hour default to 28 days. This reduces the number of fetches (reducing your S3 cost), and the metadata will be passed by CloudFront to the client - further reducing fetches from the client. The net effect is reduced S3/CloudFront costs, and a faster experience for the end user. I have set the above ASTER tiles to 365 days as they are for all intents and purposes static. I can implement versioning (eg. have a directory "ASTER_2") if I need to deploy an update.
0
Issues with code
written by Charles Brown, August 28, 2009
We were using the functions SetMapSize and GetSize

We are getting an object expected error on "div.style.width = (browserSize[0] - 25) + "px";" and of course the map will not load at that point.

function SetMapSize(){
var browserSize = GetSize();
var div = document.getElementById("Map");

if (browserSize != null)
{
div.style.width = (browserSize[0] - 25) + "px";
div.style.height = (browserSize[1] - 25) + "px";
}


Any clue as to why?
62
Do you know which variable is causing the error?
written by Richard Marsden, August 28, 2009
Do you know which variable is not set to an object? If it is the div, then this is probably because you don't have a "Map" div (or it isn't named properly)? Similarly, if browserSize is not set, then there's probably a problem with GetSize().

Write comment

security code
Write the displayed characters


busy