The links below provide an outline of the material for this lesson. Be sure to carefully read through the entire lesson befor returning to Canvas to submit your assignments.
Note: You can print the entire lesson by clicking on the "Print" link above.
This week's lesson focuses on tiled web maps. Tiles are relatively small square-shaped "chunks" of data (either rasterized map images or raw vector coordinates) that have been pregenerated by the server and stored in a directory called a cache. When web users navigate the map, the server can just hand out the tiles rather than generating the map on the fly.
In this lesson, you'll learn the pros and cons of tiled maps, as well as strategies for building and maintaining a tile cache. You will learn about traditional rasterized tiled image formats, as well as a newer generation of tiles that store vector coordinates. Because rasterized image tiles have been around much longer and have mature support from tile generating engines, tile servers, and clients, most of the lesson content focuses on these types of tiles. The newer vector tile format is discussed near the end of the lesson content.
There are two walkthroughs associated with this lesson, both of which involve rasterized image tiles because these currently have the most mature tools for end-to-end FOSS workflows. The first walkthrough shows how to create a simple cache of your Philadelphia neighborhoods map using the GeoWebCache software that is integrated into GeoServer. In the second walkthrough, you'll use TileMill and a markup language called CartoCSS to create a Philadelphia basemap cache with some nicer cartography than you would get using GeoServer.
As mentioned in previous lessons, the earliest web maps were typically drawn on the fly by the server, no matter how many layers were available or requested. These are the types of maps you just created using GeoServer and WMS. As you may have noticed, the symbol sets and labeling choices for this type of map are relatively limited and complex to work with. In fact, for many years, web cartographers had to build a map with minimal layer set and simple symbols to avoid hampering performance. In many cases, a cartographer was not even involved; instead, the web map was made by a server administrator tweaking XML files that defined the layer order, symbol sizes, and so forth. This was the case with both open specification web services (like WMS) and proprietary web services (like Esri ArcIMS).
Part of this approach stemmed from early efforts to make web GIS applications look exactly like their desktop counterparts. Sometimes these are referred to as “Swiss Army Knife applications” because they try to do everything (you may know one!). People expected that in a web GIS they should be able to toggle layer visibility, reorder layers, change layer symbols on the fly, and do everything else that they were accustomed to doing on the desktop. Ironically, this mindset prevailed at a time when web technology was least suited to accommodate it.
In the mid-2000s, after Google Maps, Microsoft Virtual Earth (now Bing Maps), and other popular mapping applications hit the web, people started to realize that maybe they didn't need the ability to tinker with the properties of every single layer. These providers had started fusing their vector layers together in a single rasterized image that was divided into 256 x 256 pixel images, or tiles. These tiles were pregenerated and stored on disk for rapid distribution to clients. This was done out of necessity to support hundreds or thousands of simultaneous users, a burden too great for drawing the maps on the fly.
The figure below shows how a tiled map consists of a "pyramid" of images covering the extent of the map across various scales. Tiled maps typically come with a level, row, and column numbering scheme that can be shared across caches to make sure that tile boundaries match up if you are overlaying two tile sets.
Cartographers loved the tiled maps, because now they could invest all the tools of their trade into making an aesthetically pleasing web map without worrying about performance. Once you had created the tiles, you just had a set of images sitting on disk, and the server could retrieve a beautiful image just as fast as it could retrieve an ugly one. And because the tiled map images could be distributed so quickly by a web server, Google and others were able to employ asynchronous JavaScript and XML (AJAX) programming techniques to retrieve the tiles with no page blink as people panned.
This was revolutionary. Which would you rather have: a slippy map with stunning cartography and no layer control, or a clunky and ugly map with the ability to reorder layers and adjust the color of a school? Some longtime GIS geeks had to stop to think about this, but for the common web user, the choice was a no-brainer.
Within a year or two of Google Maps' release, commercial GIS software began offering the ability to build map tiles. For many, ArcGIS Server was desirable because the map could be authored using the mature map authoring tools in ArcMap; however, cost was a concern for some. Arc2Earth was another commercial alternative. The free and open source Mapnik library could also build tiles, but it wasn't until recent years that projects like TileMill wrapped a user-friendly GUI around Mapnik.
Tiled maps were the only model that could reasonably work for serving complex web maps to thousands of simultaneous users. However, they eliminated the ability for users to change layer order or symbols. People started working around this by serving out their general-purpose basemap layers as tiles and then overlaying a separate layer with thematic information. The general-purpose basemap tiles could be re-used in many applications. The thematic layers could also be tiled if the data didn't change too quickly or cover too broad an area at large scales. For example, if you examine Google Maps with a developer tool, you will see that the basemap and the thematic layers (such as Panoramio photographs) are both retrieved as tiles.
If you want faster navigation of your basemap or you feel that more than a couple of users will be requesting maps simultaneously from your server, you should create a tile cache of your basemap. You may also choose to cache thematic layers if their features are not constantly changing attributes or position.
In both cases, be aware that the tile cache represents a snapshot of your data at the time the cache was created. To put it more bluntly, your tiles are “dumb images” that will not automatically update themselves when the back end data changes. You are responsible for periodically creating new tiles in order to update the map. With large caches, sometimes server administrators target the cache updates at just the changed areas rather than rebuilding tiles for the entire map. This requires keeping some kind of log about which places were edited, or comparing “before” and “after” versions of a dataset.
Building a tiled basemap requires lots of rich data, high-end map authoring software, cartographic skills, and potentially enormous amounts of time and disk space. You may still need to do it at one point or another, which is why you will get a taste of this experience in Geog 585. However, because of these challenges, general-purpose web mashups often use tiles made by somebody else. OpenStreetMap is an attractive option if you want free tiles with no restrictions. If you want to use Google's, Microsoft's, or Esri's tiles, you may be able to use them for free or you may have to pay, depending on the nature of your map (commercial or not for profit) and how many people use your app. Other companies such as CloudMade and MapBox have marketed their own versions of tiles using OpenStreetMap data.
If you are going to build your own basemap, it's helpful to have an experienced cartographer on staff who is experienced with designing at multiple scales. The symbols, colors, and details must be adjusted appropriately at each of the scales where tiles are created. Tiled basemaps can quickly get complicated with layer and labeling scale suppression rules. Cartographers may also need to design one set of tiles to stand on its own and another set of tiles to overlay remotely sensed imagery, a task that requires very different colors and symbols.
If you are going to overlay your tiles with any of the tiles from OpenStreetMap, Google, Microsoft, or Esri (or even just attempt to look like them), you must warp your most precious GIS data into a modified spherical Mercator projection that was created solely for the convenience of fitting the world onto a set of square tiles. GIS purists balked at this idea and predicted it would fail to achieve mass uptake, but many people (at least at mid latitudes) now hold their nose and move forward with it.
Be aware that Esri, Google, and other organizations have used other code numbers and variations of this projection in the past: things can get very confusing if you are using older software or APIs. In the past couple of years, people seem to have standardized on EPSG:3857 although even the subparameters of this projection can be interpreted in diverse ways that lead to offsets. As a side note, can you figure out the humorous reason that Google once used the code 900913 for this projection?
Even when you display your maps in EPSG:3857, you do not ever want to perform measurements in this projection. The results will be largely skewed even at mid latitudes. It's best to make sure that geometries are projected into a more local coordinate system before performing any measurements. The Esri blog post Measuring distances and areas when your map uses the Mercator projection [1] gives a good overview of the problem and an example solution that uses the ArcGIS API for JavaScript, although the concepts can also be applied with FOSS APIs.
Not only must you match the projection in order to overlay, you must also match the scales. These are not nice well-rounded numbers; rather, they were derived mathematically from the starting point of putting the whole world on a 2x2 grid of tiles. For example, one of the scales is 1:36,111.98 and as you zoom in, the next one is 1:18,055.99. So much for your simple USGS 1:24000 series! Partly for this reason (and partly because many laypeople don't understand map scales), the common web map scaleset is often referred to with simple numbers such as “Level 14,” “Level 15,” etc., that increase as you zoom in. You just have to get a feel for which levels correspond to national scale, provincial scale, city scale, neighborhood scale, etc. The table in the Bing Maps Tile System [2] article is helpful for this.
Map tiles often have a simple folder structure, which makes them easy to serve. However, they can become complex to manage due to their sheer size and number. There are several different ways people have devised to serve map tiles:
At large (zoomed in) scales, the number of tiles to cache can be overwhelming, especially if you are covering a broad area such as a state or country. The irony is that at such large scales, many tiles will convey very little information. Zooming in to a neighborhood block scale at 1:2250 may show plenty of interesting features, but pan out into the desert or ocean at the same scale, and the tile may be completely blank. Do you want to spend the hours creating these tiles and gigabytes of disk space to store them?
In these situations, you may want to find software that can create tiles on demand, meaning at the time they are first visited by a user. The first person to navigate to a region will need to wait for the server to create the tiles, but subsequent visitors will enjoy the full benefits of created tiles. The most popular areas will fill up with tiles, but you won't spend resources creating and storing tiles that are never visited. Obviously, this approach varies in effectiveness based on how fast the server can actually draw the tiles on demand.
An alternative is to use a “Data not available” tile to denote areas where tiles have not been created. Web map administrators are sometimes loathe to do this, but it is often a common enough practice that lay users blame themselves when they see the tile (“Oops, I zoomed in too far!”) rather than the administrator (“Why didn't they make the map available at this scale?”).
The best approach may be to strategically create a subset of the most interesting tiles and leave the less interesting tiles to be created on demand (or returned using a “Data not available” tile). As a geographer, it may hurt your soul to call any place “less interesting,” but the bitter truth is that not all tiles of the map experience an equal number of views. Fisher (2007) showed how early visitors to Microsoft Virtual Earth (now Bing Maps) stuck mostly to the big cities, coastlines, and transportation corridors. Quinn and Gahegan (2010) built models taking into account these patterns, showing how the majority of map requests could likely be satisfied by creating a fraction of the full number of tiles that it would ordinarily take to cover the full rectangular extent of the map. Their models were cobbled together using datasets like buffered roads, coastlines, and points of interests, but more recent feeds from social media such as geotagged tweets and Flickr photos may prove even more accurate in revealing the most interesting regions of the map for the majority of users. Note that some types of specialized maps (for mineral exploration, or wilderness conservation) may have very different usage patterns than general-use basemaps.
The ability to selectively cache a subset of tiles like this depends on the tile creation software's flexibility in allowing the administrator to designate custom regions for caching. Most software just allows the submission of a rectangular bounding box for tile creation; however, interesting areas of the map such as cities and coastlines are usually not shaped like rectangles. If you identify an irregularly-shaped region where you want to create tiles, you may have to abstract it into a series of rectangles and run multiple tile creation jobs, using each rectangle as an input. If you are luckier, your tile creation software will accept a spatial dataset (such as a shapefile) as a bounding region.
Creating tiled web maps is a common task that has been addressed by various FOSS packages. The most accessible one for you at this point is GeoWebCache, because it's integrated directly into GeoServer. Others include TileCache and TileStache.
The Mapnik library is a FOSS tile creation library that binds to Python and other languages. It allows a lot of advanced drawing options not found in your typical WMS layer. Working with Mapnik typically required some Linux knowledge and some trial and error; however, a couple of years ago, the for-profit company Mapbox released an open source program called TileMill that can run on Mac and Windows and puts a nice GUI around Mapnik. This simplifies the cartographic process and puts Mapnik within the scope of Geog 585. In the second walkthrough in this lesson, you'll use TileMill to create a basemap of Philadelphia using some of the layers you processed earlier. Of course, feel free to also explore Mapbox Studio if you are interested, and share your experience with it on the forums.
Although the rasterized tile sets we have discussed in this lesson are able to deliver nice-looking maps in a relatively rapid format, they can be cumbersome to keep updated and they require enormous amounts of computing resources at large map scales. To work around these challenges, a data storage format called "vector tiles" has gained popularity in the past several years. Mapbox has led development efforts on vector tiles and has shared a vector tiles specification [4] under a Creative Commons license.
Vector tiles are exactly what you would guess: they store chunks of vector data instead of storing a map image. The idea behind vector tiles is that it is more efficient to keep data styling separate from the data coordinates and attributes. The client can use a predefined set of styling rules to draw tiles of raw vector coordinate and attribute data sent by the server. This allows the restyling of data on the fly, which is another serious limitation of rasterized tiles. Think about it: If you want to change the shade of green used to draw parks with your rasterized tiles, you must rebuild every tile containing a park. If you want to do the same thing with vector tiles, you just update your styling instructions in one place and the tiles themselves stay the same. Other display operations such as rotating the map also become easier to implement with vector tiles.
Vector tiles are designed to be small on disk, and employ a number of optimization approaches designed to reduce the amount of characters needed to store the geographic data and attributes, some of which are described in this video by Mapbox engineer Dane Springmeyer [5]. He also introduces a product called Mapbox Studio [6] which works with vector tiles only and is being promoted by Mapbox as a replacement for Tile Mill. The .mbtiles file format, which originally stored rasterized tiles, now only stores vector tiles when exported from Mapbox Studio.
In reality, there continue to be use cases [7] for vector and rasterized tile formats, although it is likely that a number of organizations will see performance benefits from rebuilding some of their originally rasterized tile sets as vector tiles in the future. This is even more likely as popular commercial software packages such as ArcGIS introduce tools to work with the Mapbox vector tile specification, a strategic decision that Esri announced in a 2015 blog post [8]. Open source tools are also recognizing the staying power of vector tiles, exemplified by the VectorTile layer format [9] built into OpenLayers 3 and plugin support for Mapbox vector tiles in Leaflet [10]. At the time of this writing, QGIS does not natively support viewing vector tiles, although some have been developing plugins [11] for this purpose.
Suppose you're satisfied with the layers and symbols in your WMS, but you want it to draw faster and be available to many simultaneous users. In this situation, it might make sense to use GeoWebCache to create your tiles, because GeoWebCache is built directly into GeoServer. In this walkthrough, you'll use GeoWeb Cache to create a tile cache for the Philadelphia NeighborhoodMap group layer WMS that you published in the previous lesson.
Although performance is improved with the tile cache, you may notice some duplicate labels appearing. This is a difficult problem to avoid with map tiling, because each tile does not "know" about the labels on the adjacent tiles. To mitigate this problem, tile caching software typically draws an area much larger than a tile and then cuts it up into individual tiles. GeoWebCache calls this large area a "metatile" (Esri calls it a "supertile"). If you like, you can experiment with adjusting the metatile size; although duplicate labels can still appear at the metatile boundaries, the duplicates will be fewer and farther between. You may also find that the settings and options in the next walkthrough with TileMill make it easier to get the labeling you want.
In this walkthrough, you will use Mapnik (wrapped by TileMill) to create a general-use basemap of Philadelphia. The idea is that you will be able to overlay thematic layers on top of this map in lessons to come. The data for this walkthrough is the base data for Philadelphia that you preprocessed in Lesson 3. If you followed all instructions, you should have this data in a folder called c:\data\Philadelphia or something similar. It should be using the mercator projection EPSG:3857.
Mapnik [12] is a FOSS map-drawing engine that is often used for the purpose of making sets of map image tiles. Mapnik incorporates techniques like antialiasing that make the edges of lines and labels appear smooth, not pixelated. It is not tied to the overhead of any other GIS software framework, a fact that improves its performance. Various types of file-based data and spatial databases are supported as data sources for the maps.
To use Mapnik, you define a set of data sources and then connect them to a bunch of styling rules. You can set up the styling rules using an XML file [13], or you can write them programmatically [14] using a language like Python. Once you've configured the data sources and style rules, you can instruct Mapnik to export pictures of the map.
This is easy in theory but is not the simplest process for a beginner, especially if your programming skills are thin. To work with Mapnik in this course, you'll use a program called TileMill that makes the process a lot easier. TileMill puts a GUI around Mapnik, allowing you to browse to the datasets you want to use and see the results of your work as you apply styling rules for each layer. Instead of using an XML file or code to define the rules, you'll use a fairly intuitive language called CartoCSS that borrows from the structure of web stylesheets.
TileMill is FOSS, but was developed by Mapbox to integrate with the company's for-profit tile hosting service. This is evident when you export tiles with TileMill and the resulting file has the extension .mbtiles. If you don't want to host the tiles on Mapbox's servers, you can "unpack" the tiles out of the .mbtiles file and host them as individual images on your own web server. In this walkthrough, you'll unpack the images and host them on your Penn State PASS web space. Then you'll test the tiles by supplying their URL structure to a web map.
As you complete this walkthrough, you may see occasional software messages that Tile Mill is not under active development and that Mapbox has shifted its focus to Mapbox Studio. Tile Mill continues to be available as a FOSS tool for building tile sets of rasterized map images. In contrast, Mapbox Studio is geared towards creating vector tiles which Mapbox will then host in exchange for a fee. Be aware that Mapbox Studio-generated .mbtiles files you encounter professionally may contain vector tiles instead of containing rasterized images like the Tile Mill-generated .mbtiles files we will use in this walkthrough.
Map { background-color: #FFFFFF; }The background should update to white as soon as you save your code. Now, let's add some layers and follow this process to style them.
#CityLimit { line-color:#88789e; line-width:3; }When you save, you should see the following:
#Waterways { line-width:1; line-color:#89aceb; }
#NaturalFeatures{ [type='park']{ polygon-opacity:1; polygon-fill:#ae8; } [type='riverbank']{ polygon-opacity:1; polygon-fill:#89aceb; } [type='water']{ polygon-opacity:1; polygon-fill:#89aceb; } }Notice above that you can add query terms if you only want to display a subset of features in the dataset. We are displaying just the parks, riverbanks, and water features, even though there are a lot more types of features in this shapefile.
#Neighborhoods[zoom>12] { text-name:[NAME]; text-face-name:"Arial Black"; text-fill:#88789e; text-size: 12; text-character-spacing: 2; text-transform: uppercase; }In this case, we only specify text properties. This has the effect of hiding the neighborhood boundaries so that just the label of the neighborhood name is shown. This is more fitting to neighborhoods anyway, since they don't tend to have rigid boundaries.
#MajorRoads{ [type='motorway']{ line-width:3; line-color:#606060; } [type='trunk']{ line-width:3; line-color:#606060; } [type='primary'] { line-width:2; line-color:#838383; } }This will symbolize just the major roads. OpenStreetMap has a lot of different road types, and there are different ways you could apply styling here. In our situation, we will display minor streets by adding all the roads again and putting them beneath the major roads. This introduces some redundancy but keeps the code simpler.
#Roads[zoom>12]{ line-width:1; line-color:#b6b6b6; } #Roads[zoom>14]{ line-width:1; line-color:#b6b6b6; text-name:[name]; text-face-name:"Arial Regular"; text-fill:#838383; text-size: 11; text-placement: line; text-min-path-length:100; text-avoid-edges:true; text-min-distance:50; text-dy: 6; text-max-char-angle-delta: 15; }This puts some labels on the roads when zoomed in.
#Railroads{ line-width:1; line-color:#d2bcb0; } #Railroads[zoom>15] { ::line, ::hatch { line-color: #d2bcb0; } ::line { line-width:1; } ::hatch { line-width: 4; line-dasharray: 1, 24; } }
Map { background-color: #FFFFFF; } #CityLimit { line-color:#88789e; line-width:3; } #Waterways { line-width:1; line-color:#89aceb; } #NaturalFeatures{ [type='park']{ polygon-opacity:1; polygon-fill:#ae8; } [type='riverbank']{ polygon-opacity:1; polygon-fill:#89aceb; } [type='water']{ polygon-opacity:1; polygon-fill:#89aceb; } } #Neighborhoods[zoom>12] { text-name:[NAME]; text-face-name:"Arial Black"; text-fill:#88789e; text-size: 12; text-character-spacing: 2; text-transform: uppercase; } #MajorRoads{ [type='motorway']{ line-width:3; line-color:#606060; } [type='trunk']{ line-width:3; line-color:#606060; } [type='primary'] { line-width:2; line-color:#838383; } } #Roads[zoom>12]{ line-width:1; line-color:#b6b6b6; } #Roads[zoom>14]{ line-width:1; line-color:#b6b6b6; text-name:[name]; text-face-name:"Arial Regular"; text-fill:#838383; text-size: 11; text-placement: line; text-min-path-length:100; text-avoid-edges:true; text-min-distance:50; text-dy: 6 text-max-char-angle-delta: 15; } #Railroads{ line-width:1; line-color:#d2bcb0; } #Railroads[zoom>15] { ::line, ::hatch { line-color: #d2bcb0; } ::line { line-width:1; } ::hatch { line-width: 4; line-dasharray: 1, 24; } }
Once you've finished the map design phase and your map looks good at each scale, you can start thinking about generating the tiles. For large maps, plan for this to take some time, taking into account some of the factors discussed earlier in the lesson such as the shape of the map and the scale levels you choose to generate.
This section of the walkthrough explains how to generate tiles of your Philadelphia basemap, unpack them, and host them on your web space. You will then test them out in a web page.
Map { background-color: #FFFFFF; buffer-size: 512; }
c:\python27\python.exe c:\mbutil\mb-util c:\data\Philadelphia\PhillyBasemap.mbtiles c:\data\Philadelphia\PhillyBasemapNotice that the parameters are, in order, the path to Python, the path to the mb-util utility, the path of your compressed tiles, and the new folder where you want your tiles uncompressed. If any of your paths are different, you will need to adjust them when you type this command. After you run this command, you should see a set of uncompressed tiles in the folder c:\data\Philadelphia\PhillyBasemap.
If you navigate around your folder of unpacked tiles, you'll notice that the images are extracted into a highly organized structure of level\column\row. This structure is understood by various mapping programs and APIs, so all you have to do at this point is put your tiles onto a web-facing server. A convenient place for you to experiment with this is the www folder in your PASS space, a public-facing directory that all Penn State students are given in their personal web space. We will place the tiles there and then test them in a web map.
Now that you've gone through a walkthrough and built your own tiled map, hopefully your appreciation has increased for the amount of cartographic design and effort required for producing a web basemap. In this week's assignment, you'll look at some existing tiled maps, then get some practice building your own.
Do the following:
Please produce a report containing all the information requested in the first bullet above, as well as a URL to your tileset on your PASS space that I can test in a web map as shown in the walkthrough. Then submit the report to the Lesson 5 drop box on Canvas.
Links
[1] http://blogs.esri.com/esri/arcgis/2010/03/05/measuring-distances-and-areas-when-your-map-uses-the-mercator-projection/
[2] http://msdn.microsoft.com/en-us/library/bb259689.aspx
[3] https://mts0.google.com/vt/lyrs=m@241289412&hl=en&src=app&x=74&y=96&z=8&s=Galile
[4] https://github.com/mapbox/vector-tile-spec
[5] https://www.youtube.com/watch?v=se2cd3BMYRY
[6] https://www.mapbox.com/mapbox-studio/
[7] http://www.azavea.com/blogs/labs/2015/05/converting-mapbox-studio-vector-tiles-to-rasters/
[8] http://blogs.esri.com/esri/arcgis/2015/07/20/vector-tiles-preview/
[9] http://openlayers.org/en/master/examples/mapbox-vector-tiles.html
[10] https://github.com/SpatialServer/Leaflet.MapboxVectorTile
[11] https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin
[12] http://mapnik.org/
[13] https://github.com/mapnik/mapnik/wiki/GettingStartedInXML
[14] https://github.com/mapnik/mapnik/wiki/GettingStartedInPython
[15] https://www.mapbox.com/tilemill/
[16] https://www.mapbox.com/tilemill/docs/guides/metatiles/
[17] https://github.com/mapbox/mbutil
[18] http://www.python.org
[19] http://explorer.pass.psu.edu
[20] https://www.work.psu.edu
[21] http://personal.psu.edu/<your_PSU_ID>/tiles/PhillyBasemap/15/9555/12400.png
[22] http://www.arcgis.com/home