Published on GEOG 585: Web Mapping (https://www.e-education.psu.edu/geog585)

Home > Lessons > Lesson 9: Exploring open data, VGI, and crowdsourcing

Lesson 9: Exploring open data, VGI, and crowdsourcing

The links below provide an outline of the material for this lesson. Be sure to carefully read through the entire lesson befor returning to Canvas to submit your assignments.
Note: You can print the entire lesson by clicking on the "Print" link above.

Overview

A GIS or web map is only as useful as the data you put into it. Just as the GIS landscape offers proprietary software and open software, you will see sources of proprietary data and open data. This lesson explores the different meanings of "open data" and provides an introduction to OpenStreetMap, a growing repository of open data that is useful in a variety of projects.

Objectives

  • Define “open data” and describe some of the differences in use conditions among open data options.
  • Recognize the benefits and weaknesses of OpenStreetMap and its process of crowdsourcing.
  • Describe options for retrieving data from OpenStreetMap.
  • Edit OpenStreetMap according to community-defined tagging standards, and describe what you learned about open data sources from this experience.

Checklist

  • Read the Lesson 9 materials on this page.
  • Complete the walkthrough.
  • Complete the Lesson 9 assignment.
  • Complete the "third quiz" on Canvas. This covers material from Lessons 7 - 9.

Ways of opening data

Lately it seems that "open data" is everywhere. Reaching buzzword status, this term is often seen in tandem with phrases such as "open government," "crowdsourcing," "government transparency," and "free and open source software."  But what makes data "open"? Just as you learned in Lesson 1 that different organizations, even proprietary software companies, employ the term "open source" to their advantage, there are various nuances to the term "open data" that you should consider whenever you hear someone touting this phrase.

Consider the following means of data access and how they might be placed on a continuum of more or less "open":

  • The data is not available in any format for the public to view or download (the baseline case of "closed" data).
  • The data is distributed in static format only, such as a paper map or PDF.
  • The data can be viewed by anyone through a web map, but not downloaded.
  • The data can be accessed by anyone through a web service and displayed in a GIS or web map, but the full dataset cannot be downloaded.
  • The data can be downloaded in proprietary data formats at no cost.
  • The data can be downloaded in open formats at no cost.

Consider how these levels of data access play into the following scenarios:

  • A person exposing a dataset claims that "the data is open because I let you see the data;" however, the data itself may not be available for download due to licensing, security, technical, or human resources restrictions.
  • An "open data portal" might expose datasets for free download, yet some items in the portal may only be available in formats readable by proprietary software.

The most open types of data are those that allow complete download, re-use, and modification of the data in open formats. However, other levels of data openness may be more useful than not seeing the data at all. If you expose a useful dataset through a web map or a web service, you should prepare an answer for the question, "Can I download this data?" It won't be long before somebody asks.

Open data licenses

Even when data is freely available for download in open formats at no cost, it may still be subject to licensing restrictions. There are numerous types of open data licenses that stipulate what types of applications can use the data (personal, noncommercial, commercial, etc.) and what kind of attribution must be given. The license may also state the types of modifications that are allowed on the data, especially if the modified dataset is to be redistributed.

To get a feel for some of these licenses such as Creative Commons, Open Database License, Open Government License, and Public Domain, please take a few minutes to read pages 4 - 8 of Licensing Open Data: A Practical Guide [1] by Korn and Oppenheim, 2011. Focus especially on the chart on page 6.

Proprietary software and open data

FOSS typically excels at working with open data formats; however, FOSS is certainly not the only option for creating, exposing, or using open data. For example, Esri has invested in building open data discovery and download mechanisms into its ArcGIS Online and Portal for ArcGIS products. The idea is that government customers will be more likely to maintain their data in the proprietary software repository if the repository is easily engineered to allow free and open downloads by the public in popular formats such as KML and CSV. The video ArcGIS Open Data with Andrew Turner [2] shows how the pitch was made to federal government customers. The same type of application is achievable using FOSS, but would be less "out-of-the-box-y;" it would probably require a programmer to create and maintain.

VGI and crowdsourced data collection

If you don't have the money or means to purchase your required GIS data, or if the data doesn't exist, then you may need to collect the data yourself. If your goal is to openly share the resulting data with the public, then you may consider enlisting the public in your data collection efforts. VGI and crowdsourcing are two concepts that come into play when enlisting the public or non-domain experts in the collection of GIS data.

VGI

In 2007 Michael Goodchild published a paper in which he elaborated on the idea of volunteered geographic information (VGI). This kind of data is collected by citizens acting as sensors to gather information about the world around them. The citizens then feed this information into a centralized GIS database, often employing a user interface that has been simplified to the degree that specialized training is not required.

VGI has since become a hot term in geographic information science as thousands of people contribute to the OpenStreetMap digital map of the world (discussed later) and governments evaluate the possibilities of creating "citizen reporting" apps that allow anyone to upload information about potholes, graffiti, etc., with the objective of bringing them to the attention of local authorities.

Crowdsourcing

Crowdsourcing is the idea of using the power of a crowd to collect data that is too vast, heterogeneous, or expensive to be collected by other types of sensors. Consider how many people you would have to hire in order to write an encyclopedia with 30 million articles in 250 languages. The crowdsourced website Wikipedia [3] has been able to create a project of this scope solely through crowdsourcing. Other applications of crowdsourcing include combing remotely sensed imagery [4] to find lost people or vehicles, recording old weather measurements from ship logs [5] in order to create climate databases, and transcribing census records [6] to create searchable genealogical indexes.

Crowdsourcing is a particularly good fit for tasks that require an element of human cognition not easily performed by machines. Amazon has even made a business out of crowdsourcing through its Mechanical Turk [7] service. This allows you to hire a crowd of unknown individuals to perform tasks for a particular fee, often pennies for each task. Using an architecture that is conceptually similar to cloud computing, you can scale the task up to as many volunteers as you need.

The concept of crowdsourcing is a good fit for VGI, particularly when a vast amount of data must be collected under time pressure; however, not all VGI projects use crowdsourcing. Some of them are focused on gathering information from a small sample of people or a focused group of domain experts. Cinnamon and Schuurman (2012), for example, enlisted a set of emergency medical professionals at a single hospital to submit information about the locations of local auto accidents. Using tablet computers, the paramedics tapped the screen or typed an address to record the locations of the accidents. The researchers called this type of guided process facilitated VGI (f-VGI), after Seeger (2008). These readings are available in the Lesson 9 module on the course Canvas site if you're interested in learning more about them.

The human factor

The introduction of humans into the sensory element of data collection presents some interesting advantages and challenges. One advantage is that there are a lot of humans potentially available. Some of them even appear to have a lot of time on their hands! This means that tasks can be scaled up quickly and the data can be collected (or corrected) in a hurry. Humans also have the ability to care about projects and become passionate about them, increasing the amount and quality of data collected and creating an endless source of free organization and labor. It's not always necessary to hire the Mechanical Turk when you're enlisting people in a project they really believe in.

However, humans, by nature, make mistakes in some ways that computers may not. They get tired, they commit typos, they make subjective judgments, and so forth. Furthermore, the technical skills and physical infrastructure (e.g., Internet access) required for VGI participation may not be uniformly distributed throughout your study area. Finally, humans carry particular biases and interests that may skew the types of data collected.

Anyone employing VGI in scientific research or mission-critical applications should be aware of these limitations. The next section of this lesson provides some examples of how these advantages and limitations of VGI have affected OpenStreetMap.

References

  • Cinnamon, J., & Schuurman, N. (2013). Confronting the data-divide in a time of spatial turns and volunteered   geographic information. GeoJournal, 78(4), 657–674.
  • Goodchild, M. F. (2007). Citizens as sensors: the world of volunteered geography. GeoJournal, 69(4), 211–221.
  • Seeger, C. J. (2008). The role of facilitated volunteered geographic information in the landscape planning and site design process. GeoJournal, 72(3-4), 199–213.

OpenStreetMap and its use as open data

OpenStreetMap (OSM) is a digital map database of the world built through crowdsourced volunteered geographic information (VGI). OSM is supported by the nonprofit OpenStreetMap Foundation [8]. The data from OSM is freely available for visualization, query, download, and modification under open licenses [9].

OSM works in a style similar to Wikipedia, in which virtually all features are open to editing by any member of the user community. OSM was conceived in 2004 and has grown to over two million registered users [10] since that time. Although only a fraction of these are frequent map editors, the map has matured enough in some locations to the point where its detail and precision rival "authoritative" datasets from governments and commercial entities. This is particularly true in Western Europe and some parts of the US. The image below of the Penn State campus provides an idea of the intricate features that can be submitted to OSM.

 OpenStreetMap showing part of the Penn State campus
Figure 9.1

OSM originally gained popularity in places where government data was not freely available, but a thriving GIS community existed. For example, in the mid-2000s, the UK Ordnance Survey data was available only for purchase, and OSM grew rapidly as an attractive free alternative. In places where governments were willing to freely share their data, bulk upload negotiations were sometimes arranged. For example, the US has fairly thorough road coverage due to a US Census TIGER street data bulk upload.

OSM volunteer efforts constitute a social event and hobby for many, who gather for group data collection events known as "mapping parties." These activities organized armies of volunteers to walk, bike, and drive through sectors of a city with GPS units and notepads, returning later to a central lab to enter the data (Perkins and Dodge 2008). Although this is still useful in cases, nowadays many OSM beginners can get pretty far just through tracing aerial photographs in simple browser-based editors. In addition to a physical exploration of the city, mapping parties now offer training, awareness, and renewed enthusiasm of OSM (Hristova et al. 2013).

How OSM features are contributed and tagged

To contribute a feature to OSM, you typically digitize a geometry (a point, line, or area) and then add descriptive attributes, or tags. For example, to tag a grocery store, you trace its building footprint and tag it with shop=supermarket. There's no restriction on the tags you can use, but the data is only useful to the degree that you tag things consistent with the way other OSM users have applied the tags.

To promote consistency in tagging, the OSM community has an informal tag voting and approval process organized on the OpenStreetMap wiki [11] site. Approved tags are added to the online documentation so that others can easily find and apply them. For example, the tag shop=supermarket [12] denotes a grocery store. Before you add a tag, check the wiki to make sure that you're using the established tag and syntax. If you create your own tag and start using it on many features, consider putting it through the OSM proposal process [13].

Benefits and weaknesses of OSM

OSM is not the only crowdsourced VGI project, but it is one of the most well known. As such, it provides a useful exemplification of the pros and cons of crowdsourcing and VGI.

Some of the main benefits of OSM include:

  • There is no cost to use the data. In some locales, OSM may be the only freely available source of high-resolution GIS vector data. In other places, it may be the only source of data.
  • The source data is available for download and use in derived cartographic products. Here OSM differs from Google's similar crowdsourced map called Map Maker. Cartographers cannot download and re-use Google Map Maker data; whereas OSM is available to anyone with enough technical know-how to get the data.
  • Because OSM allows people to add any type of feature, it may include a richer and more socially valuable set of features than commercial or government maps. Possibilities include trees, wheelchair ramps, food banks, spigots for potable water, and so forth.
  • OSM data is flexible and can quickly be updated in the event that a new business opens, a bridge gets washed away, etc. In contrast, commercial and government maps tend to be updated on fixed cycles.

Some of the main challenges of OSM include:

  • There is no systematic quality check performed on the data. You use the data at your own risk and should typically avoid relying on OSM information for mission critical functions unless no other dataset is available.
  • The detail, precision, and accuracy of OSM coverage varies across space, without a simple means of detecting the variation. In the global South, some cities are missing basic street data, let alone other useful features such as parks, schools, and civic buildings.
  • OSM is subject to contributor biases. Each contributor must make decisions about the types of features he or she will place on the map and the places he or she will map. Looking at the map of any given place on openstreetmap.org, we have little idea of who created the map and why. The browser-based tool Crowd Lens for OpenStreetMap [14] attempts to offer a window into the crowd that created OSM in a particular place. It shows that some places have garnered much more attention than others.
  • The OSM community decides the types of features that are worthy of the community tagging system. This is accomplished through a semi-formal proposal and voting process, but it is a process that tends to reflect the interests of the contributors. In 2013, Stephens lamented that there were multiple tags for marking sexual entertainment venues in OSM, whereas proposals for tags denoting hospice services and daytime child care had floundered. She attributed this directly to the fact that a significant majority of OSM contributors are male. Recent speakers at OSM conferences have raised attention [15] to the consequences of gender imbalance in OSM, and have provided suggestions of how to make the OSM community more diverse.
  • Unless you're just viewing the default map tiles, getting a focused set of data out of OSM often takes a lot more technical skill than getting the data in. Data ingress and retrieval techniques are covered in the lesson walkthrough and assignment, where you can judge this point for yourself.

Uses of OSM

The most basic use of OSM is to retrieve its map tiles as a background for other thematic layers. High-profile sites using OSM in this way include Foursquare, Craigslist, and Wikipedia. Some web developers switched to OSM as a basemap [16] after the Google Maps API introduced potential fees into its terms of service.

From a technical perspective, anyone can use a rendering engine like Mapnik to draw tiles of OSM data. In fact, this is what you did in the Lesson 5 walkthrough. The image below shows how you can select various basemap renderings on OpenStreetMap.org. Other companies such as Mapbox [17] have made their own OSM renderings that can be consumed as web services. In fact, Mapbox's business model has come to rely so heavily on OSM that the company has invested in near-real-time quality monitoring of incoming OSM edits, a process you can view through their online OSM Changeset Analyzer [18] tool.

 Renderings available on OpenStreetMap.org
Figure 9.2

Let's now take a look at some of the ways OSM can be used "beyond the basemap."

Crisis response

OSM gained publicity as a disaster response aid in 2010 after the Haiti earthquake. Prior to this disaster, publicly available digital data for Haiti was sparse, and OSM was limited to major roads and a handful of other features. In the weeks following the earthquake, Internet volunteers worldwide traced imagery and referenced out-of-copyright maps to create a detailed geographic database of the country in OSM. This provided helpful basemaps for humanitarian aid workers who were flocking to the country and needed maps to get around. It also served as an inventory of hospitals, churches, civic facilities, and other resources that could be used by responders.

The growth of OSM during this period was nothing short of dramatic, and a number of animations such as this video: OpenStreetMap - Project Haiti [19] have depicted the expansion of the map in Haiti during this time period. Zook et al (2010) offer an analysis of various methods of VGI and crowdsourcing used in the earthquake response, including OSM and the crisis mapping site Ushahidi.

Crowdsourced volunteer efforts work most efficiently when there is an organizing force behind the work. Using lessons from the Haiti experience, the Humanitarian OpenStreetMap Team [20] (HOT) now provides this function. After Typhoon Haiyan hit the Philippines in 2013, HOT provided tools to explain and partition the volunteer mapping work on OSM so that the most needed features and geographic areas were given priority. Volunteers visiting the HOT site could click a map sector to work on, and were given instruction about which features to trace and how to tag them. The image below shows the OSM Tasking Manager, an application used by HOT to catalog sectors completed and sectors that need work.

 Screen capture of the OSM Tasking Manager
Figure 9.3

Providing a presence for unmapped or undermapped areas

The efforts to rapidly assemble crisis mappers in Haiti and the Philippines are admirable, but the ideal situation would be to already have the OSM data on hand. These regions only needed the mapping because sufficient information hadn't been contributed in the first place. Lack of technical infrastructure, a shortage of human and monetary capital, civil restrictions, and other factors can cause places to remain unmapped. Graham (2010) calls these places "virtual black holes" in VGI. Unfortunately commercial Internet maps may also neglect these places if it is believed the search and advertising functions related to the map will not produce sufficient revenue to justify the investment.

OSM has been used as a way to give a presence to communities that have previously remained unmapped. Hagen (2010) describes a project in Nairobi wherein local youth volunteers were enlisted and trained to map the sprawling slum of Kibera. Home to hundreds of thousands of people, this settlement was little more than a name on previous maps. The Map Kibera [21] effort used OSM tools to record water points, toilets, clinics, schools, pharmacies, places of worship, and NGO offices. The result is a map that the residents can use to find local services and lobby the government for infrastructure support. The features added through this project are immediately apparent when you navigate to Kibera using even the default OSM map.

 Kibera, Nairobi, Kenya in OpenStreetMap
Figure 9.4

Similar stories can be found elsewhere in the world. When participants in a Buenos Aires hackathon wanted to map social services in a local slum, they found the area empty in commercial maps and decided to use OSM as a basemap [22]. Even when a street network exists, other layers such as bus routes may be helpful for individuals without automobiles, opening possibilities for local travel outside of daily routines. Motivated individuals have headed up an OSM project with bus routes in India [23], noting that a detailed local map can also help with tourism promotion efforts.

Thematic mapping for the social good

One of the advantages of OSM is its flexibility to store any type of feature, given the many tags that already exist and the community-based tag proposal and voting process. In some cases, specialized thematic maps have been created around a subset of feature types. Examples of these include:

  • OpenCycleMap [24], specializing in drawing bicycle trails that people have submitted to OSM
  • OpenSkiMap [25], showing ski lifts and trails
  • Wheelmap [26], allowing the browsing and marking of wheelchair-accessible locations
  • Philly Fresh Food Map [27], displaying urban farming resources and fresh food distribution outlets in Philadelphia, Pennsylvania. (This should look familiar!)

In these maps, OSM acts as a freely accessible repository for local knowledge of useful things. Some of these mapped features provide great value to a community, but are not monetarily lucrative and may be excluded from proprietary commercial maps. Even the default OSM tiles do not show all the above types of features because to do so would cause the map to be cluttered. There is a great need for developers who can retrieve custom subsets of data from OSM and display it in thematic maps.

 Philly Fresh Food Map
Figure 9.5

Remember that thematic maps are only possible because OSM allows free download and re-use of the data. Sites that use OSM for thematic mapping often rely on one of the various query APIs available for OSM, such as the Overpass API that allows the submission of custom tag queries through a web service. Asking a web service to give you all features matching a certain tag is often more manageable than downloading the entire OSM dataset for a region. You will get a taste of the Overpass API in the lesson walkthrough.

The maps and queries depend heavily on users maintaining consistency with established tag syntax. For example, the Philly Fresh Food Map relies on tags described in the Food Security [28] page of the OSM wiki.

References

  • Graham, M. (2010). Neogeography and the palimpsests of place: Web 2.0 and the construction of a virtual earth.Tijdschrift Voor Economische En Sociale Geografie, 101(4), 422–436.
  • Hagen, E. (2010). Putting Nairobi’s Slums on the Map. Development Outreach | World Bank Institute, 41 – 43.
  • Hristova, D., Quattrone, G., Mashhadi, A., & Capra, L. (2013). The life of the party: Impact of social mapping in OpenStreetMap. In Proceedings of the AAAI International Conference on Weblogs and Social Media (ICWSM2013). Retrieved from http://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/download/6098/6362 [29]
  • Perkins, C., & Dodge, M. (2008). The potential of user-generated cartography: a case study of the OpenStreetMap project and Mapchester mapping party. North West Geography, 8(1), 19–32.
  • Stephens, M. (2013). Gender and the geoweb: Divisions in the production of user-generated cartographic information. GeoJournal, 78(6), 1–16.
  • Zook, M., Graham, M., Shelton, T., & Gorman, S. (2010). Volunteered geographic information and crowdsourcing disaster relief: a case study of the Haitian earthquake. World Medical & Health Policy, 2(2), 7–33.

Walkthrough: Getting source data from OpenStreetMap

Getting data out of OpenStreetMap (OSM) presents more technical challenges than putting data into OSM. When you put data into OSM, you can use your choice of a number of different types of editors. You can use any tags that you want, attempting to stick to tagging conventions of course.

In contrast, when you get data out of OSM, you have to deal with the following:

  • Retrieving only the tags you need
  • Retrieving the data format you need
  • Not overwhelming yourself or the server by requesting too much data

Complicating matters is the fact that OSM returns data in its own structure of XML, which is not immediately readable by many GIS applications. Therefore, getting data from OSM often involves converting from this XML into some other format.

There are a variety of mechanisms for downloading OSM data. The easiest ones address the challenges by providing a way to filter the tags you want, allowing you to specify the output format, and allowing you to specify a geographic bounding box for the requested data, so you don't retrieve too much.

One of the most user-friendly GUI-oriented ways that I have found for retrieving OSM data is a server at BBBike.org [30] (http://extract.bbbike.org [30]). This little web-based tool allows you to draw a bounding box interactively and specify the output format you want. After a while, you receive an e-mail with a link to download your data.

 OSM downloads from BBBike extract service
Figure 9.6

In the walkthrough, however, we'll use the OSM download mechanism that is build directly into QGIS. Although this way is a little more advanced than the BBBike extract service, it is more immediate and allows greater flexibility for the amount of data and tags selected.

Downloading OSM data using QGIS

Examine the image below of Cayenne, French Guiana. You'll notice that the city has detailed building footprint polygons available. Let's suppose that we want to get a shapefile of these building footprints using QGIS.

Map of buildings in Cayenne
Figure 9.7

Note that we have defined our three pieces of essential information to filter the OSM data we want:

  • The tags we want: any polygon with the building tag populated as anything other than building=no (a somewhat rare value but one that is occasionally used)
  • The format we want: a shapefile
  • The bounding box of data we want: just the city of Cayenne

Follow these steps to get the data using QGIS:

  1. Create a new data folder such as c:\data\Cayenne.
  2. Launch QGIS and click Vector > OpenStreetMap > Download data.
  3. Choose Manual and specify the bounding coordinates of the area you want to download. In this case, use the bounding coordinates of Cayenne, which are shown below.
     Download OpenStreetMap data dialog box in QGIS
    Figure 9.8

    When doing this, be careful that you don't specify a bounding box larger than you need, or you could end up with an extraordinary amount of data.

    The bounding coordinates must be supplied in WGS 1984 lat/lon format or the tool will not work. It may take a bit of detective work to figure out these coordinates before you launch QGIS.

  4. Specify the Output file with a name such as
    cayenne.osm
    as shown above and click OK. Wait while your data is downloaded. At the time of this writing, the size was around 23 MB.
    If the download fails in the middle, delete your .osm file and try it again.
  5. Click Close to close the download window.
    You currently have a .osm file containing XML. You will now convert this into a SpatiaLite database that can be used in QGIS and other programs.
  6. Click Vector > OpenStreetMap > Import topology from XML and fill out the dialog box as shown below.
     OpenStreetMap Import dialog box
    Figure 9.9
  7. Click OK and wait for the import to occur. Then click Close to close this dialog box.

    You've now brought the data into a SpatiaLite database, but now you need to create a useful layer out of it with just the geometries and tags of interest.
     
  8. Click Vector > OpenStreetMap > Export topology to SpatiaLite.
  9. Complete the dialog box as follows:

    - For Input DB file, browse to your
    cayenne.osm.db
    SpatiaLite file. Be aware that the .db extension may not be visible in Windows Explorer, but if the file shows up in the file browser dialog, then you are okay.
    - For Export type choose Polygons (closed ways)
    - For Output layer name use
    cayenne_polygons

    - For Exported tags, click Load from DB and then check some tags pertinent to buildings. In our scenario, check source, building, amenity, addr:housenumber, and addr:street.
     Export OpenStreetMap topology dialog box
    Figure 9.10
  10. Click OK and then click Close to close the dialog box. You should now see a layer in QGIS containing all the polygons. If the layer is not added automatically, you can do so manually by using the Add SpatialLite Layer button, connecting to cayenne.osm.db, selecting the cayenne_polygons table, and then clicking on Add.
     Cayenne OSM polygons in QGIS
    Figure 9.11
    Now you need to select only the building polygons.
  11. In the map table of contents, right-click cayenne_polygons and click Open Attribute Table.
  12. At the top of the attribute table, click the Select features using an expressionExpression button button.
  13. Paste the following query in the Expression box including all quote marks: "building" != 'NULL' AND "building" != 'no'
     Select buildings using an expression
    Figure 9.12
    This expression filters out everything that's not a building. When you do this with your own data of interest, you'll need to create some expression that selects only the tag combinations that you want.
  14. Click Select. You should see the building features selected in the map.
  15. In the map table of contents, right-click the cayenne_polygons layer and click Save As...
  16. Choose Esri shapefile as the format and specify an output location. Select the Save only selected features option. Then click OK.
     Save selection as shapefile
    Figure 9.13
  17. Use QGIS to verify that your exported shapefile contains only the buildings.
     Final view of Cayenne buildings in QGIS
    Figure 9.14

Downloading data using the Overpass OpenStreetMap query API

Behind any data retrieval mechanism from OSM is a web service request. You can send these requests directly from your web browser or an automated program using an OSM query API. One of the most powerful of these APIs is called Overpass [31]. Try the following:

  1. Paste the following URL in a web browser and wait for a minute until prompted to save a file:
    http://www.overpass-api.de/api/xapi_meta?*[building=yes][bbox=-52.35,4.88,-52.25,4.98]
    Notice what this is requesting...It should look familiar.
  2. When prompted to save the file, save it as buildings.osm.
  3. Open buildings.osm in a text editor and see what all the buildings in Cayenne look like when expressed as OSM-formatted XML.

You can use Python or other scripting languages to make these requests automatically. For example, here's how you could use Python to query OSM for all the farmers' markets in Philadelphia and save them to a .osm file. (You're not required to run this code).

import urllib
	
workspace = "C:\\data\\OSMdev\\"
	
# Make data queries to jXAPI
marketsXml = urllib.urlopen("http://www.overpass-api.de/api/xapi_meta?*%5Bshop=farm%5D%5Bbbox=-75.29,39.86,-74.95,40.15%5D").read()
	
# Make farmers markets file
marketsPath = workspace + "markets.osm"
marketsFile = open(marketsPath, 'w')
marketsFile.write(marketsXml)
marketsFile.close()

For Python junkies: The above code uses a library called urllib which is able to make web requests and read the responses. You just have to provide the URL for the request. So as not to be interpreted as defining a list, the "[" and "]" characters are escaped using the %5B and %5D sequences, respectively, but otherwise the query has the same syntax as the one you issued above for Cayenne buildings. The resulting XML is then written to a file using the standard Python write method.

A script like this might be useful if you wanted to update one or more datasets on a periodic basis. The script could be combined with GDAL processing to get the data into a format suitable for your web map. Recent versions of GDAL (1.10 and later) can read OSM XML and convert it to different formats, such as GeoJSON or shapefiles. (Be careful with shapefiles though, because GDAL plops most of the less common "other tags" into one field that gets cut off at 256 characters, a limitation of the shapefile format).

As an exclamation point at the end of all this geekiness, play around with the graphical tool overpass turbo [32] for a few minutes. This gives you an interactive environment for querying OSM and seeing the results on the map. You can save any interesting result in popular formats, such as KML. This is helpful if you just want to make a one-off query to OSM for some particular feature type.

There are many circumstances and needs that can affect the way you retrieve data from OSM. Hopefully, this walkthrough has provided enough options that you can make an informed decision about how to best get the scope and scale of data you need. Now let's go to the lesson assignment where you'll get some experience with the other side of things: putting data into OSM.

Lesson 9 assignment: Evaluate OpenStreetMap usage and contribute to OpenStreetMap

The Lesson 9 assignment has two parts: reporting on a web map that uses OpenStreetMap (OSM), and actually editing OSM yourself. You will produce a single document describing these efforts.

Reporting on a web map that uses OSM

Find an Internet map that uses some element of OSM. Produce a writeup of several paragraphs describing the following:

  • What is the purpose and URL of the site, and who built it?

  • How is OSM being used? (i.e., Is the site simply pulling the OSM tiles, or is the source data used for creating thematic layers, etc.?)

    • Include at least one screenshot showing the OSM data.

  • What advantages and disadvantages are introduced into this map by using OSM data?

  • Do you see any other appropriate ways that OSM data could be used in this site?

Editing OSM yourself

In this part of the assignment, you'll get some practice with adding data to OSM in your town or some other place that you know well. You'll take some "before" and "after" screenshots to demonstrate the things you added to the map.

The easiest way to get started with editing OSM is using the in-browser editor at OpenStreetMap.org [33], which is called iD.

  1. Visit OpenStreetMap.org [33] and register for an account. This requires creating a name and password and supplying your e-mail address (so you can prove you're a person and not a robot).
  2. After you've created the account, return to OpenStreetMap.org and log in. Do not use Internet Explorer for this part of the exercise, because it cannot display the iD editor.
  3. Navigate to your hometown or another place you want to edit and click the Edit dropdown button. If asked which editor you want to use, choose iD.
  4. Click Start the Walkthrough and carefully follow all instructions to complete the iD training.

    ID includes a walkthrough for beginners that gives you some hands-on practice with tracing and tagging features. Thus, full editing instructions for using iD are not included in this lesson. Follow the walkthrough, and you should have the basics down.
  5. After completing the walkthrough, scan the features available in your hometown and identify some things you want to add. The easiest way to start is probably by tracing a building. Choose to create an Area and trace around a building that you see in the imagery.
     Tracing a building in iD
    Figure 9.15
    The first thing to do is tag it with building=yes. Also, if the building has a name, supply the name tag.
     Placing tags on a building
    Figure 9.16
    Now you can provide other tags further specifying the purpose of the building, if these are known. An appropriate tag to add in the above image might be amenity=restaurant.
    You can optionally supply addresses for features, but this involves several OSM tags. The easiest way to do it in iD is to fill in the Address form. This ensures all the address parts get the correct tags.
     Address field editor in iD
    Figure 9.17
    Adding addresses is not required for this assignment.
  6. Continue with your assignment by adding (or modifying) a total of at least 7 features, making sure the following criteria are satisfied:
  • Five different types of features must be represented. In other words, don't just trace 7 building footprints. Use the OpenStreetMap wiki to learn the correct way to tag different types of features. You can also use the left hand Search menu in iD to get hints about tags, but, during this assignment, you should verify that the tags placed by iD match what you intended to map as described in the OpenStreetMap wiki documentation.
  • Point, Line, and Area features must be represented. You can satisfy this requirement by adding new lines or adjusting existing lines. For example, sometimes roads need more detail to produce smooth curves, line up with the imagery, etc.
  • Before you add the features, go to the main OSM page at OpenStreetMap.org and take some screen captures.
    If you have trouble thinking what to add, consider mapping parks, schools, churches, restaurants, civic buildings, clinics, ponds, wetlands, etc. New subdivisions are also a great opportunity for mapping roads. You can get more ideas by looking around OSM in a city that's been mapped in detail, such as Seattle or State College. Another way to get good ideas is by walking or biking around your town.
  1. Produce a list of the features you edited and add it to your report that you created in the first part of the project. Include a list of all tags that you placed or edited on each feature. These tags must comply with the community-approved documentation on the OpenStreetMap wiki.
  2. Wait for about 10-15 minutes after your edit session and then go to OpenStreetMap.org and take some screen captures of the new features. Note that OSM updates their tile levels at different times, so your feature may only be visible at certain tile levels. Choose any tile level that works for the screenshot. You don't have to supply 7 screenshots; just provide enough to show some of the "before" and "after" effects of your edits.
  3. Paste the screenshots in your report.

Deliverable

Submit a single document containing the two assignment parts above to the Lesson 9 assignment drop box on Canvas.

 


Source URL: https://www.e-education.psu.edu/geog585/node/735

Links
[1] http://discovery.ac.uk/files/pdf/Licensing_Open_Data_A_Practical_Guide.pdf
[2] http://video.esri.com/iframe/3138/000000/width/480/0/00:00:00
[3] http://wikipedia.org
[4] http://www.cnn.com/2014/03/11/us/malaysia-airlines-plane-crowdsourcing-search/
[5] http://www.oldweather.org/
[6] https://familysearch.org/indexing/
[7] http://aws.amazon.com/mturk/
[8] http://wiki.osmfoundation.org/wiki/Main_Page
[9] http://www.openstreetmap.org/copyright
[10] https://blog.openstreetmap.org/2015/03/12/two-million-contributors/
[11] http://wiki.openstreetmap.org/wiki/Main_Page
[12] http://wiki.openstreetmap.org/wiki/Tag:shop=supermarket
[13] http://wiki.openstreetmap.org/wiki/Proposal_process
[14] http://sterlingquinn.net/apps/crowdlens
[15] http://lanyrd.com/2013/sotm/scphhf/
[16] https://www.techdirt.com/articles/20120405/17321218398/google-maps-exodus-continues-as-wikipedia-mobile-apps-switch-to-openstreetmap.shtml
[17] https://www.mapbox.com/data-platform/
[18] https://osmcha.mapbox.com/
[19] https://www.youtube.com/watch?v=BwMM_vsA3aY
[20] http://hot.openstreetmap.org/
[21] http://mapkibera.org/
[22] http://blog.ilabamericalatina.org/2013_06_01_archive.html
[23] http://bitterscotch.wordpress.com/2010/04/29/mapping-a-new-way-forward-for-openstreetmap-in-india/
[24] http://www.opencyclemap.org/
[25] http://openskimap.org/
[26] http://www.wheelmap.org
[27] http://www.geovista.psu.edu/phillyfood/
[28] http://wiki.openstreetmap.org/wiki/Food_security
[29] http://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/download/6098/6362
[30] http://extract.bbbike.org
[31] http://wiki.openstreetmap.org/wiki/Overpass_API
[32] http://overpass-turbo.eu/
[33] http://www.openstreetmap.org