This is the first part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML for a given boundary in OpenStreetMap. Update: the second part is now available.
As mentioned in a previous post, we’re looking at ways of making it smoother to deploy FixMyStreet for a new country, city or use-case. Essentially there are two fundamental bits of data that you need for this:
- a mapping between a latitude and longitude (or postcode) to all the adminstrative areas that cover that point.
- a mapping between problem type and adminstrative areas to an appropriate email address for reporting that problem.
The first of those is typically provided by a service called MapIt, an open source GeoDjango web application written by Matthew Somerville. In the UK we are fortunate that official boundary data and the postcode database have now been released under an open data license from the Ordnance Survey. However, in other many other countries similar data is unavailable, or not available under reasonable licensing conditions. In such cases, though, all is not lost thanks to the extraordinary work of contributors to the OpenStreetMap project. OpenStreetMap contains high-quality administrative boundary data for many countries of the world, and we know that data submitted to the project is available under the Creative Commons Attribution-ShareAlike license, so we can reuse it in a web service like MapIt.
The first step towards being able to build an instance of MapIt based on OpenStreetMap boundary data is to be able to generate a shapefile that represents a boundary in the project. (In OpenStreetMap’s data model, boundaries are represented as either ways or relations, and those that we are interested in are tagged with boundary=administrative.) Matthew had previously written code to generate KML files for boundaries in Norway in order to help to set up an instance of MapIt for Norway, which is used by FiksGataMi. However, that script was quite specific to the organization of boundaries in that country, and it did not deal with more complex boundary topologies (e.g. enclaves), or different representations of boundaries (e.g. multiply nested relations).
So, Matthew and I wrote a new version of the code to extract a boundary from OpenStreetMap and generate a KML representation of it. The new version uses the Overpass API instead of XAPI, since it allows us to specify multiple predicates in the query and recursively fetch the ways and nodes that are contained in a relation. Once all the ways that make up a relation have been fetched (ignoring those with roles like “defaults” or “subarea”), the script tries to join each unclosed way to any other with which it shares an endpoint. We should end up with a series of closed polygons – the script exits in error if there are any unclosed ways left. We can then directly create KML from these polygons, the only subtlety being that we need to mark certain boundaries as being an inner boundary (i.e., creating a hole in a boundary) if they had the role “enclave” or “inner” in an OpenStreetMap relation. For example, the South Cambridgeshire District Council boundary has a Cambridge City Council-shaped hole in it:
Similarly, the script has to cope with multiple distinct polygons, such as the boundary of Orkney.
If you want to use this code to generate a KML representation of a closed way or boundary relation from OSM, just clone the MapIt repository and run bin/boundaries.py:
$ bin/boundaries.py --help Usage: boundaries.py [options] Options: -h, --help show this help message and exit --test Run all doctests in this file --relation=<RELATION_ID> Output KML for the OSM relation <RELATION_ID> --way=<WAY_ID> Output KML for the OSM way <WAY_ID>
For example, to generate a KML boundary for the Hottingen area of Zürich, you can do:
$ bin/boundaries.py --relation=1701449 > hottingen.kml
In the next blog post in this series, we will discuss extracting such boundaries en masse and creating a service based on them.
Would be awesome to have a global tile set with only admin boundaries…
I am looking for part 2 – has this been published already?
Hi Derick: Apologies, I haven’t written the second part yet – unfortunately, I had to move to work on other projects shortly after we launched MapIt Global. However, I’ll I’ll make sure I get the post done next week. This might also be of interest, in the meantime: http://www.mysociety.org/2012/07/05/new-mapit-global-an-administrative-boundaries-web-service-for-the-world/
Hi Derick: the second part is up now – sorry about the delay!
awesome, its something Im looking for, will definitely give it a try. Thanks!
Hello, and thanks for all this information.
When I try to run the example:
$ bin/boundaries.py –relation=1701449 > hottingen.kml
I receive the following error:
raceback (most recent call last):
File “bin/boundaries.py”, line 2160, in
kml, bbox = get_kml_for_osm_element(element_type, element_id)
File “/Users/thiagorpp/Documents/mapit/bin/generate_kml.py”, line 385, in get_kml_for_osm_element
e = fetch_osm_element(element_type, element_id)
File “/Users/thiagorpp/Documents/mapit/bin/boundaries.py”, line 1874, in fetch_osm_element
parsed = parse_xml(filename, fetch_missing)
File “/Users/thiagorpp/Documents/mapit/bin/boundaries.py”, line 1828, in parse_xml
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/__init__.py”, line 33, in parse
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/expatreader.py”, line 107, in parse
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/xmlreader.py”, line 125, in parse
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/expatreader.py”, line 220, in close
self.feed(“”, isFinal = 1)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/expatreader.py”, line 214, in feed
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/handler.py”, line 38, in fatalError
xml.sax._exceptions.SAXParseException: /Users/thiagorpp/Documents/mapit/bin/../data/new-cache/relation/449/relation-1701449.xml:1:0: no element found
Hi Thiago, could you check the contents of the file /Users/thiagorpp/Documents/mapit/bin/../data/new-cache/relation/449/relation-1701449.xml ? It sounds as if somehow that file may have been created empty. You could try deleting it and rerunning the command, to see if that makes a difference.
As a further comment, this typically happens when the execution of osm3s_query fails (e.g. because it’s not on your PATH) – that will leave an empty file in the cache https://github.com/mysociety/mapit/commit/b939d1b06280996c965cd02d4767b1ff34ad8ea1 This is a bug, which I’ll fix as soon as I have time https://github.com/mysociety/mapit/issues/107
If you want to use a remote overpass server instead of overpass installed locally, you could cherry-pick this commit into your version: https://github.com/mysociety/mapit/commit/b939d1b06280996c965cd02d4767b1ff34ad8ea1