OpenStreetMap is a project that creates and maintains maps all over the world, putting them out as open data that anyone can use.
While many additions are made by on-the-ground volunteer contributors, the input of other data sources allows OpenStreetMap to make leaps and bounds in its coverage, as you might imagine. But using such data is only possible if it can be reused within the terms of OSM’s share-alike open data licence, the ODbL.
And that’s where we pick up the story of Robert Whittaker, who used WhatDoTheyKnow in the hope of augmenting the OpenStreetMap offerings for Cambridgeshire, UK.
Rights of way
Robert saw a chance to add better data on public footpaths, bridleways, and byways in the county to OpenStreetMap. He explains the background:
“Councils have a legal duty to maintain an official list and physical map of rights of way, but most councils — including Cambridgeshire — also maintain an unofficial digital map as well. It was the underlying data behind the digital map that I was after.”
Not just for OpenStreetMap, though — the project’s reuse policy means that once they’ve put the data in place, it’s available for others, too.
“Having this data — and the right to re-use it — will allow people to create their own maps of the Rights of Way and mix the data with information from other sources. This would then allow, for example, routing software to plan walks using Public Rights of Way and other roads.
“Cambridgeshire was one of the few councils, until recently, that was not making the data freely available.”
The right to ask
So, how do you go about obtaining something like this? If you’re familiar with Freedom of Information or its close neighbour EIR (Environmental Information Regulations), they provide an obvious route, as these pieces of legislation provide us all with the right to request data from public authorities. Robert was very familiar:
“I’ve made quite a few FOI and EIR requests over the years, mostly through the excellent WhatDoTheyKnow.com. A lot have been for data that will be useful to OpenStreetMap mappers, but I’ve also made requests to gain information about the workings of public authorities, either to inform campaigns, increase transparency, or expose poor decision-making.
“I think the first FOI request I sent personally was in 2006 to my university to ask for the specification and testing details for an out-sourced student-facing web-app that had a particularly poor user interface. It was to inform a campaign by the Student Union to get improvements made.”
With this experience in his background, EIR and FOI were the natural routes for Robert in obtaining this data. He made three requests: first, asking for the GIS data, then, to request permission for its reuse; and finally for the related written descriptions.
The right to refuse
Unfortunately, the requests did not go as smoothly as he might have hoped. That first request was back in August 2014, and if you read through the stream of responses and annotations, you’ll see that Robert experienced almost the full set of obstacles that can get in the way of an FOI response — from the council simply not responding in time, to their responding with only parts of the data he had asked for, and citing rules which didn’t apply to the situation in hand.
It’s a good thing that Robert is both well-informed and tenacious, as surely these hurdles would have proved discouraging, if not completely off-putting, to many requesters.
Much of his argument pivoted around a specific exemption — a clause which allows an authority not to provide data under certain circumstances, in this case, the enticingly named EIR 6(1)(b).
“EIR 6(1)(b) allows public bodies to refuse to provide information in a specific form or format, if it’s already publicly available and easily accessible in another form.
“The council argued that because they had an online map available on their website, the information about the rights of way was already available and so 6(1)(b) meant they could refuse to release the underlying data.
“I successfully argued that the map was only a summary or approximation of the underlying data I’d requested. That data contained the actual coordinates of the points and the lines joining them to make up the routes. I think one of the key arguments was that given the data you could generate the map, but given the map you could not recreate the full underlying dataset, you could only obtain an approximation to it.”
The (almost) right outcome
Robert was ultimately successful in his first two requests, two and a half years after making that initial request. The third is still being contested.
“It’s been frustrating, but eventually worthwhile. I’m annoyed at how long it has taken to get to the end, and also annoyed at the public money that the Council has wasted in prevaricating and trying to withhold the information.
“I think the ICO probably needs more resources to be be able to investigate cases more promptly. I also think it should take a stricter line with public authorities that frustrate requesters or the ICO’s investigations. The ICO already has some additional powers that would help here, but they seem reluctant to use them, even though doing so could speed things up significantly.”
But even while we await the outcome of the final request, Robert’s patience has already begun to pay off:
“I’ve already loaded the data into my comparison tool to help mappers improve OSM. Also, thanks to Barry Cornelius, the Cambridgeshire data is now available from his site in a number of different standard formats, for anyone else who wants to use or view it.”
We run WhatDoTheyKnow so it’s easy for anyone to make an FOI or EIR request — and your contributions help us carry on doing so.Donate now
All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.
Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.
As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.
Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.
We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.
We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.
The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.
From Britain to the World
For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.
MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.
If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.
About the Data
The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.
The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global. We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.
If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.
Commercial Licenses and Local Copies
MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.
Last week, FixMyStreet gained a number of new features that we hope you will find useful.
Firstly, we’ve thrown away our old maps and replaced them with new, shiny, zoomable maps. This should make it easier for people to find and report problems, especially in sparser locations. We’re using the OS StreetView layer (hosted internally) when zoomed in, reverting to Bing Maps’ Ordnance Survey layer when zoomed out, as we felt this provided the best combination for reporting problems. In urban areas, you can still see individual houses, whilst in more rural areas the map with footpaths and other such features is probably of more use. FixMyStreet tries to guess initially which map would be most appropriate based upon population density, meaning a search for Stroud looks a bit different from that for Birmingham.
OpenStreetMap fans, don’t worry – as part of our mapping technology upgrade, you can now use osm.fixmystreet.com to access your favourite mapping instead.
Secondly, we now have user accounts. We’ve rolled these out alongside our current system of email confirmation, and it’s up to you which you use when reporting a problem or leaving an update. This means that those who come to the site one time only to report a pothole can continue to do so quickly, but have the option of an account if they want. Having an account means you no longer have to confirm reports and updates by email, and you have access to a page listing all the reports you’ve made through FixMyStreet, and showing these reports on a (obviously new and shiny) summary map.
Other improvements include a much nicer All Reports section, so you can see all reports to Adur District Council on a map, paginated and with the boundary of the council marked – and individual wards of councils now each have their own pages too.
I’ll follow up this post with another, more technical, look at the maps and how they work, for anyone who’s interested 🙂
At mySociety we love our site FixMyStreet – it’s the epitome of a web tool that gives simple tangible benefits whilst generating a little accountability at the same time. Reports through the site were up 40% last year, so it’s clear that users quite like it too.
FixMyStreet has been copied in many different countries, which makes everyone in mySociety very happy, too, even apparently appearing in a slide deck the White House uses to show innovative services. However, it turns out that the cheerfully minimalist, almost wantonly unfashionable user-interface has an unfortunate down side: most people who copy the site look at it, think “That looks easy!” and then cheerfully start coding their own clone.
Alas – the very simplicity that makes the site good hides the fact that making a site like FixMyStreet really work well is actually way harder than it looks. What will you do when a government email inbox fills up? What about when administrative boundaries change, due to an election or restructuring? How do you know you’re not scaring users away with careless wording? All the hard-won lessons from these questions have been baked into the FixMyStreet codebase, and we’re only too keen to talk to people about them.
We were therefore particularly pleased when the Nowegian Unix User’s Group (NUUG) came to us to ask if we could help improve FixMyStreet to make it easier for them to install. Over the last month mySociety Senior Developer Matthew Somerville has been working hard with Petter Reinholdtsen and Christer Gundersen of NUUG, and here’s what they’ve managed in just a handful of weeks.
- The launch of a Norwegian FixMyStreet called Fiksgatami, covering nearly every corner of Norway’s 300,000 square kilometers.
- Problems reported anywhere within Norway will be correctly directed to any of the 400+ responsible municipalities, thanks to Petter and Christer’s amazing data sourcing skills.
- As a necessary side-effect of developing this, Norway now has a free, public administrative web service gazeteer – http://mapit.nuug.no. If Norway is anything like the UK this will soon become an indispensable service for many other web sites and mobile tools.
- The open source FixMyStreet codebase has been upgraded to make it easier to translate into other languages, easier to use different mapping with, and easier to install. These efforts will continue, as we realise this has been one reason why others have made their own versions.
- All this has been done without forking, so various major upgrades we have planned for the UK version will be exportable later in the year.
NUUG’s Fiksgatami is the epitome of what makes civic open source at its best so unmatchably good. It was developed incredibly quickly: just a month to create what is effectively a fully fledged, best-of-breed nationwide e-government service – albeit an unofficial one. Thanks to the hard work of the public servants who fix problem reports, it will make small but meaningful improvements to the lives of a lot of people in Norway. And it has made the free FixMyStreet codebase better and easier for other people to use to help them do the same thing in other countries.
I know that at mySociety we are all looking forward to working with NUUG again. And I hope that this story inspires others to look at our code, and to work with mySociety to make FixMyStreet a service that can help everyone who would benefit from it.