1. Where Next for Open Standards

    Earlier this week we hosted our Open Standards in Local Government workshop at Newspeak House in London, with the aim of unpicking where open standards might be of benefit and what might be stopping us from making more progress.

    We were joined by 20 smart people representing a bunch of local councils across the UK and it’s fair to say we made a good bit of progress. A number of consistent themes arose through our discussions.

    It was widely agreed that Open Standards are key to getting the basics right, and standardising the ability of different services to speak to one another is a prerequisite for a sustainable local authority service strategy. The insistence on compliance with open standards at the procurement stage should place an imperative on suppliers to build-in interoperability and reduce the fear of vendor lock in – councils shouldn’t inadvertently replace one set of closed systems for another.

    This link between adoption of open standards and the procurement process was fundamental.

    In our opinion demanding compliance from suppliers to agreed open standards up front, is probably the single most important thing that central government could do to help local government.

    Phil Rumens from LocalGovDigital introduced recent progress on the development of the Local Government Digital Standard. Notably, it goes further than the equivalent in central government, with an emphasis on reuse of existing data and services, and commitment to make more data open and reuseable.

    Both the LGA through LG Inform, and GDS via standards.data.gov.uk already look to gather standards for use in central and local government; however adoption by local government often lags substantially behind. Simply put this is a conversation that doesn’t really happen outside a small number of web or digital staff within councils, and the wider group of service staff don’t yet understand the opportunity that open standards represent.

    Indeed, Tom Symons from Nesta who introduced the Connected Council’s reporthighlighted that the councils furthest ahead are those that have both put in the hours to achieve proper internal Governance standards, and have benefitted from leadership by the Chief Exec and Senior management team.

    The biggest need we identified was to showcase great examples of how open standards can lead to better outcomes in practice.

    Showing what’s possible, both with case studies and live services that can be adopted was seen as essential, especially when this leads to actual financial savings and better outcomes for the citizen. This is something we’re keen to put some time into in the future.

    Sarah Prag and Ben Cheetham shared their experiences of collaborating on the DCLG led Waste Standards project. The most interesting thing for me was how a group of committed individuals just decided to get on with it and find some funding to make it happen – a proper coalition of the willing.

    Practical Next Steps

    The second half of the workshop looked at what we should focus on next.

    We were particularly keen to build on the progress that we’ve made in popularising the Open311 standard, which we can use to integrate FixMyStreet.com with each local authorities’ systems.

    We heard two contrasting experiences, firstly from Chris Fairs at Hertfordshire, who employ an extensive internal management system for issue reporting including individual definitions for fault types. They discovered that citizens are not so good at judging the severity of potholes – and through triage inspection, around 40% of reports are downgraded due to misreporting.

    This contrasted secondly with the experience of Nigel Tyrell and his team at Lewisham who have recently adopted an Open311 enabled service, now linked into both FixMyStreet.com and LoveCleanStreets.

    Lewisham Open311 App Results

    Perhaps the most surprising aspect of Lewisham’s experience is that well over half of reports actually come from their own internal staff using the system. This peer to peer approach has been transformative for them, with frontline staff motivated, more in control, more engaged with and connected to residents, and better able to integrate citizen reports into their own workflow – a very neat solution.

    From this discussion we identified three specific actions that we’re going to help take forward;

    1. Identify local authority service areas that would benefit from the development of open standards
    2. Review output from the DCLG Waste Standards project, to determine how a similar approach can be applied elsewhere
    3. Feed back with suggested improvements to Open311.org for non-emergency reporting and update the list of UK Open311 endpoints

    As with any such event the real value comes in the following weeks and months as we look for ways to collaborate together and opportunities to put into practice some of the things that we discussed.

    We’ll certainly be planning follow-up events in the future, so if you’d like to get involved sign up for our newsletter, post a comment below or get in touch at hello@mysociety.org.

  2. Calling all teachers: downloadable lesson plans

    If you’re a teacher, looking to spend the Easter holidays planning lessons, our latest news could save you a little time.

    mySociety has collaborated with the Citizenship Foundation on the creation of materials for use in schools.

    These activities, written and tested in consultation with teachers, introduce students to concepts of democracy, citizenship and community. A number of the materials also show students how they can use mySociety websites such as WriteToThem, WhatDoTheyKnow or FixMyStreet to bring about change.

    We hope that you will find these activities useful. They span years 1 to 13, will fit into a variety of curricula from Politics to Geography, and are completely free to download and use. Access them here – and please do pass the word on to your teacher colleagues.

    Image: Rachel (CC)

  3. Open Data Day resources

    Image by OpenSourceWay

    Note (June 2016): This post is now slightly out of date. FixMyTransport is no longer running, though all of the other APIs and tools listed are still available.

    There is also one significant addition which developers should find useful: EveryPolitician, which provides data on all current politicians around the world (and, in the future, we hope, all past ones too). See more here.


     

    Much of what we do here at mySociety relies on Open Data, so naturally we support Open Data Day. In case you haven’t come across this event before, here’s the low-down:

    Open Data Day is a gathering of citizens in cities around the world to write applications, liberate data, create visualizations and publish analyses using open public data to show support for and encourage the adoption open data policies by the world’s local, regional and national governments.

    If you’re planning on being a part of Open Data Day, you may find some of mySociety’s feeds, tools and APIs useful. This post attempts to put them all in one place. (more…)

  4. An Italian Case Study: Fatequalcosa.it

    Despite being home to the beautiful La Fiorita festival, one man still believes the authorities could do more to fix street problems in his part of Italy. Platforms existed in Venice and across other regions of Italy, so a precedent was set.

    Thanks to local Claudio Carletti, the La Marche region will soon get its very own FixMyStreet platform, fatequalcosa.it, which translates as Do Something! Claudio has shared his experience of using the mySociety code and his plans for the site.

    The beginning

    “I heard about FixMyStreet [after] reading a book called Wikicrazia where the author talked several times about mysociety.org,” Claudio explains. After exploring the mySociety sites he came across a blog post offering free developer time to help people start their freedom of information or street problem reporting projects. “I contacted [mySociety] and told them I wanted to create fixmystreet in Italy, and asked them for help. They responded immediately.”

    Why FixMyStreet?

    Claudio explains, “I chose this platform after trying fixmystreet.com and discovering a really well done service and a user friendly site experience.” He tells us there is already an Italian site running a similar service aimed at councils: decorourbano.org. So Claudio’s plan is to target his site at the Italian youth, 14 to 18 year olds. “I think only a cultural revolution that uses concrete tools (like FixMyStreet) can really change the way you think and act, but I think [it’s] mainly young people who must have the tools to do so.” Plus, he tells us, the youth of today will be the older generation of tomorrow.

    Installation

    After an initial Skype chat with mySociety’s international team Claudio felt confident enough to try a basic install of the software himself.

    “I’m not a website developer but I could do it,” Claudio explains. “My experience was using the installation through the Amazon web service. So most of the work was already done and I just ran the installation following the tutorial.”

    That’s not to say it was all plain sailing. “Most of my issues came from the Ubuntu terminal commands, which I didn’t know how to use. After the release of the 1.2 platform version most of my problems were resolved. One important improvement was the automatic email service set up and the MAPIT_ID_WHITELIST feature.”

    But the mailing list service run by mySociety to help self-installs was an invaluable tool for solving these problems. “Matthew [lead developer for FixMyStreet] always wrote back to my emails, spending a lot time helping me understand the problems,” Claudio says.

    Customisation

    One of the perks of the FixMyStreet code is that it’s quite simple to change basic design elements of the site, such as colours and logos, so it can look unique from other installations. Claudio tells us, “After understanding the conf/general.yml file everything went smoothly.”

    Launch plans 

    Claudio has recently begun collaborating with another user on this service, and is presenting his idea to the Projetto Kublai platform where he hopes to gain more support through their community. He’ll also be taking the project to a EU meeting in Berlin in the hopes that there might be some funding there. The launch date isn’t set but he has plans for collaboration with local schools and youth organisations. “[The youth] are more familiar with smartphones and they spend more time living in the city’s streets, because they take the bus or walk or cycle, but they don’t use cars.” And Claudio envisions these will be his superusers.

    Moving forward

    As Claudio tells the community on Projetto Kublai, “If you look at this site http://www.emptyhomes.com you understand how in reality there are no limits to the possibilities of use of the platform.” Other instances in other countries have reported any number of different types of problems. And it doesn’t just have to be email based, FixMyBarangey in the Philippines allows users to report issues using SMS.

    The bigger picture behind Claudio’s installation, he tells us, is this: “Let’s say that the platform fixmystreet is a good starting point, because it works and is easy to install and use. I’d like to have the opportunity not only to bring a problem to solve; for example organizing task forces where people will agree to go and clean up an area filled with concrete waste or to sort out a pitch in order to organize a [football] tournament. This will be the site that will give the tips (video tutorial) to fix that problem. Having said so it sounds pretty utopian idea, but I also believe that if you let people participate (and give them the means to do so) they’ll believe in it and do everything they can to achieve it.”

    An inspirational idea!

    Image credits:
    Romantica 2010 by Jukka Heinonen, Artisaniaflorae CC BY-NC-ND 2.0

  5. Co-brands, code additions and pull requests

    More and more people are starting to build websites to help people become more powerful in their civic and democratic lives. Some of these are on codebases that mySociety has created which is so great. There are some things which we would love to happen when you take our code and re-use it.

    We want people using our code to keep it as up to date as they can, so that they gain the benefits of any changes made to the code by us or by other users. There are a few reasons for this:

    • You can co-brand the site without breaking anything.

    Dave, one of our developers, explains how you do this. “So suppose, instead of calling it FixMyStreet you want to call it FixMyBorchester with a Borchester logo. Obviously this is a very real requirement, because people want to rebrand. One very feasible (but wrong! As you’ll see…) way of doing this is downloading the FixMyStreet code, finding the bit that paints the FixMyStreet logo and replacing it with the words <h1>FixMyBorchester</h1> and an image. This would work as far as the FixMyBorchester branding would appear on the site.

    But if you then saved and committed your change to git and passed it back to us as a push request, we would reject it. This is for the obvious reason that if we didn’t, next time we deployed FixMyStreet in the UK it would have your logo on it.

    However, say we suddenly discover there is a bug with FixMyStreet. For (a bizarre) example, if someone put the number 0 in instead of a postcode and the site returns a huge picture of a kitten. We love kittens, but that’s not what the site is trying to do. So, we make some fixes to the code that rejects zeros, commit it, update the repo, and it’s now there on the master branch. We write to everyone saying “really everyone, update to the latest (most up-to-date) place on the master branch” And you think, “yeah OK!” and you download the latest version.

    If you just download it and copy it into place, you’re going to lose your FixMyBorchester changes, because there’s a more recent version of that file from us that hasn’t got them. If you did a “git pull” (which roughly means, “git! get me the latest version of master branch”) then git will refuse because there’s a conflict on that file.

    So, instead of inserting your FixMyBorchester stuff over ours, which can’t work, you make a new directory in the right place called ‘FixMyBorchester‘, put your stuff in there and switch the FixMyStreet config — which knows this is something people want to do — to use that cobrand. Any templates FixMyStreet finds in there will now be used instead of ours. You can now safely update the codebase from our repo from time to time and FixMyStreet and git will never damage your templates, because they are in a place it doesn’t mess with.”

    • You can add new features

    Dave continues. “Say when someone uses FixMyBorchester it’s essential that you have their twitter handle, because every time a problem is updated, FixMyBorchester direct-tweets them a kitten for fun. Right now there is no capacity to store a twitter handle for a user in FixMyStreet.

    You simply add a column to the users table in your database and add some code for accepting that twitter handle when you register, and sending the kittens etc. That’s new code that isn’t in FixMyStreet at all. Sooner or later you’ll need to put at least one line into the main FixMyStreet program code to make this happen. As soon as you do that you have the same problem we had before, only this time it’s in code not in an HTML template.

    What we would encourage you to do is put all your new code in a branch that we can look at, and maybe set it to run only if there’s a config setting that says USE_TWITTER=true. That way any implementation that doesn’t want to use twitter, which is — at this point — every other FixMyStreet installation in the world — won’t be affected by it. You send that to us as a pull request and a developer checks it’s not breaking anything, and is up to scratch in quality, and has good test coverage. Then we’ll accept it.

    Even though currently nobody else in the world wants your twitter feature, it’s not breaking anything and it’s now in the repo so you can automatically update from our master when we change bits of our files, and the installation/overwrite/git-pull will work. Plus anyone that does decide they want this feature will now be able to enable it and use it.

    And all of this helps everyone using the code; you have a secure website that can be patched and updated each time we release something, other people have access to features you’ve built and vice versa. And overall, the project becomes more feature rich.

    Please do make changes and push them back to the main codebase!

    Image credit: US Coast Guard CC BY-NC-ND

  6. The Flexibility of FixMyStreet

    A lot of people come to mySociety to reuse our code having seen the UK websites, which is great! Then you can see what we’re trying to do in the UK and how you could replicate it abroad. But what I wonder, and what lead me to write this blog post, is are we reining in your imagination for what these platforms could be used for?

    9 times out of 10, when someone contacts me about FixMyStreet, it’s for street reporting problems. Naturally, it’s in the name of the platform!  But we do get the occasional request to use it differently, which is something we’re really keen to explore. Here are some things I think it could be used for, that aren’t street related:

     

    1) Antiretroviral Drug shortages in clinics in Africa.

    The background: 34% of the world’s HIV positive population currently live in Southern or Eastern Africa [1]. These people need antiretroviral drugs to survive, some of which could be supplied by the Government’s medical stores, some of which could be supplied by charities, but it is often reported that there are shortages of drugs at some clinics [2][3]

    The concept: A mobile responsive FixMyStreet site which health clinic staff can use to report the status of their stock to the relevant supplier. The site would instantly send an email to the clinic supplier when the staff member dropped a pin on their clinic on a map in the site. There could be different alert categories such as “stock running low”, “stock critically low” and “Out of stock”

    Impact it would hope to achieve: The aim would be to enable clinics to report on the status of their stock far enough in advance that the supplier could order and deliver stock before they hit the Critically low or Out of Stock status. This would mean that people would always be supplied with ARVs if they need them. Another point would be that patients could check the map to see if the clinic in their area has stock of the ARVs they need, and potentially choose another clinic if there is a shortage.

     

    2) Contributing data on endangered wildlife

    The background: It’s no surprise to anyone to hear that some species of wildlife are under threat. Wildlife conservation charities, like the Royal Society for the Protection of Birds (RSPB), annually monitor population levels for endangered species [4] to ensure they have accurate data on population growth or decline and the lifestyles and habitats of the wildlife they are aiming to preserve.

    The concept: A mobile responsive FixMyStreet site which allows people to report sightings of endangered animals to wildlife conservation charities. The site would be tailored for area (eg the endangered animals native to certain countries) or could simply be per species (eg mammals, avians etc). The public would then be able to take a picture of the animal, attach it to the report and leave a short message, like “2 adult bitterns accompanied by young seen at 10:41am). The report will give the charities the location the animal was spotted in and they will be able to add this to their research data.

    Impact it would hope to achieve: Hopefully this idea would contribute valuable data to the research of Wildlife Conservation charities. Another hope is that it would make people more interested in the wildlife in their surrounding area, thus more involved in conserving it and its habitat.

     

    3) Reporting polluted Waterways

    The background: You may have seen the reports from China earlier this year about the dead pigs found in the Huangpu River [5]. It’s not just a Chinese phenomenon: around the world rivers, canals and lakes are becoming more and more polluted. [6] In fact the statistics coming from the UN are quite shocking[7]. This not only has a harmful effect on wildlife in the river, but could lead to longer term issues with clean drinking water, especially in countries where cleaning polluted water is an expensive option.

    The concept: This is very similar to the classic FixMyStreet. A website would be set up where a person could submit a photo and report of a polluted waterway by dropping a pin on a map at the position of the river. This report would then get sent to the local council or persons responsible for caring for the waterway.

    Impact it would hope to achieve: Similarly to FixMyStreet in the UK, this would help to get citizens more actively involved in their local area and government. The idea would also be that the council would hopefully start dedicating more resources to clear rivers and waterways. Or local residents could form a group to remove litter themselves. In the case of chemical or oil spills this would obviously not be advised. However if chemical waste or oil spillages were noticed to be originating from specific buildings then the council would have the opportunity to bring this up with the residents or companies in these buildings.

     

    So those are some of my ideas! What are yours?

    We’re actively looking to support non-street uses of FixMyStreet so please do get in contact  on international@mysociety.org with your ideas and we’ll work together to see how we can achieve them!

    Oh, and, don’t worry if you still want a classic FixMyStreet, we’ll help you with that too!

     

    References:

    [1]http://www.unicef.org/esaro/5482_HIV_AIDS.html

    [2]http://allafrica.com/stories/201307070100.html

    [3]http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3194149/

    [4]http://www.rspb.org.uk/ourwork/projects/details/258718-annual-bittern-monitoring-

    [5]http://behindthewall.nbcnews.com/_news/2013/03/18/17357810-china-rivers-dead-pig-toll-passes-13000-but-officials-say-water-quality-is-normal?lite

    [6]https://www.gov.uk/government/consultations/tackling-water-pollution-from-the-urban-environment

    [7]http://www.unwater.org/statistics/en/

    [8] Orangutan by Matthew Kang

    [9] Primary colours by Vineet Radhakrishnan

  7. Open311: Extended

    This is the third of our recent series of Open311 blog posts: we started by explaining why we think Open311 is a good idea, and then we described in a non-techie way how Open311 works. In this post we’ll introduce our proposed extension to Open311, and show how we use it in FixMyStreet.

    The crux of our suggested improvement is this: normal people want to know what has happened to their problem, and Open311 currently isn’t good enough at telling them whether or not it has been dealt with. To be more specific, our additions are all about reports’ status change, by which we mean something like this:

    That pothole?

    I just totally fixed it.

    That’s robot-311 from the previous post, if you’ve dropped in here without reading the previous posts. Once again we’re blurring the distinction between client and user (the girl you’ll see below) a little, to make things simpler to follow.

    Create→send→fix→update

    Every month in the UK, thousands of problems are reported on www.fixmystreet.com and, moments later, sent on to the councils who will fix them. Here’s what happens with a problem report for something like a pothole or a flickering streetlight:

    1. You create the report on FixMyStreet.
    2. FixMyStreet sends that report to the right department at the right council.
    3. That body puts it into its own back-end system.
    4. Later, when the council fixes the problem, FixMyStreet is updated, and everyone knows it’s fixed.

    On the face of it, you might think we need only care about 1 and 2. But really, FixMyStreet isn’t just about dispatching reports, it’s about helping to get things like potholes actually fixed. And neither citizens nor local governments benefit if work gets done but nobody finds out about it – which is part 4 on the list above.

    What do we mean by “status change”?

    The example at the top of the page shows the robot effectively changing a problem’s status to “fixed”.

    Actually, statuses can be simple, such as either OPEN or CLOSED, or more detailed, such as “under investigation”, “crew has been dispatched”, “fixed”, and so on. But since we’re only concerned here with the status changing, that specific vocabulary deployed doesn’t really matter – it can be anything.

    In situations where FixMyStreet is not integrated with council systems (i.e we just send email problem reports) FixMyStreet problems still frequently get marked as fixed, because anyone can change the status of a report just by visiting the page and clicking the button.  Obviously, though, we prefer to have FixMyStreet directly connected to the local government back-end databases, so that news of a fixed report can be automatically bubbled from the back-office up into FixMyStreet and out onto the net.

    And here’s where the problem lies: Open311 doesn’t quite support this business of getting problem updates from the back office out to the public. So first, we’ll show you how it can be done today, using Open311, and we’ll explain why this isn’t a good option. Then we’ll show our preferred solution, which we’ve proposed as an extension.

    Looking at everything just to spot one change (bad)

    One way to notice if any problems’ statuses have changed is to use Open311 to ask for every single service request, and see if any of them have a different status since the last time you checked.

    Tell me all the service requests you’ve ever received

    OK:

    request 981276 the pothole on the corner by Carpenter Street is now CLOSED (I filled in the pothole)

    request 988765 the pothole by bus stop on Nigut Road is now CLOSED (I filled in the pothole)

    request 998610 gaping hole at the end of Sarlacc Road is now OPEN (the pothole fell through)

    request 765533 where the street was cracked outside Taffey’s Snake Pit is now CLOSED (I filled in the pothole)

    . . .

    continues for thousands of requests

    Um, OK. Now I’ll look at all these and see if any have changed since I last asked *sigh*

    Obviously there are some problems with this. Even though Open311 lets you ask for quite specific service requests, you have to ask for all of them, because by definition you don’t know which ones might have changed. Remember, too, that problems can potentially change status more than once, so just because it’s been marked as CLOSED once doesn’t mean it won’t become OPEN again later. This exchange is very wasteful, very slow and ultimately (with enough reports) may become de facto impossible.

    Asking for just the changes (good)

    So here’s a better way of doing it. We’ve actually been doing this for some months, and now seems the time to share.

    The client asks the server for just the updates on a regular basis, so any requests that have recently changed get updated on FixMyStreet automatically, usually just a few minutes later.

    Have you changed the status of any of service requests today?

    Yes, request 981276 was CLOSED at 3 o’clock (I filled in the pothole)

    Or, more practically for keeping FixMyStreet up to date:

    Have you changed the status of any of service requests in the last 15 minutes?

    Nope.

    This is handled by our extension to Open311, GET Service Request Updates. There’s also an optional equivalent call for putting updates into the server (POST Service Request Update), which would apply if the client changed the status after the service request had been submitted.

    Note that the server identifies the problem with its own reference (that is, 981276 is the council’s reference, not a FixMyStreet ID, for example). This is important because not all these requests necessarily came from this particular client. Remember that all service requests are available through the Open311 GET Service Requests call anyway (as shown above). So the server doesn’t send each service request back in its entirety: just its ID, the new status, when it changed, and a brief description.

    In practice the client wouldn’t usually ask for “today”. In fact, we typically send a request asking for any updates in the last 15 minutes, and then at the end of the day ask for the whole day’s updates, just to check none were missed.

    The technical bit

    From a client’s point of view, this is simply an extra call like others in the Open311 API. So it’s just a request over HTTP(S) for XML (or JSON, if required).

    We deliberately make the client poll the server for updates and pull them in, rather than expecting the server to push updates out. This frees the server from any obligation to track which clients (for there may be more than just one) care about which updates. The requests themselves are sent with unique IDs, allocated by the server, so the client can dismiss duplicates. It’s also robust in the event of connection failures, so if there are timeouts or retry logic, that’s for the clients to worry about, not the server. Basically, this is all to make it as light on the server as possible: the only real issue is that it must be able to provide a list of updates. This usually means adding a trigger to the database, so that when a problem’s status is updated a record of that update is automatically created. It’s the table of those “service request update” records that incoming requests are really querying.

    We have published our own recommendation of this mechanism, which FixMyStreet already implements, alongside the FixMyStreet codebase.

    Is that it?

    Yup, that’s it.

    This extension is in addition to the Open311 specification — it doesn’t break existing implementations in any way. Obviously this means FixMyStreet’s Open311 implementation is compatible with existing Open311 servers. But we hope that others working on Open311 systems will consider our extension so that clients are kept better informed of the status of the problems being fixed.

    Why are statuses so important that it is worth extending the Open311 spec?

    mySociety didn’t originally build FixMyStreet because we wanted to get potholes fixed. We built it because we wanted nervous, politically inexperienced people to know what it felt like to ask the government to do something, and to be successful at that. We wanted to give people the buzz of feeling like they have a bit of power in this world, even if the most tiny amount.

    If the government fixes a problem and the citizen doesn’t find out it’s a double loss. The citizen becomes disillusioned and weakened, and the government doesn’t get the credit it is due. Everyone loses. We think that Open311 is a key mechanism for making large numbers of people feel that the government does respond to their needs. It just needs a bit of an upgrade to do it better. We hope very much that the wider community tests and endorses our extensions, and it can be folded in to the next official version of the Open311 standard.

    Find out more about FixMyStreet for Councils

  8. Extracting Boundaries from OpenStreetMap – Part 2

     

    Hadrian's Wall by Joe Dunckley

    This is the second part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML files for every useful administrative and political boundary in OpenStreetMap.

    The previous post on this subject described how to take the ID for a particular relation or way that represents a boundary in OpenStreetMap, and generate a KML file for it. That’s much of the work that we needed to create MapIt Global, but there are a few more significant steps that were required:

    Efficiently extracting boundaries en masse

    The code I previously described for extracting a boundary from OpenStreetMap used a public Overpass API server.  This is fine for occasional boundaries, but, given that there are hundreds of thousands of boundaries in OSM, ideally we don’t want to be hitting a public server that many times – it puts a large load on that server, and is extremely slow. As an alternative, I tried parsing the OSM planet file with a SAX-based parser, but this also turned out to be very slow – multiple passes of the file were required to pick out the required nodes, ways and elements, and keeping the memory requirements down to something reasonable was tricky. (Using the PBF format would have helped a bit, but presented the same algorithmic problems.) Eventually, I decided that a better approach was simply to set up a local Overpass API server, and to query that.  This is a great improvement – it allows very fast lookups of the data we need, and the query language is flexible enough to be able to retrieve huge sets of relations and ways that match the tags we’re interested in.  It also means we would no longer have problems if connectivity to the remote server went down.

    Another question that arose when scaling up the boundary extraction was, “Which set of tags we should consider as boundaries of interest?” On the first import, we considered any relation or way with the tag boundary=”adminstrative” and where the admin_level tag was one of “2”, “3”, “4”, … “11”.  At the time, there were about 225,000 such elements that represented closed boundaries. Afterwards, it was pointed out to me that we should also include elements with the tag boundary=”political”, which includes parliamentary constituencies in the UK, for example.  For later import purposes, I gave each of these boundary types a 3-letter code in MapIt, which are as follows:

    • O02 (boundary="administrative", admin_level="2")
    • O03 (boundary="administrative", admin_level="3")
    • [...]
    • O11 (boundary="administrative", admin_level="11")
    • OLC (boundary="political", political_division="linguistic_community")
    • OIC (boundary="political", political_division="insular_council")
    • OEC (boundary="political", political_division="euro_const")
    • OCA (boundary="political", political_division="canton")
    • OCL (boundary="political", political_division="circonscription_législative")
    • OPC (boundary="political", political_division="parl_const")
    • OCD (boundary="political", political_division="county_division")
    • OWA (boundary="political", political_division="ward")

     

    Importing Boundaries into MapIt

    The next step in building our service was to take the 236,000 KML files generated by the previous step and import them into MapIt.

    The code that creates the KML file for an element includes all its OpenStreetMap tags in the <ExtendedData/> section.  On importing the KML into MapIt, there are only a few of those tags that we’re interested in – chiefly those that describe the alternative names of the area.  We have to pick a canonical name for the boundary, which is currently done by taking the first of name, name:en and place_name that exists. If none of those exist, the area is given a default name of the form “Unknown name for (way|relation) with ID (element ID)”. There are also tags for the name of a country in different languages, which we also import into the database so that localized versions of the name of the boundaries will be available through MapIt with their ISO 639 language code.

    Another tricky consideration when importing the data is how to deal with boundaries that have changed or disappeared since the previous import. MapIt has a concept of generations, so we could perfectly preserve the boundaries from the previous import as an earlier generation. This would certainly be desirable in one respect, since if someone is depending on the service they should be able to pick a generation that they have tested their application against, and then not have to worry about a boundary disappearing on the next import. However, with quarterly imports the size of our database would grow quite dramatically: I found that approximately 50% of the boundaries in MapIt Global had changed over the 5 months since the initial import. Our proposed compromise solution is that we will only keep the polygons associated with areas in the two most recent generations, and notify any known users of the service when a new generation is available for them to test and subsequently migrate to.

    For reference, you can see the script that extracts boundaries and generates the KML files and the Django admin command for importing these files into MapIt.

    The end result: MapIt Global

    The aim of all this work was to create our now-launched web service, MapIt Global. As far as we know, this is the only API that allows you to take the latitude and longitude of any point on the globe, and look up  the administrative and political boundaries in OpenStreetMap that surround that point. We hope it’s of use to anyone trying to build services that need to look up administrative boundaries – please let us know!

    Photo credit: Hadrian’s Wall by Joe Dunckley,

  9. mySociety design tips: avoiding duplicate messages

    duplication

    One of the common elements you will find across mySociety’s sites is that they have features designed to reduce duplicate messages or reports being sent to politicians, governments or companies. We often do this in quite a subtle way, so it is worth spelling out here how we do this across several sites:

    • If you start to report a broken street light or pothole on FixMyStreet, you’ll see local problems before you start to type in your own details. This means if the problem is already there, you can see before you waste any effort.
    • If you use WhatDoTheyKnow to send a Freedom of Information request to a public body, we provide a facility which encourages users to search through other people’s requests before they type their new request in.
    • If the 08:10 train you take to work is always late, when you go to report it on FixMyTransport, we show you all the other problems already reported on that route. If someone else has already set up a page, you can press the big green ‘join’ button, and show your support.
    • If more than a handful of people try to use WriteToThem to send an exact duplicate of the same message to a politician, it will prevent it. This is because we know that politicians listen much, much more to individual messages from constituents than bulk mails.

    This pattern – trying to intervene before people write identical messages or reports – is a design decision that makes a big difference to the way these sites operate. As usual with mySociety sites, this little feature seems like the sort of thing that would be quite tempting to skip when building a copy. But it really matters to the long term success of the sites. There are three reasons why.

    First, there is a simple public benefit that comes from saving time. There’s no point us wasting your time if a report or request has already been sent, especially around minor issues. Saving your users time makes them happier and more likely to enjoy their experience.

    Second, if you can spot that someone is about to send a duplicate message, we may be able to encourage that user to support the existing report instead of making a new one. For example, on FixMyStreet you can add an update to an existing pothole report (“it’s getting worse!”).

    This feature is most visible, and most mature, on FixMyTransport, where users are clearly encouraged to ‘support’ pre-existing reports, rather than making new copies.  By discouraging duplicate reports, we let people with a shared problem work together, even if this only means adding themselves as a “supporter” and doing nothing else. We know that many people search for, and find, problem reports which have turned into these little campaigns, which they then join and help. So even if they are only reading them (not joining them) that exposure can have some value to the people affected. This would be diluted if we created lots of similar reports about the same problem.

    Third, we discourage duplicates for the benefit of the governments and companies receiving messages. We don’t think FixMyStreet is effective because it lets people moan: we think it’s effective because it helps local government to be effective by giving them good quality reports about local problems, in formats that area easy to handle. This good quality reporting increases the chance that the government will understand the problem and act on it, which leads to our main goal – citizen empowerment. Recipients are unlikely to help users if many of the messages they get are confused, inaccurate or duplicates, so we work on all these fronts.

    So if you haven’t thought about this before, notice how the “work flow” through our sites makes you see similar problems before you’ve finished reporting your own. This is the implicit way to prevent duplication. We don’t have “Stop! Warning! Check this is a new problem!” messages, because we never want to discourage genuine users. But the careful design of the interface gently discourages, successfully, duplicate reports, and encourages supporting of other items.

    It’s never possible to entirely prevent duplication. But we try hard, because it’s always better to join people together around common causes, than it is to let them struggle alone.

  10. mySociety design tips: how we choose the best map zoom level

    When you report a problem on FixMyStreet.com, the site displays a map for you to click on to indicate its exact location. Of course, you can zoom in and out of that map, but when it is first displayed, FixMyStreet needs to use an initial ‘default’ zoom level. Ideally, this is a zoom level that reduces the number of clicks required before a user can pinpoint the location of their problem.

    map_zoom_by_-kathiemcl_pty

    “In the middle of America” by Kathie M Ceballos (CC BY-NC-ND licensed from Flickr)

    And here’s where we encounter a tricky problem. The world is a varied place – some towns are very dense with buildings and streets crammed close together. In these areas you need to default to a zoom that’s quite ‘close in’, otherwise it can be hard to locate your problem.

    But out in the countryside, we have the opposite problem. You can have huge areas where there’s nothing but blank fields or moorlands. If the default map zoom is ‘close in’ here then users are likely to see a big map full of nothingness, or maybe just a single stretch of unidentifiable road.

    So, what is to be done?

    The answer is this – every time you search for a location in FixMyStreet the website does a check to see whether the location you typed is in an area where a lot of people live, or very few people live.

    mySociety has been storing this population density data in a webservice which we call Gaze. If the area you searched for is in a densely populated area we assume that it must be an urban location, and the map starts with a helpfully zoomed-in map. But if you’re in a sparsely populated area then it’s probably rural, so FixMyStreet starts zoomed out, making it easier to get an overview of the whole area.

    Where do we get the data from? Our late colleague Chris discusses this in a blog post from 2005 — the short answer is NASA SEDAC and LandScan. It’s an interesting example of how unexpected things can happen when data is made public — if population density wasn’t available to us, we wouldn’t have been able to put this small but clever detail into FixMyStreet’s interface.