1. Hello, is it me you’re searching for?

    There’s a common theme to a lot of mySociety sites: enter your postcode, see something that relates to you.

    From FaxYourMP—the mySociety project so old it predates mySociety itself (paradox!)—through to TheyWorkForYou, FixMyStreet, and WriteToThem, as well as a few of our commercial projects like Mapumental and Better Care, we’ve discovered that asking for a visitor’s location is a super effective way of unlocking clear, relevant information for them to act on.

    So perhaps it shouldn’t have come as a surprise that, while doing some regular monitoring of traffic on this website, we noticed a fairly significant number of people attempting to search for things like postcodes, MP names, and the topics of recent debates.

    Random sample of search terms, July–December 2017
    animal sentience corbyn
    germany CR0 2RH
    theresa may facebook
    EN3 5PB fire
    ruth davidson HG5 0UH
    eu withdrawal bill diane abbott

    By default, the search box on this site delivered results from our blog post archive (it goes all the way back to 2004 don’t you know!)… which is pretty much what you’d expect if you know how we do things here at mySociety. We have this centralised website to talk about ourselves as an organisation; then each of our projects such as TheyWorkForYou or FixMyStreet is its own separate site.

    But, looking at these search terms, it was pretty clear that an awful lot of people don’t know that… and, when you think about it, why should they?

    The most obvious solution would just have been to direct visitors towards the individual sites, so they could repeat their searches there. Job done.

    But we figured, why inconvenience you? If you’ve made it this far, we owe it to you to get you the information you need as quickly as possible.

    Handily, we’ve got rather good at detecting valid postcodes when our users enter them, so programmatically noticing when a user was searching for a location wasn’t hard. And equally handily, TheyWorkForYou offers a powerful API that lets developers exchange a user’s postcode for detailed data about the boundaries and representatives at that location.

    What do you get when you combine the two? Automatic search suggestions for TheyWorkForYou, FixMyStreet, and WriteToThem, when you enter your postcode on www.mysociety.org.

    The search page is also aware of the most frequently searched-for MPs on our site, and will offer a direct link to their TheyWorkForYou profile if you search for their names.

    And finally, if you search for something other than a postcode, we give you a single-click way to repeat your search, automatically, on TheyWorkForYou, opening up decades of parliamentary transcripts to you, with a single tap of your finger.

    It’s not a big, glamorous feature. But it’s something we know will come in useful for the few hundred people who search our site every week—possibly without their ever noticing this little bit of hand-holding as we steer them across to the site they didn’t even know they wanted. And most importantly, it should introduce a few more people to the wealth of data we hold about the decision-makers in their lives.

    Header image, Flickr user Plenuntje, CC BY-SA 2.0

  2. Standing up for erectile dysfunction care: a digital empowerment tool

    What’s the best way to get your supporters to campaign, when the finer details of what they’re pressing for may vary from place to place? That’s the issue that faced Prostate Cancer UK as they call for better provision for men across the country with erectile dysfunction as a result of prostate cancer.

    There are five core treatments for tackling erectile dysfunction, but whether all of them will be offered to you depends on your postcode. In some areas, all are offered as standard, while in others there may be none.

    The tool we built for Prostate Cancer UK used several of mySociety’s areas of expertise, from mapping to user testing — we even used Freedom of Information. And putting it all together, we have a powerful campaigning platform that responds to users’ location, while raising awareness and pushing for improvement.

    Prostate Cancer UK’s Erectile Dysfunction campaign site informs people about what care should be available to those who experience the condition as a result of prostate cancer treatment, and urges them to write to their local health commissioner if provision is poor in their area.

     

    Prostate Cancer tool, built by mySociety

    Educating, campaigning, sharing

    The user is first informed: they are shown the five factors which constitute good treatment of erectile dysfunction. After that, they are prompted to input their postcode to see how many of those measures are provided by the NHS body responsible for their region.

    If provision is poor, they are encouraged to help campaign: users can opt to write to their Clinical Commissioning Group (CCG), Health Board or Health and Social Care Board to ask them to improve what’s available. They are given the choice between writing a letter from scratch, or using a pre-composed template which also contains a section for the writer to add a paragraph of their own words — a pragmatic balance that avoids an influx of identical form letters, while still addressing fact that when users are faced with a completely blank page, many will drop out of the process.

    When you’ve done that, for those in England there’s also an opportunity to contact Jeremy Hunt, Secretary of State for Health to highlight the variation in treatment for erectile dysfunction and establish which organisation is responsible for the national commissioning guidelines.

    Finally, the user is invited to share what they’ve learned, via Facebook, Twitter or email. Our user testing revealed that, contrary to our worries, people were happy to do this without embarrassment.

    How it works

    Like most of mySociety’s own sites, the ‘Better Care’ site uses MapIt to match the user’s postcode with a boundary, in this case the boundaries of the CCGs, Health Boards and Health & Social Care Boards. That’s how we deliver the information about what’s available in their local area.

    When you input your postcode to see how your local provisioners are doing, MapIt also delivers information for other areas, including a couple of close neighbouring ones. This allows us to provide a nice comparison, along with the statistic that shows whether your provisioner is better, worse, or within the same range as the average.

    PCUK comparison screen

    But how did we gather the data to tell you how well each CCG, Health Board or Health and Social Care Board is catering for erectile dysfunction patients? Well, fortunately, thanks to our own WhatDoTheyKnow website, it was relatively easy to send a Freedom of Information request to every one in the country — 235 of them in total. The WhatDoTheyKnow volunteer admin team were able to help with this large batch request.

    Once we had all the data and a general idea of how the tool would work, we took an early version out to test it with users. The insights we gained from this process were, as always, extremely useful, and led to us altering page layouts and other elements that made the whole process as clear as it could be.

    Finally, we incorporated quite a bit of statistics-gathering into the whole tool, so that Prostate Cancer UK would be able to see where their campaign might benefit from further optimising in the future.

    All in all, we’re very glad to have been part of this important campaign to help men understand what’s available to them, and where they might need to push for more.

     

    Image: Brad Hagan (CC-by/2.0)

  3. Extracting Boundaries from OpenStreetMap – Part 2

     

    Hadrian's Wall by Joe Dunckley

    This is the second part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML files for every useful administrative and political boundary in OpenStreetMap.

    The previous post on this subject described how to take the ID for a particular relation or way that represents a boundary in OpenStreetMap, and generate a KML file for it. That’s much of the work that we needed to create MapIt Global, but there are a few more significant steps that were required:

    Efficiently extracting boundaries en masse

    The code I previously described for extracting a boundary from OpenStreetMap used a public Overpass API server.  This is fine for occasional boundaries, but, given that there are hundreds of thousands of boundaries in OSM, ideally we don’t want to be hitting a public server that many times – it puts a large load on that server, and is extremely slow. As an alternative, I tried parsing the OSM planet file with a SAX-based parser, but this also turned out to be very slow – multiple passes of the file were required to pick out the required nodes, ways and elements, and keeping the memory requirements down to something reasonable was tricky. (Using the PBF format would have helped a bit, but presented the same algorithmic problems.) Eventually, I decided that a better approach was simply to set up a local Overpass API server, and to query that.  This is a great improvement – it allows very fast lookups of the data we need, and the query language is flexible enough to be able to retrieve huge sets of relations and ways that match the tags we’re interested in.  It also means we would no longer have problems if connectivity to the remote server went down.

    Another question that arose when scaling up the boundary extraction was, “Which set of tags we should consider as boundaries of interest?” On the first import, we considered any relation or way with the tag boundary=”adminstrative” and where the admin_level tag was one of “2”, “3”, “4”, … “11”.  At the time, there were about 225,000 such elements that represented closed boundaries. Afterwards, it was pointed out to me that we should also include elements with the tag boundary=”political”, which includes parliamentary constituencies in the UK, for example.  For later import purposes, I gave each of these boundary types a 3-letter code in MapIt, which are as follows:

    • O02 (boundary="administrative", admin_level="2")
    • O03 (boundary="administrative", admin_level="3")
    • [...]
    • O11 (boundary="administrative", admin_level="11")
    • OLC (boundary="political", political_division="linguistic_community")
    • OIC (boundary="political", political_division="insular_council")
    • OEC (boundary="political", political_division="euro_const")
    • OCA (boundary="political", political_division="canton")
    • OCL (boundary="political", political_division="circonscription_législative")
    • OPC (boundary="political", political_division="parl_const")
    • OCD (boundary="political", political_division="county_division")
    • OWA (boundary="political", political_division="ward")

     

    Importing Boundaries into MapIt

    The next step in building our service was to take the 236,000 KML files generated by the previous step and import them into MapIt.

    The code that creates the KML file for an element includes all its OpenStreetMap tags in the <ExtendedData/> section.  On importing the KML into MapIt, there are only a few of those tags that we’re interested in – chiefly those that describe the alternative names of the area.  We have to pick a canonical name for the boundary, which is currently done by taking the first of name, name:en and place_name that exists. If none of those exist, the area is given a default name of the form “Unknown name for (way|relation) with ID (element ID)”. There are also tags for the name of a country in different languages, which we also import into the database so that localized versions of the name of the boundaries will be available through MapIt with their ISO 639 language code.

    Another tricky consideration when importing the data is how to deal with boundaries that have changed or disappeared since the previous import. MapIt has a concept of generations, so we could perfectly preserve the boundaries from the previous import as an earlier generation. This would certainly be desirable in one respect, since if someone is depending on the service they should be able to pick a generation that they have tested their application against, and then not have to worry about a boundary disappearing on the next import. However, with quarterly imports the size of our database would grow quite dramatically: I found that approximately 50% of the boundaries in MapIt Global had changed over the 5 months since the initial import. Our proposed compromise solution is that we will only keep the polygons associated with areas in the two most recent generations, and notify any known users of the service when a new generation is available for them to test and subsequently migrate to.

    For reference, you can see the script that extracts boundaries and generates the KML files and the Django admin command for importing these files into MapIt.

    The end result: MapIt Global

    The aim of all this work was to create our now-launched web service, MapIt Global. As far as we know, this is the only API that allows you to take the latitude and longitude of any point on the globe, and look up  the administrative and political boundaries in OpenStreetMap that surround that point. We hope it’s of use to anyone trying to build services that need to look up administrative boundaries – please let us know!

    Photo credit: Hadrian’s Wall by Joe Dunckley,

  4. MapIt Global

    introducing-mapit-global

    We’ve mentioned Components before on this blog – they’re modules which you can slot into your website, and which should save you a lot of time and effort. Today, we’re pleased to announce that a fundamental Component is ready for use – MapIt Global.

    This Component will match geographical points to administrative areas anywhere in the world. So for example, you can use it on sites like FixMyStreet, where we ask the user for a zipcode/postcode, and then automatically knows which council to send their report to.

    mySociety’s Director Tom has written an in-depth blog post about MapIt Global. You’ll want to read it if you’re thinking of building a site or app for reporting street faults, for contacting elected representatives, for parliamentary monitoring… or just maybe you have ideas for a type of website that we haven’t even thought of. We look forward to seeing how MapIt Global will be used.

    Questions? Thoughts? We’d love to hear them, either on this post or on Tom’s.

  5. Extracting Administrative Boundaries from OpenStreetMap – Part 1

    Boundary Stone between Yorkshire and Lancashire

    "Boundary Stone 1" taken by Tim Green (CC-BY-licensed from Flickr)

    This is the first part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries.  This part describes how to generate KML for a given boundary in OpenStreetMap. Update: the second part is now available.

    As mentioned in a previous post, we’re looking at ways of making it smoother to deploy FixMyStreet for a new country, city or use-case.  Essentially there are two fundamental bits of data that you need for this:

    1. a mapping between a latitude and longitude (or postcode) to all the adminstrative areas that cover that point.
    2. a mapping between problem type and adminstrative areas to an appropriate email address for reporting that problem.

    The first of those is typically provided by a service called MapIt, an open source GeoDjango web application written by Matthew Somerville.  In the UK we are fortunate that official boundary data and the postcode database have now been released under an open data license from the Ordnance Survey.  However, in other many other countries similar data is unavailable, or not available under reasonable licensing conditions.  In such cases, though, all is not lost thanks to the extraordinary work of contributors to the OpenStreetMap project.  OpenStreetMap contains high-quality administrative boundary data for many countries of the world, and we know that data submitted to the project is available under the Creative Commons Attribution-ShareAlike license, so we can reuse it in a web service like MapIt.

    The first step towards being able to build an instance of MapIt based on OpenStreetMap boundary data is to be able to generate a shapefile that represents a boundary in the project.  (In OpenStreetMap’s data model, boundaries are represented as either ways or relations, and those that we are interested in are tagged with boundary=administrative.)  Matthew had previously written code to generate KML files for boundaries in Norway in order to help to set up an instance of MapIt for Norway, which is used by FiksGataMi.  However, that script was quite specific to the organization of boundaries in that country, and it did not deal with more complex boundary topologies (e.g. enclaves), or different representations of boundaries (e.g. multiply nested relations).

    So, Matthew and I wrote a new version of the code to extract a boundary from OpenStreetMap and generate a KML representation of it. The new version uses the Overpass API instead of XAPI, since it allows us to specify multiple predicates in the query and recursively fetch the ways and nodes that are contained in a relation.  Once all the ways that make up a relation have been fetched (ignoring those with roles like “defaults” or “subarea”), the script tries to join each unclosed way to any other with which it shares an endpoint.  We should end up with a series of closed polygons – the script exits in error if there are any unclosed ways left.  We can then directly create KML from these polygons, the only subtlety being that we need to mark certain boundaries as being an inner boundary (i.e., creating a hole in a boundary) if they had the role “enclave” or “inner” in an OpenStreetMap relation. For example, the South Cambridgeshire District Council boundary has a Cambridge City Council-shaped hole in it:

    South Cambridgeshire District Council boundary

    Similarly, the script has to cope with multiple distinct polygons, such as the boundary of Orkney.

    If you want to use this code to generate a KML representation of a closed way or boundary relation from OSM, just clone the MapIt repository and run bin/boundaries.py:

    $ bin/boundaries.py --help
    Usage: boundaries.py [options]
    
    Options:
      -h, --help            show this help message and exit
      --test                Run all doctests in this file
      --relation=<RELATION_ID>
                            Output KML for the OSM relation <RELATION_ID>
      --way=<WAY_ID>        Output KML for the OSM way <WAY_ID>

    For example, to generate a KML boundary for the Hottingen area of Zürich, you can do:

    $ bin/boundaries.py --relation=1701449 > hottingen.kml

    In the next blog post in this series, we will discuss extracting such boundaries en masse and creating a service based on them.