UPDATE: SocialCareInfo has now been rolled into the all-encompassing AdviceLocal service, which covers all areas of social welfare law (benefits, housing, debt, employment, immigration and social care), and has the same 408 local authority look-ups powered by MapIt to help display maps showing the local provisions.

It’s great to see the launch of SocialCareInfo, a new website which helps people in the UK find local & national social care resources.

All the more so because it uses one of our tools, MapIt, to match postcodes with the relevant local authorities. The site’s builders, Lasa, came to us when it became clear that MapIt did exactly what they needed.

Socialcareinfo.net covers the whole of the UK. Users begin by typing in their postcode, whereupon they are shown the range of services available to them.

That’s also how many of our own projects (think FixMyStreet, WriteToThem or TheyWorkForYou) begin, and there’s a good reason for that: users are far more likely to know their own postcode than to be certain about which local authority they fall under, or even who their MP is.

MapIt is really handy for exactly this kind of usage, where you need to match a person to a constituency or governing body. It looks at which boundaries the geographic input falls within, and it returns the relevant authorities.

We’re glad to see it working so well for SocialCareInfo, and we feel sure that the site will prove a useful resource for the UK.

You might not be in the ward you think you are. Due to ongoing boundary changes, many people will be voting within new wards this year. Confused? Fortunately, you can use our new MapIt-powered ward comparison site to see whether you’ll be affected by any new boundaries.

Pop your postcode in at 2015wards.mysociety.org and if your boundary is changing you’ll see your old and new wards.

These alterations are generally put in place by the Local Government Boundary Commissions, in an aim to even out the number of constituents represented by each councillor.

So we wanted to build an app for FixMyStreet. Easy: we just had to make a cut-down version of the website, right?

Hmm, not quite.

In our two recent posts, we announced the new FixMyStreet apps, and their developer Struan went into some technical detail about why and how we built them.

Now he explains a little more about what informed the decisions he made during the apps’ development.

Moving the map, not the pin

When you use the desktop version of FixMyStreet, the first thing it asks for is your location, and there’s a good reason for that. It’s such a good reason that it needed to apply to the app as well.

On the desktop site we ask you to input your postcode or street name. With a mobile app, it’s much more likely that you’ll be reporting a problem that’s right in front of you, so we can usually skip that step and show you a map of your current location.

However, while the accuracy of geolocation technology is pretty good, it’s not perfect, so we wanted to let users fine-tune the location.

On the website you click on the map to drop a pin where the problem is, but we’ve found this isn’t the best solution on a small screen. Fingers are big and the end of a pin is small so it can take several clicks to correctly position the pin.

We quickly realised that having a central static crosshair, and moving the map to the location was a much easier and more accurate way to set a location.

Sending reports made offline

As we explained in the previous post, one of the benefits of the apps over the mobile site is that you can make reports even if you have no phone coverage. The app stores all the details until you get back within range and you’re ready to send it off.

One decision we made, which might seem initially puzzling, is that these offline reports don’t automatically get sent off to the council once you’re back within range of a phone or wifi signal.

There are two linked reasons for this, and they’re both related to the fact that FixMyStreet lets you report a problem even if you don’t know who to send it to.

Simply, before we can work out who to send the report to, we need to know exactly where you are – and that you are within FixMyStreet’s area of coverage (ie, within the UK).

Your location also dictates the categories that we show you. Each council has its own categories, and in areas covered by two tiers of government, each council will deal with different types of report. So for example, your county council might deal with potholes, while your district council handles dog fouling.

Once you’re back online we can check that the location is one we can accept a report about, and then fetch the list of categories for you to pick from.

In effect, this delay is also a second chance for you to check your report before you send it off, although that was never the reason for the decision!

The constant map

The final decision we made was to leave the map in the background at every step of the app process.

We initially designed it with the map only appearing when you needed it, but having the map underlying the reporting process provides a nice bit of continuity with the website, and seemed to make the app cohere better too. So, while there’s no particular reason for it to be there, we made the decision to keep things uniform.

If anything else about the app has got you wondering, do feel free to leave a comment below!

Photos by Cali4Beach and Christopher Bulle, and adapted from an image by Premshree Pillai (CC)

Note (June 2016): This post is now slightly out of date. FixMyTransport is no longer running, though all of the other APIs and tools listed are still available.

There is also one significant addition which developers should find useful: EveryPolitician, which provides data on all current politicians around the world (and, in the future, we hope, all past ones too). See more here.

Much of what we do here at mySociety relies on Open Data, so naturally we support Open Data Day. In case you haven’t come across this event before, here’s the low-down:

Open Data Day is a gathering of citizens in cities around the world to write applications, liberate data, create visualizations and publish analyses using open public data to show support for and encourage the adoption open data policies by the world’s local, regional and national governments.

If you’re planning on being a part of Open Data Day, you may find some of mySociety’s feeds, tools and APIs useful. This post attempts to put them all in one place. (more…)

All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.

Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.

As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.

Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.

We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.

Thanks OpenStreetMap!

We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.

The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.

From Britain to the World

For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.

MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.

If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.

About the Data

The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.

The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global.  We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.

If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.

Commercial Licenses and Local Copies

MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.

 Feedback

As with any new service, we’re sure there will be problems that need sorting out. Please drop us an email, or tweet us @mySociety.

This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

Progressive enhancement

During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

Zooming

We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

OpenStreetMap

Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

OS StreetView tile server

Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

[sv]
type=WMS
url=path/to/mapserv.fcgi?map=os.map&
layers=streetview
tms_type=google
spherical_mercator=true

MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration* – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked 🙂 ).

Other benefits of OpenLayers

As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

    if ( fixmystreet.area ) {
        var area = new OpenLayers.Layer.Vector("KML", {
            strategies: [ new OpenLayers.Strategy.Fixed() ],
            protocol: new OpenLayers.Protocol.HTTP({
                url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                format: new OpenLayers.Format.KML()
            })
        });
        fixmystreet.map.addLayer(area);
        area.events.register('loadend', null, function(a,b,c) {
            var bounds = area.getDataExtent();
            if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
        });
    }

Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

*Broken link removed, Nov 2014

As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.

To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).

Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.

Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.

There don’t appear to be any central official lists of candidates in these elections. Amnesty.org.uk has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations 🙂

Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15” to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.

We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

• Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
• As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
• From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
• Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
• As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
• Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

I’m very pleased to announce that mySociety’s upgraded point and postcode lookup service, MaPit, is public and available to all. It can tell you about administrative areas, such as councils, Welsh Assembly constituencies, or civil parishes, by various different lookups including name, point, or postcode. It has a number of features not available elsewhere as far as I know, including:

• Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! 🙂
• Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
• Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
• History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

In my last blog post, I explained the new service TheyWorkForYou offers to show you what constituency you will be in at the next general election. Now I’m going to show you why you shouldn’t use anything else.

The defintions of the boundaries for the forthcoming constituencies in England were originally published in The Parliamentary Constituencies (England) Order 2007 (SI 2007/1681), based on ward boundaries as they were on 12th April 2005. However, due to some local government changes since that date, The Parliamentary Constituencies (England) (Amendment) Order 2009 (SI 2009/698) was published changing the boundaries for four constituencies – Daventry, South Northamptonshire, Wells, and Somerton & Frome – to be based on the new council wards as they were on 3rd May 2007.

The forthcoming constituencies in Northern Ireland were defined in The Parliamentary Constituencies (Northern Ireland) Order 2008 (SI 2008/1486). In this, Derryaghy ward was split between two constituencies – Belfast West is given “that part of Derryaghy ward lying to the north of the Derryaghy and Lagmore townland boundary.”

All of which means that other sites that try to tell you what constituency you will be in at the election invariably get it wrong.

Both Labour and the Conservatives say that BA6 8NJ is in Wells at the next election, when it will be in Somerton & Frome. Both say that NN12 8NF will be in Daventry, when it will be in South Northamptonshire. I assume that both sites are using boundary data predating the Amendment Order from March 2009. The Conservatives also say that BT17 0XD will be in Lagan Valley when it will be in Belfast West; Labour simply say “Northern Ireland” for any Northern Irish postcode you provide.

The Liberal Democrats site currently returns no results for any postcode, which I assume is a bug 🙂

The /current/ boundary between Belfast West and Lagan Valley. (Image produced from the Ordnance Survey electionmap service. Image reproduced with permission of Ordnance Survey and Land and Property Services)

The official election-maps.co.uk service (from where TheyWorkForYou gets its boundary maps) returns the correct results for BA6 8NJ and NN12 8NF, but doesn’t have future boundaries for Northern Ireland. It’s not clear that it doesn’t, as searching for Lagan Valley with “future boundaries” selected returns a result, but that result is the current boundary. This can be seen from the picture on the right – as is clear from the quote I gave above, everything within Derryaghy ward north of the Lagmore/Derryaghy boundary will be in Belfast West at the next election. Plus the site doesn’t work without JavaScript.

TheyWorkForYou’s “constituency at the next election” service gives BA6 8NJ in Somerton & Frome, NN12 8NF in South Northamptonshire, and BT17 0XD in Belfast West. There is enough confusion with the changes to boundaries for everywhere except Scotland, that it is somewhat frustrating to have it compounded by sites giving incorrect information. The lack of any official service also doesn’t help.