We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

• Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
• As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
• From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
• Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
• As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
• Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

They could perhaps have picked a better day, as it was quite serious – at the stroke of midnight on the 1st of April, 37 district councils and 7 county councils in England ceased to exist, replaced by 9 new unitary authorities. This means people in Durham, Northumberland, Cornwall, Shropshire, Wiltshire, Chesire, and Bedfordshire only have one principal local authority to deal with now. The Wikipedia article on the changes has more information on the background to this change.

Obviously this meant some work for WriteToThem and FixMyStreet, both of which require up-to-date local council information. Our database of voting areas, MaPit, has “generations”, so we can keep old areas around for various historical purposes. So firstly, I created a new generation and updated all the areas that weren’t affected to the new generation. Next, six of the new unitary authorities (all the counties except Cheshire and Bedfordshire, plus Bedford) share their boundaries and wards with the coterminous councils they’re replacing, so for them it was a simple matter of updating those councils to be unitary authorities.

That left Bedfordshire and Cheshire. I created areas for the three new councils (Cheshire West and Chester, Cheshire East, and Central Bedfordshire), and transferred across the relevant wards from the old county councils – basically a manual process of working out the list of correct ward IDs.

WriteToThem was now dealt with, but FixMyStreet needed a little more work. The councils that no longer existed had understandably disappeared from the all reports table, so I had to modify the function that fetches the list of councils to optionally return historical areas so they could be included. And lastly, FixMyStreet needs a way of mapping a point on a map to the relevant council. For this, it needs to know the area covered by a council, which was missing for the new authorities I’d manually created. Thankfully, each of the three new authorities are made up of the areas of either 2 or 3 district councils (e.g. Cheshire East is the area covered by Congleton, Macclesfield, and Crewe and Nantwich), so I just had to write a script that stuck those areas together to create the area of the new council. It all seems to work, and I’m sure our users will be in touch if it doesn’t 🙂

So goodbye to Alnwick, Bedfordshire, Berwick-upon-Tweed, Blyth Valley, Bridgnorth, Caradon, Carrick, Castle Morpeth, Cheshire, Chester, Chester-le-Street, Congleton, Crewe and Nantwich, Derwentside, Durham City, Easington, Ellesmere Port and Neston, Kennet, Kerrier, Macclesfield, Mid Bedfordshire, North Cornwall, North Shropshire, North Wiltshire, Oswestry, Penwith, Restormel, Salisbury (which is getting a new town council), Sedgefield, Shrewsbury and Atcham, South Bedfordshire, South Shropshire, Teesdale, Tynedale, Vale Royal, Wansbeck, Wear Valley, and West Wiltshire. RIP.

We updated our boundary and postcode database at the start of the week (apart from two wards in Scotland that I misspelled and updated on Tuesday, sorry), so hopefully everyone in the country can contact their representatives at WriteToThem or have their postcode recognised on HearFromYourMP or TheyWorkForYou. This applies especially to a small number of councils, such as Bradford, for which the boundaries had completely changed at their last election and which we were unable to get working until now – apologies for the inconvenience.

Related to this, and for interest, on 1st April, a number of councils are being abolished as their county councils become unitary authorities. The district councils within Durham, Northumberland, Cornwall, Wiltshire, Shropshire, and Cheshire/Chester all disappear – Cheshire becomes two unitary authorities called Cheshire West and Chester, and Cheshire East. Lastly, Bedford borough council becomes a unitary authority, and Central Bedfordshire council covers the area previously covered by Mid Bedfordshire and South Bedfordshire.

Parliamentary boundaries in England and Northern Ireland are changing, but these do not take effect until the next general election – until then, your constituency and MP remains the same.

Last month saw the launch of not one but two websites asking the public to report empty properties to the relevant council. First off the blocks was mySociety’s ReportEmptyHomes.com, commissioned by the Empty Homes Agency, followed shortly afterwards by EveryHomeCounts.info from a group of eight councils in Surrey and Hampshire. Since mySociety claims to want to show the public sector how to use the internet properly, I thought it might be interesting to compare the two sites, at least from a user’s perspective.

I’m going to imagine I was walking down, say, Fosters Lane in Knaphill, Surrey, and I noticed that the house on the corner next to the chip shop was in a state of disrepair.* I snapped a picture on my mobile phone, and I want to send it to the council to see if they can do something about it.
[*I ought to just add that this is entirely fictional. I’ve never been to Knaphill, I’ve no idea whether there’s a chippy on Fosters Lane, and even if there is, the house next to it probably belongs to a lovely couple. Please don’t go taking pictures of their house for the council.]

So, first up: EveryHomeCounts.info. Clicking the big red REPORTING AN EMPTY PROPERTY button takes me to a page of text telling me why the council might like people to report empty properties, although presumably if I’ve got as far as finding the website and clicking the big red button, I’m already convinced of the case. At the bottom of the text I’m invited to “click here” to report an empty property.

On the next page I’m asked for… a whole load of personal information. I want to tell you about an empty house; do I really need to declare my title, first name, surname, house name, house number, street, locality, town, county, postcode, country, telephone number, and email address before doing so? Well, as it turns out, no — they only insist on an email address (although the single letter “f” was accepted as a valid email address).

On to page four, and I’m finally asked for the address of the house. I suppose “house on the corner next to the chippy, Queens Road, Knaphill” would probably be enough for the council to identify it. But then — get this — they want me to tell them which borough council might be responsible for this address, so that the report can be sent to the right place! Unless I happen to live in that street, how would I know? Even if I could have an educated guess, it might be near a boundary, or just over a boundary… Leaving the field blank isn’t allowed, and there’s no option that says “I’m not sure, sorry” — I’m told in red ink that I must specify a council if I want to continue filing my report.

Finally, I reach a screen that says at the top, “Thank you. You have reached the end of this form. blah blah” The second paragraph says, “What will happen next? The council will process your form. You will receive an email blah blah.” So, I pat myself on the back, turn off the computer and go for a walk. Except that if I’d scrolled down the page, I would have seen “submit” button, along with the “review” and “cancel” buttons. My form hasn’t been submitted, and I’ve wasted half an hour filling in a form that’s been thrown away.

Now, what would have happened if I’d gone to ReportEmptyHomes.com instead?

The top of the front page asks me for a postcode, street name or area. I enter “Fosters Lane, Knaphill” and hit enter. This brings up an Ordnance Survey map with Fosters Lane in the middle of it, and I click on the offending property. The text on the page immediately changes and tells me that this problem falls in the area of Woking Borough Council, and I’m asked for a description of the property, a photo if I’ve got one to upload, my name, email and phone number.

Having filled in the information and clicked “submit”, I’m told to go off and check my emails, where I’ll find a confirmation link to click. This finalises the report.

So, how do the two sites compare? The mySociety site certainly gets the user through the process quicker, and offers maps and photos to boot. It helps the user greatly by taking responsibility for finding the right council, and does so for the whole country too, not just for a couple of counties in the south. On the down side, one could question why it’s so important to verify the user’s email address before filing the report; waiting for a confirmation link by email adds an extra hurdle which will probably trip at least some users, so why do it?

Also, EveryHomeCounts.info isn’t just for filing reports about empty homes; it contains information on buying, selling, owning and letting them too, providing ways for local people to perhaps make use of empty properties without enlisting the council’s help at all, which can only be a good thing.

To be fair, the councils concerned should be applauded for taking the initiative to launch this service, and I hope it proves to be a worthwhile use of council tax money. It’s great to see public bodies using the internet in innovative ways to try to make concrete improvements in people’s immediate environment. It appears though that mySociety have shown that it can be done better, and for the whole country, and probably more cheaply to boot.

Recently I gave a talk at a conference where I told a group of local government officials that FixMyStreet was built not just to provide cleaner streets for their citizens, but also to force the hands of councils to procure and contract internal IT systems fit for the 21st century. In particular I pointed out that companies like Google seek to have people use their service from any site, any browser and device – they don’t just demand that everyone goes to www.google.com. And, I said, it’s only through building nice interfaces (APIs) that you can become an organisation that realises the benefits for yourself and other organisations from taking this ‘we’re happy to interoperate with anyone’ approach.

Less than three weeks later Michael Houlsby from East Hants council has single-handedly built an external facing API for their faults and problems database. So now FixMyStreet posts problems in that council direct into their database, without them first being translated into emails.

This is fantastic, especially as Michael clearly knocked it together in his spare time, and helps confirm what we’ve said before – if government builds nice interoperable APIs people like mySociety will use them to improve citizens’ experiences, whist simultaniously keeping everyone’s unnecessary workloads and expenses to a minimum. Plus it shows that if your IT supplier tells you you need to sign a new five or six figure contract to add an API to a CRM system you’ve already bought – you’re being jerked around.

Hats off to Michael – you’re a great example of a pro-active public servant using your skills to make government both better and more efficient.

Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.

We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.

I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.

• TheyWorkForYou opens up access to conventional, representational democracy, between and during elections. In 10 years time, I asked for Parliament to publish all information about its work in a structured way, as hinted at in our Free Our Bills campaign. So it is much easier for everyone to help make new laws better.
• FixMyStreet is local control of the things people care about, a very practical democracy. In 10 years time I’d like to see all councils running their internal systems (planning, tree preservation orders… everything that isn’t about individuals) in public, so everyone can see and be reassured about what is being done, why and where.
• WhatDoTheyKnow shows the deep interest that there is by the public in the functioning of all areas of government. In 10 years time, I’d like to see document management systems in wide use by public authorities that publish all documents by default. Only if overridden for national security or data protection reasons would they be hidden.

Charles Armstrong, from the Circus Foundation, has written up the workshop.

Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort – Charles is going to work on it for another time.

Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).

Obviously it’s always great when any paper gives mySociety coverage – it helps get the word about our services out and helps more people get things done that help their lives.

However, today’s look at mySociety’s 5 years in the Guardian makes a few claims I think it’s important to challenge, so instead of writing to the readers editor I thought I’d just seize the power of Citizen Media(TM) to note them here.

First, has the No10 petitions site had “little notable impact” on government policy? Given that that project appears almost single handedly to have bounced Parliament into developing an online petitioning system and devoting debate time to major petitions, I’d say that it certainly has had some impact. But there is indeed a bigger problem of pointing at No10 petitions and going “That one changed policy.” It’s a problem of two halves: scale, and deniability.  Governments almost never acknowledge that they were forced into anything, ever. Policy announcements are almost always framed as if the right course of action was being followed all along. So apart from the fact that I don’t know how one could possibly assess the impacts of so many thousands of petitions without a huge research project, I would expect that even those that do have in impact will still usually be denied by the government, even when shifting policy. I would encourage No10 and the whole of Government to take a look at directly challenging this culture, and employ someone whose job it is to find out which petitions are having an impact, and shout about them in plain English.

Second, the majority of mySociety’s sites are programmed by staff and contractors, not volunteers. The volunteers are super-essential to mySociety running every day, but the sheer size of some of our projects makes it unlikely a volunteer could have built them without giving up their day job for many months. This needs mentioning to explain why it matters if our finances are precarious!

Next – do councils find FixMyStreet an irritation or an asset? Well, last time we did a count a few weeks ago, we had 4 complaining emails from councils, and 62 supportive ones, with several linking directly to us. As for the Customer Relationship Management at councils, we’d be delighted to send reports straight into their databases without going via email first, it’s just that only one council has set up such an interface so far. I hope that FixMyStreet can put pressure on councils and their suppliers to build a small number of standardised interfaces for the good of everyone. And yes, we are building FixMyStreet for iPhone and Android, and I’m happy to talk to anyone who wants to build UIs for any other phones.

There – hope that doesn’t come across as too ungrateful to Michael Cross et al. See you at the next birthday party, I hope!

Update: I also meant to mention that I’ve never been a ‘Downing Street Insider’. I was a junior civil servant in the Prime Minister’s Strategy Unit, which is not in Downing Street and more loosely affiliated than the name might suggest.

FixMyStreet has a lot of RSS feeds. There’s one for every one-tier council (170), one for every ward of every one-tier council (another 5,044), two for every two-tier (county and district) council (544), and two for every ward of every two-tier council (20,296) – two per two-tier council because you might want either problems reported to one council of a two-tier set-up in particular, or all reports within the council’s boundary.

Then there’s an RSS feed every 162m across Great Britain in a big grid, returning all reports within a radius of that point, the radius by default being automatically determined by that point’s population density, but customisable to any distance if preferred. That’s, at a very rough approximation assuming Great Britain is a rectangle around its extremities, which it’s not, 19 million RSS feeds, lots of which will admittedly be very similar. 🙂

Every single one of those feeds can be subscribed to by email instead if that’s preferable to you, and are all accessible through a simple interface at http://www.fixmystreet.com/alert.

However, none of these RSS feeds was suitable for the person who emailed from a Neighbourhood Watch site and said that all they had was a postcode and they wanted to display a feed of reports from FixMyStreet. Given you could obviously look up a FixMyStreet map by postcode, it did seem odd that I hadn’t used the same code for the RSS feeds. Shortly thereafter, this anomaly was fixed, and if you now go to a URL of the form http://www.fixmystreet.com/rss/pc/postcode you will be redirected to the appropriate local reports feed for that postcode (I could say that adds another 1.7 million RSS feeds to our lot, but given they’re only redirects, that’s not strictly true). And after a couple more emails, I also added pubDate fields to the feeds which should make displaying in date order easier.

It’s great to see our RSS feeds being used by other sites – other examples I’ve recently come across include Brent Council integrating FixMyStreet into their mapping portal (select Streets, then FixMyStreet), or the Albert Square and St Stephen’s Association listing the most recent Stockwell problems in their blog sidebar. If you’ve seen any notable examples, do leave them in the comments.

The first new councillor details have begun to automatically arrive in our database, thanks to GovEval. 34 councils were reactiviated on WriteToThem today, from Alnwick to Wokingham. It would have been 38 but the other 4 councils have had boundary changes that we don’t have the data for yet.

16 of the 40 Welsh Assembly constituencies did not change their boundaries at the election (this took some time to work out, as the Press Association said it was 18, and the official report from the Boundary Commission for Wales said it was 17 🙂 ). Those 16 Assembly Members are now also reactiviated on WriteToThem, along with their regional AMs.

Other than that, I’ve continued tweaking Neighbourhood Fix-It and started some work on TheyWorkForYou – the first step of which is to deal with the large backlog of mail that’s accumulated, leading to a number of bugfixes. Apologies to anyone who was trying to look at Brian Wilson MLA‘s page and found themselves stuck in an infinite loop of being told there were two Brian Wilsons. We also had a couple of emails asking us why Gordon Brown didn’t have a voting record on equal gay rights like other MPs. This was easy to answer – he’s never voted in any division that is included by that PublicWhip policy – and so an MP’s page now states if they’ve been absent from every vote on a particular policy.

Whereas new sites are lovely, and I talk about Neighbourhood Fix-It improvements further down, there’s still quite a bit of work that needs to go into making sure our current sites are always up-to-date, working, and full of the joys of spring. Here’s a bit of what I’ve been up to recently, whilst everyone else chats about database upgrades, server memory, and statistics.

The elections last week meant much of WriteToThem has had to be switched off until we can add the new election results – that means the following aren’t currently contactable: the Scottish Parliament; the Welsh Assembly; every English metropolitan borough, unitary authority, and district council (bar seven); and every Scottish council. The fact that the electoral geography has changed a lot in Wales means there will almost certainly be complicated shenanigans for us in the near future so that our postcode lookup continues to return the correct results as much as possible.

Talking of postcode lookups, I also noticed yesterday that some Northern Ireland postcodes were returning incorrect results, which was caused by some out of date entries left lying around in our MaPit postcode-to-area database. Soon purged, but that led me to spot that Gerry Adams had been deleted from our database! Odd, I thought, and tracked it down to the fact our internal CSV file of MLAs had lost its header line, and so poor Mr Adams was heroically taking its place. He should be back now.

A Catalan news article about PledgeBank brought a couple of requests for new countries to be added to our list on PledgeBank. We’re sticking to the ISO 3166-1 list of country codes, but the requests led us to spot that Jersey, Guernsey and the Isle of Man had been given full entry status in that list and so needed added to our own. I’m hoping the interest will lead to a Catalan translation of the site; we should hopefully also have Chinese and Belarussian soon, which will be great.

Neighbourhood Fix-It update

New features are still being added to Neighbourhood Fix-It.

Questionnaires are now being sent out to people who create problems four weeks after their problem is sent to the council, asking them to check the status of their problem and thereby keep the site up-to-date. Adding the questionnaire functionality threw up a number of bugs elsewhere – the worst of which was that we would be sending email alerts to people whether their alert had been confirmed or not. Thankfully, there hadn’t yet been any such alert, phew.

Lastly, the Fix-It RSS feeds now have GeoRSS too, which means you can easily plot them on a Google map.