In January last year, at our yearly staff and volunteers retreat, we decided that TheyWorkForYou should do something special for the general election. We decided that we wanted to gather information on where every candidate in every seat stood on what most people would think were the biggest issues, not just nationally but locally too.

Our reasons for setting this ambitious goal were two fold. First, we thought that pinning people down to a survey that didn’t reward rhetorical flourishes would help the electorate cut through the spin that accompanies all elections. But even more important was to increase our ability to hold new MPs to account: we want users of TheyWorkForYou in the future to be able to see how Parliamentary voting records align with campaign statements.

This meant doing quite a lot of quite difficult things:

1. Working out who all the candidates are (thousands of them)
2. Working out how to contact them.
3. Gathering thousands of local issues from every corner of the country, and quality assuring them.
4. Developing a balanced set of national issues.
5. Sending the candidates surveys,  and chasing them up.

The Volunteer Army

This has turned out to be a massive operation, requiring  the creation of the independent Democracy Club set up by the amazing new volunteers Seb Bacon and Tim Green,  and an entire candidate database site YourNextMP, built by another new volunteer Edmund von der Burg.  Eventually we managed to get at least one local issue in over 80% of constituencies, aided by nearly 6000 new volunteers spread from Lands End to John O’Groats. There’s at least one volunteer in every constituency in Great Britain, and in all but three in Northern Ireland. Volunteers have done more than just submit issues, they’ve played our duck house game  to help gather thousands of email addresses, phone numbers, and postal addresses.

The Survey

What we ended up with is a candidate survey that is different for every constituency – 650 different surveys, in short. The survey always contains the same 15 national issues (chosen by a politically balanced panel held at the Institute for Government)  and then anything between zero and ten local issues. We’ve seen everything from cockle protection to subsidies for ferries raised – over 3000 local issues were submitted, before being painstakingly moderated, twice, by uber-volunteers checking for for spelling, grammar, obvious bias and straightforward interestingness (it isn’t really worth asking candidates if they are in favour of Good Things and against Bad Things).

In the last couple of days we’ve started to send out the first surveys – we’ve just passed 1000 emails, and there are at least 2000 still to be sent.

The Output

We’re aiming to release the data we are gathering on candidates positions on 30th April. We’ll build a nice interface to explore it, but we also hope that others will do something with what we are expecting to be quite a valuable dataset.

The Pressure

Candidates are busy people, so how do we get their attention? Happily, some candidates are choosing to answer the survey just because TheyWorkForYou has a well know brand in the political world, but this has limits.

The answer is that we are going to ask Democracy Club, and it’s army of volunteers to help. We’ll shortly roll out a tool that will tell volunteers which of their candidates haven’t taken the opportunity to go on the record , and provide a range of ways for them to push for their candidates to fill it in.

It would be a lie to say we’re confident we’ll get every last candidate. But we are confident we can make sure that no candidate can claim they didn’t see, or didn’t know it was important to their constituents. And every extra voice we have makes that more likely.

Join Democracy Club today

In my last blog post, I explained the new service TheyWorkForYou offers to show you what constituency you will be in at the next general election. Now I’m going to show you why you shouldn’t use anything else.

The defintions of the boundaries for the forthcoming constituencies in England were originally published in The Parliamentary Constituencies (England) Order 2007 (SI 2007/1681), based on ward boundaries as they were on 12th April 2005. However, due to some local government changes since that date, The Parliamentary Constituencies (England) (Amendment) Order 2009 (SI 2009/698) was published changing the boundaries for four constituencies – Daventry, South Northamptonshire, Wells, and Somerton & Frome – to be based on the new council wards as they were on 3rd May 2007.

The forthcoming constituencies in Northern Ireland were defined in The Parliamentary Constituencies (Northern Ireland) Order 2008 (SI 2008/1486). In this, Derryaghy ward was split between two constituencies – Belfast West is given “that part of Derryaghy ward lying to the north of the Derryaghy and Lagmore townland boundary.”

All of which means that other sites that try to tell you what constituency you will be in at the election invariably get it wrong.

Both Labour and the Conservatives say that BA6 8NJ is in Wells at the next election, when it will be in Somerton & Frome. Both say that NN12 8NF will be in Daventry, when it will be in South Northamptonshire. I assume that both sites are using boundary data predating the Amendment Order from March 2009. The Conservatives also say that BT17 0XD will be in Lagan Valley when it will be in Belfast West; Labour simply say “Northern Ireland” for any Northern Irish postcode you provide.

The Liberal Democrats site currently returns no results for any postcode, which I assume is a bug 🙂

The /current/ boundary between Belfast West and Lagan Valley. (Image produced from the Ordnance Survey electionmap service. Image reproduced with permission of Ordnance Survey and Land and Property Services)

The official election-maps.co.uk service (from where TheyWorkForYou gets its boundary maps) returns the correct results for BA6 8NJ and NN12 8NF, but doesn’t have future boundaries for Northern Ireland. It’s not clear that it doesn’t, as searching for Lagan Valley with “future boundaries” selected returns a result, but that result is the current boundary. This can be seen from the picture on the right – as is clear from the quote I gave above, everything within Derryaghy ward north of the Lagmore/Derryaghy boundary will be in Belfast West at the next election. Plus the site doesn’t work without JavaScript.

TheyWorkForYou’s “constituency at the next election” service gives BA6 8NJ in Somerton & Frome, NN12 8NF in South Northamptonshire, and BT17 0XD in Belfast West. There is enough confusion with the changes to boundaries for everywhere except Scotland, that it is somewhat frustrating to have it compounded by sites giving incorrect information. The lack of any official service also doesn’t help.

Parliamentary boundary changes appear to be a source of confusion to many people and organisations. The facts are quite simple – parliamentary boundary changes, proposed by the various Boundary Commissions, do not take effect until the next general election. Until then, your MP remains whoever they have been, no matter what literature you may get through your letter box, or what anyone may tell you.

As one example, take Birmingham City Council. Their page on constituencies and wards correctly states that Birmingham is divided into eleven parliamentary constituencies, but then goes on to list only ten – they are listing the new constituencies which do not yet exist, as Birmingham is losing one constituency at the next election. It appears that they have organised themselves along the new boundaries in advance – which is fine, but this doesn’t affect current Parliamentary representation, and so they should explain this clearly, as otherwise members of the public get confused (and blame us for giving them the “wrong” MP, when we haven’t done so). As you can see from the maps above (which highlight Birmingham, Hall Green), the constituencies will be changing their boundaries quite a bit, and we have had reports of people receiving letters from candidates in the next election who are MPs of different neighbouring constituencies, simply referring to themselves as an MP, which is a great source of confusion.

An inhabitant of St Josephs Avenue, Birmingham (behind the Royal Orthopaedic Hospital), which is currently within the Selly Oak parliamentary constituency (red), and the Northfield ward of Birmingham City Council (green), would, on looking at Birmingham City Council’s website, assume they’re in a parliamentary constituency called Northfield. Northfield is currently the constituency to the west of Selly Oak; at the next election, its boundary with Selly Oak will change to the blue line, at which point St Josephs Avenue will be in the Northfield constituency. But not until then.

As another example (chosen purely as it has come up in user support), the Labour candidate’s website for Streatham had a page about the constituency – obviously you would expect a candidate to be talking about the future constituency, but would it hurt to add some explanation that Streatham is currently a slightly different shape?

Boundaries of different things are all independent – if a ward boundary moves due to some local issue, the corresponding Parliamentary boundary does not necessarily change with it (probably not, in fact). So when Birmingham changed its ward boundaries back in 2003, they became out of sync with the Parliamentary constituencies. From the next election, things will be more in sync as the new Parliamentary boundaries are based on more recent ward boundaries, but this will again separate over time. All we can do is always clearly explain the current situation, and ask that others do the same.

Image from jaygoldman

Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites

I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.

Identifying and categorising these types of site turns out to be surprisingly useful.  First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.

Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.

1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.

2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou in NZ, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch,  Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.

3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.

4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.

5. Government spending – where the big money goes. In the US the dominant site is FedSpending.org, and in the UK we have ukpublicspending.co.uk.

6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and  Statute Law Database. Much of the legal database field, however, remains essentially private.

7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet‘s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.

8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places,  the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others.  mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.

9. Political document archives – This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.

10. Bulk data – Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.

——

Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.

Last week we seemed to spend all week in London. Partly interviewing people, partly redesigning PledgeBank, partly plotting the overthrow of Parliament (joke), partly preparing for the election (thank god it didn’t happen – we’d be far too busy). We even did some general work, scurrying wifi out of the ICA and at one of our trustee’s offices.

As if that wasn’t enough, Stef gave the first of our disruptive technology talks, mainly about Farm Subsidy.org and UNDemocracy. It was interesting, engaging, fantastically attended, and turned into beer and sushi. Adam’s posted up a recording of the talk (scroll down in the comments). Make sure you come to the next one on 1st November.

Whereas new sites are lovely, and I talk about Neighbourhood Fix-It improvements further down, there’s still quite a bit of work that needs to go into making sure our current sites are always up-to-date, working, and full of the joys of spring. Here’s a bit of what I’ve been up to recently, whilst everyone else chats about database upgrades, server memory, and statistics.

The elections last week meant much of WriteToThem has had to be switched off until we can add the new election results – that means the following aren’t currently contactable: the Scottish Parliament; the Welsh Assembly; every English metropolitan borough, unitary authority, and district council (bar seven); and every Scottish council. The fact that the electoral geography has changed a lot in Wales means there will almost certainly be complicated shenanigans for us in the near future so that our postcode lookup continues to return the correct results as much as possible.

Talking of postcode lookups, I also noticed yesterday that some Northern Ireland postcodes were returning incorrect results, which was caused by some out of date entries left lying around in our MaPit postcode-to-area database. Soon purged, but that led me to spot that Gerry Adams had been deleted from our database! Odd, I thought, and tracked it down to the fact our internal CSV file of MLAs had lost its header line, and so poor Mr Adams was heroically taking its place. He should be back now.

A Catalan news article about PledgeBank brought a couple of requests for new countries to be added to our list on PledgeBank. We’re sticking to the ISO 3166-1 list of country codes, but the requests led us to spot that Jersey, Guernsey and the Isle of Man had been given full entry status in that list and so needed added to our own. I’m hoping the interest will lead to a Catalan translation of the site; we should hopefully also have Chinese and Belarussian soon, which will be great.

Neighbourhood Fix-It update

New features are still being added to Neighbourhood Fix-It.

Questionnaires are now being sent out to people who create problems four weeks after their problem is sent to the council, asking them to check the status of their problem and thereby keep the site up-to-date. Adding the questionnaire functionality threw up a number of bugs elsewhere – the worst of which was that we would be sending email alerts to people whether their alert had been confirmed or not. Thankfully, there hadn’t yet been any such alert, phew.

Lastly, the Fix-It RSS feeds now have GeoRSS too, which means you can easily plot them on a Google map.

Lots has happened since I last posted, which was months ago, before Chris died.

Luckily for us, in the last few months Chris had only been working one day a week for us, so it hasn’t been as difficult in practical terms as it could have been. There were various mySociety things running in his flat, such as the WriteToThem fax server, which had to be set up quickly elsewhere.

We miss Chris’s expertise most days (only yesterday I was swearing at gnuplot). Matthew is twice as much for me to handle; he works so quickly, he has hard questions to ask at a ferocious rate that I can’t keep up with. I think before Chris used to handle most of them, so it was much easier for me.

I highly recommend Chris Lightfoot’s obituary by Martin Keegan. Also, Chris’s obituary on Last Word, which was on Radio 4 on 23rd March, is well worth listening to.

We have a few new members of staff.

Keith Garrett is working for us now, mainly tending our servers, but he’s also been working on the E Petitions site. He’s trying to bully us into documenting all our internal processes so it’s easier for new people.

Heather Cronk has started working for us in the US, evangelising PledgeBank. This is funded by the Omidyar Foundation. As well as getting the word out about PledgeBank in the states, this is extra good for us, as she’s forcing us to give PledgeBank the TLC that it deserves.

Deborah Kerr is now doing customer support for us part time. She’s been busy with Neighbourhood Fix-it which has had lots of traffic and attention the last few weeks.

OK, back to the present.

Last week Ben Campbell and I gave a seminar at Technology for a Small Nation in Llandudno. We split into two groups, I got people to add Google maps to an HTML page on their website, and Ben got people to call the TheyWorkForYou API from PHP. It went down well, very satisfying to do practical exercises, and answer all the niggling questions (how to install a testing webserver on Windows, how to use FTP) which are the real things that stop people doing what they want to do.

And Thursday? Just a reminder, unless you’re not apathetic, that there’s an election today.

Despite being strictly non-partisan, mySociety is still indisputibly a political entity that seeks to encourage new and better sorts of things to happen in our democracy. But ‘new’ and ‘better’ are vague words without a clear understanding of what democracy is, or what it could become.

Consequently, mySociety needs a theory of British democracy of the same robustness and quality as the code written by our developers. I am not as good a theorist as my team are programmers, but I will do my best to spell out where we see the weaknesses and opportunities for improvement in the current system.

What I am publishing here is explicity my view, not mySociety’s, and it is most definitely a first draft. I would be very grateful if you would be kind enough to leave comments containing your thoughts and suggestions, so that eventually (perhaps) it can be published as a proper essay or pamphlet and, with luck, enjoy the endorsement of the mySociety developers, trustees and volunteers.

Part 1. Democracy is about far more than voting

Once every four years a majority of all those elegible to vote in the UK make their way to schools and community centres to cast their ballots at the general election. These voters, picking from a range of options usually somewhat smaller than their supermarket choice of canned tuna, put their cross next to their preferred candidate and then make their way home.

The crosses that the voters leave on their ballot papers are painstakingly aggregated to determine the next occupents of 646 prized jobs; few of the jubilant winners understanding at the time what a seismic career mistake they have just made. But the individual constituency results are only the side-show to the main attraction. The real purpose of spending all that time and money on holding an election is to send two bits of information back to Parliament, the public and the Queen: who gets to govern, and how easily. When written down, the main output of the huge logistical exercise of a general election is astonishingly small: Lab 343, Con 196, Lib 63. Just 24 characters to dictate who gets to run the country, and how.

This seemingly esoteric observation highlights of the most remarkable things about democracies: despite the tiny amount of formal instruction given by the public to the political classes, democracies tend to be very different places compared to non-democracies. Clearly these 24 characters (bytes) are very important. With few exceptions democratic states come with a host of other institutions which are rare or unheard of in countries without them: fair courts, freedom of speech, freedom of association and so on.

These institutions haven’t just been created from scratch by democratically formed laws: they’re also the foundations upon which democracies have grown – they’re both the chicken and the egg. But why mention them here?

The answer is that these institutions attempt to provide guidance as to what government should actually do between elections. There are many millions of political decisions that need to be made in between our widely spaced elections, and the 24 bytes of instruction data generated at a general election just doesn’t contain enough detail. The non-electoral institutions of a democratic society are history’s solution to getting as many of those millions of decisions as possible right.

And what a bewildering array there are: from Quangos, to newspapers, to community groups, to industry lobbyists, to PTAs and NGOs to blogs. All ceasely pour out decisions that are vary from pure punditry to powerful statutory judgement.

Collectively these institutions and the decisions they make create a democratic culture which actually shapes our lives more frequently than bellweather issues at elections. Whether a school accepts our child, whether we get to add a skylight to our roof, and whether or not we feel afraid on our streets are determined by organisations that have almost never been subject to elections. And yet these institutions and the decisions they make are normally thought of as ‘democractic’, even though most of them are not formally elected. A community group of a dozen pensioners somehow seems far more democratic than Pravda, despite the latter being read by millions, and neither being elected.

The culture of democratic states isn’t some wishy washy backdrop against which real power politics happens, either. The institutions that shape our democratic culture often act as some of the hardest checks and balances against which politicians bang their heads. Just speak to a politician who has ever been foiled by a concerted media campaign to understand the power of democratic culture.

Where does mySociety fit into this? Well, we build tools that enable people to get better decisions made between elections. Whether that means writing direct to your politicians, or forming a campaign, or keeping politicians on their toes through publishing data on what they say and do, most of the things we build are about enabling our users to get the decisions made that they want to be made.

However, we’re far from alone in the arena of decision-influencing. The next section looks at how it has happened in the past, and how it may change in the future.

So, I’m about to start work on loading the next copy of the BoundaryLine electoral geography data into MaPit, which will give us 100% coverage of county councillors and fix some problems which we weren’t able to work around when we did this after last year’s election. But this is a tedious job and so I’m not going to talk about it now.

Instead I’ll draw your attention to Ratty, our rate-limiting service, which is of general usefulness but (so far as we know) hasn’t been used by anyone outside mySociety. Our major use for it is in the anti-abuse rules in the WriteToThem back-end. I’m about to head out for lunch,so I won’t explain how the thing works in detail, but if this is the sort of thing that you think might be useful to you, leave a comment or drop a mail to hello@mysociety.org with any questions or comments. Like almost all our code, Ratty is licenced under the Affero GPL.

Things have been quiet here recently, but are now getting busy again. Tom’s back from America, Chris is back from holiday, I’m better after being ill for most of last week.

Earlier in the week we finally managed to load new county boundaries into MaPit. So WriteToThem once again has county councils working. Please try it out with your postcode. Let us know of any problems.

This required lots of work from Chris, because a new version of BoundaryLine (from Ordnance Survey) has not yet been released with the updated boundaries. He’s done it using lists of the district council wards which make up the county electoral divisions.
These lists were taken from the Statutory Instruments. This has covered most postcodes, but there are still some where the boundaries were specificed in text (walk along this river etc.) rather than wards. And we don’t have those.

The last couple of days I’ve been turning on lots of things to automate updating of WriteToThem. A cron job now grabs new data on councillors from GovEval once a day, and merges their changes with any changes we’ve made.

It’s automatically emailing GovEval with user submitted corrections to councillor data (the “Have you spotted a mistake in the above list?” link on WriteToThem). Hopefully this will create a virtuous feedback loop of ever improving data quality goodness. Or at least let us keep up with council by-elections without having to lift a finger.

Finally I’ve made it send a mail once a week to the mailing list where WriteToThem admins (mostly volunteers) hang out. This describes what needs doing – such as missing contact details to gather, or messages in the queue which need human attention.

Next up, wiring up the new screenscrapers Richard and Jonathan contributed last week, so the Welsh and London Assemblies automatically update…