My colleague Alex has already written about looking forward from this election, so here I am going to look back at the technical work that was involved for the election, and in getting all the new MPs into TheyWorkForYou.

Boundary changes

This election was the first UK Parliament election with boundary changes since 2010. Due to the long-running nature of TheyWorkForYou, which has been around now for over 20 years, this can throw up some interesting challenges. In this particular case, it turned out we were using two different JSON data lists of constituencies – both containing the same data, but one also included the other Parliaments and Assemblies, whilst the other included alternative names for some constituencies. I took the opportunity presented to merge these together and update the bits of code to use the one consolidated dataset, and then added in the 650 new constituencies to the JSON data.

Loading the new constituency data into TheyWorkForYou then threw up another historical problem – the constituency table was still using the very old Latin-1 character set encoding, rather than a more modern encoding such as UTF-8, that almost everything we have uses. This had been fine until now, with even Ynys Môn covered by that encoding, but the new constituency of Montgomeryshire and Glyndŵr contained a letter that Latin-1 could not cope with, leading to a quick emergency upgrade of the table to UTF-8 (thankfully this is a backwards compatible encoding, so worked without issue).

We had already generated data of the new constituencies and loaded these into our lookup service MapIt before Christmas. Ordnance Survey more recently published the official dataset of the boundaries, which we could then import via our usual processes, though even this raised a small issue to be resolved. It turned out in the last data release OS had given the parts of two county council electoral divisions with detached parts (Lightwater, West End and Bisley and Thorpe St Andrew) different identifiers, which they had reverted in their new release, causing our import script to get a bit confused – resolved with a small manual script.

Displaying on TheyWorkForYou

In the period before the election, we knew people would be using our site as a postcode lookup, perhaps to look up their previous MP but perhaps also expecting something useful for the upcoming election, which we wanted to provide, and so we used Democracy Club’s API to show election candidates and link to their WhoCanIVoteFor and WhereDoIVote services. We also displayed your boundary changes using the new constituency data mentioned above.

TheyWorkForYou isn’t just the UK Parliament, though, it also covers the Scottish and Welsh Parliaments, and the Northern Ireland Assembly, so we also had to maintain the provision of that information to people – email alerts for those bodies continued throughout as usual, and the postcode lookup kept showing people their representatives in the devolved nations.

Once the election closed, we automatically updated our messaging, and the next day switched back to our normal behaviour of taking you directly to your MP page in England, and showing you your MP and other representatives elsewhere.

We had a fun issue where some people were getting their new MP, whereas some were getting the old MP – during the period of dissolution, when there are no MPs, we have a configuration flag to enable the site to know it should return the latest result even if it’s not current (you don’t want this all the time, when e.g. an MP has resigned or died), but once new data was being loaded in, one database query was returning results in a random order; fixed by adding some sorting by descending end date.

Election result data

At the last election in 2019, we took a live feed of election results from Democracy Club, who have collected all the candidate information for their Who Can I Vote For service – which all began as the result of a mySociety project back in 2010.

Democracy Club were performing the same service this time, and gratifyingly it was quite a small change to have our 2019 code work with any 2024 changes to the source information (incidentally, there aren’t a lot of narrative doctests in our codebase, but I quite like the one in use there!).

This script would do half the job, of taking in some source data (who has been elected, and including their TheyWorkForYou identifier if they already had one due to being a previous representative of some sort) and amending our source JSON data to add the newly elected representative.

The other half is loading that source data into the TheyWorkForYou database for display on the site. Our normal loading script works fine, but looks through all the source data to see if there have been any changes to take account of. For the election, we don’t need it to do all that, so I tweaked the script to only do the minimal necessary to load in newly created information.

These two scripts were then added to a cron on our server, running every few minutes through the night. I did stay up long enough to check that the first few worked okay, before leaving it to itself from then on. I also set it up to pipe its output to our Slack channel, so people could see it operating:

This also meant as the final few trickle through, it’s popping up reminding us it’s still doing its job:

All the results (bar the one we’re still waiting for) are now committed to the repository, joining all our other open data.

Support TheyWorkForYou and our work

TheyWorkForYou and WriteToThem are run by mySociety, a small UK charity. We’re a very efficient operation and do a lot with a small team; if we had bit more money, we could achieve a lot more.

We want to see a transparent, resilient democracy, with equal access to information, representation and voice for citizens. If you believe in this vision please donate today to enable greater transparency and accountability of the next government.

—

Image: Moritz Kindler

This a guest blog by Joe Mitchell from Democracy Club, a non-profit whose aim is to create the digital foundations to support everyone’s participation in democratic life.

The TL; DR

Democracy Club has produced an election information widget that you can add to any website. It’s free. It provides candidate and, where we have it, polling location information for any postcode.

Check it out:

The backstory

As you may be aware, the UK Parliamentary General Election will take place on 12 December.

You may be less aware that no public body takes responsibility for accurate, locally relevant digital information about elections.

Local governments publish election and candidate data as PDFs on their websites (or, in one notorious case, simply printed out and stuck up on the noticeboard outside the council offices).

Polling location information is printed on cards and sent, in theory, to every voter. It sometimes never arrives. And good luck if you live in a house of multiple occupancy, if you struggle to read the print, if you lose it or if you don’t have it with you when you need to refer to it.

Digital technology has massively improved access to information in many other areas of our lives, so a group of volunteer developers and digital types got together to try to apply the approach to elections in the UK.

That group formed Democracy Club, which is now several thousand volunteers and a small core team constantly working to bring together election, candidate and results data. We also work with local government to aggregate local polling location data and make it available online.

The candidate data we produce is published openly and is used by news media, campaign organisations and, ultimately, voters to learn more and participate in the campaigns. The polling location data we produce is available via an API.

Not everyone has the time to develop a stand-alone product with our data. So we produced a polling location finder widget, which has been popular among local newspapers and local councils. Today, we’ve introduced a widget which includes  candidates data too. Users pop in their postcode and away they go. They can click through to a candidate’s page on WhoCanIVoteFor.co.uk for more information.

The civic need for this information is clear. At the last general election, polling location data was accessed (via our website, The Electoral Commission’s website or via the widget) over 1.8m times at the last general election. Candidates information was accessed over 1m times, but this doesn’t count all the uses powered by a one-off CSV download.

The fact that this information — increasingly critical to our functioning as a democratic society — is managed and produced by a tiny non-profit is not a ringing endorsement of our democratic institutions’ fitness for the 21st century.

Democracy Club is working hard to convince public bodies to take on the basic open data elements of our work: when are elections happening, for which area, who are the candidates, what is their preferred contact method, what were the results, etc. And civic user needs go beyond elections — we can’t get to a world of user-friendly, accessible information about democratic processes until the raw data exists for local democracy too.

But for now, at least there’s a widget.

Let us know what you think! You’re welcome to hop into the Democracy Club Slack — or reach out via Twitter or email.

—

Image: Justgrimes (CC by-sa/2.0)

As you’ll know if you’re a regular reader of this blog, YourNextMP crowd-sourced details of every candidate who stood in the UK general election.

But, just because our own election is over, doesn’t mean we’ll be letting YourNextMP gather dust. On the contrary—we want to see it being re-used wherever there are elections being held, and citizens needing information! We’re already seeing the first re-use case, and we’d love to see more.

Opening up data

YourNextMP’s main purpose was to provide a free, open database of candidates, so that anyone who wanted to could build their own tools on top of it, and it was very successful with that aim.

We heard of more than twenty projects which used the data, some small scale operations built by a single developer, some big names such as Google, and national newspapers like the Guardian.

The traditional source of candidate data for such projects has been through expensive private providers, not least because the official candidate lists are published just a few days before the election.

Thanks to YourNextMP’s wonderful crowd-sourcing and triple-checking volunteers, we reckon that we had the most complete, most accurate data, the earliest. And it was free.

Directly informing over a million citizens

YourNextMP also came into its own as a direct source of information for the UK’s electorate. This hadn’t been the priority when the project was launched, but it was helped greatly by the fact that constituency and candidate pages ranked very highly in search engines from early on, so anyone searching for their local candidates found the site easily.

Once they did so, they found a list of everyone standing in their constituency, together with contact details, links to their online profiles such as web pages, social media and party websites, and feeds from spin-off projects (themselves built on YourNextMP data) such as electionleaflets.org and electionmentions.com.

YourNextMP had more than a million unique users. In the weeks just prior to the UK general election, it was attracting approximately 20,000 visitors per day, and on the day before the election, May 6th, there was suddenly a massive surge: that day the site was visited by nearly 160,000 people.

So, in a nutshell: YourNextMP has not only enabled a bunch of projects which helped people become more informed before our election—it also directly informed over a million citizens.

A reusable codebase

YourNextMP was built on Poplus Components as a Democracy Club project: PopIt (for storing the candidates’ names) and MapIt (for matching users’ postcodes with their constituencies).

And, in the spirit of Poplus, the codebase is open for anyone to re-use in any country.

It’s already being pressed into use for the upcoming elections in Argentina, and we hope that developers in many other countries will use it to inform citizens, and inspire great web tools for the electorate, when their own elections come around.

If that’s something that interests you, please come and talk, ask questions and find out what’s involved, over on the Democracy Club mailing list.

—

Image: KayVee (CC)

Thanks to the work of thousands of volunteers across the country, we’ve now launched our survey of candidates to be your MP.

View the survey results

It tells you the views of candidates on a range of national and local issues. What’s particular exciting is that this is individual views – we separately surveyed all the candidates.

About 1/3rd of them have replied. The survey has a tool to let you ask the other candidates in your constituency to respond. Please give it a go, as we’d like to the survey to get as complete as possible over the weekend, to be most use to people in the last days leading up to the election.

Competition! Have you found an inventive way to ask your candidates to respond to the survey? Maybe you doorstepped them, or sent them a cake. Post your ideas and things you’ve done in the comments below.

As you may know, TheyWorkForYou are conducting a survey of candidates for Parliament.

Quite a few people have been asking how we worked out the questions. There are two parts to this, one local and one national.

Local questions

We used the power of volunteers.

Thousands of DemocracyClub members were asked to suggest local issues in there area. These were then edited by other volunteers, to have consistent grammar, and be worded as statements to agree/disagree with, and filtered to remove national issues. The full criteria and examples are available.

You can view the issues for any constituency on the DemocracyClub site. They are in the “local questions” tab.

We’ve ended up with local issues for about 85% of constituencies. They’re really interesting and high quality, and quite unique for a national survey.

Thank you to all the volunteers who helped make this happen!

National questions

This was hard, because we felt that asking more than 15 questions would make the survey too long. We also wanted to be sure it was non-partisan.

We convened a panel of judges, either from mySociety/Democracy Club or with professional experience in policy, and from across the political spectrum. They were:

• James Crabtree, chair of judges, trustee of mySociety, journalist for Prospect magazine
• Tim Green, Democracy club developer, Physics student, Cambridge University.
• Michael Hallsworth, senior researcher, Institute for Government.
• Will Davies, sociologist at University of Oxford, has worked for left of centre policy think tanks such as IPPR and Demos.
• Andrew Tucker, researcher at Birkbeck, worked for Liberal Democrats from 1996-2000.
• Robert McIlveen, research fellow, Environment and Energy unit at Policy Exchange, did PhD on Conservative party election strategy.

They met at the offices of the Institute for Government, and had a 3 hour judging session on 29th March 2010. They were asked to think of 8-15 questions, with multiple choice answers, which could usefully be answered both by members of the public and prospective candidates for national office.

To ensure maximum transparency, the discussions of the judges were recorded. You can download the recordings in two parts: part 1, part 2 (2 hours, 20 mins total).

Details of the broad framework the judges operated under are given by the chair of judges, James Crabtree, a trustee of mySociety, in the opening to the recordings.

Please do ask any questions in the comments below.

The following is a message that we’d like to see emailed around within political parties of all stripes. If you work for a party, or know anyone who does, please send it along:

———-

Hi there,

TheyWorkForYou.com has sent online surveys to nearly 3000 candidates across the UK, including most of your party’s candidates. If you don’t know it, TheyWorkForYou is probably the largest politician transparency website in the UK, with about 3m visitors last year.

The survey we’ve sent is a rigorously neutral attempt to clarify candidates positions on many of the biggest issues at the election. It is also a long-term document – the data that comes from candidate responses will be viewed millions of times between now and the general election after this one. It also contains both local and national questions.

There are 6000+ volunteers now nagging non-responsive candidates. You can help your party improve its responsiveness rating, here, by passing on the word that TheyWorkForYou’s survey is not push-polling, not single issue, not short-termist.

Please help us by passing on the message that TheyWorkForYou will be one of the main ways that new MPs from all parties (and none) will be scrutinised and neither we nor new MPs want to start our relationship with a “refused to go on the record” badge on their pages.

If you are a candidate, and you want to do the survey, check your email for TheyWorkForYou (no spaces). If you don’t have it, drop a mail to developers@democracyclub.org.uk and it’ll be sent along shortly.

many thanks,

The staff and volunteers at TheyWorkForYou and Democracy Club

In January last year, at our yearly staff and volunteers retreat, we decided that TheyWorkForYou should do something special for the general election. We decided that we wanted to gather information on where every candidate in every seat stood on what most people would think were the biggest issues, not just nationally but locally too.

Our reasons for setting this ambitious goal were two fold. First, we thought that pinning people down to a survey that didn’t reward rhetorical flourishes would help the electorate cut through the spin that accompanies all elections. But even more important was to increase our ability to hold new MPs to account: we want users of TheyWorkForYou in the future to be able to see how Parliamentary voting records align with campaign statements.

This meant doing quite a lot of quite difficult things:

1. Working out who all the candidates are (thousands of them)
2. Working out how to contact them.
3. Gathering thousands of local issues from every corner of the country, and quality assuring them.
4. Developing a balanced set of national issues.
5. Sending the candidates surveys,  and chasing them up.

The Volunteer Army

This has turned out to be a massive operation, requiring  the creation of the independent Democracy Club set up by the amazing new volunteers Seb Bacon and Tim Green,  and an entire candidate database site YourNextMP, built by another new volunteer Edmund von der Burg.  Eventually we managed to get at least one local issue in over 80% of constituencies, aided by nearly 6000 new volunteers spread from Lands End to John O’Groats. There’s at least one volunteer in every constituency in Great Britain, and in all but three in Northern Ireland. Volunteers have done more than just submit issues, they’ve played our duck house game  to help gather thousands of email addresses, phone numbers, and postal addresses.

The Survey

What we ended up with is a candidate survey that is different for every constituency – 650 different surveys, in short. The survey always contains the same 15 national issues (chosen by a politically balanced panel held at the Institute for Government)  and then anything between zero and ten local issues. We’ve seen everything from cockle protection to subsidies for ferries raised – over 3000 local issues were submitted, before being painstakingly moderated, twice, by uber-volunteers checking for for spelling, grammar, obvious bias and straightforward interestingness (it isn’t really worth asking candidates if they are in favour of Good Things and against Bad Things).

In the last couple of days we’ve started to send out the first surveys – we’ve just passed 1000 emails, and there are at least 2000 still to be sent.

The Output

We’re aiming to release the data we are gathering on candidates positions on 30th April. We’ll build a nice interface to explore it, but we also hope that others will do something with what we are expecting to be quite a valuable dataset.

The Pressure

Candidates are busy people, so how do we get their attention? Happily, some candidates are choosing to answer the survey just because TheyWorkForYou has a well know brand in the political world, but this has limits.

The answer is that we are going to ask Democracy Club, and it’s army of volunteers to help. We’ll shortly roll out a tool that will tell volunteers which of their candidates haven’t taken the opportunity to go on the record , and provide a range of ways for them to push for their candidates to fill it in.

It would be a lie to say we’re confident we’ll get every last candidate. But we are confident we can make sure that no candidate can claim they didn’t see, or didn’t know it was important to their constituents. And every extra voice we have makes that more likely.

Join Democracy Club today