-
My colleague Alex has already written about looking forward from this election, so here I am going to look back at the technical work that was involved for the election, and in getting all the new MPs into TheyWorkForYou.
Boundary changes
This election was the first UK Parliament election with boundary changes since 2010. Due to the long-running nature of TheyWorkForYou, which has been around now for over 20 years, this can throw up some interesting challenges. In this particular case, it turned out we were using two different JSON data lists of constituencies – both containing the same data, but one also included the other Parliaments and Assemblies, whilst the other included alternative names for some constituencies. I took the opportunity presented to merge these together and update the bits of code to use the one consolidated dataset, and then added in the 650 new constituencies to the JSON data.
Loading the new constituency data into TheyWorkForYou then threw up another historical problem – the constituency table was still using the very old Latin-1 character set encoding, rather than a more modern encoding such as UTF-8, that almost everything we have uses. This had been fine until now, with even Ynys Môn covered by that encoding, but the new constituency of Montgomeryshire and Glyndŵr contained a letter that Latin-1 could not cope with, leading to a quick emergency upgrade of the table to UTF-8 (thankfully this is a backwards compatible encoding, so worked without issue).
We had already generated data of the new constituencies and loaded these into our lookup service MapIt before Christmas. Ordnance Survey more recently published the official dataset of the boundaries, which we could then import via our usual processes, though even this raised a small issue to be resolved. It turned out in the last data release OS had given the parts of two county council electoral divisions with detached parts (Lightwater, West End and Bisley and Thorpe St Andrew) different identifiers, which they had reverted in their new release, causing our import script to get a bit confused – resolved with a small manual script.
Displaying on TheyWorkForYou
In the period before the election, we knew people would be using our site as a postcode lookup, perhaps to look up their previous MP but perhaps also expecting something useful for the upcoming election, which we wanted to provide, and so we used Democracy Club’s API to show election candidates and link to their WhoCanIVoteFor and WhereDoIVote services. We also displayed your boundary changes using the new constituency data mentioned above.
TheyWorkForYou isn’t just the UK Parliament, though, it also covers the Scottish and Welsh Parliaments, and the Northern Ireland Assembly, so we also had to maintain the provision of that information to people – email alerts for those bodies continued throughout as usual, and the postcode lookup kept showing people their representatives in the devolved nations.
Once the election closed, we automatically updated our messaging, and the next day switched back to our normal behaviour of taking you directly to your MP page in England, and showing you your MP and other representatives elsewhere.
We had a fun issue where some people were getting their new MP, whereas some were getting the old MP – during the period of dissolution, when there are no MPs, we have a configuration flag to enable the site to know it should return the latest result even if it’s not current (you don’t want this all the time, when e.g. an MP has resigned or died), but once new data was being loaded in, one database query was returning results in a random order; fixed by adding some sorting by descending end date.
Election result data
At the last election in 2019, we took a live feed of election results from Democracy Club, who have collected all the candidate information for their Who Can I Vote For service – which all began as the result of a mySociety project back in 2010.
Democracy Club were performing the same service this time, and gratifyingly it was quite a small change to have our 2019 code work with any 2024 changes to the source information (incidentally, there aren’t a lot of narrative doctests in our codebase, but I quite like the one in use there!).
This script would do half the job, of taking in some source data (who has been elected, and including their TheyWorkForYou identifier if they already had one due to being a previous representative of some sort) and amending our source JSON data to add the newly elected representative.
The other half is loading that source data into the TheyWorkForYou database for display on the site. Our normal loading script works fine, but looks through all the source data to see if there have been any changes to take account of. For the election, we don’t need it to do all that, so I tweaked the script to only do the minimal necessary to load in newly created information.
These two scripts were then added to a cron on our server, running every few minutes through the night. I did stay up long enough to check that the first few worked okay, before leaving it to itself from then on. I also set it up to pipe its output to our Slack channel, so people could see it operating:
This also meant as the final few trickle through, it’s popping up reminding us it’s still doing its job:
All the results (bar the one we’re still waiting for) are now committed to the repository, joining all our other open data.
Support TheyWorkForYou and our work
TheyWorkForYou and WriteToThem are run by mySociety, a small UK charity. We’re a very efficient operation and do a lot with a small team; if we had bit more money, we could achieve a lot more.
We want to see a transparent, resilient democracy, with equal access to information, representation and voice for citizens. If you believe in this vision please donate today to enable greater transparency and accountability of the next government.
—
Image: Moritz Kindler
-
It’s a more common problem than you might think: given a list of postcodes, how can you match them to the administrative and electoral areas, such as wards or constituencies, that they sit within?
MapIt’s data mapping tool gives a quick, easy and cheap solution: just upload your spreadsheet of postcodes, tell it which type of area you want them matched to, and the data is returned to you — complete with a new column containing the information you need.
The tool can match your postcodes to every type of data that MapIt offers in its API, including council areas, Westminster constituencies, parish wards and even NHS Clinical Commissioning Groups (CCGs).
If that doesn’t sound like something you can imagine being useful, let’s look at a few hypothetical use cases (and if you have an actual case that you’d like to tell us about, please do let us know — we’re always keen to hear how our tools are being used).
Organisations, charities and campaigns sometimes need to match postcodes to administrative areas
Membership organisations, charities and campaigns usually collect the addresses of supporters, but don’t commonly ask them who their MP is (even if they did ask, most people in the UK don’t actually know the name of their MP).
But when a campaign asks followers to contact their MPs, it’s helpful to be able to suggest an angle based on whether the MP is known to be sympathetic to their cause, or not — indeed, there’s arguably no point in contacting MPs who are already known to be firmly on board.
So: input a spreadsheet of supporters’ postcodes, and get them matched to the associated Westminster constituencies.
For more advanced usages, organisations might match the MapIt tool’s output of postcodes with other datasets to discover the answers to questions like:
- Which members in a disability group have fewest GPs in their area, and might be finding it difficult to get help for their condition?
- Which supporters of a transport charity live in regions less served by public transport, and would be likely to take action to campaign for improved bus and train services?
- Which people affiliated to an ecological organisation live in predominantly rural areas and could help with a wildlife count?
Researchers sometimes need to match postcodes to administrative areas
Researchers often need to correlate people, institutions or locations with the boundaries they fall within.
They might have a list of postcodes for, say, underperforming schools, and want to find out whether they are clustered within authorities that have similar characteristics, like cuts to their funding or an administration that has a political majority one way or the other.
Teamed with other datasets, MapIt can help towards answering important questions like the number of people each CCG serves, how unemployment rates vary in different European regions, or average house prices within parliamentary constituencies.
Journalists sometimes need to match postcodes to administrative areas
Investigative or data journalists may obtain long spreadsheets full of postcodes in the course of their work, perhaps as a result of having submitted Freedom of Information requests to one or more authorities.
Perhaps they have the address of every university in the country, and there’s an election coming up — during the summer holidays. Knowing that students will mostly be in their home constituencies, they might be able to make informed predictions about how votes in the university towns will be affected.
Or let’s say that a journalist has gathered, from local councils, an address for every library scheduled to close. This could be compared with another dataset — perhaps literacy or crime rates — to draw conclusions over what impact the closures would have.
Part of a wider service
The one-off data mapping tool is just one service from mySociety’s MapIt, which is best known for its API.
This provides an ongoing service, typically for those running websites that ask users to input geographical points such as postcodes or lat/longs, and return tailored results depending on the boundaries those points fall within.
MapIt powers most mySociety sites, for example:
- When you drop a pin on the map while using FixMyStreet, MapIt provides the site with the administrative boundaries it falls within, so that the site can then match your report with the authority responsible for fixing it.
- When you type your postcode into WriteToThem, Mapit gives the site the information it needs to to display a list of every representative, from local councillor up to MEP, who represents your area.
- If you search for your postcode on TheyWorkForYou, MapIt tells the site what your Westminster constituency is and the site matches that to your MP. You can then be taken to their page with a record of how they have voted and everything they’ve said in Parliament.
Give it a try
Find out more about MapIt here or have a go at uploading a spreadsheet into the data mapping tool.
If you’re not sure whether it’s the right tool for your needs, feel free to drop us a line — and, as we said before, if you are already using it to good effect, please do let us know.
—
Image: Thor Alvis
-
It’s something we’ve been wanting for a long time, and it’ll very soon be a reality: FixMyStreet reports will, where appropriate, be channeled to Highways England. Look out for this functionality in the coming week.
My way or the highway
Previously, if you reported a problem on one of the country’s motorways or major A roads, we had no way of identifying whether it was the responsibility of the government department rather than the council. We had to rely on whichever council the report fell within, and hope that they would forward it on.
But now, we can send reports off to just the right authority. What’s changed to make this improvement possible?
Well, FixMyStreet uses our MapIt software, which matches points (in this case, the pin you put in the map when you make a report) with the boundaries they fall within (mainly, until now, council boundaries). That’s how it knows which council to send your issue to, even if you have no idea yourself when you make the report.
Motorways and A roads have boundaries too, of course, but that data wasn’t previously available under an open licence that would allow us to use it on the site. That all changed with GOV.UK’s release of the Highways England Pavement Management System Network Layer — just what we needed!
So now, if you make a report that falls within a small distance from one of the relevant roads, FixMyStreet will use MapIt in combination with this data layer. You’ll see a message asking for confirmation that your report actually does pertain to the highway: where roads cross a motorway, for example, a pin could relate to the road on a bridge, or the motorway below.
Confirm either way and boom: off it goes to either Highways England or to the council, as appropriate.
So that’s a big thumbs up for open data: thanks, GOV.UK! It’s also a good example of how our commercial work, providing FixMyStreet Pro to councils as their default street reporting system, has a knock-on benefit across the open source FixMyStreet codebase that runs not only FixMyStreet.com, but sites run by other folk around the world.
As you may remember, we recently added red routes to Bromley for FixMyStreet Pro, and it was this bit of coding that paved the way for the highways work. We can only prioritise not-for-profit development if we have the funding for it; but being able to improve FixMyStreet for everyone on the back of work done for commercial clients is a win for everyone.
Or, as our developer Struan says, in a metaphor perhaps better suited to shipping routes than highways, “a rising tide raises all boats”.
—
Image: Alex Kalinin
-
We, and Open Knowledge International, are looking for the digital files that hold electoral boundaries, for every country in the world — and you can help.
Yeah, we know — never let it be said we don’t know how to party.
But seriously, there’s a very good reason for this request. When people make online tools to help citizens contact their local politicians, they need to be able to match users to the right representatives.
So head on over to the Every Boundary survey and see how you can help — or read on for a bit more detail.
Data for tools that empower citizens
If you’ve used mySociety’s sites TheyWorkForYou — or any of the other parliamentary monitoring sites we’ve helped others to run around the world — you’ll have seen this matching in action. Electoral boundary data is also integral in campaigning and political accountability, from Surfers against Sewage’s ‘Plastic Free Parliament’ campaign, to Call your Rep in the US.
These sites all work on the precept that while people may not know the names of all their representatives at every level — well, do you? — people do tend to know their own postcode or equivalent. Since postcodes fall within boundaries, once both those pieces of information are known, it’s simple to present the user with their correct constituency or representative.
So the boundaries of electoral districts are an essential piece of the data needed for such online tools. As part of mySociety’s commitment to the Democratic Commons project, we’d like to be able to provide a single place where anyone planning to run a politician-contacting site can find these boundary files easily.
And here’s why we need you
Electoral boundaries are the lines that demarcate where constituencies begin and end. In the old days, they’d have been painstakingly plotted on a paper map, possibly accessible to the common citizen only by appointment.
These days, they tend to be available as digital files, available via the web. Big step forward, right?
But, as with every other type of political data, the story is not quite so simple.
There’s a great variety of organisations responsible for maintaining electoral boundary files across different countries, and as a result, there’s little standardisation in where and how they are published.
How you can help
We need the boundary files for 231 countries (or as we more accurately — but less intuitively — refer to them, ‘places’), and for each place we need the boundaries for constituencies at national, regional and city levels. So there’s plenty to collect.
As we so often realise when running this sort of project, it’s far easier for many people to find a few files each than it would be for our small team to try to track them all down. And that, of course, is where you come in.
Whether you’ve got knowledge of your own country’s boundary files and where to find them online, or you’re willing to spend a bit of time searching around, we’d be so grateful for your help.
Fortunately, there’s a tool we can use to help collect these files — and we didn’t even have to make it ourselves! The Open Data Survey, first created by Open Knowledge International to assess and display just how much governmental information around the world is freely available as open data, has gone on to aid many projects as they collect data for their own campaigns and research.
Now we’ve used this same tool to provide a place where you can let us know where to find that electoral boundary data we need.
Where to begin
Start here — and please feel free to get in touch if anything isn’t quite clear, or you have any general questions. You might want to check the FAQs first though!
Thanks for your help — it will go on to improve citizen empowerment and politician accountability throughout the world. And that is not something everyone can say they’ve done.
—
Image credit: Sam Poullain
-
It’s that time of year again. Local elections are on the 4th of May and we have updated our boundary change checker. It also helpfully lets you know if your ward is not having elections (not that your author was unaware of course.)
On May 4th elections will be taking place across English, Welsh and Scottish councils as well as the the elections for the new ‘combined authority’ Mayors.
Ward boundaries are changing
You might think you already know where to vote, and who’s standing for election in your area.
But both are dictated by which ward you live in — and that may not be the one you’re used to, thanks to ongoing changes in ward boundaries.
There’s no need to worry, though. As before, we’ve got the data that will tell you whether your ward has changed. Just enter your postcode here.
Once again the Democracy Club’s project Who Can I Vote For? will allow you to see whatever information is available for candidates in your ward.
—
Image: Roger (CC by-sa/2.0)
-
MapIt has had a bit of a refresh to bring the look into line with the rest of the mySociety projects. At the same time, we thought we’d take the opportunity to make it a bit easier for non-technical folk to understand what it offers, and to make the pricing a little less opaque.
You may not be familiar with MapIt, but all the same, if you’ve ever found your MP on TheyWorkForYou, written to your representatives on WriteToThem, or reported an issue through FixMyStreet, you’re a MapIt user!
That’s because MapIt does the heavy lifting in the background when you enter a postcode or location, matching that input to the boundaries it falls within (ward, constituency, borough, etc). It is, if you like, the geographic glue that holds mySociety services together.
Like most of mySociety’s software offerings, MapIt is available for others to use. So for example, the GOV.UK website uses it to put users in touch with the right council for a number of services, and Prostate Cancer UK uses it on their campaign site, using MapIt’s knowledge of CCG (Clinical Commissioning Group) region boundaries.
And you can use MapIt too: if your app or website needs to connect UK locations with areas like constituencies or counties, it will save you a lot of time and effort.
Simple payment
Pricing and payment is a lot slicker now: while it was previously managed manually, you can now purchase what you need online, quickly and without the need for human intervention. It’s also quite simple to see the pricing options laid out.
We hope that this will make it easier for people to make use of the service, and better understand what level of usage they need. But if you need to experiment, there’s a free ‘sandbox’ to play about with!
As ever, we’re happy to provide significant discounts for charity and non-profit projects: see more details on the licensing page.
The new MapIt pricing
If you have any questions or comments please do get in touch.
-
There’s a new piece of data on MapIt, and it wasn’t added by us. It’s tiny but useful, and it’s slightly esoteric, so bear with us and we’ll explain why it’s worth your attention.
Local Authority codes come from the government’s set of canonical registers. They may not look much, but they’re part of a drive to bring consistency across a wide range of data sets. That’s important, and we’ll try to explain why.
One name can refer to more than one thing
If you try to buy a train ticket to Gillingham in the UK, and you are lucky enough to be served by a conscientious member of staff, they will check whether you are going to the Gillingham in Kent (GIL), or the one in Dorset (GLM).
The names of the two towns might be identical, but their three-letter station codes differ, and quite right too — how, otherwise, would the railway systems be able to charge the right fare? And more importantly, how many people would set off confidently to their destination, but end up in the wrong county?
I mention this purely to illustrate the importance of authoritative, consistent data, the principle that is currently driving a government-wide initiative to ensure that there’s a single canonical code for prisons, schools, companies, and all kinds of other categories of places and organisations.
Of particular interest to us at mySociety? Local authorities. That’s because several of our services, from FixMyStreet to WriteToThem, rely on MapIt to connect the user to the correct council, based on their geographical position.
One thing can have more than one name
I live within the boundaries of Brighton and Hove City Council.
That’s its official name, but when talking or writing about my local authority, I’m much more likely to call it ‘Brighton’, ‘Brighton Council’, or at a push, ‘Brighton & Hove Council’. All of which is fine within everyday conversation, but which is an approach which could cause mayhem for the kind of data that digital systems need (“machine readable” data, which is consistent, structured and in a format which can be ‘understood’ by computer programs).
Registers of Open Data
The two examples above go some way towards explaining why the Department for Local Government & Communities, with Government Digital Services (GDS), are in the process of creating absolute standards, not just for councils but for every outpost of their diverse and extensive set of responsibilities, from the Food Standards Agency to the Foreign & Commonwealth Office, the Land Registry and beyond.
Where possible, these registers are published and shared as Open Data that anyone can use. It’s all part of GDS’ push towards ‘government as a platform’, and in keeping with the work being done towards providing Open Data throughout the organisation. Where possible these registers are openly available, and can be used by anyone building apps, websites and systems.
And now we come to those Local Authority codes that you can find on MapIt.
Anyone can contribute to Open Source code
Like most mySociety codebases, MapIt is Open Source.
That means that not only can anyone pick up the code and use it for their own purposes, for free, but that they’re also welcome to submit changes or extensions to the existing code.
And that’s just how GDS’ Sym Roe submitted the addition of the register.
What it all means for you
If you’re a developer, the addition of these codes means that you can use MapIt in your app or web service, and be absolutely sure that it will integrate with any other dataset that’s using the same codes. So, no more guessing whether our ‘Plymouth’ is the same as the ‘Plymouth’ in your database; the three-letter code tells you that it is.
Plus, these register codes identify a local authority as an organisation, or a legal entity, as opposed to setting out the boundary, so that’s an extra layer of information which we are glad to be able to include.
—
Image (showing another example of uniformly-used three-letter codes): Jim Linwood (CC by/2.0)
-
What’s the best way to get your supporters to campaign, when the finer details of what they’re pressing for may vary from place to place? That’s the issue that faced Prostate Cancer UK as they call for better provision for men across the country with erectile dysfunction as a result of prostate cancer.
There are five core treatments for tackling erectile dysfunction, but whether all of them will be offered to you depends on your postcode. In some areas, all are offered as standard, while in others there may be none.
The tool we built for Prostate Cancer UK used several of mySociety’s areas of expertise, from mapping to user testing — we even used Freedom of Information. And putting it all together, we have a powerful campaigning platform that responds to users’ location, while raising awareness and pushing for improvement.
Prostate Cancer UK’s Erectile Dysfunction campaign site informs people about what care should be available to those who experience the condition as a result of prostate cancer treatment, and urges them to write to their local health commissioner if provision is poor in their area.
Educating, campaigning, sharing
The user is first informed: they are shown the five factors which constitute good treatment of erectile dysfunction. After that, they are prompted to input their postcode to see how many of those measures are provided by the NHS body responsible for their region.
If provision is poor, they are encouraged to help campaign: users can opt to write to their Clinical Commissioning Group (CCG), Health Board or Health and Social Care Board to ask them to improve what’s available. They are given the choice between writing a letter from scratch, or using a pre-composed template which also contains a section for the writer to add a paragraph of their own words — a pragmatic balance that avoids an influx of identical form letters, while still addressing fact that when users are faced with a completely blank page, many will drop out of the process.
When you’ve done that, for those in England there’s also an opportunity to contact Jeremy Hunt, Secretary of State for Health to highlight the variation in treatment for erectile dysfunction and establish which organisation is responsible for the national commissioning guidelines.
Finally, the user is invited to share what they’ve learned, via Facebook, Twitter or email. Our user testing revealed that, contrary to our worries, people were happy to do this without embarrassment.
How it works
Like most of mySociety’s own sites, the ‘Better Care’ site uses MapIt to match the user’s postcode with a boundary, in this case the boundaries of the CCGs, Health Boards and Health & Social Care Boards. That’s how we deliver the information about what’s available in their local area.
When you input your postcode to see how your local provisioners are doing, MapIt also delivers information for other areas, including a couple of close neighbouring ones. This allows us to provide a nice comparison, along with the statistic that shows whether your provisioner is better, worse, or within the same range as the average.
But how did we gather the data to tell you how well each CCG, Health Board or Health and Social Care Board is catering for erectile dysfunction patients? Well, fortunately, thanks to our own WhatDoTheyKnow website, it was relatively easy to send a Freedom of Information request to every one in the country — 235 of them in total. The WhatDoTheyKnow volunteer admin team were able to help with this large batch request.
Once we had all the data and a general idea of how the tool would work, we took an early version out to test it with users. The insights we gained from this process were, as always, extremely useful, and led to us altering page layouts and other elements that made the whole process as clear as it could be.
Finally, we incorporated quite a bit of statistics-gathering into the whole tool, so that Prostate Cancer UK would be able to see where their campaign might benefit from further optimising in the future.
All in all, we’re very glad to have been part of this important campaign to help men understand what’s available to them, and where they might need to push for more.
Image: Brad Hagan (CC-by/2.0)
-
If you’ve visited the MapIt site this week, you might have noticed a change: we’ve introduced key-based authentication for API users.
This enables us to be more flexible about how we provide our service, which means you can be more flexible about how you serve your users.
MapIt is both an open source application and, via https://mapit.mysociety.org, a web service. Use of the API is free for low-volume, charitable use, while all other uses require a licence.
For the moment API keys are optional. We’ll always offer a free level of service to support independent developers and charities.
We’ll have more details soon about the increased flexibility this change will bring.
—
Image: Clare Black (CC)
-
Are you still in the same ward? Check whether your ward boundaries have changed here.
May 5 is election day
If you’re a UK citizen, you have an election in your near future. We can say that with confidence.
May 5 sees elections not only for the Scottish Parliament, the National Assembly of Wales and the Northern Ireland Assembly, but also for many local councils. Londoners will be picking their London Assembly representatives and their Mayor. As if all that isn’t enough, there are also Police and Crime Commissioner Elections.
Ward boundaries are changing
You might think you already know where to vote, and who’s standing for election in your area.
But both are dictated by which ward you live in — and that may not be the one you’re used to, thanks to ongoing changes in ward boundaries.
There’s no need to worry, though. As before, we’ve provided a nifty tool that will tell you whether your ward has changed. Just enter your postcode here.
Thanks to Democracy Club’s new project Who Can I Vote For? you’ll also be able to click through to see whatever information is available for candidates in your ward.
—
Image: Matt Brown (CC)