1. Face the general election like a pro, with mySociety’s tools

    There’s going to be a UK general election on 4 July. We’ve written a 10-point guide to explain how the election works.

    Here’s how our tools can help you cut through the noise and find out what’s happening in your constituency:

    Assess your previous MP’s activities 

    Parliament was dissolved on Thursday 30 May. After this point there are technically no MPs. 

    Instead, your former MPs just become candidates (if they’ve chosen to stand again – many haven’t). 

    That doesn’t stop you from looking up your previous MP’s voting record and register of interests on TheyWorkForYou, and comparing it with the way other parties’ MPs voted. 

    In case you missed it, we recently changed the way we calculate voting summaries to prioritise actions, not words – making our summaries even more accurate and even more useful. 

    Consider your new candidates in your new constituency

    We’ve made some changes so that when you enter your postcode into TheyWorkForYou, you’ll be taken to a new General Election page that will give you an up-to-date list of candidates standing in your constituency. 

    This page links to a much more detailed breakdown from WhoCanIVoteFor, made by our friends at Democracy Club. On WhoCanIVoteFor, you can find information about your candidates’ previous attempts to run for office, any statements or election materials they’ve made, and links to their social media pages. Once you’ve looked up your postcode, bookmark that link; it’s ideal for answering people on your neighbourhood Facebook or Next Door groups who will inevitably be asking who’s standing in your area.

    On TheyWorkForYou and WhoCanIVoteFor you’ll find a handy map comparing your new constituency (pink) with your old one (grey). Here’s what that looks like for me, in Leeds:

    What impact will the new boundaries have on this election? 

    We can’t know for sure until after the election, but don’t forget you can also check out the Local Intelligence Hub for loads more info about both your old and new constituency. Just put in your postcode and you’ll find public opinion polls, candidate information, nearby campaigning groups and more. The hub is made in partnership with the Climate Coalition, so you’ll find a wealth of climate and nature data too. 

    This information is absolutely invaluable for when canvassers come knocking at your door and ask what your priorities are. You can hit them with stats about things like what support there is for sustainable energy or net zero in your constituency; or share your opinions on how your previous MP voted on an issue that matters to you. Maybe even give them the link – www.localintelligencehub.com – so they can explore for themselves.

    Build your own clever things using our free APIs

    Want to dig into the data yourself? Maybe even build your own tools using the new boundaries? For those with a little coding knowledge, we’ve made the building blocks available in a number of formats

    Mapit, our our geographical postcode lookup website has the old and new constituencies, many other geographies, and the register of members interest for the previous Parliament is available as one big spreadsheet


    Help us do more of this work

    Whoever is elected, they need to understand the importance of transparency and accountability — and we’ll be making sure that happens. Please consider donating.


    Header image by Element5 Digital on Unsplash

  2. Off by one: How Parliament counts votes is out of date

    Last night there was a vote to allow MPs to be excluded from Parliament (after a risk assessment) if arrested on suspicion of a serious offence. This vote passed by a single vote.

    The problem is, looking across several sources of voting information, there’s not a good agreement on what the actual totals were. Ultimately the tellers count is authoritative, but this problem reflects the complicated way that MPs vote.

    The result(?)

    SourceDescribed resultCount of names
    votes.parliament.uk170 Ayes, 169 Noes169 Ayes, 169 Noes
    hansard.parliament.uk170 Ayes, 169 Noes169 Ayes, 168 Noes
    (teller result)
    169 Ayes, 168 Noes

    170 Ayes, 169 Noes (speech with teller result)
    169 Ayes, 168 Noes

    What’s going on here?

    In the voting lobby, there are two different systems going on to record votes:

    • An electronic pass based voting system – run by the clerks, that feeds into votes.parliament.uk and Hansard.
    • A counting system run by the tellers – a MP for each side is in each lobby, and if they agree the count, that’s the count used to make the decision.

    Meanwhile, at TheyWorkForYou, we use tidied up division names created by votes.parliament.uk, but the division lists from Hansard, and add the names to get the number of people on each side. 

    Votes.parliament.uk will be quickest with who voted – this feeds into the Hansard list, but the two can get out of sync if one is updated but not the other. 

    In this case, Rebecca Harris is counted in votes.parliament.uk but not in Hansard. This could be for a few reasons, for instance she may not have been able to use the pass system for some reason but was recorded manually and added as a correction but after it was fed into Hansard. We’ve queried this with her. In any case, what the tellers counted is the authoritative result for the vote. They could also have been right – and someone else forgot/was not able to tap in who should have done.

    But if the votes.parliament.uk count was right, it would mean the tellers in the Aye lobby overcounted by one. This would make it a draw, and in a draw the speaker will cast a deciding vote against the motion (as there isn’t a majority for it).  When it’s down to one vote – you want to have faith the system got the right answer. 

    Better ways are possible

    We think it should be easier for MPs to vote, and have previously recommended that:

    • The House of Commons should in normal circumstances, defer votes to a standardised voting time (within ‘core hours’), where multiple votes are held in succession.
    • These votes should be held through a fast electronic means – whether through terminals, voting pass systems, or apps.
    • Current proxy voting schemes should be extended to personal discretion to designate a proxy – e.g. a set number of days a year a proxy vote can be allocated, no questions asked.

    Electronic voting and a voting time would be bringing back good practice from the devolved Parliaments and help MPs make better use of their time than standing in division lobbies. But as well as being slow – there are clear questions to ask about the accuracy of the current approach.

    How MPs vote has big impacts on how our country works – getting it right matters.

  3. Update: some APPGs are back

    WhoFundsThem is our new project looking to uncover the influence of money in politics. You can donate or volunteer to support this project.

    Last month, we asked “What happened to all the APPGs?” because between March and April over a third of All Party Parliamentary Groups were deregistered, from 722 down to just 445. This story was covered in the Byline Times and the Parliament Matters podcast.

    On Monday, we got a partial answer to our question. 

    The May register shows an increase of 90 groups – up to 535.

    We’ve crunched the numbers, and found that 86 of the 277 groups that were removed in April have been re-registered for the May edition. We can’t know for sure why this happened, but we know that Parliamentary authorities did an audit of compliance ahead of the April register, which might have contributed to lots of groups being removed. It’s possible that these groups have since passed the necessary requirements to be re-registered in time for the May edition.

    Taking into account the last three registers, we found:

    Dive into the data yourself

    We’ve updated our public spreadsheet with the new register and an ‘All groups’ tab that shows which groups fall into the six categories above.

    What next?

    We’ve launched our WhoFundsThem project which is requesting information from all APPGs (yes, our job just got a bit bigger!).

    We need your help – please consider volunteering, or donating £10 to help make this work happen. 

    Photo by Zetong Li on Unsplash

  4. And we’re off: gathering information on financial interests and APPGs

    We’ve kickstarted the WhoFundsThem project, and now we have a (tight!) timeline of work

    WhoFundsThem is our new project looking to uncover the influence of money in politics. You can donate or volunteer to support this project.  

    On Friday, we sent our first batch of requests for information to 25 All Party Parliamentary Groups (APPGs) as part of our WhoFundsThem work. 

    This is a test batch to see how well the template we’ve made works as a method for getting information back from APPGs. The new rules require them to make quite a lot of different kinds of information available, and there are 445 APPGs — so we want to ask in a way that makes sense for them, and for us.

    We’re asking for this information because we think it’s important to have it openly available for the public benefit. There are loads of possible uses for it: for example, we’d like to improve the APPG membership information we include on the Local Intelligence Hub, but once the information is public, it will be available for all sorts of other projects and individuals to use

    To select the lucky 25 APPGs who would make up our test batch, we took Parliament’s A-Z list of all of the APPGs, numbered them, and then randomly generated 25 numbers. The selected APPGs were:

    1. Africa
    2. Denmark
    3. Japan
    4. Poland
    5. South Africa
    6. Tibet
    7. Artificial Intelligence
    8. Arts and Heritage
    9. Biodiversity in the UK Overseas Territories and Crown Dependencies
    10. Children of Alcoholics
    11. Deafness
    12. Disability
    13. Ethnic Minority Business Owners
    14. First Do No Harm
    15. Future of Work
    16. Human-Relevant Science
    17. Internet, Communications and Technology
    18. Life Sciences
    19. Microplastics
    20. Packaging Manufacturing Industry
    21. Responsible Vaping
    22. SME (Small and Medium-sized Enterprises) House Builders
    23. Sport
    24. Taxation
    25. United Nations Global Goals for Sustainable Development

    On Friday,  we emailed these groups a copy of the template, and informed them that as per the rules they’ve got 28 days to get back to us, making a deadline of Friday 7 June 2024. After this deadline we’ll review the feedback and responses, make any adjustments necessary, and then email the template to all of the remaining 420 APPGs. This should give us responses from every APPG by the middle of July.

    Don’t forget, this is just one of the two parts of the WhoFundsThem project. While we’re waiting for APPG responses, we’ll spend the month of May recruiting volunteers, and then in June we’ll begin answering questions for the other stream of the project which looks at the Register of Members’ Financial Interests (RMFI). By mid-July, we’re hoping to have turned those answers into individual summaries for each MP. Then the right of reply process begins: MPs will have a month to respond to our summary of their financial interests.

    All being well, as we send off these summaries to MPs, we’ll be able to switch back to looking at APPGs, as the returns from the second batch should be back ready for us to clean and analyse. By the end of August, we should have both clean APPG data and RFMI summaries with MP feedback. We’ll then spend some time auditing this data ready for publication in the autumn.

    Well, that’s the plan at least!

    If you’re interested in being one of the volunteers who will work on this exciting new project, you have until 28 May to fill in our short application form! On Tuesday evening (14th), we’re hosting a Q&A event to explain more about the project and answer any questions about volunteering. We know not everyone can give up their time, though, so if you want to support projects like these in another way, please consider financially supporting us.

    Want to find out more about APPGs? I wrote a blog post last month explaining what APPGs are, how the rules changed, and the impact that change had. 

    As ever, if you’re interested in the work we do, make sure you’re signed up to our newsletter. Thanks!

  5. Improving TheyWorkForYou’s voting summaries

    If you value the work mySociety and TheyWorkForYou do, please consider whether you can make a donation.

    We have a good track record of making Parliament more open, provide essential tools to civil society and small charities, and with our platform a little support can go a long way.  If you would like to make a larger donation to support specific work, or to match-fund other donations – please get in touch.

    Our MPs in Parliament have many roles, but one of the most important is that they make decisions on the laws that govern us, and these decisions can affect every aspect of how we live our lives. 

    TheyWorkForYou’s voting record summaries are part of a number of different arguments about what the role of MPs is, and how Parliament should work.

    As well as listing individual votes in Parliament, our voting summaries give an overview of how MPs have voted on policies that come up in multiple votes. We strongly stand by the principle of our summaries, but don’t think there’s only one way of doing it. Using a grant from the Newby Trust, we’ve been reviewing our methods and refining our approaches to voting records. 

    The key headline is that we’re going to sharpen the focus in our approach. The main changes are:

    We have written a longer document explaining how these changes achieve our goals.

    This change is being applied alongside a backlog of new policy lines that we’ve been reviewing with our new criteria for inclusion. While these may be big shifts in principle, in practice most existing summaries stay exactly the same. It’s a progression and simplification rather than a revolution.

    To see our voting summaries for your MP, search for your postcode on TheyWorkForYou and click ‘Voting Summaries’.

    What we want to achieve with our summaries

    In thinking about our voting summaries, we wanted to clearly define what we’re trying to accomplish. This has led to two headline goals:

    We want to present clear and accurate summaries of how individual MPs have voted, for use by the public.

    • As a point of principle, it should be possible and straightforward to find out how MPs have acted on behalf of their constituents. 
    • The top-line display of information should be a good reflection of the data that was used to create it – balancing clarity and accuracy.  We should provide options for people to learn or explore more, with the expectation that most won’t, and so the clarity of the summary matters.
    • While we aspire to produce information that is also of use to people with a professional interest in Parliament, this need might be better met through other tools or summaries. For instance, while it is possible to compare different MPs through voting records, it is not the main purpose of these summaries.  

    In line with our general approach, we want to align with and amplify citizen perspectives of how MPs should work, as voiced by Citizens’ Assemblies (in particular the Democracy in the UK Citizens Assembly) and polling.

    • Historically, we’ve seen that making the actions of MPs more visible changes their behaviour.
    • We need to be conscious of the likely effects of our summaries, and ensure they reflect our values, democratic principles and approach. We want to anchor our approach in wider ideas of how our democracy works rather than our own opinions.
    • We also need to be aware of when pressure on individual MPs is not the best way to achieve systemic change. As such we need to consider where our work reinforces rather than changing parliamentary systems that are hostile to MPs from groups historically excluded from Parliament (e.g. women, ethnic minorities, disabled MPs). 

    A longer document explaining how these changes achieve these goals can be read here

    The impact of this change

    Most of the top-level summaries on the site (73%) are completely unaffected by these changes. 82% of MP ‘scores’ are either the same, or have a stronger/weaker version of the same alignment (i.e. the adjustment has not affected our assessment of whether the MP is for or against a policy). About 14% of connections between MP and policies are removed, which is a combination of removing seven policy lines that were made up entirely of votes that did not directly use Parliamentary power, and longer running policies being confined to a narrower time frame. The remaining changes are “a mix for and against” assignments becoming more clear (or the reverse), and a small group (about 120 out of 80,000) where the direction of the score has changed (i.e. where someone was voting for and is now seen as voting against – this is mostly concentrated in two policies). You can read more about this in our longer summary.

    This is good because we don’t generally want these to be too sensitive to the exact formula used: the kind of broad points we’re making should be reachable no matter which method is applied. Ultimately only a small group of votes have been removed from policies, and the positions we were displaying before were mostly driven by votes that already passed the “use of powers” criteria. The goal of this process is to simplify how we work and enable clearer explanations of what we’re doing – but the general end product isn’t massively changed by adopting these new rules.

    A process, not a destination

    This isn’t where we stop. This update is a step in the journey.

    There is a growing clarity issue that for long-serving MPs there are now quite a lot of policies — and part of our work creating summaries should be helping people find the relevant information they’re looking for. 

    There is also a pending question about presenting a retrospective on the current Parliament during the next election. With the technical work we’ve done, it is now much easier to explore alternate approaches to displaying this data. 

    We are considering how we can best do this, and how we work with others to ensure we are capturing the important issues of the last Parliament. 

    Making other tools available 

    One kind of complaint about voting summaries is that they do not provide an easy way of drawing out small differences between two MPs on how they voted. This is true – we might say two MPs voted a mixture of for and against a policy, but in practice they took opposite positions on different votes. 

    In our voting summaries we’ve made the decision to focus on providing information that makes sense for a constituent looking at their MP – we produce better summaries by focusing on specific kinds of users we want to make sure it works for. But for our own work as well as to support others, we want to provide a wider range of tools and information for both citizens and specialists. 

    Our previous approach to voting was deeply tied technically with the Public Whip (originally a companion project to TheyWorkForYou, but not run by mySociety). This means we had limited ability to take big swings in our approach: while we indirectly maintain it through the data feed, we can’t change the basic functioning of the Public Whip.  

    To implement the changes described above, we have internally created a Public Whip replacement (TheyWorkForYou Votes) that we’re using to update voting records, and provide new analysis tools to help us understand votes, giving us easy understanding of the parliamentary dynamics of a vote and basic analysis of the motion.  In the next year, we want to talk to more people who want better tools for working with raw voting information, to help shape this tool for a public release. 

    Supporting our work

    In our voting records work we have an approach that has public support, and we think serves an important purpose. But we don’t think this is the only or best way of creating voting summaries: we want to be able to be reactive to how Parliament is changing, and always making our coverage and approach better. We also want to work to encourage better transparency and public understanding at the source, through improving how Parliament works.

    If you value the work we do, please consider whether you can support us financially. We have a good track record, and with our platform a little support can go a long way.  If you would like to make a larger donation to support specific work, or to match-fund other donations, please get in touch. 

    In the next few weeks we will be announcing a new project involving volunteers and the register of members interests. If you’re interested in hearing more about that, please sign up to our volunteer mailing list.

    Header image: Photo by Aaron Du on Unsplash

  6. What happened to all the APPGs?

    Over Easter, some groups went missing in Parliament.

    No, not lost tourists: of the 722 All Party Parliamentary Groups registered in March, only 444 are left – a 39% decrease in the space of a month. What caused this, which groups have been removed, and what happens next?

    Tl;dr: we’ve published the changes as a spreadsheet.

    What is an APPG?

    All Party Parliamentary Groups (APPGs) are self-selecting groups of MPs and Lords with an interest in a particular policy area. Browsing the list might help you find out that you have more in common with MPs than you think; subject-based APPGs include Craft, Jazz, and Parkrun, and country APPGs range from Albania to Zimbabwe. Most groups are supported by a secretariat, which is usually a charity, membership body or consultancy organisation.

    The logic behind APPGs is to create legitimate avenues for experts and interested parties from outside Parliament to discuss policy with MPs – but unfortunately they can also be vehicles for corruption. As  Transparency International argue: “While APPGs can help inform debate, time and time again we see examples of MPs and Peers exercising poor judgement by accepting all-expenses-paid trips from regimes with highly questionable records on corruption and human rights.”

    Why were so many groups removed?

    New rules came into place on 31st March 2024 that required:

    • Increased financial reporting 
    • A ban on funding from foreign governments
    • Increased reporting on secretariat support 
    • A minimum of 20 members 
    • Exactly four officers, two of whom must be MPs

    How did the Register change?

    Parliament maintains the Register of all APPGs that gets updated approximately every six weeks. The last edition before the rule change, published on 6th March 2024, showed 722 groups in total – 130 country groups and 592 subject groups. The 8th April edition shows 444 in total – 74 country groups and 370 subject groups. In total, 39% (278) groups were removed, with the countries list shrinking by 43% and the subjects list by 38%.

    Why does this matter?

    We don’t know exactly why each group was removed from the register. In some cases they simply may not meet the new 20 member threshold, but in others, deregistering might be an attempt to evade scrutiny.

    Deregistered “unofficial” groups can operate in very similar ways to registered APPGs (and there is some evidence they are already doing so) but will not have to abide by the same rules. This means that the only way to track the activities and spending of these groups, and the outside interests that fund them, is through individual Members’ Registers of Financial Interests. Parliament’s rules are clear that MPs are supposed to declare all benefits received through group membership (whether or not a group is an official APPG) but in practice this can be inconsistent.

    Which groups were removed?

    We’ve published the full list of groups from the last two registers, the changes, and the list of removed groups as a spreadsheet.

    What next?

    TheyWorkForYou has a long history of making MPs financial interests data easier to access and understand. We make it easier to see changes in MPs’ declarations over time and are now publishing this information as a big spreadsheet

    We have a lot more work in the pipeline around both APPG data and Register of Members Financial Interests data (stay tuned for details in our newsletter).

    If you think what we’ve done so far is valuable, and want to help us go further: please donate

    Photo by Zetong Li on Unsplash

  7. Guest post: What are the questions MPs ask that don’t get answered?

    This blog post is part of our Repowering Democracy series. We are publishing a series of short pieces of writing from mySociety staff and guest writers who are thinking about how our democracy works and are at the frontlines of trying to improve it.

    This week, we’re re-publishing a blog post from Anna Powell-Smith at the Centre for Public Data, which is a new, non-partisan non-profit working for stronger public data. We’re previously worked together on recommendations to avoid fragmented public data. This blog post touches on several issues close to our hearts: Parliamentary written questions, and where there isn’t enough data to understand what’s going on.

    Data gaps are under-reported, because it’s hard to write about data that doesn’t exist.

    As we’ve written about before, newspapers publish endless stories on house prices, where there’s lots of data – but few on rental costs, even though millions of people rent. That’s partly because the Office for National Statistics doesn’t collect much data on rentals.

    To tackle this problem, I’ve been thinking about how to map data gaps, and make them more visible.

    And I think the best way is actually to think about questions, instead of data. What are the important questions that the government can’t answer?

    Obviously, ‘important’ is subjective! But one source of clearly important questions is Parliamentary written questions, which are the formal questions that MPs and peers ask the government. Where the government doesn’t have the data to answer them, it has to say so.

    So this post introduces new research: a data analysis of 200,000 Parliamentary written questions, and what they tell us about the UK’s missing numbers.

    Our modest goal: to find the UK’s biggest data gaps.

    What we did

    Building on some previous research of ours, we strapped on our coding hats 🪖, and did the following:

    • First, we scraped all the written questions in Parliament from December 2019 to February 2023, from TheyWorkForYou, which gaves us about 200,000 questions.
    • Next, we flagged questions asking for quantitative information, with phrases like “how many” or “how much” – which showed that about a fifth of questions wanted data, just under 40,000.
    • Then we flagged questions where the government apparently said the data was “not held”, “not collected”, etc. About a quarter of quantitative questions were answered like this.

    And we ended up with a dataset of around 10,000 questions where MPs apparently both (i) asked for data, and (ii) were told it was not available. So: missing numbers.

    Then we spot-checked the questions to check our method. It wasn’t perfect, but it was very decent. (It helps that Parliament uses formal, consistent language.) You can download the full dataset here.

    Sometimes, MPs ask about strange things, like jobs for clowns. But most are extremely serious, covering the issues that affect MP’s constituents. And overall, they tell us what MPs need to know.

    Data gaps by department

    Firstly, we looked at how often each government department said that data wasn’t available. (See the code.) And there were were huge differences:

    • At the Department of Health & Social Care, around 40% of quantitative requests were unanswered (though we can cut them some slack, as this was during the Covid pandemic).
    • At the Home Office and the Department for Work & Pensions, around a third were; at the Ministry of Justice the proportion of unanswered quantitative requests was 30%, and the Department for Education 27%.
    • But the proportion was much lower at other big departments – almost all others were below 20%.

    Of course, we need to be cautious here, as the numbers are approximate. Without reading each question, we can’t be sure that we’ve tagged it correctly, or if the MP was asking something impossible. It’s probably most useful to consider the differences between departments.

    Given that, it’s not surprising that the health, benefits, justice and education departments would get requests for data, since they run massive operational services that affect people’s lives. (The Foreign Office, by contrast, largely seems to get asked about wine.) It’s more surprising that they seem to struggle to answer them more than other departments.

    Now let’s dive into what these unanswered questions were about.

    The topics with the biggest data gaps

    Each question scraped has a title. We can use this to see which topics were least likely to get an answer.

    Other than Covid-related topics, the major topics with the highest proportion of unanswered questions were:

    1. Benefits – grouping together benefits like Universal Credit and PIP
    2. Asylum, refugees and migrants
    3. Child maintenance
    4. Energy meters
    5. Armed forces housing

    This seems plausible. The DWP Select Committee has repeatedly criticised the government for the lack of visibility over the benefits system; the statistics regulator has expressed concerns about the use of asylum statistics, while the National Audit Office has noted gaps in the data available on smart meters.

    We also used GPT-4 to try tagging questions, which worked quite well. We used it to tag questions to the Department of Health & Social Care. This helped us identify major clusters of unanswered questions in these areas.

    In healthcare, MPs often struggled to get basic prevalence information, whether:

    Also, funding is a topic it’s surprisingly difficult to get information about, e.g.

    Following on from this, hospital-level information in general often seems to be poor, e.g.:

    And finally, workforce is a huge one, with topics like:

    You can see the tagged questions here – there are many more examples under each topic.

    This gets really worrying when you look at the dataset over time. It’s immediately clear that MPs often ask the same thing over and over again – yet the information doesn’t seem to improve.

    What next?

    We think statistics producers should be monitoring Parliamentary questions, to tell them where data needs to be better. After all, MPs deserve answers to their questions, and so do we all.

    If you can help us make this happen, we’d love to talk.

    If you’re interested in this research – or even better, if you can fund us to do more of it! – please do get in touch.

    Image: Tom Chen on Unsplash.

  8. Democracy month notes: February

    Previously: January!

    Gaza ceasefire blog post

    I wrote a blog post about the Gaza Ceasefire opposition day votes – especially focusing on how there ended up being no recorded votes. 

    This is the kind of responsive work we’d like to do more of. We don’t need to duplicating every explainer out there, but we want to be able to better articulate “this is how Parliament works, but there’s something wrong with that” when there’s currently something confusing/going wrong in the news. 

    Asking for money to do good things

    Alice, Julia and I have been putting together a more structured version of the idea I talk about at the bottom of this blog post about our new spreadsheet of the register of interests — using crowdsourcing to create good, understandable summaries of MPs interests. Will let you know how that goes. 

    Something we’d like to get better at is being more public when these applications for funding do not work out (spoiler: this happens a lot!). There’s a lot of work and creativity that goes into our ambitions for TheyWorkForYou, and ideally these wouldn’t just be locked away in various virtual desk drawers. 

    Oflog consultation

    Julia worked with our friends at the Centre for Public Data on a joint response to an Office for Local Government (OFLOG) consultation – read more about that

    This is a continuation of our work around public data fragmentation

    Small API updates

    Matthew has added Parliament’s unique identifier to the response to the ‘getMPInfo’ API call, making it easier to jump from our data to query the Parliament API.

    Server upgrades

    Sam and Matthew have been upgrading the servers that run TheyWorkForYou and WriteToThem.

    We need to do this periodically for security reasons: the organisations that distribute the server software (and other packages we depend on, like those that distribute the programming languages) only provide security and bug fixes for a certain period, after which they only provide it for newer versions. 

    Running software on the web — where there are *constantly* bad people testing for weaknesses — means taking this seriously. But upgrading the lower levels of the “stack” often means small changes further up where features we use have been deprecated and replaced with other approaches. Some of this work is running just to stay in the same place, but it does also enable us to adopt new approaches in how we code and the packages we use. 

    This is one of the massive benefits of the same organisation running TheyWorkForYou AND WhatDoTheyKnow AND FixMyStreet AND (many more) – we have excellent people thinking hard about our technical infrastructure across all our work. 

    Voting summary update

    We’ve done some of the trickiest technical work required to enable the voting summary update we’re planning.

    We’ve moved TheyWorkForYou from pointing at the Public Whip website, where it used to get voting summary calculations, to an instance of a new,experimental “twfy-votes” platform. This is doing the work Public Whip was originally doing, but also taking over the party comparison calculations that were being done in TheyWorkForYou itself previously. 

    TheyWorkForYou has become simpler, and more of the relevant code is now in the same place. We’re not yet completely independent of the Public Whip because twfy-votes currently uses the database dump to populate itself — but soon we’ll be able to move that to an export from TheyWorkForYou’s own database. 

    The goal in this set of changes is to move from this:

    Diagram showing the flow of data from the Hansard XML, through Parlparse, into both TheyWorkForYou and the PublicWhip - with that then reentering theyworkforyou and additional calculations being done to calculate voting summaries

    To this:

    Diagram showing the different flow of data from the Hansard XML - through ParlParse to TheyWorkForYou, and a feedback look between TWFY and TWFY-VOTES

    Which is… still a lot of boxes and arrows, but is better than it was. This could in principle then be simplified even further, but this brings the whole process under our control and simplifies some of the back and forth steps. 

    Currently, all this work should have resulted in almost no visible changes to the site. But we now can flip a switch and it will switch the underlying algorithm used from the one in the Public Whip to the new (simplified) approach.  One of the motivations behind this shift is to be fully in control of that algorithm (which is effectively a number-based editorial policy). 

    One of the things I’ve been doing this month is running the analysis to clearly map what exactly the public effect of this will be. Broadly, most things stay the same, which is good because we don’t want the headline messages to be hugely affected by different methodologies behind the scenes – At the same time we’ll end up with something that is easier to explain. 

    The final stage before full release is a set of less technical changes, consolidating the voting summary information on one page, and adding a rewritten page describing both how Parliamentary voting works in different places across the UK, and what our approach is in the data we publish. Making good progress on these, and hope to have this project completed soon. 

    That’s all for now

    As ever, if you’re the kind of person who reads to the end of these (I’m going to assume a generally nice person who is also a fan our our work) – donations are welcome. But also get in touch if you’ve got something to chat with us about!

    Header image: Photo by yasin hemmati on Unsplash

  9. Gaza Ceasefire votes and voting records

    What happened?

    Yesterday in the House of Commons, there was an SNP Opposition Day debate about a ceasefire in Gaza. This meant that the SNP had an opportunity to put forward a motion for the House to vote on.

    The Labour Party’s preferred wording of a ceasefire motion replaced the SNP motion and was passed by the House of Commons based on what is known as a voice vote. 

    This is when the speaker (in this case the Deputy Speaker who was in the chair) judges the result of the vote based on the volume of shouts in the chamber. As such, there is no record of how individual MPs voted.

    This is not the same as saying the vote was unanimous – and listening to the recordings there is a clear ‘no’ present on both votes (the Deputy Speaker does later claim that “nobody called against it”, which is then contested).

    From the Speaker’s point of view, the goal is taking a read on the decision of the House (and this may have been correct in that one side was louder, if not unopposed) – and a vote in the lobbies (division), which takes around 15 minutes, serves no purpose. 

    But votes also serve the purpose of putting the opinions of individual MPs on the record, which several were frustrated to have been denied. Votes are part of the public facts about MPs’ impact in Parliament, and part of how actions are communicated to constituents. This is a factor in the democratic process that also needs balancing in these decisions. 

    This decision followed a long division for a motion to sit in private – and votes that seemed clear on a voice vote may have been seen as costly in terms of time to take to a full division. In general, it is possible to have voting processes that are much faster and fairer to MPs, that would allow getting two votes on the record without taking most of an hour of parliamentary time. 

    What does this mean for TheyWorkForYou?

    Because there was no recorded division – the approval of the motion does not appear in the recent votes tab for MPs.

    The full debate is worth a read – the general sense is of a long debate where MPs engage with a complicated situation, and reflect that the UK’s role can only be part of any solution. 

    We’re in the process of updating the processes behind our voting summaries, which includes ways to include what we’re calling “agreements” (decisions without a “division”) in summaries. But issues like last night’s decision reflect that we need to take a cautionary approach – as there is clear evidence that it was not an unambiguous decision. We will publish more on this approach soon.

    Why was the amendment process controversial?

    As it was an SNP Opposition day, (a day when an Opposition party gets to choose the main debate) they got to propose the motion. Both Labour (another Opposition party) and the Conservative Party (Government party) proposed amendments to the motion.

    The Speaker went against previous convention and allowed both a Government and Opposition amendment – which was unexpected. The motivation of this was to give most MPs a chance to vote for a motion on their preferred wording – the problem is that the amendment process is not really set up for this.

    The thinking makes sense given how Opposition day voting is supposed to work: the Opposition by definition is not supposed to win because they have fewer MPs than the government.

    What’s supposed to happen is that MPs debate a topic, hold a vote, and the motion is rejected. If the topic strikes closer to home, the Government will amend it to say “this is an important issue but the Government is doing a great job”, and that is the motion that is passed because the Government should have the numbers to win the vote.

    Government amendments come after the vote on the main motion to respect the purpose of Opposition day debates, while reflecting the reality that the government can amend the motion and win. This sequencing allows for a vote on the pure motion on the record before the amended one wins.

    The same applies for amendments from other parties or backbenchers in the Opposition – these votes should also lose, and can be put before the motion without disrupting the flow.

    So what could have happened is: Labour amendment rejected (mostly by Government MPs), SNP motion rejected (mostly by Government MPs), Conservative amendment approved and adopted (mostly by Government MPs). In this scenario, most MPs have had a chance to vote for their party’s preferred wording, but this is only possible because the first few votes are rejected.

    In practice what seems to have fallen apart is the government approach – exactly why is still unclear but one suggestion is not enough Government MPs wanted to vote against the Labour wording, so to avoid an internal conflict they pulled their amendment and stopped opposing other votes. 

    This meant that Labour’s amendment won, it replaced the SNP motion and was passed as the main motion. 

    This outcome was the opposite of the one the Speaker’s choice was intended to facilitate. The SNP (and anyone who preferred the contents of their motion) didn’t get the chance to vote on their version, and no one generated a voting record either. A ceasefire motion passed, but no individual votes were recorded for it.

    The role of the Speaker

    The core issue is different ideas of what the Speaker is supposed to do. 

    In one reading the Speaker is supposed to be an agent to draw out the collective will of MPs, in another, the emphasis is on being non-partisan and reflecting a settled (cross-party) view of how the House of Commons operates. 

    The Clerk of the House advised not to allow both amendments, but also said that this was allowed by the rules, it went against previous approaches and risked that the SNP motion wouldn’t be voted on. The Speaker didn’t do anything inherently wrong by the rulebook, but has upset the sense that he was supposed to be a speaker who “innovated” less than his predecessor John Bercow.

    The virtue of deferring to the dead hand of precedent is that it shields the Speaker from the accusations of political bias. The outcome of this decision was good for Labour in that it avoided a split over the SNP vote, leading to a perception the rules were being bent in Labour’s favour. If this had threaded the needle and everyone had got the votes they wanted, this might have paid off. As it is, there’s a big question mark over whether the Speaker is trusted by MPs to be fulfilling the role. 

    The argument made by Owen Thompson (SNP) was that “the purpose of an Opposition day is for our party to have the ability to put forward our business”.  In general, Labour has a lot more Opposition days, where they haven’t chosen to propose their version of the motion. One of the SNP’s few days has resulted in SNP MPs not being able to put their views on the record. 

    But also if a motion would be preferred by the House it doesn’t seem undemocratic to include it. The amendment process is not meant to allow voicing opinions on three different things – but working towards a single statement that has majority support. If including more amendments changes the outcome, it is reasonable to include them on this basis. 

    This gets at different ideas of what voting in Parliament is for – is it for Parliament to come together and agree a consensus view, or for political actors to signal their divergent views? Both of these are legitimate purposes for a political body – especially when the goal of the motion is signally internationally (and also domestically) what the UK political establishment’s views are. 

    What does this mean for foreign policy?

    There’s a view that this kind of vote is navel gazing – and what MPs yell about over here doesn’t affect things over there. But this view is too narrow and misunderstands Parliament’s role as a political institution and how that relates to international politics.

    Motions can be broadly “doing something” or “saying something” motions, and this was a “saying something” motion. It doesn’t commit the government to do anything, and if it did, the government doesn’t have the power to impose a ceasefire tomorrow.

    In practical terms, it doesn’t matter what the Opposition thinks except in terms of the approach it signals in a possible next government. It does matter what government MPs are thinking however, and these motions seem to have flushed out some fault lines within the Conservative Party. Even if this isn’t on the voting record, it shapes internal discussion and policy making. 

    It is broadly good for the long term project of British diplomacy and coalitions with other countries where there is widespread consensus in Parliament on an action. Even partial support for bigger approaches within the governing party gives the Foreign Secretary more flexibility, and alignment with the likely next party of government similarly empowers the kind of statements and alliances that can be made. 

    So the vote does nothing in itself, but helps reveal what the political lie of the land actually is, and empowers actors working within it. Just because something is partial and political doesn’t mean it’s pointless.

    Political violence is shaping how representatives behave

    Another running thread here is the idea of political violence impacting decisions on the parliamentary agenda. The Speaker explicitly said the decisions he made on amendments were based on conversations “about the security of Members, their families and the people involved” – where MPs were considering their personal safety in weighing up if they could oppose motions by other parties. Regardless of whether you think it would be fine if more MPs had supported the SNP motion, it’s not good that this is part of the thinking in either direction. 

    This is part of a wider problem where political violence and threats of violence are collapsing political trust and openness – making politicians more suspicious of each other (seeing each other as whipping up mobs rather than engaging in politics), and less likely to give the benefit of the doubt to what may be passionate but legitimate participation of citizens in politics. 

    The parliamentary rulebook cannot take the weight of this – there are arguments about the extent to which allowing individual expression is an important purpose, but it can’t take the weight of allowing individual expression for the purposes of safety. 

    Here is where the recent Jo Cox Foundation report  No place in politics: tackling abuse and intimidation gives constructive steps. A key argument in the JCF report is that proportionate reactions to political violence can damage the relationship between representatives and their constituents. More safety measures and less public visibility make representatives less accessible. Its impact is not just in the one act, but the chilling effect it extends through the whole system that makes us more distant and suspicious of each other. 

    Reflecting this, the report puts a lot of time into a series of very practical measures to improve policing and reporting of threats and abuse, tying together different systems of support across Parliament, parties and policing. The clearest way to take abuse and intimidation seriously is to join up support and action on the least ambiguous cases. Politicians feeling that they are safe, and that threats against them are taken seriously, helps an environment where trust and openness support a better democratic system.

    Photo: UK Parliament – Central Lobby 

  10. By-Election Briefing: Understanding boundary changes with the Local Intelligence Hub

    Last Thursday saw two by-elections and two new MPs elected. When the Kingswood and Wellingborough voters go to the polls for the upcoming general election, many will be voting for candidates in brand new constituencies, and won’t have the MP they’ve just elected on their ballot paper. What can the Local Intelligence Hub tell us about how these constituencies will change?

    The times boundaries, they are a’changing

    Both of the constituencies that went to the polls on Thursday are being divided up to form multiple new constituencies at the next general election. The total number of constituencies and MPs (650) isn’t changing, but the boundaries are moving, and there are lots of new (and long) constituency names. In the case of Kingswood, no constituency of that name will exist anymore, instead being replaced by four brand new constituencies. Wellingborough, meanwhile, will be divided into three new constituencies. Let’s dive into the detail 👇

    So, who goes where?

    At the top of our new constituency pages, you’ll find the candidates that have been announced for that seat, thanks to our friends at The Democracy Club. This isn’t an official data set, it’s crowdsourced by Democracy Club and their wonderful volunteers.

    We can see that Kingswood’s new MP, Damian Egan, is standing as the candidate in the new Bristol North East constituency. We also know that just 36% of the constituency’s current population will have the opportunity to vote for him next time round. Here’s how Kingswood will change:

    • Bristol North East will cover approximately 36% of this constituency’s population, and 15% of this constituency’s area. 
    • Filton and Bradley Stoke will cover approximately 18% of this constituency’s population, and 10% of this constituency’s area.
    • North East Somerset and Hanham will cover approximately 45% of this constituency’s population, and 60% of this constituency’s area.
    • Thornbury and Yate will cover approximately 1% of this constituency’s population, and 14% of this constituency’s area.

    What about Wellingborough? We don’t have as much candidate information, but we do know that Wellingborough will become:

    • Daventry, which will cover approximately 4% of this constituency’s population, and 24% of this constituency’s area.
    • South Northamptonshire, which will cover approximately 5% of this constituency’s population, and 24% of this constituency’s area.
    • Wellingborough and Rushden, which will cover approximately 90% of this constituency’s population, and 51% of this constituency’s area.

    What does that mean for our data?

    As we explain here, it depends on how the data comes to us in the first place.

    Over time, statistics agencies will release more information for future constituencies, which we will be able to import straight into the Local Intelligence Hub. But during the changeover we want to keep as much of the value of datasets for the outgoing constituencies as possible.

    What can we say about how these constituencies will change?

    For datasets where we have the original data at a very granular level (eg: LSOA or point-based data), we’ve started creating new datasets using future constituencies. We’ve already done that for the Index of Multiple Deprivation dataset, and we’ll let you know as we make more progress on this. 

    Where we only have data at the level of current constituencies, we’ve created a process to approximately convert information from current to future constituencies. The big assumption of this method is that, for either people or area, the thing being measured is evenly distributed across that metric. As such, we think it’s fair to say that while the data is fuzzy in comparison between neighbours, overall it will capture trends across wider areas or regions.

    You can also dig into the new constituencies data yourself.


    The Local Intelligence Hub is brand new, and we’re still working out how to make it as useful as possible – for old constituencies, and new ones. Please try the hub out for yourself, and let us know how you get on!

    Photo by Red Dot on Unsplash

    P.S. We’ve also published this on our LinkedIn page – why not connect with us there?