1. Off by one: How Parliament counts votes is out of date

    Last night there was a vote to allow MPs to be excluded from Parliament (after a risk assessment) if arrested on suspicion of a serious offence. This vote passed by a single vote.

    The problem is, looking across several sources of voting information, there’s not a good agreement on what the actual totals were. Ultimately the tellers count is authoritative, but this problem reflects the complicated way that MPs vote.

    The result(?)

    SourceDescribed resultCount of names
    votes.parliament.uk170 Ayes, 169 Noes169 Ayes, 169 Noes
    hansard.parliament.uk170 Ayes, 169 Noes169 Ayes, 168 Noes
    theyworkforyou.com
    (teller result)
    169 Ayes, 168 Noes

    170 Ayes, 169 Noes (speech with teller result)
    169 Ayes, 168 Noes

    What’s going on here?

    In the voting lobby, there are two different systems going on to record votes:

    • An electronic pass based voting system – run by the clerks, that feeds into votes.parliament.uk and Hansard.
    • A counting system run by the tellers – a MP for each side is in each lobby, and if they agree the count, that’s the count used to make the decision.

    Meanwhile, at TheyWorkForYou, we use tidied up division names created by votes.parliament.uk, but the division lists from Hansard, and add the names to get the number of people on each side. 

    Votes.parliament.uk will be quickest with who voted – this feeds into the Hansard list, but the two can get out of sync if one is updated but not the other. 

    In this case, Rebecca Harris is counted in votes.parliament.uk but not in Hansard. This could be for a few reasons, for instance she may not have been able to use the pass system for some reason but was recorded manually and added as a correction but after it was fed into Hansard. We’ve queried this with her. In any case, what the tellers counted is the authoritative result for the vote. They could also have been right – and someone else forgot/was not able to tap in who should have done.

    But if the votes.parliament.uk count was right, it would mean the tellers in the Aye lobby overcounted by one. This would make it a draw, and in a draw the speaker will cast a deciding vote against the motion (as there isn’t a majority for it).  When it’s down to one vote – you want to have faith the system got the right answer. 

    Better ways are possible

    We think it should be easier for MPs to vote, and have previously recommended that:

    • The House of Commons should in normal circumstances, defer votes to a standardised voting time (within ‘core hours’), where multiple votes are held in succession.
    • These votes should be held through a fast electronic means – whether through terminals, voting pass systems, or apps.
    • Current proxy voting schemes should be extended to personal discretion to designate a proxy – e.g. a set number of days a year a proxy vote can be allocated, no questions asked.

    Electronic voting and a voting time would be bringing back good practice from the devolved Parliaments and help MPs make better use of their time than standing in division lobbies. But as well as being slow – there are clear questions to ask about the accuracy of the current approach.

    How MPs vote has big impacts on how our country works – getting it right matters.

  2. Improving TheyWorkForYou’s voting summaries

    If you value the work mySociety and TheyWorkForYou do, please consider whether you can make a donation.

    We have a good track record of making Parliament more open, provide essential tools to civil society and small charities, and with our platform a little support can go a long way.  If you would like to make a larger donation to support specific work, or to match-fund other donations – please get in touch.


    Our MPs in Parliament have many roles, but one of the most important is that they make decisions on the laws that govern us, and these decisions can affect every aspect of how we live our lives. 

    TheyWorkForYou’s voting record summaries are part of a number of different arguments about what the role of MPs is, and how Parliament should work.

    As well as listing individual votes in Parliament, our voting summaries give an overview of how MPs have voted on policies that come up in multiple votes. We strongly stand by the principle of our summaries, but don’t think there’s only one way of doing it. Using a grant from the Newby Trust, we’ve been reviewing our methods and refining our approaches to voting records. 

    The key headline is that we’re going to sharpen the focus in our approach. The main changes are:

    We have written a longer document explaining how these changes achieve our goals.

    This change is being applied alongside a backlog of new policy lines that we’ve been reviewing with our new criteria for inclusion. While these may be big shifts in principle, in practice most existing summaries stay exactly the same. It’s a progression and simplification rather than a revolution.

    To see our voting summaries for your MP, search for your postcode on TheyWorkForYou and click ‘Voting Summaries’.

    What we want to achieve with our summaries

    In thinking about our voting summaries, we wanted to clearly define what we’re trying to accomplish. This has led to two headline goals:

    We want to present clear and accurate summaries of how individual MPs have voted, for use by the public.

    • As a point of principle, it should be possible and straightforward to find out how MPs have acted on behalf of their constituents. 
    • The top-line display of information should be a good reflection of the data that was used to create it – balancing clarity and accuracy.  We should provide options for people to learn or explore more, with the expectation that most won’t, and so the clarity of the summary matters.
    • While we aspire to produce information that is also of use to people with a professional interest in Parliament, this need might be better met through other tools or summaries. For instance, while it is possible to compare different MPs through voting records, it is not the main purpose of these summaries.  

    In line with our general approach, we want to align with and amplify citizen perspectives of how MPs should work, as voiced by Citizens’ Assemblies (in particular the Democracy in the UK Citizens Assembly) and polling.

    • Historically, we’ve seen that making the actions of MPs more visible changes their behaviour.
    • We need to be conscious of the likely effects of our summaries, and ensure they reflect our values, democratic principles and approach. We want to anchor our approach in wider ideas of how our democracy works rather than our own opinions.
    • We also need to be aware of when pressure on individual MPs is not the best way to achieve systemic change. As such we need to consider where our work reinforces rather than changing parliamentary systems that are hostile to MPs from groups historically excluded from Parliament (e.g. women, ethnic minorities, disabled MPs). 

    A longer document explaining how these changes achieve these goals can be read here

    The impact of this change

    Most of the top-level summaries on the site (73%) are completely unaffected by these changes. 82% of MP ‘scores’ are either the same, or have a stronger/weaker version of the same alignment (i.e. the adjustment has not affected our assessment of whether the MP is for or against a policy). About 14% of connections between MP and policies are removed, which is a combination of removing seven policy lines that were made up entirely of votes that did not directly use Parliamentary power, and longer running policies being confined to a narrower time frame. The remaining changes are “a mix for and against” assignments becoming more clear (or the reverse), and a small group (about 120 out of 80,000) where the direction of the score has changed (i.e. where someone was voting for and is now seen as voting against – this is mostly concentrated in two policies). You can read more about this in our longer summary.

    This is good because we don’t generally want these to be too sensitive to the exact formula used: the kind of broad points we’re making should be reachable no matter which method is applied. Ultimately only a small group of votes have been removed from policies, and the positions we were displaying before were mostly driven by votes that already passed the “use of powers” criteria. The goal of this process is to simplify how we work and enable clearer explanations of what we’re doing – but the general end product isn’t massively changed by adopting these new rules.

    A process, not a destination

    This isn’t where we stop. This update is a step in the journey.

    There is a growing clarity issue that for long-serving MPs there are now quite a lot of policies — and part of our work creating summaries should be helping people find the relevant information they’re looking for. 

    There is also a pending question about presenting a retrospective on the current Parliament during the next election. With the technical work we’ve done, it is now much easier to explore alternate approaches to displaying this data. 

    We are considering how we can best do this, and how we work with others to ensure we are capturing the important issues of the last Parliament. 

    Making other tools available 

    One kind of complaint about voting summaries is that they do not provide an easy way of drawing out small differences between two MPs on how they voted. This is true – we might say two MPs voted a mixture of for and against a policy, but in practice they took opposite positions on different votes. 

    In our voting summaries we’ve made the decision to focus on providing information that makes sense for a constituent looking at their MP – we produce better summaries by focusing on specific kinds of users we want to make sure it works for. But for our own work as well as to support others, we want to provide a wider range of tools and information for both citizens and specialists. 

    Our previous approach to voting was deeply tied technically with the Public Whip (originally a companion project to TheyWorkForYou, but not run by mySociety). This means we had limited ability to take big swings in our approach: while we indirectly maintain it through the data feed, we can’t change the basic functioning of the Public Whip.  

    To implement the changes described above, we have internally created a Public Whip replacement (TheyWorkForYou Votes) that we’re using to update voting records, and provide new analysis tools to help us understand votes, giving us easy understanding of the parliamentary dynamics of a vote and basic analysis of the motion.  In the next year, we want to talk to more people who want better tools for working with raw voting information, to help shape this tool for a public release. 

    Supporting our work

    In our voting records work we have an approach that has public support, and we think serves an important purpose. But we don’t think this is the only or best way of creating voting summaries: we want to be able to be reactive to how Parliament is changing, and always making our coverage and approach better. We also want to work to encourage better transparency and public understanding at the source, through improving how Parliament works.

    If you value the work we do, please consider whether you can support us financially. We have a good track record, and with our platform a little support can go a long way.  If you would like to make a larger donation to support specific work, or to match-fund other donations, please get in touch. 

    In the next few weeks we will be announcing a new project involving volunteers and the register of members interests. If you’re interested in hearing more about that, please sign up to our volunteer mailing list.

    Header image: Photo by Aaron Du on Unsplash

  3. Democracy month notes: February

    Previously: January!

    Gaza ceasefire blog post

    I wrote a blog post about the Gaza Ceasefire opposition day votes – especially focusing on how there ended up being no recorded votes. 

    This is the kind of responsive work we’d like to do more of. We don’t need to duplicating every explainer out there, but we want to be able to better articulate “this is how Parliament works, but there’s something wrong with that” when there’s currently something confusing/going wrong in the news. 

    Asking for money to do good things

    Alice, Julia and I have been putting together a more structured version of the idea I talk about at the bottom of this blog post about our new spreadsheet of the register of interests — using crowdsourcing to create good, understandable summaries of MPs interests. Will let you know how that goes. 

    Something we’d like to get better at is being more public when these applications for funding do not work out (spoiler: this happens a lot!). There’s a lot of work and creativity that goes into our ambitions for TheyWorkForYou, and ideally these wouldn’t just be locked away in various virtual desk drawers. 

    Oflog consultation

    Julia worked with our friends at the Centre for Public Data on a joint response to an Office for Local Government (OFLOG) consultation – read more about that

    This is a continuation of our work around public data fragmentation

    Small API updates

    Matthew has added Parliament’s unique identifier to the response to the ‘getMPInfo’ API call, making it easier to jump from our data to query the Parliament API.

    Server upgrades

    Sam and Matthew have been upgrading the servers that run TheyWorkForYou and WriteToThem.

    We need to do this periodically for security reasons: the organisations that distribute the server software (and other packages we depend on, like those that distribute the programming languages) only provide security and bug fixes for a certain period, after which they only provide it for newer versions. 

    Running software on the web — where there are *constantly* bad people testing for weaknesses — means taking this seriously. But upgrading the lower levels of the “stack” often means small changes further up where features we use have been deprecated and replaced with other approaches. Some of this work is running just to stay in the same place, but it does also enable us to adopt new approaches in how we code and the packages we use. 

    This is one of the massive benefits of the same organisation running TheyWorkForYou AND WhatDoTheyKnow AND FixMyStreet AND (many more) – we have excellent people thinking hard about our technical infrastructure across all our work. 

    Voting summary update

    We’ve done some of the trickiest technical work required to enable the voting summary update we’re planning.

    We’ve moved TheyWorkForYou from pointing at the Public Whip website, where it used to get voting summary calculations, to an instance of a new,experimental “twfy-votes” platform. This is doing the work Public Whip was originally doing, but also taking over the party comparison calculations that were being done in TheyWorkForYou itself previously. 

    TheyWorkForYou has become simpler, and more of the relevant code is now in the same place. We’re not yet completely independent of the Public Whip because twfy-votes currently uses the database dump to populate itself — but soon we’ll be able to move that to an export from TheyWorkForYou’s own database. 

    The goal in this set of changes is to move from this:

    Diagram showing the flow of data from the Hansard XML, through Parlparse, into both TheyWorkForYou and the PublicWhip - with that then reentering theyworkforyou and additional calculations being done to calculate voting summaries

    To this:

    Diagram showing the different flow of data from the Hansard XML - through ParlParse to TheyWorkForYou, and a feedback look between TWFY and TWFY-VOTES

    Which is… still a lot of boxes and arrows, but is better than it was. This could in principle then be simplified even further, but this brings the whole process under our control and simplifies some of the back and forth steps. 

    Currently, all this work should have resulted in almost no visible changes to the site. But we now can flip a switch and it will switch the underlying algorithm used from the one in the Public Whip to the new (simplified) approach.  One of the motivations behind this shift is to be fully in control of that algorithm (which is effectively a number-based editorial policy). 

    One of the things I’ve been doing this month is running the analysis to clearly map what exactly the public effect of this will be. Broadly, most things stay the same, which is good because we don’t want the headline messages to be hugely affected by different methodologies behind the scenes – At the same time we’ll end up with something that is easier to explain. 

    The final stage before full release is a set of less technical changes, consolidating the voting summary information on one page, and adding a rewritten page describing both how Parliamentary voting works in different places across the UK, and what our approach is in the data we publish. Making good progress on these, and hope to have this project completed soon. 

    That’s all for now

    As ever, if you’re the kind of person who reads to the end of these (I’m going to assume a generally nice person who is also a fan our our work) – donations are welcome. But also get in touch if you’ve got something to chat with us about!

    Header image: Photo by yasin hemmati on Unsplash

  4. Gaza Ceasefire votes and voting records

    What happened?

    Yesterday in the House of Commons, there was an SNP Opposition Day debate about a ceasefire in Gaza. This meant that the SNP had an opportunity to put forward a motion for the House to vote on.

    The Labour Party’s preferred wording of a ceasefire motion replaced the SNP motion and was passed by the House of Commons based on what is known as a voice vote. 

    This is when the speaker (in this case the Deputy Speaker who was in the chair) judges the result of the vote based on the volume of shouts in the chamber. As such, there is no record of how individual MPs voted.

    This is not the same as saying the vote was unanimous – and listening to the recordings there is a clear ‘no’ present on both votes (the Deputy Speaker does later claim that “nobody called against it”, which is then contested).

    From the Speaker’s point of view, the goal is taking a read on the decision of the House (and this may have been correct in that one side was louder, if not unopposed) – and a vote in the lobbies (division), which takes around 15 minutes, serves no purpose. 

    But votes also serve the purpose of putting the opinions of individual MPs on the record, which several were frustrated to have been denied. Votes are part of the public facts about MPs’ impact in Parliament, and part of how actions are communicated to constituents. This is a factor in the democratic process that also needs balancing in these decisions. 

    This decision followed a long division for a motion to sit in private – and votes that seemed clear on a voice vote may have been seen as costly in terms of time to take to a full division. In general, it is possible to have voting processes that are much faster and fairer to MPs, that would allow getting two votes on the record without taking most of an hour of parliamentary time. 

    What does this mean for TheyWorkForYou?

    Because there was no recorded division – the approval of the motion does not appear in the recent votes tab for MPs.

    The full debate is worth a read – the general sense is of a long debate where MPs engage with a complicated situation, and reflect that the UK’s role can only be part of any solution. 

    We’re in the process of updating the processes behind our voting summaries, which includes ways to include what we’re calling “agreements” (decisions without a “division”) in summaries. But issues like last night’s decision reflect that we need to take a cautionary approach – as there is clear evidence that it was not an unambiguous decision. We will publish more on this approach soon.

    Why was the amendment process controversial?

    As it was an SNP Opposition day, (a day when an Opposition party gets to choose the main debate) they got to propose the motion. Both Labour (another Opposition party) and the Conservative Party (Government party) proposed amendments to the motion.

    The Speaker went against previous convention and allowed both a Government and Opposition amendment – which was unexpected. The motivation of this was to give most MPs a chance to vote for a motion on their preferred wording – the problem is that the amendment process is not really set up for this.

    The thinking makes sense given how Opposition day voting is supposed to work: the Opposition by definition is not supposed to win because they have fewer MPs than the government.

    What’s supposed to happen is that MPs debate a topic, hold a vote, and the motion is rejected. If the topic strikes closer to home, the Government will amend it to say “this is an important issue but the Government is doing a great job”, and that is the motion that is passed because the Government should have the numbers to win the vote.

    Government amendments come after the vote on the main motion to respect the purpose of Opposition day debates, while reflecting the reality that the government can amend the motion and win. This sequencing allows for a vote on the pure motion on the record before the amended one wins.

    The same applies for amendments from other parties or backbenchers in the Opposition – these votes should also lose, and can be put before the motion without disrupting the flow.

    So what could have happened is: Labour amendment rejected (mostly by Government MPs), SNP motion rejected (mostly by Government MPs), Conservative amendment approved and adopted (mostly by Government MPs). In this scenario, most MPs have had a chance to vote for their party’s preferred wording, but this is only possible because the first few votes are rejected.

    In practice what seems to have fallen apart is the government approach – exactly why is still unclear but one suggestion is not enough Government MPs wanted to vote against the Labour wording, so to avoid an internal conflict they pulled their amendment and stopped opposing other votes. 

    This meant that Labour’s amendment won, it replaced the SNP motion and was passed as the main motion. 

    This outcome was the opposite of the one the Speaker’s choice was intended to facilitate. The SNP (and anyone who preferred the contents of their motion) didn’t get the chance to vote on their version, and no one generated a voting record either. A ceasefire motion passed, but no individual votes were recorded for it.

    The role of the Speaker

    The core issue is different ideas of what the Speaker is supposed to do. 

    In one reading the Speaker is supposed to be an agent to draw out the collective will of MPs, in another, the emphasis is on being non-partisan and reflecting a settled (cross-party) view of how the House of Commons operates. 

    The Clerk of the House advised not to allow both amendments, but also said that this was allowed by the rules, it went against previous approaches and risked that the SNP motion wouldn’t be voted on. The Speaker didn’t do anything inherently wrong by the rulebook, but has upset the sense that he was supposed to be a speaker who “innovated” less than his predecessor John Bercow.

    The virtue of deferring to the dead hand of precedent is that it shields the Speaker from the accusations of political bias. The outcome of this decision was good for Labour in that it avoided a split over the SNP vote, leading to a perception the rules were being bent in Labour’s favour. If this had threaded the needle and everyone had got the votes they wanted, this might have paid off. As it is, there’s a big question mark over whether the Speaker is trusted by MPs to be fulfilling the role. 

    The argument made by Owen Thompson (SNP) was that “the purpose of an Opposition day is for our party to have the ability to put forward our business”.  In general, Labour has a lot more Opposition days, where they haven’t chosen to propose their version of the motion. One of the SNP’s few days has resulted in SNP MPs not being able to put their views on the record. 

    But also if a motion would be preferred by the House it doesn’t seem undemocratic to include it. The amendment process is not meant to allow voicing opinions on three different things – but working towards a single statement that has majority support. If including more amendments changes the outcome, it is reasonable to include them on this basis. 

    This gets at different ideas of what voting in Parliament is for – is it for Parliament to come together and agree a consensus view, or for political actors to signal their divergent views? Both of these are legitimate purposes for a political body – especially when the goal of the motion is signally internationally (and also domestically) what the UK political establishment’s views are. 

    What does this mean for foreign policy?

    There’s a view that this kind of vote is navel gazing – and what MPs yell about over here doesn’t affect things over there. But this view is too narrow and misunderstands Parliament’s role as a political institution and how that relates to international politics.

    Motions can be broadly “doing something” or “saying something” motions, and this was a “saying something” motion. It doesn’t commit the government to do anything, and if it did, the government doesn’t have the power to impose a ceasefire tomorrow.

    In practical terms, it doesn’t matter what the Opposition thinks except in terms of the approach it signals in a possible next government. It does matter what government MPs are thinking however, and these motions seem to have flushed out some fault lines within the Conservative Party. Even if this isn’t on the voting record, it shapes internal discussion and policy making. 

    It is broadly good for the long term project of British diplomacy and coalitions with other countries where there is widespread consensus in Parliament on an action. Even partial support for bigger approaches within the governing party gives the Foreign Secretary more flexibility, and alignment with the likely next party of government similarly empowers the kind of statements and alliances that can be made. 

    So the vote does nothing in itself, but helps reveal what the political lie of the land actually is, and empowers actors working within it. Just because something is partial and political doesn’t mean it’s pointless.

    Political violence is shaping how representatives behave

    Another running thread here is the idea of political violence impacting decisions on the parliamentary agenda. The Speaker explicitly said the decisions he made on amendments were based on conversations “about the security of Members, their families and the people involved” – where MPs were considering their personal safety in weighing up if they could oppose motions by other parties. Regardless of whether you think it would be fine if more MPs had supported the SNP motion, it’s not good that this is part of the thinking in either direction. 

    This is part of a wider problem where political violence and threats of violence are collapsing political trust and openness – making politicians more suspicious of each other (seeing each other as whipping up mobs rather than engaging in politics), and less likely to give the benefit of the doubt to what may be passionate but legitimate participation of citizens in politics. 

    The parliamentary rulebook cannot take the weight of this – there are arguments about the extent to which allowing individual expression is an important purpose, but it can’t take the weight of allowing individual expression for the purposes of safety. 

    Here is where the recent Jo Cox Foundation report  No place in politics: tackling abuse and intimidation gives constructive steps. A key argument in the JCF report is that proportionate reactions to political violence can damage the relationship between representatives and their constituents. More safety measures and less public visibility make representatives less accessible. Its impact is not just in the one act, but the chilling effect it extends through the whole system that makes us more distant and suspicious of each other. 

    Reflecting this, the report puts a lot of time into a series of very practical measures to improve policing and reporting of threats and abuse, tying together different systems of support across Parliament, parties and policing. The clearest way to take abuse and intimidation seriously is to join up support and action on the least ambiguous cases. Politicians feeling that they are safe, and that threats against them are taken seriously, helps an environment where trust and openness support a better democratic system.

    Photo: UK Parliament – Central Lobby 

  5. Democracy month notes – January

    What are month notes?

    They are notes on what we’ve been working on… each month. It’s like weeknotes for lazy people. 

    We’ve been writing them for our Climate programme, and we’re building up to the point where we’ve got enough going on in our Democracy work that it’s worth establishing the habit of being clear about what’s going on.

    It’s about being open with what we’re working on, and if we’re lucky that helps spark conversations that help move things along. 

    What is “democracy” at mySociety?

    “We should do some Democracy monthnotes” is a sentence that makes perfect sense internally, but for some quick scene setting…

    In principle, everything we do is in some way about democracy —  we consider our FOI and transparency work to be important in part because it enables and grows civic ideas of democracy. Key to our climate work is the idea that democratic and climate problems are linked, and so our climate work is very engaged with the kinds of problems of democracy we’ve been thinking about all along — but with a sharper focus.

    Internally, we tend to think about our democracy work as being around TheyWorkForYou  and WriteToThem, and internationally looking at similar “parliamentary monitoring organisations”. These are some of our longest running services, widely used, and with a long potential future ahead of them. One of the things we’ve been doing over the last year is creating a clearer idea of what we want to accomplish with PMO work. Lots of this work has been behind the scenes in funding bids – but can be seen in the work adding the Senedd as a general direction of travel. More on what we’re currently working on (and some things that didn’t work out) in future. 

    Behind the scenes, there isn’t a Democracy “team” as such because we don’t currently have the funding available for that. My estimate is that last year we probably had 1-1.5 full time equivalent (FTE) people working on Democracy – but that was spread over 5-6 actual people. Given the porous lines between the different things we do, the immediate goal isn’t to get a big team, but to be increasing the consistency with which we can use the wide range of skills already in the organisation — and in making links and making the most of opportunities across our wider work. 

    So “Democracy” at mySociety is always going to be a little fluid —  we’ll use these monthnotes to be clearer about what that means in practice. 

    All the conferences

    The Democracy Network held its second conference this January, attended by 10% of mySociety (three people). 

    This is an interesting crowd that is, for obvious reasons, moving into being quite election focused. By contrast, a lot of mySociety’s work is about an effective civic democracy between elections. Many WriteToThem and TheyWorkForYou features become less useful in an election, while traffic increases and usage changes. Once Parliament dissolves ,there are no MPs until new ones are elected, but what those MPs have been up to is important. We’re doing some thinking on our options for running the most useful version of the site during an election, and have picked up some conversations at the conference to continue. 

    Julia also went to the Democracy Classroom strategy day, where she spoke on a panel about using data in campaigning alongside our friends at Generation Rent and the Democracy Club. Our hosts, the Politics Project, brought together organisations of all sizes and types, from all across the UK. It was great to have a really practical conversation about the data needs of organisations working with young people, and after a follow up chat with Gaibhin from United Response, we’re already working on adding census disability data to the Local Intelligence Hub service we’re launching soon. 

    We’ve also had the mySociety quarterly team meeting, where we all head somewhere in the country (this time, Leeds!) to talk for a few days with colleagues we generally only see on the internet. They’re nice! 

    Register of interests

    Great to put out our spreadsheet version of the register of members interests

    Fun fact: this has been mostly ready to go since October —  part of not having a lot of funded time for Democracy work means there’s a backlog of 99% finished work to get out the door. 

    But the response to the blog post shows the value in getting that out, and in being transparent in general. Lots of nice comments from people who think they’ll find it useful – but it’s also leading to more conversations with people with an interest in the register that can help us get a better sense of what’s currently happening where, and what our role in the picture might be. 

    For keeping track: On my bit at the end about the kinds of questions people might want answered, Stuart from Open Innovations has linked me to some of their old weeknotes on work they did with PDS and the HoC Library on the kind of questions that can be answered through the current data.

    Voting records

    One of the big things we were doing behind the scenes last year is reviewing and updating our approach to voting records. This is one of TheyWorkForYou’s most notable features, and we got a grant from Newby Trust to have a good look at these and used some of our grant from the Porticus Foundation to do some more involved technical work than we may have otherwise been able to manage. January has been the fiddly final stages of getting this to launch. 

    I’m not going to go into a lot of details here (there are long blog posts to come) – but the big task has been thinking through what we’re trying to achieve, and then untangling our technical systems to make that sustainable over the long run. 

    Our current system is based on various data flows in and out of the Public Whip – which has a complicated history with TheyWorkForYou. It has some overlap with the people who founded it, it’s not run by us, but at the moment is substantially kept updated by Matthew’s work unclogging the ParlParse system the two sites have in common. Some of the things we want to change would need changes deep in the Public Whip, which we can’t do, and that’s bad for what’s such an important feature of the site. 

    As such, we’ve made a transitional replacement for the Public Whip, where we can build in the kind of analysis tools we need to have more visibility and control of the whole process.Over this year we’re going to be talking to people who want more of the kind of number-y analysis the Public Whip does well to tidy up what we’re using internally — and set it up as a useful specialist complement to TheyWorkForYou. 

    TICTeC/PMO Communities of Practice

    TICTeC is back in London! See the call for proposals about the conference itself – where we’re especially trying to think about how/if/where civic tech is relevant to themes of democracy in crisis, and democratic approaches to the climate crisis.

    As part of this funding, we are also setting up communities of practice around access to information and parliamentary monitoring organisations. A big bit of January for me has been (working with the Civic Tech Field Guide) making a good list of PMO organisations from around the world to approach to get a sense of what problems we might discuss over the next year. I’m currently working through the survey responses to that. 

    New combined authority

    Welcome to the new “York and North Yorkshire Mayoral Combined Authority”  — added to our big list of local authorities, our IMD dataset, and nearest neighbour dataset – and to CAPE – our local climate action tracker – where you can see some of these datasets in practice.

    CAPE has some features helping navigate the connections between authorities and combined authorities, and in general we’re trying to think about how we can better reflect Combined Authorities in our core work. 

    A key use of WriteToThem is a “here is the structure of government where you are” — and we haven’t added the new CAs because unlike the London Assembly they don’t have as clear a public facing representative. The long term solution here is either to lean into WriteToThem having information for people you don’t necessarily “write to” or building a clearer page for this into TheyWorkForYou – which does some version of this for devolved Parliaments/Assemblies. We’re thinking about it. 

    Making progress

    In any given month, we’re generally making incremental progress on things we think are good ideas, that might also importantly be fundable ideas on where we’re well placed to make something better

    Julia’s been developing more about what our approach to training might be — making more of the fact our tools are already used by educators, and building a better loop between that and our service development. 

    I’ve been developing our thoughts on the register of interests further, and reading through the Jo Cox Foundation’s new report “No place in politics: tackling abuse and intimidation” (which I thought was measured, and well-thought through) and making some notes on how it applies to our work. 

    We’re also thinking more about how practically we can try and increase support from the public for our services. Here moving a bit away from “Save TheyWorkForYou” language to being clearer about how what we do is part of making things better (regardless of who wins elections), and that we want to be far more ambitious than keeping the lights on.

    Anyway, if you’ve read this far — donations are welcome. But also get in touch if you’ve got something to chat with us about!

  6. Improving the register of MPs interests

    Tl;dr: We’re now releasing our register of interests data as a spreadsheet.

    High quality data about the external interests of our MPs and ministers is vital to identifying conflicts of interests, and discouraging politicians from having conflicts of interest in the first place.

    Lack of clarity on the interests and income streams of MPs is a corruption risk. The problem with second jobs and outside interests is less that MPs might be distracted from their main job – but that when they stand in Parliament, they may be representing groups beyond their constituents, asking questions (or not asking questions) depending on their outside work.

    When outside interests exist, it’s vital they are clear and transparent. The Register of Members Interests contains a list of disclosures MPs are required to make of financial interests or benefits which “others might reasonably consider to influence his or her actions or words as a Member of Parliament”. Following the Owen Patterson scandal, there was renewed interest in this data, as it was clear that there were a number of potential stories and scandals hidden in plain sight – just requiring someone to join up the data.



    Building a data ecosystem

    A key problem is that the data is not easy to work with. The data is released (roughly fortnightly) on the parliament.uk website as a HTML document for each MP. This process technically releases the information, but makes it hard to compare releases of the same MP over time, or to make comparisons between different MPs.

    TheyWorkForYou improves on this by creating structured data from the HTML release. Using this we can highlight the changes in each release from the previous release. This is useful for journalists and campaigners in quickly understanding what has changed in each release. For instance, the change in Rishi Sunak’s register over time can be seen here.

    We want to avoid people doing the same work of cleaning the data over and over. We make our version of the data available publicly, so other people can use our work to do things that we haven’t done ourselves. For instance, Open Innovations have built on top of the data we publish to link the data to other datasets and create a Register of Members’ Financial Interests Explorer.

    While projects like the Tortoise/Sky News Westminster Accounts create new value in joining up datasets and cleaning the data for their own work – ultimately the new datasets they have created are only usable by those organisations. That’s their right as the people doing the work – but we think there is a bigger (and more sustainable) impact to be had in improving the data in public.

    Making our data more accessible

    Previously, we have published our interests data as a series of XML files, which is useful for programmers, but harder for other specialists to work with. We did some thinking with OpenDemocracy last year to explore if there were small changes we could make that would make the work we already do more useful.

    As well as the XML files, we now publish an experimental spreadsheet version of all data since 2000, and the register for the current 2019 Parliament.

    These sheets show the earliest and latest disclosure of an interest, and include some (very) basic NLP analysis to extract mentioned orgs from the free text and make it easier to quickly parse when scrolling.

    This data can also be explored through Datasette, which can be used to query the datasets in the browser, and save the queries as links that can be shared.

    For instance, the following links go to specific queries (we’re using an in-browser version for prototyping and this might take a minute to load):

    We want to continue to improve our approach here – and welcome feedback from anyone this spreadsheet helps.

    Parliament can do better data publication

    A key problem run into by everyone working with the data is that it’s broken to start with. MPs fill things out in inconsistent ways that makes the overall data different to analyse without cleaning first (see both the Open Innovations and Tortoise/Sky News methodology notes). Fixing this up is a key first step towards aggregate analysis – and the easiest place to fix it is with validation when the data is collected at the start.

    While work can be done to improve the data after the fact (and experiments with Generative AI have found it to be quite good at fixing inconsistent formatting), improving the initial data collection is the most effective way of improving the quality of the data. There are active moves in Parliament to fix some of these problems. Producing more information in machine readable formats, and adding methods to make sure the data is correct to start with, will make the transparency process simpler at every stage.

    Similar issues apply to the register published for All-Party Parliamentary Groups (APPGs), which should publish as “machine readable” data the range of data that the groups are formally supposed to make publicly available. APPGs are semi-official groups that MPs can form around specific interests or issues. Many of these are useful ways of having discussions, but these can also be an avenue for corruption, with outside interests supporting the group and its activities. The register includes the officers of groups and financial assistance and gifts received by the group – but not the overall membership. APPGs are separately required to disclose their wider membership on their website (or if they don’t have a website, if someone asks) but this isn’t included in the register, and so can’t be consistently scraped to produce data. While MPs are supposed to disclose benefits from groups on their individual disclosure, clearer data on what is officially “public” memberships would help ensure that there is nothing missed between these two datasets.

    Separately there is a register of ministerial interests that applies to MPs who also have government positions. This is in principle more strict, requiring disclosures of relevant interests of family members, and avoiding even perceived conflicts of interest. However, in practice the information does not contain the specific financial value of gifts or benefits, just that they exist. The disclosure cycle is also longer, being published every six months rather than monthly. In practice – this means that relevant interests may not be public for a significant time after a minister is appointed (and potentially never published, if the minister has again moved on by then).

    There is a lot of work that can be done from the outside to build on official data. But the more Parliament does things that it is uniquely able to do, the more we can focus on analysis and data comparisons that are best done outside.

    What mySociety can do

    A very basic thing we can do is beat the drum (and work with those who have been doing this for ages) for better publication of data from Parliament.

    But if this happens or not, we can do work to make the data better. If it looks like Parliament’s data is unlikely to be fixed at the source, then a project of improving the data in public in a way that multiple projects could then build on would be useful. But if the data gets better, then we can better spend our time doing more work on top of this data. This might include joining up the official data with other datasets (including those of the UK’s other Parliaments and Assemblies) to draw out connections and better analysis.

    But our work here isn’t just about producing good data – it’s about displaying it in a way that’s useful and understandable by people. Chris Bryant MP (former Chair of the Standards Committee) has argued that Parliament’s own display of the history of registers should match what’s provided by TheyWorkForYou. If Parliament improved its own display to the public of registers of members’ interests this would be fantastic news – and we in turn would need to think about if there are new approaches that would be useful on top of that.

    One approach we are thinking about would be to find out what people wanted to know the answers to about their MPs interests, and then using volunteers to answer a set of common questions. This is the kind of editorialising that Parliament itself would find much harder to do – while providing something different from aggregate analysis of the data all together. This is something we could do with the data as it exists, but is something where better data would let us create new tools so volunteers could answer more complicated questions.

    Making MPs’ interests clearer and easier to understand is key to spotting conflicts of interest and keeping politicians accountable. We hope our new spreadsheet version of the data helps make the work we’re already doing more useful and accessible – while we think about the road we want to take in future to improve TheyWorkForYou and the project of a transparent democracy.





    If you'd like to see us extending our work in democracy further, please consider making a contribution.
    Donate now

    Image: Wilhelm Gunkel on Unsplash.

  7. How we improved CAPE this year

    CAPE is our data-packed website that makes it easier for everyone to track and improve local action on climate change, built in collaboration with Climate Emergency UK.

    When CAPE launched, it was basically a list of councils’ climate action plans — but we’ve continued to make regular improvements and additions, and it’s better, and more useful, than ever.

    If you’re a council officer with climate duties; a researcher looking for climate data; an activist needing facts to back up your climate campaign; or simply an interested citizen, you’ll find some truly invaluable facts and figures on CAPE.

    And if you haven’t visited for a while, you’re in for a surprise. Here’s what we’ve added this year:

    Improved search

    Search results for 'flooding' - these include terms such as 'drainage' and 'river'

    Now you don’t need to think of all the different ways that councils might have described the concept you’re looking for. We’ve integrated machine learning into the search function, to help surface your search term even in documents that use different words for the same concept. Interested to know more? We wrote it up in detail here

    Updated for new councils

    In April, a set of UK local authorities was abolished and merged into new unitary authorities.  When this happens, we do work to update the data that CAPE depends on. You can make use of it too!

    We’ve updated:

    And this has all fed back into CAPE to make changes of local government structure transparent in the site. 

    Local polling

    We wanted to make it easier to see where there is support for climate action, so we took a few sets of MRP polling on people’s attitudes to net zero and energy sources and converted it from per-constituency to per-local council.

    For each local authority (except in Northern Ireland) – see Croydon, for example – you can now see estimates for support for net zero (even when it is described as expensive), and for different kinds of renewable energy project.

    Support for net zero in Croydon

    Not sure what MRP polling is? We wrote something up about that

    Climate Assemblies

    Local climate assemblies at CAPEIn 2019, we were part of the UK-wide Climate Assembly, and we’ve kept an interest in citizens assemblies about climate change. There’s been one held for the whole of Scotland, and more widely we’ve found 17 (so far) held by local authorities across the UK.

    We’ve uploaded the final reports of those assemblies into CAPE. It is now easy to see which authorities have held climate assemblies, and to search the results of those assemblies. Find all this at cape.mysociety.org/assemblies.

    Hundreds of new documents have a fresh coat of paint

    Thanks to people telling us about new plans, and us conducting an extensive search for new documents across council domains, we’ve updated and added hundreds of new documents to councils’ pages. 

    With all that content, we’ve gone back and tidied up the design of our council page layout, giving us more space to add explainers and summaries to each section. Why not take a look for your area?

    Council Climate Action Scorecards

    Let’s not forget that while all this activity has taken place on CAPE, we were also helping Climate Emergency UK with the Council Climate Action Scorecards. At the time of their launch in October, we wrote about the part we played in building the site, wrangling the data, advising on the methodology and building a scoring interface.

    The result of all this is, of course, that whatever your climate interests – be you council staff, researcher, journalist, campaigner or interested member of the public – you can dive in to any council’s page to see how they’re doing against several markers of climate action.

    Since launch there has also been the significant addition of Question Pages, allowing a never before seen view of how each council scores on every individual action point. Julia wrote about that in detail, in this post.


    Your donations keep our sites running.
    Donate now

  8. Providing tech for effective crowdsourcing

    An important idea in mySociety’s history and work is that combining people-power and technology is a powerful method to make change.

    Recently we’ve had the chance to work with Climate Emergency UK on the council climate scorecards. As part of this we’ve been learning from how CE UK works with volunteers, and thinking how best we can be a technical partner to crowdsourcing projects.

    But we’ve also been thinking how we can better incorporate crowdsourcing into our core services. This has led to improving our approach to crowdsourcing within WhatDoTheyKnow, and developing a new platform (GRACE) to help volunteers crowdsource information for the scorecards.

    Developing our technical approach

    In the first round of the Council Climate Scorecards,back in 2021, the process was managed on a big Google Sheet. Each answer to a question has a line, which tracks the answer from a dropdown and keeps track of notes, and we use macros to populate and move information around. This worked and let us get started at the pace we wanted, but introduced a lot of room for error, and meant more work was needed at the end of the process to validate the data.

    For the second round of scorecards, with a more complicated set of questions, we wanted to move away from the spreadsheet approach, into something that could validate the data as we went, and integrated the right of reply process into the same platform.

    Working with CE UK, we built a crowdsourcing platform GRACE (named after CE UK’s first volunteer) to replace the old spreadsheet approach.

    The overall process worked like this:

    • Working with their advisory group, CE UK created questions across a range of different areas, with different sections for different kinds of councils.
    • These questions and data from FOI responses were uploaded into the platform.
    • Volunteers were given access to the platform, and could answer questions in the topic areas assigned to them. This would include an answer from a dropdown, a public “evidence” field, and a private notes field to share between volunteers.
    • Overall progress across all areas could be seen by the volunteer coordinators.
    • Local authorities were given a right of reply per question. Emails were sent to all local authorities, giving them access to the platform to review the scores and evidence for their section. This gave them the opportunity to submit new evidence where they felt the score did not reflect their actions. 74% submitted responses to the right of reply.
    • Responses from the right of reply were reconciled with the original answers by a group of volunteers and CE UK.
    • This data could then be reviewed and loaded into councilclimatescorecards.uk

    How did CE UK find the tool? Isaac Beevor, Co-director of CE UK:

    GRACE – our online data collection system was incredible. We would not have been able to collect the thousands of data points on local authorities’ climate action without it. It was simple to use and effective so all of our volunteers were able to use it, with minimal training. Furthermore, it allowed us to coordinate and track progress. This meant we were able to encourage volunteers who needed it and track progress in sections allowing us to plan ahead for staff and volunteer capacity. We then used it to allow access for councils in the Right of Reply and all of the feedback from officers, particularly its ease of use, was positive. The best thing is we can now use this for every Scorecards, which saves us so much time and energy.

    Effective crowdsourcing

    As we wrote in our report about public fragmented data the issue with the idea of ‘armchair auditors’ is not that they do not exist, but that they were thought about in the wrong way. People can and do use their time to support civic accountability through looking at spreadsheets. But they need to be given support and structure to work effectively together.

    Reflecting on the CE UK process and other crowdsourcing approaches we admire, like Research for Action and Democracy Club, what these projects have in common is that they involve knowledge sharing and collaboration across the country, with volunteers themselves contributing a local or specialist focus. “Armchair auditors” aren’t atomised individuals, but work together as a community.

    The Effective Crowdsourcing diamond (below) describes the aspects we think are key to successful projects, and helps us shape our thinking about future crowdsourcing/citizen science projects.

    A blue diamond with the following text "Effective crowdsouricng: working together to answer big questions". The four points of the diamond have: Volunteers: A pool of motivated supporters break the problem up Expertise: Experts and advisors to frame the questions to ask Technology: specialist tools streamline information gathering and quality assurance steps Impact: clear paths form gathering data to making change.

    In this framework, an effective process will have four features that all interrelate and reinforce each other:

    • Expertise – Helping move from high level principles to concrete questions and approaches that inform how work can be split up for volunteers, and the technical tools needed.
    • Volunteers – People who care about an issue, with the skills to contribute to answering questions.
    • Technology – Technology makes it easier for people to work together, and validation and cross checking improves the accuracy of the process, enabling more complex approaches.
    • Impact – What is the path from the result of a project to change the world? Clear routes to impact means clearer benefits of engaging to experts and volunteers.

    For Scorecards and other projects, we have provided the technical side of the diamond. But there is lots of potential for us to run crowdsourcing projects that improve our core Democracy and Transparency services. Starting with the tools we now have, we’re thinking about how we can strengthen our skills and approaches across the diamond – and how we might partner on projects to bring in other expertise and skills.

    GRACE screenshots

    Initial volunteer screen showing sections assigned:

    Screenshot of a page showing the assignments and completion rates assigned to a specific indivudal. A series of progress bars next to sections.

    Screen showing councils and progress

    Screenshot of a pages showing the overall progress for a section. It shows by council the number of questiosn completed for this group of questions

    Marking screen

    This shows the marking screen - which is a form capture the answer to a question, with boxes and links for evidence

    Right of Reply screen

    This is the right of reply screen that will be visible by councils - it shows the answers to the question in uneditable boxes -while providing options to agree/disagree with response and submit further evidence

    Admin screen so you can check that how much has been assigned to volunteers

    Volunteer management screen - showing how many sections are currently assigned to volutneerws

    Admin screen to show progress on an authority level

    Progress screen, showing the number of questions completed by council, and the overall progress

    Or on a section level

    Progress screen by question section, showing the number of councils with a complete set of questions for that section.

    Image: Clay Banks on Unsplash.

  9. CE UK and mySociety are using people power and Freedom of Information to bring transparency to local climate action

    A story in this week’s Financial Times [paywalled] has brought the EPC ratings of council-owned properties into the public conversation. This story was based on data obtained through FOI requests as part of the Council Climate Action Scorecards project, which we’ve been working on in partnership with Climate Emergency UK (CE UK).

    What you can read in the FT is one story pulled from a wealth of data, but there’s more to come. Our WhatDoTheyKnow Projects tool allowed CE UK’s team of volunteers to conduct a nationwide survey of every council through well-placed FOI requests covering the use of renewable energy, plans for retrofitting, green skills training, road expansion and more. 

    The data they gathered has allowed for the understanding of councils’ action on a nationwide scale. This level of oversight has not previously been possible: as with so much about the Scorecards project, it is allowing councils to take more informed action on climate, and individuals to clearly understand what is being done.

    Why local action matters

    In the UK, it is estimated that around one third of carbon emissions are in some way under the influence of local authorities. 80% of UK councils have declared a ‘climate emergency’ to indicate they recognise the scale of the problem of climate change, and are in a position to take practical steps to be part of the solution. To help local authorities achieve the goals they set themselves (and to push them to go further), we need to engage with the plans that local authorities are making, and the actions they are starting to take. 

    In 2021, CE UK and mySociety worked together to produce the first Council Climate Plan Scorecards. CE UK’s upcoming launch is the second iteration of the Scorecards. It is much bigger and more ambitious in scope than the last: it scores not the plans, but the climate actions of every local authority in the UK. 

    FOI requests were just one part of the process. As well as giving CE UK access to WhatDoTheyKnow Projects, we developed a crowdsourcing tool for volunteers to use while marking across the 90+ datapoints collected for each council. 

    How do you score action?

    CE UK moved from scoring plans to scoring actions. That required new approaches to gathering the information. 

    The questions CEUK used in the new Scorecards are the result of a long and thorough process of research and refinement. Building on their own research and expertise, they conducted one-on-one consultations with approximately 80 organisations and sector-specific experts. An advisory group of environmental and local government experts provided further discussion and refinement, to help build a list of questions that would practically be possible to answer, and that would reveal important information about the climate actions of councils. 

    The aim was to identify areas where information was publicly accessible; but also where gaps existed, especially in operational matters that aren’t often made public. Additionally, CE UK wanted to investigate whether councils are truly implementing the actions outlined in their climate action plans, including aspects like lobbying for additional powers.

    Making use of Freedom of Information

    Freedom of Information laws means that a huge range of information held by public authorities (including local councils) can be requested by any person who asks. This provides a legal tool to create greater transparency where information is not being published proactively.

    For CE UK, the potential of FOI for the Scorecards project was clear – but there were concerns. In consultations with council staff, there was pushback regarding the use of FOI requests due to the potential time and financial burden on council officers who work on climate – with some requests for a more informal survey approach to be used. But the drawback of that would be making good data dependent on goodwill everywhere. FOI requests provided a way to make sure the scorecards were not just effective for councils who engaged with the process and provide an approach that was fair across the country. 

    To balance a process where they want to encourage positive engagement from councils, with one that works without that, CE UK’s approach was to plan out the most efficient and least burdensome use of FOI requests. 

    Based on feedback from the advisory group, and trial runs to a small number of councils, they eliminated questions that were less important and useful, made more ‘yes/no’ or ‘single number’ responses, and learned where certain questions weren’t relevant to certain areas or groups of councils. 

    The subsequent FOI requests became more streamlined, and this resulted in quicker response times for the final requests than they had in the trial – as the information sought was more direct and concise.

    In the end, CE UK submitted a total of over 4,000 FOI requests to councils across the UK. The questions were divided into 11 categories, with some being specific to certain types of councils, such as district councils or combined authorities. The next stage was taking these 4,000 requests and getting them into a form that can be used for the scorecards. 

    Crowdsourcing and review process

    CE UK used WhatDoTheyKnow to manage their FOI request process. mySociety’s WhatDoTheyKnow acts as a public archive for requests – requests made through the site have the responses shown in public to bring more information into the open  – making it more discoverable by other people interested in the information, and reducing the need for duplicate requests being made. As of 2023, a million requests for information have been made through the site, with hundreds of thousands of pieces of information being released. 

    A feature we are trialling with a range of organisations is WhatDoTheyKnow Projects, which integrates crowdsourcing tools into WhatDoTheyKnow, and allows the task of extracting information into a new dataset to be spread out. The goal is that this helps organisations be more ambitious in finding out information and helps people work together to create genuinely new and exciting datasets, that no single organisation has ever seen. 

    As CE UK’s approach already made heavy use of volunteers and crowdsourcing, this was a natural fit.  Alongside a wider group of 200 volunteers working on getting answers to the other questions, 15 volunteers specifically worked on the FOI requests. These volunteers were a mixture of people with prior experience or professional interest in FOI requests, campaigners well-versed in FOI processes, and individuals new to the concept but eager to engage in activism.

    After the crowdsourcing of FOI data was complete, it joined the rest of the data in the new tool mySociety had developed for helping volunteers crowdsource information for the Scorecards.  

    From here, councils were given access to the data collected about them and given a right of reply to correct any inaccuracies or point towards information not previously discovered or disclosed. The results of this process will then be reviewed to produce the final Scorecards data, which will be launched this month.

    But the Scorecards data will not be the only useful thing that will come out of this process. Because of how WhatDoTheyKnow was used, to see evidence supporting the final Scorecards, people will be able to click through and see the original responses, for instance, to see what councils have lobbied on support for their climate work. 

    Some of the FOIs are being used to construct datasets that have a broader impact, and here we come back to that FT story on the Energy Performance Certificate (EPC) ratings of council-owned houses. Building these new public datasets will be useful for councils to understand their own situation, and as we see with the news story, more broadly to understand the challenges ahead for local governments to meet net zero emissions goals. 

    Onwards

    The original Scorecards project has already been influential on how local governments understand their own plans, and how organisations like the UK’s Climate Change Committee understand the role and progress of local government in the challenges ahead. When the next generation of Scorecards is released, we hope that they continue to be useful in shaping and improving local government action around climate change.

    mySociety believes that digital technology can be used to help people participate more fully in democracy, make governments and societies more transparent, and bring communities together to address societal challenges.

    The Scorecards project showcases how the combination of digital tools, people power, and the right to information produces powerful results. We hope that the impact of this project can inspire and make possible similar approaches for other problems, or in other countries.

  10. ICO advisory note on publishing spreadsheets

    Following the PSNI and other recent data breaches, the ICO has issued guidance to public authorities. This guidance suggests a temporary stop on publishing Excel-style spreadsheets in response to FOI requests made via online platforms like WhatDoTheyKnow. The full advisory note is available online

    The advisory note emphasises that this is not a reason not to disclose requested information. Instead, the ICO says to release the information from original source spreadsheets as a CSV file – a simpler format than Excel Workbooks, with less potential for including hidden sheets or metadata that can lead to an accidental breach.

    A focus on file formats is a blunt measure, and one that will need to be superseded by better procedures and technical processes.

    We support authorities releasing data in the most appropriate format for the information being requested. This may sometimes mean an extract from a table, and sometimes a complete document. Excel spreadsheets are legitimate public documents, and information released in this format can be hugely valuable. It’s important to develop processes where they can be released safely. 

    Significant data breaches involving Excel files clearly show the risks when data management and release processes fail. These include not just breaches we see through WhatDoTheyKnow, but through disclosure logs and releases made directly to requesters. This is an opportunity for public authorities, the ICO and us at WhatDoTheyKnow to reflect on how we can best deliver the huge benefits of public transparency while safeguarding personal data. 

    Modern authorities need to be good at handling data. Data breaches happen at the intersection of technical and human processes. The FOI team can be the last link in the chain of a data breach when they release the information, but the root cause often goes back to wider organisational issues with the handling of sensitive data.

    In the short run, the ICO has recommended training for staff involved with disclosing data. Many teams already have excellent processes and do excellent work, but all authorities should take this opportunity to consider their responsibility on the data they hold, and have appropriate processes in place.

    Long term progress means developing good universal processes that keep data safe, regardless of the format of the data or how the data is released. All FOI releases should in principle be treated as if they are being released to the public, because the authority’s ability to stop a data breach ends when the information is released. Making FOI responses public produces huge efficiencies for the public sector, increasing transparency in practice, and multiplying the benefit to society of the information released. 

    Technology can also be part of the solution – we need to understand more about why existing technical ways of removing hidden information from Excel spreadsheets are not being used (as described in the ICO’s established guidance on disclosing information safely), and how new tools or guidance can make it easier to release data safely. 

    A core part of our work at WhatDoTheyKnow is dealing with the practical reality of promoting public transparency while protecting personal information. We take data breaches seriously and have processes in place for dealing with them as promptly as possible. We continue to plan and work to help reduce the occurrences and impact of personal data breaches through both our procedures and technical approach. 

    By monitoring how authorities respond to requests on WhatDoTheyKnow, we will seek to understand how this guidance is working in practice, and engage with the ICO and other organisations to promote effective long term approaches to this problem. 


    Notes on the content of the advisory

    Below is our understanding of the advisory note by subject matter:

    Freedom of Information requests

    • Continue to comply with FOI responsibilities. This guidance is about releasing information in a way that reduces risk of accidental disclosure. 
    • Temporarily, do not release original source spreadsheets to online platforms like WhatDoTheyKnow. Instead – convert and release to CSV files.
    • If that is not possible, then:
      • Ask if the Excel sheet can be sent to a separate (non-public) address. Proceed with the original address if they ask for this. 
      • In all releases, go through processes to ensure there is no data breach in the material. 

    General data management

    • Excel files are unsuitable working environments when they become very large (hundreds of thousands of rows). Authorities need to switch to appropriate data management systems that are more appropriate for managing larger amounts of data.  
    • Staff who use data software and are involved in disclosing information need continuous training.  
    • Understanding of pivot tables and their risks should be incorporated into data management.

    The ICO plans to update their guidance on Disclosing Information Safely

    The checklist released accompanying the advisory has several useful steps on checking for hidden data in Excel sheets. However, on the ‘considered alternative ways to disclose’ step, refer back to the steps in the advisory note. Information converted to CSV can be released to WhatDoTheyKnow in compliance with the advisory note. The advisory note says that the source dataset should continue to be released to WhatDoTheyKnow if it cannot be converted, the requester does not want to use an alternative route, and the authority is confident it does not contain a data breach.