In celebration of our millionth public FOI request on WhatDoTheyKnow, we’ve been examining many aspects of the service — and now we come to that huge archive of released information, built up by members of the public asking for what they wanted, and receiving data in response.

So, what’s in there?

The thing is, we don’t know. 

Well, that’s not to say we have no idea: of course we do occasionally look at what’s being requested and released via WhatDoTheyKnow. When we find something interesting, we often tweet about it. But with requests being submitted at an ever-increasing rate, there’s no way that we can inspect every single one.

We do, however, have some thoughts about what you can find in the massive public archive that is WhatDoTheyKnow, and it’s not just ‘information that people have asked for’. The meta-benefit of having a big corpus like this free and online is that it provides a lens for examining our own society.

So for example, some of the information just waiting to be analysed by curious researchers, linguists or historians include:

What sort of things people want to know Machine learning technology can now allow a researcher to take a huge dataset like WhatDoTheyKnow’s million public requests, and analyse it. What topics do the most people ask about, which authorities do they make what type of requests to, which type of wording is most likely to gain a response, and all sorts of other questions are just waiting to be answered — should a curious enough researcher want to get their teeth into it.

And incidentally – if an authority analyses the requests made to it over the years, it would also find useful intel on what data it could be publishing proactively, so people don’t even have to submit an FOI request, and they don’t have to expend effort responding to them.

Lessons about how language changes When you have a million questions, and a vast number of responses from authorities, you can start to understand shifts in our language. For example: when did information officers start including pronouns in their signatures? That may not be relevant to the information being released, but it still marks a significant change in social history. And when did society settle on ‘covid’ rather than ‘coronavirus’, ‘Brexit’ rather than ‘leaving the EU’? We reckon there’s a huge value to any linguist or historian who takes a look.

Widely useful information There’s at least one type of information that we know is useful to hundreds of people every year: the acceptance rates at various universities. 

Every year, prospective students start to worry about their chance of getting into their preferred institution; and every year (usually on Reddit) they are pointed towards previous releases showing the data; or, where it doesn’t exist, advised to put in their own request for it.

That’s just one example of how information, once published, has a spread far beyond the single person who requested it. There are many more, not least in Wikipedia citations.

When contracts are due for renewal Another common use of the FOI Act is by companies or startups looking for commercial information — such as when a contract is up for renewal so they can submit a tender; or whether certain authorities have a need for the product they’re developing.

A request sent across a number of authorities in the area, or even across the country, can be a very efficient source of intel. 

Your esoteric pet subject What are you into? Politics?, Public transport? Cookery, bats, or seagulls perhaps? Whatever it is, you can search WhatDoTheyKnow and see if anyone’s uncovered interesting information about it. 

If not, maybe you’ll think of something you’d like to know — and don’t forget you can set up an alert, so you’ll receive an email whenever someone mentions your chosen keyword in a future request or response.

What authorities don’t actually have a record of. We blogged a while back about what it means when your request comes back as ‘information not held’. Sometimes this can be as revealing as the information itself.

Datasets behind news stories. When journalists or researchers use WhatDoTheyKnow or WhatDoTheyKnow Pro to gather data that helps them break a story or write a paper, we always encourage them to link their article or report back to the responses on the site. Because, yes, they’ve found one story, but there may be more to discover in there, and there are always people motivated enough to look. Equally, we’ll link back from the site to their story – look out for the ‘in the news’ section in the right hand column of every request page.

–

These are just a few examples of the riches to be found in plain view on WhatDoTheyKnow. If it wasn’t for the UK’s Freedom of Information Act, and for WhatDoTheyKnow’s ability to make information truly free, none of this would be available. But it is, and that’s great, so why not dive into the search bar and see what you can find?

Tomorrow, in the final post of this series, we’ll be looking at what we’re doing to bring WhatDoTheyKnow’s benefits to the communities that need them most.

 

—

Image: Fabio

Yesterday we shared the news that WhatDoTheyKnow has processed its millionth public request. 

The site’s been around since 2008, nearly as long as the UK’s right to information, and we think it’s fair to say that we’ve had some impact on the world during that time.

Let’s go back, just for a moment, to 2006 when mySociety ran its open call for suggestions of new websites we could build. Imagine we’d bypassed the ‘Freedom of Information Filer and Archive’ suggested by both Francis Irving and Phil Rodgers, and instead had plumped for one of the easier ideas. And in this scenario, let’s imagine that no-one else went ahead and made an FOI site either.

So, in a world without WhatDoTheyKnow:

Information would be released to the requester only. Here’s the most obvious difference: instead of being automatically published on WhatDoTheyKnow, any information received would come directly to the person who requested it. 

If someone else wanted the same information, they’d have to ask for it again. And every time it was requested, authorities would have to send it out all over again.

This one simple thing that WhatDoTheyKnow does – publishing responses – both puts information into the public domain, and saves authorities from the bother of duplicating their efforts.

Information might not be released by email. Of course, when you make a request on WhatDoTheyKnow, it goes to the authority by email, and, almost always, the response is sent by the same means. But in our alternate universe without WhatDoTheyKnow, information might come much more regularly through the request-maker’s own letterbox.

In WhatDoTheyKnow’s early days, one of the big battles we had to fight was for email to be accepted as a valid FOI request — not to mention email that came from a WhatDoTheyKnow-generated email address. Guidance from both the Ministry of Justice and the Information Commissioner now confirms that such requests are not only valid — and in 2016 an independent commission concluded that publishing responses to FOI requests “should be the norm”.

Many fewer people would have heard of FOI, and FOI would be the preserve mainly of journalists and researchers. Let’s face it, FOI still isn’t as well-understood as we might like it to be — even though our research found that one in ten adults in the UK has put in a request at some time.  

But without WhatDoTheyKnow, we believe the concept of FOI would be even less recognised. Fewer people would have stumbled across it when looking for answers; even those who had heard of the Act might find it difficult to figure out how to access it. It’s probable that only trained professionals such as journalists and researchers would be using FOI on a regular basis. 

We wouldn’t be there to help people with FOI issues. WhatDoTheyKnow’s amazing team of volunteers answers a massive number of queries every day — questions from users of the site who are puzzled about how to make a request, what to do when they receive a refusal, or what an exemption means. 

If it wasn’t for WhatDoTheyKnow, the chances are that the small part of the general population who did figure out how to make a request would give up as soon as they received a refusal or a request for clarification.

People around the world wouldn’t have access to FOI sites, either. If we hadn’t built WhatDoTheyKnow, we’d never have packaged it up as the open source Alaveteli codebase  — and motivated individuals around the world wouldn’t have had a simple way to set up their own access to information websites. We’re proud to say that Alaveteli sites are running in more than 25 jurisdictions globally, from Argentina and Australia, to Ukraine, Uganda and Uruguay. 

Our right to information would be weaker. We’ve defended the FOI Act through successive governments, with winds blowing FOI in and out of favour. We’ve given evidence in Parliament, stood up for FOI via inquiries and fought against its erosion with campaigns. 

We believe in the right to information as a basic tenet of democracy and accountability, and we’re prepared to do whatever it takes to defend it.

So, with all that in mind, aren’t you glad that WhatDoTheyKnow does exist? 

Come back tomorrow to find out how WhatDoTheyKnow can be used to tackle the overarching issue of our times: climate.

—

Image: Fons Heijnsbroek

Pop open the bubbly — this is huge! Yes, it’s a big day for us, as the number of Freedom of Information requests on WhatDoTheyKnow ticks over to a mahoosive one million. That milestone was reached at 05:34 this morning, when a request to Kent Police was published.

A million public requests! It’s proof of the value of FOI, and of the need for WhatDoTheyKnow. In essence, this big round number represents the vast archive of publicly-available information, built up by hundreds of thousands of individual users over the site’s 15 year lifetime. They’ve asked — and continue to ask — for information from public authorities, at the current rate of two-and-half thousand requests a week.

Why? Because, thanks to the Freedom of Information Act, they can; and, perhaps more importantly, thanks to WhatDoTheyKnow, it’s easy. Normal. Unintimidating.

According to our polling, one in ten UK citizens have used FOI. People are doing good things with WhatDoTheyKnow  — we celebrated several of them at our recent awards, and over the years we’ve written about the varied and often surprising ways in which people have used our service to change the world. As a small sample of the many amazing uses we’ve seen, here’s how WhatDoTheyKnow has helped people to:

• get to the truth of  the Post Office Horizon scandal
• reveal vital environmental information
• investigate into missing children
• protect social housing
• understand what happens to victims of modern slavery
• fight against LOBO loans to local councils
• uncover discrimination towards wheelchair users
• stand up for erectile dysfunction care
  

But the impact doesn’t stop there. We know from the massive ratio of visitors to requesters that the main use of WhatDoTheyKnow is in viewing information that others have made public. This means that for the same cost to the public purse of processing an FOI request, information has been made much more public and discoverable. 

Over the past nine  years, 660,000 requests have had 107 million page views (160x). WhatDoTheyKnow is, in systematic terms, a cheap way of getting more benefit from the hundreds of thousands of pieces of public information that have been released through FOI. That benefit will multiply, long into the future, with an archive that will always be available.

And that’s what we mean when we say that information can be free. Free, as in free to fly; and free as in provided at absolutely no cost to anyone who can make use of it. 

Thank you to everyone who’s played a part in WhatDoTheyKnow reaching this meaningful milestone: the volunteers who help run the site; the developers who helped to build it and those who continue to refine it; the information officers who gather and respond with information; the funders who understand the worth of our service; and of course all those citizens who, collectively, have asked for information and, together, built up this unparalleled library of knowledge. 

Here’s to you all, and here’s to the ten millionth request — which given the exponential rate of growth, will not take ten times as long for us to reach.

If you’d like to assure the future for easy access to information, then please do make a donation. Thank you.

Next post in this series: what the world would look like if WhatDoTheyKnow had never been launched.

—

Image: Ivan Lopatin

The ways in which people and organisations have used mySociety’s services through the lifetime of the organisation have been impressive, inspiring and sometimes astonishing.

So, to celebrate our 20th anniversary, on 15 November we’ll be presenting awards in five categories, showcasing impactful usage of their services through the years.

• Driving Institutional Change
• Accelerating Climate Action
• Exposing Truth
• Impactful International Reuse
• Campaigning for Justice

The shortlist is as follows:

Driving Institutional Change

• The Give Them Time campaign used WhatDoTheyKnow to get the law changed over funding for nursery care in Scotland.
• John Graham-Cumming In 2009, John used the petitions website that mySociety had built for 10 Downing Street, resulting in Gordon Brown apologising on behalf of the British Government for its treatment of the computer scientist Alan Turing.
• Richard Bennett used WhatDoTheyKnow, coupled with the Equality Act, to make pathways more accessible for wheelchair users, sharing his methods so that others could do the same.
• Privacy International The ‘Neighbourhood Watched’ project used WhatDoTheyKnow to reveal the unchecked use of surveillance technology by police forces across the UK.

Accelerating Climate Action

• Zero Hour Using mySociety’s WriteToThem software, they’ve garnered the backing of over 150 MPs for their draft Climate and Ecology Bill.
• Sustain used data from CAPE, our Climate Action Plans Explorer, to analyse the degree to which local authorities are including food within their strategies to cut emissions. 
• Save the Trees of Armada Way Plymouth’s grassroots campaign fought against the removal of much-loved trees in the city centre, using WriteToThem to send emails to the local councillors — apparently, the most emails they had ever received on a single subject. 

 Exposing Truth

• Jenna Corderoy Jenna is shortlisted for her investigation — using WhatDoTheyKnow — of the Cabinet Office’s controversial Clearing House, a secretive unit that screened  and blocked FOI requests made by journalists and campaigners, often on matters of serious public interest.
• The Bureau of Investigative Journalism Their Sold From Under You project used crowdsourced and FOI data to reveal how much publicly-owned property was sold off by councils across England, in an attempt to fill funding gaps caused by austerity measures. 
• Lost in Europe worked with people running FOI sites on our Alaveteli platform, in 12 different countries, to uncover previously unknown statistics around how many children disappear at borders. 

Impactful International Reuse 

• Dostup do Pravda/Access to Truth The Ukrainian Freedom of Information site continues providing access to information even in the difficult circumstances of war.
• vTaiwan, Public Digital Innovation Space, and the Taiwanese Ministry of Digital Affairs The Taiwanese government uses mySociety’s SayIt software to make deliberations on difficult subjects public and accessible to citizens.
• DATA Uruguay The organisation has built both FixMyStreet and Freedom of Information sites on mySociety’s codebases, changing the way their governments  communicate with citizens at both local and national levels.

Campaigning for Justice 

• Doug Paulley is a lifelong campaigner for rights for disabled people, using FOI to fight against access discrimination, especially around public transport.
• Eleanor Shaikh has dedicated hours and hundreds of FOI requests to finding out the truth behind the Post Office Horizon scandal, with her findings making front page headlines.
• After Exploitation use Freedom of Information to uncover the failings of the government’s measures to protect vulnerable detainees.

—

Of course, every single user of our services is a winner in our eyes – but watch this space to find out who takes home the award in each category!

—

Image: Rene Böhmer

Today, we’re happy to join in the excitement around the launch of the Council Climate Action Scorecards.

Just over 18 months ago, we were pointing at the first iteration of this work by Climate Emergency UK (CE UK), which marked every council’s climate action plan according to a detailed schema. Back then, we were impressed by the scale and quality of what they’d pulled off, and pleased to have been a partner in delivering the work.

But if that was impressive, what’s been achieved this time around is even more so. While climate action plans are simple documents, with all the information in one place, unpicking how climate action is progressing at the local government level is a much more complicated matter. 

Once again, CE UK amassed a large cohort of volunteers, trained them up and set them the task of obtaining information about the state of play in every council area via a variety of means: news stories, meeting minutes, websites and strategy docs; and where the information couldn’t be found by any publicly-available source, with FOI requests. It is a real testament to people power, coupled with one of mySociety’s longtime favourite methods of breaking daunting tasks into more manageable chunks by crowdsourcing.

To ensure the data is meaningful, CE UK have completed the work with the oversight of an advisory panel, of which our Head of Research Alex Parsons was one member. mySociety have again played an active part in the project, building a tool on which volunteers assessed action, developing and designing the website, and helping send the FOI requests to multiple councils via our WhatDoTheyKnow Projects tool.

We hope that the Council Climate Action Scorecards will help councils and other key actors such as central government to see where they could be doing more, and to knowledge-share with others. More than that, we hope that campaigners, researchers, journalists — and individuals who want to understand how their councils are doing on climate — will dig into the data and learn more about both the local and nationwide pictures.

At mySociety, our Climate team‘s focus is set by the fact that around a third of all emissions are estimated to be within the power of local authorities. That’s why we have gladly put time and resource into supporting CE UK’s fantastic work.

—

Image: Markus Krisetya

“Freedom Of Information. Three harmless words.”

The Post Office Horizon scandal serves as a prime example of how, when official channels have failed, determined investigators can eventually unpick the truth and ensure that justice is served.  

Horizon, the computerised system on which sub-postmasters were required to balance their tills, was riddled with technical faults that led to inaccurate accounting. These faults could be exacerbated by technicians undertaking remote access without the knowledge of staff, overnight. But incompetence and denials from the top — even as far as government — meant that the blame fell on innocent sub-postmasters. 

Justice has been a long time coming. Staff were subject to fines and even unfair imprisonment; families and friendships were destroyed. Years of diversions and cover-ups have now finally come to light, as you may have seen in the extensive media stories.

One person who has been persistent in uncovering the details of the case over the years is Eleanor Shaikh, who has used WhatDoTheyKnow extensively in the pursuit of truth — you can see her requests here. 

We asked Eleanor to tell us more about her ongoing work in this area, starting with why she got involved: in response, she pointed us towards a letter on Nick Wallis’ Post Office Scandal website.

It begins: “I learned of the Post Office Horizon scandal through being a regular customer at my local post office in Farncombe, Surrey. My ex-Sub-Postmaster, Chirag Sidhpura, was hit by an alleged £57,000 shortfall in October 2017 and it did not take long to see that his case belonged to a more widespread and disturbing pattern.” 

She goes on to explain, “I have seen first hand the silent devastation that this scandal has wrought upon three generations of this decent, hard-working family.” You can read the full letter here.

Findings through FOI

Freedom of Information has been a very effective mechanism for many of those involved in this investigation — a cover-up like this is a perfect demonstration of why we need the rights it confers. 

As Eleanor explains, “Over the years, a number of campaigners and journalists have turned to FOI as a tool for obtaining more information and joining the dots of this vast but well-concealed miscarriage of justice. Chipping away at the cover-up was one small way that outsiders could assist Sub-Postmasters on their long road to the truth, exoneration and redress.”

Eleanor’s 150+ requests have led to some significant findings. She started off with a curiosity about how much had been known, but not publicly shared, by government.

“My initial focus was on what government knew of Horizon’s flaws”, she says; “how responsibly it monitored the unfolding scandal, and what was the extent of its involvement in the group litigation.

“The FOI disclosures I received suggested that central government was not as oblivious to events as it would have us believe; the heavy redactions on these reports themselves bore witness to the fear of reputational damage to the department to which the documents alluded.”

As time went on, Eleanor started uncovering more and more salient facts:

“Probably the two most significant documents I’ve been able to unearth have been the 2016 Swift Review and an undated Post Office Security Team Compliance Document.

“The review would seem to have suggested a moment of alignment when both government and the Post Office were committed to investigating Horizon issues beyond all doubt. 

“It took seven months to receive a response, but disclosure was made by both parties along with requested email correspondence which showed that the PO’s CEO, Paula Vennells, was aware that the review was being undertaken in 2015 but that the PO Chair, Tim Parker, declined to share its findings with the Post Office Board even as the spectre of litigation loomed in 2016. 

“This was despite the review’s strong warnings that miscarriages of justice might have taken place, its clear identification of Horizon’s fundamental operational problems and an acknowledgment of the possibility of remote access.” 

These were all crucial details in understanding the full picture, and fed into news coverage of the scandal. There was also something Eleanor hadn’t been specifically looking for that recently made headline news.

While requesting information on the way in which the Post Office monitored its investigations, Eleanor happened across something rather shocking:

“What was disclosed was a document which used deeply offensive racial identification codes, so inflammatory that it attracted widespread media coverage.” You can see, for example, the BBC’s coverage of this revelation here.

“Thanks to the WhatDoTheyKnow website, once it caught the attention of social media, any journalist could get instant access to the original document.”

FOI was necessary 

One thing that this investigation highlights is that the checks and balances built into our justice system are sometimes inadequate. The Horizon case was examined in High Court, and  has been the subject of two governmental inquiries, one of which is ongoing. But it is citizen reporters using their Right to Information that have filled some of the gaps.

“Despite much crucial information having been released into the public domain during the 2018-19 High Court litigation, many details of the scandal remained hidden”, explains Eleanor. 

“The BEIS Select Committee Inquiry, which heard evidence from early 2020, afforded a brief window of opportunity. But its independent work was abandoned soon after it began when the Department for BEIS (now DBT) put in place its own inquiry.

“The Government was adamant that this inquiry would not have statutory powers, meaning it had no authority to command witnesses to give evidence, nor powers to demand the disclosure of documents. Moreover BEIS could keep its own failures of oversight beyond public scrutiny by restricting the scope of the inquiry. 

“At this moment there was a very real danger that the true depths and extent of the scandal might never reach public consciousness and that those who’d facilitated the miscarriage of justice may never be identified.

“Ministers and Whitehall officials were attempting to shield themselves from scrutiny and to diffuse an information time-bomb by behaving as if it wasn’t there. But questions needed answers and if the government was refusing to launch a full public inquiry in 2020, then it had to be inquiry by FOI.

WhatDoTheyKnow played a part

We asked Eleanor how integral WhatDoTheyKnow was to her investigation. 

“The WhatDoTheyKnow website proved itself to be an invaluable resource. Through its gateway, information and correspondence with authorities is released directly into the public domain in a way which is both transparent and accessible to all; that’s really important in the context of the Horizon scandal.

“Disclosures are easily accessed by others and can be shared on social media. WhatDoTheyKnow has enabled intrepid journalists who follow the scandal — such as Nick Wallis, Karl Flinders and Tony Collins — to extend the reach of any new and significant information.” 

A turning point

“Thankfully”, continues Eleanor, “in June 2021 the inquiry was elevated to a statutory footing in response to ground-breaking rulings at the Court of Appeal. This was a turning point and teams of formidable lawyers set to work in supporting Sir Wyn Williams in his long-awaited public inquiry; the Chair now had the power to determine its scope of issues and assumed far greater powers to elicit the disclosure of evidence. 

“But there was still a role for FOI research in areas which may lie beyond the inquiry’s remit or which haven’t yet come under its scrutiny.“

And FOI also had another significant, but unexpected outcome: 

“One release prompted a review of the Post Office’s entire disclosure processes to the public inquiry. That a key document had never been passed to inquiry lawyers triggered a remedial disclosure exercise so significant that it resulted in the release of thousands more documents and a temporary suspension of the inquiry’s work. 

“A single FOI disclosure triggered a wholly unforeseeable domino effect.”

Three harmless words

At the start of all this, Eleanor says, “FOI was an unfamiliar avenue for me.” Since then, she’s clearly become something of an expert, with FOI being a major part of her investigations.

Reflecting on the rights that FOI confers, she says: “There is some irony that the Government which oversaw the launch of the doomed Horizon project in the late 1990s was at the same time drawing up legislation which granted our right to Freedom of Information. It’s poignant too that Tony Blair who decided that the flawed Horizon project must proceed at all costs, was at the same time doing all he could to delay the implementation of the FOI Act.

“In his memoirs, Blair revealed his regret over the decision which gave the public the right to probe the Government’s shortcomings. He feared information would be weaponised by opponents:

‘Freedom of Information. Three harmless words. I look at those words as I write them, and feel like shaking my head till it drops off my shoulders. You idiot. You naive, foolish, irresponsible nincompoop. There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it…What I failed to realise is that we would also have our skeletons rattling around the cupboard…The Freedom of Information Act was then being debated in Cabinet Committee. It represented a quite extraordinary offer by a government to open itself and Parliament to scrutiny. Its consequences would be revolutionary; the power it handed to the tender mercy of the media was gigantic. We did it with care, but without foresight. Politicians are people and scandals will happen’.¹

“Thankfully, many individuals have been making full use of FOI to help prise the rattling skeletons of the Horizon scandal from their cupboard. Each disclosure adds to our understanding of how things went so deeply and disastrously wrong. And, as sub-postmasters continue to dig for information across their multiple battle fronts, long may FOI continue to make its small contribution to their uphill struggle for justice.”

Many thanks to Eleanor for sharing this detailed account. You can read her findings on Horizon’s early years in her report Origins of a Disaster.

 —

Image: Mark Percy (CC by-sa/2.0)

¹Quote from Blair’s memoirs Tony Blair The Journey Hutchinson, 2010, cited in an article The Blair Memoirs and FOI, also 2010 by Maurice Frankel, Director of the Campaign for Freedom of Information, accessed 10.10.23

A story in this week’s Financial Times [paywalled] has brought the EPC ratings of council-owned properties into the public conversation. This story was based on data obtained through FOI requests as part of the Council Climate Action Scorecards project, which we’ve been working on in partnership with Climate Emergency UK (CE UK).

What you can read in the FT is one story pulled from a wealth of data, but there’s more to come. Our WhatDoTheyKnow Projects tool allowed CE UK’s team of volunteers to conduct a nationwide survey of every council through well-placed FOI requests covering the use of renewable energy, plans for retrofitting, green skills training, road expansion and more. 

The data they gathered has allowed for the understanding of councils’ action on a nationwide scale. This level of oversight has not previously been possible: as with so much about the Scorecards project, it is allowing councils to take more informed action on climate, and individuals to clearly understand what is being done.

Why local action matters

In the UK, it is estimated that around one third of carbon emissions are in some way under the influence of local authorities. 80% of UK councils have declared a ‘climate emergency’ to indicate they recognise the scale of the problem of climate change, and are in a position to take practical steps to be part of the solution. To help local authorities achieve the goals they set themselves (and to push them to go further), we need to engage with the plans that local authorities are making, and the actions they are starting to take. 

In 2021, CE UK and mySociety worked together to produce the first Council Climate Plan Scorecards. CE UK’s upcoming launch is the second iteration of the Scorecards. It is much bigger and more ambitious in scope than the last: it scores not the plans, but the climate actions of every local authority in the UK. 

FOI requests were just one part of the process. As well as giving CE UK access to WhatDoTheyKnow Projects, we developed a crowdsourcing tool for volunteers to use while marking across the 90+ datapoints collected for each council. 

How do you score action?

CE UK moved from scoring plans to scoring actions. That required new approaches to gathering the information. 

The questions CEUK used in the new Scorecards are the result of a long and thorough process of research and refinement. Building on their own research and expertise, they conducted one-on-one consultations with approximately 80 organisations and sector-specific experts. An advisory group of environmental and local government experts provided further discussion and refinement, to help build a list of questions that would practically be possible to answer, and that would reveal important information about the climate actions of councils. 

The aim was to identify areas where information was publicly accessible; but also where gaps existed, especially in operational matters that aren’t often made public. Additionally, CE UK wanted to investigate whether councils are truly implementing the actions outlined in their climate action plans, including aspects like lobbying for additional powers.

Making use of Freedom of Information

Freedom of Information laws means that a huge range of information held by public authorities (including local councils) can be requested by any person who asks. This provides a legal tool to create greater transparency where information is not being published proactively.

For CE UK, the potential of FOI for the Scorecards project was clear – but there were concerns. In consultations with council staff, there was pushback regarding the use of FOI requests due to the potential time and financial burden on council officers who work on climate – with some requests for a more informal survey approach to be used. But the drawback of that would be making good data dependent on goodwill everywhere. FOI requests provided a way to make sure the scorecards were not just effective for councils who engaged with the process and provide an approach that was fair across the country. 

To balance a process where they want to encourage positive engagement from councils, with one that works without that, CE UK’s approach was to plan out the most efficient and least burdensome use of FOI requests. 

Based on feedback from the advisory group, and trial runs to a small number of councils, they eliminated questions that were less important and useful, made more ‘yes/no’ or ‘single number’ responses, and learned where certain questions weren’t relevant to certain areas or groups of councils. 

The subsequent FOI requests became more streamlined, and this resulted in quicker response times for the final requests than they had in the trial – as the information sought was more direct and concise.

In the end, CE UK submitted a total of over 4,000 FOI requests to councils across the UK. The questions were divided into 11 categories, with some being specific to certain types of councils, such as district councils or combined authorities. The next stage was taking these 4,000 requests and getting them into a form that can be used for the scorecards. 

Crowdsourcing and review process

CE UK used WhatDoTheyKnow to manage their FOI request process. mySociety’s WhatDoTheyKnow acts as a public archive for requests – requests made through the site have the responses shown in public to bring more information into the open  – making it more discoverable by other people interested in the information, and reducing the need for duplicate requests being made. As of 2023, a million requests for information have been made through the site, with hundreds of thousands of pieces of information being released. 

A feature we are trialling with a range of organisations is WhatDoTheyKnow Projects, which integrates crowdsourcing tools into WhatDoTheyKnow, and allows the task of extracting information into a new dataset to be spread out. The goal is that this helps organisations be more ambitious in finding out information and helps people work together to create genuinely new and exciting datasets, that no single organisation has ever seen. 

As CE UK’s approach already made heavy use of volunteers and crowdsourcing, this was a natural fit.  Alongside a wider group of 200 volunteers working on getting answers to the other questions, 15 volunteers specifically worked on the FOI requests. These volunteers were a mixture of people with prior experience or professional interest in FOI requests, campaigners well-versed in FOI processes, and individuals new to the concept but eager to engage in activism.

After the crowdsourcing of FOI data was complete, it joined the rest of the data in the new tool mySociety had developed for helping volunteers crowdsource information for the Scorecards.  

From here, councils were given access to the data collected about them and given a right of reply to correct any inaccuracies or point towards information not previously discovered or disclosed. The results of this process will then be reviewed to produce the final Scorecards data, which will be launched this month.

But the Scorecards data will not be the only useful thing that will come out of this process. Because of how WhatDoTheyKnow was used, to see evidence supporting the final Scorecards, people will be able to click through and see the original responses, for instance, to see what councils have lobbied on support for their climate work. 

Some of the FOIs are being used to construct datasets that have a broader impact, and here we come back to that FT story on the Energy Performance Certificate (EPC) ratings of council-owned houses. Building these new public datasets will be useful for councils to understand their own situation, and as we see with the news story, more broadly to understand the challenges ahead for local governments to meet net zero emissions goals. 

Onwards

The original Scorecards project has already been influential on how local governments understand their own plans, and how organisations like the UK’s Climate Change Committee understand the role and progress of local government in the challenges ahead. When the next generation of Scorecards is released, we hope that they continue to be useful in shaping and improving local government action around climate change.

mySociety believes that digital technology can be used to help people participate more fully in democracy, make governments and societies more transparent, and bring communities together to address societal challenges.

The Scorecards project showcases how the combination of digital tools, people power, and the right to information produces powerful results. We hope that the impact of this project can inspire and make possible similar approaches for other problems, or in other countries.

Following the PSNI and other recent data breaches, the ICO has issued guidance to public authorities. This guidance suggests a temporary stop on publishing Excel-style spreadsheets in response to FOI requests made via online platforms like WhatDoTheyKnow. The full advisory note is available online. 

The advisory note emphasises that this is not a reason not to disclose requested information. Instead, the ICO says to release the information from original source spreadsheets as a CSV file – a simpler format than Excel Workbooks, with less potential for including hidden sheets or metadata that can lead to an accidental breach.

–

A focus on file formats is a blunt measure, and one that will need to be superseded by better procedures and technical processes.

We support authorities releasing data in the most appropriate format for the information being requested. This may sometimes mean an extract from a table, and sometimes a complete document. Excel spreadsheets are legitimate public documents, and information released in this format can be hugely valuable. It’s important to develop processes where they can be released safely. 

Significant data breaches involving Excel files clearly show the risks when data management and release processes fail. These include not just breaches we see through WhatDoTheyKnow, but through disclosure logs and releases made directly to requesters. This is an opportunity for public authorities, the ICO and us at WhatDoTheyKnow to reflect on how we can best deliver the huge benefits of public transparency while safeguarding personal data. 

–

Modern authorities need to be good at handling data. Data breaches happen at the intersection of technical and human processes. The FOI team can be the last link in the chain of a data breach when they release the information, but the root cause often goes back to wider organisational issues with the handling of sensitive data.

In the short run, the ICO has recommended training for staff involved with disclosing data. Many teams already have excellent processes and do excellent work, but all authorities should take this opportunity to consider their responsibility on the data they hold, and have appropriate processes in place.

Long term progress means developing good universal processes that keep data safe, regardless of the format of the data or how the data is released. All FOI releases should in principle be treated as if they are being released to the public, because the authority’s ability to stop a data breach ends when the information is released. Making FOI responses public produces huge efficiencies for the public sector, increasing transparency in practice, and multiplying the benefit to society of the information released. 

Technology can also be part of the solution – we need to understand more about why existing technical ways of removing hidden information from Excel spreadsheets are not being used (as described in the ICO’s established guidance on disclosing information safely), and how new tools or guidance can make it easier to release data safely. 

–

A core part of our work at WhatDoTheyKnow is dealing with the practical reality of promoting public transparency while protecting personal information. We take data breaches seriously and have processes in place for dealing with them as promptly as possible. We continue to plan and work to help reduce the occurrences and impact of personal data breaches through both our procedures and technical approach. 

By monitoring how authorities respond to requests on WhatDoTheyKnow, we will seek to understand how this guidance is working in practice, and engage with the ICO and other organisations to promote effective long term approaches to this problem. 

---

Notes on the content of the advisory

Below is our understanding of the advisory note by subject matter:

Freedom of Information requests

• Continue to comply with FOI responsibilities. This guidance is about releasing information in a way that reduces risk of accidental disclosure. 
• Temporarily, do not release original source spreadsheets to online platforms like WhatDoTheyKnow. Instead – convert and release to CSV files.
• If that is not possible, then:
  • Ask if the Excel sheet can be sent to a separate (non-public) address. Proceed with the original address if they ask for this. 
  • In all releases, go through processes to ensure there is no data breach in the material. 

General data management

• Excel files are unsuitable working environments when they become very large (hundreds of thousands of rows). Authorities need to switch to appropriate data management systems that are more appropriate for managing larger amounts of data.  
• Staff who use data software and are involved in disclosing information need continuous training.  
• Understanding of pivot tables and their risks should be incorporated into data management.

The ICO plans to update their guidance on Disclosing Information Safely. 

The checklist released accompanying the advisory has several useful steps on checking for hidden data in Excel sheets. However, on the ‘considered alternative ways to disclose’ step, refer back to the steps in the advisory note. Information converted to CSV can be released to WhatDoTheyKnow in compliance with the advisory note. The advisory note says that the source dataset should continue to be released to WhatDoTheyKnow if it cannot be converted, the requester does not want to use an alternative route, and the authority is confident it does not contain a data breach.

On the afternoon of Tuesday 8 August 2023, we ​​were contacted by the Police Service of Northern Ireland (PSNI) with a request to remove a response that they had made to a Freedom of Information request submitted through WhatDoTheyKnow.

As many news stories have since reported, the response contained personal information pertaining to a large number of Police Officers and civilian staff. In line with our policies around serious data breaches, we took rapid action to hide the material from public view and subsequently deleted it. We also submitted cache removal requests to ensure that any copies of the information would be removed from search engines as soon as possible.

The overwhelming majority of requests and responses that we process through WhatDoTheyKnow are unproblematic. However, we take the responsibilities that come with operating a large platform extremely seriously, especially around the personal data breaches that can occur when authorities’ release processes fail. 

We recognise the significant impact that serious data breaches like this one have on the people affected and on their families, and we are assessing whether there is more we can do to help authorities avoid making such breaches in future.

—

Image: Jason Leung

By law, a public authority usually has to respond to a request made under the Freedom of Information Act 2000 promptly, and within 20 working days. However, there are instances when this deadline may be extended. 

This usually happens when the authority needs to conduct a public interest test to determine whether it’s in the public interest to apply one of the many exemptions in the Act, or whether they should release the information.

We’ve observed a recurring trend where the Foreign, Commonwealth and Development Office (FCDO) has been frequently extending the response deadline to consider the public interest. This has resulted in significant delays in the provision of responses.

We’ve tracked 35 requests where the FCDO has taken over 100 days to respond. Here are four of the most notable delays that we’ve come across:

• This request for briefing notes prepared for then Prime Minister David Cameron’s 2013 trip to China is still unanswered after more than 430 working days and 21 Public Interest extensions.
• A request concerning UK-AIS commercial deals took 388 working days to answer.
• In another case, 274 working days after the requester asked for the information, the department sent them a letter giving them just seven days to respond if they still wanted their request to be processed. 
• A request about various projects in Turkey took 239 working days to be answered.

While the Information Commissioner’s Office (ICO) suggests that the process of considering the public interest shouldn’t typically take more than an additional 20 working days, the law itself doesn’t set a time limit, which restricts the options available to requesters to challenge this practice.The situation is better if you have asked for information from a Scottish public authority, as these requests are handled under different legislation. The Freedom of Information (Scotland) Act does not allow for additional time to be taken when assessing the public interest in releasing information, which means that Scottish public authorities must do so within the original 20-working-day timeframe. The Independent Commission on Freedom of Information that the Public Interest Test extension be abolished, and replaced with a time limited extension that covered requests that were complex to handle, a call which we echoed.  

If you find that an authority has sought to extend the response deadline for one of your requests multiple times, we would strongly recommend that you lodge a complaint with the ICO. This is quick and easy to do using their online complaint form. They will typically write to the authority, asking them to respond to your request within 10 working days. You can find examples on WhatDoTheyKnow where requesters have successfully done this. Although the ICO is continuing to work through a large casework backlog, our experience is that they usually handle complaints of this nature promptly.

We are continuing to collect examples of delays caused by authorities conducting public interest tests. If you know of any that we’ve missed, please contact us and let us know, especially if you have personal experiences to share.

—

Image: Lucian Alexe