Following the PSNI and other recent data breaches, the ICO has issued guidance to public authorities. This guidance suggests a temporary stop on publishing Excel-style spreadsheets in response to FOI requests made via online platforms like WhatDoTheyKnow. The full advisory note is available online.
The advisory note emphasises that this is not a reason not to disclose requested information. Instead, the ICO says to release the information from original source spreadsheets as a CSV file – a simpler format than Excel Workbooks, with less potential for including hidden sheets or metadata that can lead to an accidental breach.
A focus on file formats is a blunt measure, and one that will need to be superseded by better procedures and technical processes.
We support authorities releasing data in the most appropriate format for the information being requested. This may sometimes mean an extract from a table, and sometimes a complete document. Excel spreadsheets are legitimate public documents, and information released in this format can be hugely valuable. It’s important to develop processes where they can be released safely.
Significant data breaches involving Excel files clearly show the risks when data management and release processes fail. These include not just breaches we see through WhatDoTheyKnow, but through disclosure logs and releases made directly to requesters. This is an opportunity for public authorities, the ICO and us at WhatDoTheyKnow to reflect on how we can best deliver the huge benefits of public transparency while safeguarding personal data.
Modern authorities need to be good at handling data. Data breaches happen at the intersection of technical and human processes. The FOI team can be the last link in the chain of a data breach when they release the information, but the root cause often goes back to wider organisational issues with the handling of sensitive data.
In the short run, the ICO has recommended training for staff involved with disclosing data. Many teams already have excellent processes and do excellent work, but all authorities should take this opportunity to consider their responsibility on the data they hold, and have appropriate processes in place.
Long term progress means developing good universal processes that keep data safe, regardless of the format of the data or how the data is released. All FOI releases should in principle be treated as if they are being released to the public, because the authority’s ability to stop a data breach ends when the information is released. Making FOI responses public produces huge efficiencies for the public sector, increasing transparency in practice, and multiplying the benefit to society of the information released.
Technology can also be part of the solution – we need to understand more about why existing technical ways of removing hidden information from Excel spreadsheets are not being used (as described in the ICO’s established guidance on disclosing information safely), and how new tools or guidance can make it easier to release data safely.
A core part of our work at WhatDoTheyKnow is dealing with the practical reality of promoting public transparency while protecting personal information. We take data breaches seriously and have processes in place for dealing with them as promptly as possible. We continue to plan and work to help reduce the occurrences and impact of personal data breaches through both our procedures and technical approach.
By monitoring how authorities respond to requests on WhatDoTheyKnow, we will seek to understand how this guidance is working in practice, and engage with the ICO and other organisations to promote effective long term approaches to this problem.
Notes on the content of the advisory
Below is our understanding of the advisory note by subject matter:
Freedom of Information requests
- Continue to comply with FOI responsibilities. This guidance is about releasing information in a way that reduces risk of accidental disclosure.
- Temporarily, do not release original source spreadsheets to online platforms like WhatDoTheyKnow. Instead – convert and release to CSV files.
- If that is not possible, then:
- Ask if the Excel sheet can be sent to a separate (non-public) address. Proceed with the original address if they ask for this.
- In all releases, go through processes to ensure there is no data breach in the material.
General data management
- Excel files are unsuitable working environments when they become very large (hundreds of thousands of rows). Authorities need to switch to appropriate data management systems that are more appropriate for managing larger amounts of data.
- Staff who use data software and are involved in disclosing information need continuous training.
- Understanding of pivot tables and their risks should be incorporated into data management.
The ICO plans to update their guidance on Disclosing Information Safely.
The checklist released accompanying the advisory has several useful steps on checking for hidden data in Excel sheets. However, on the ‘considered alternative ways to disclose’ step, refer back to the steps in the advisory note. Information converted to CSV can be released to WhatDoTheyKnow in compliance with the advisory note. The advisory note says that the source dataset should continue to be released to WhatDoTheyKnow if it cannot be converted, the requester does not want to use an alternative route, and the authority is confident it does not contain a data breach.
On the afternoon of Tuesday 8 August 2023, we were contacted by the Police Service of Northern Ireland (PSNI) with a request to remove a response that they had made to a Freedom of Information request submitted through WhatDoTheyKnow.
As many news stories have since reported, the response contained personal information pertaining to a large number of Police Officers and civilian staff. In line with our policies around serious data breaches, we took rapid action to hide the material from public view and subsequently deleted it. We also submitted cache removal requests to ensure that any copies of the information would be removed from search engines as soon as possible.
The overwhelming majority of requests and responses that we process through WhatDoTheyKnow are unproblematic. However, we take the responsibilities that come with operating a large platform extremely seriously, especially around the personal data breaches that can occur when authorities’ release processes fail.
We recognise the significant impact that serious data breaches like this one have on the people affected and on their families, and we are assessing whether there is more we can do to help authorities avoid making such breaches in future.
Image: Jason Leung
By law, a public authority usually has to respond to a request made under the Freedom of Information Act 2000 promptly, and within 20 working days. However, there are instances when this deadline may be extended.
This usually happens when the authority needs to conduct a public interest test to determine whether it’s in the public interest to apply one of the many exemptions in the Act, or whether they should release the information.
We’ve observed a recurring trend where the Foreign, Commonwealth and Development Office (FCDO) has been frequently extending the response deadline to consider the public interest. This has resulted in significant delays in the provision of responses.
We’ve tracked 35 requests where the FCDO has taken over 100 days to respond. Here are four of the most notable delays that we’ve come across:
- This request for briefing notes prepared for then Prime Minister David Cameron’s 2013 trip to China is still unanswered after more than 430 working days and 21 Public Interest extensions.
- A request concerning UK-AIS commercial deals took 388 working days to answer.
- In another case, 274 working days after the requester asked for the information, the department sent them a letter giving them just seven days to respond if they still wanted their request to be processed.
- A request about various projects in Turkey took 239 working days to be answered.
While the Information Commissioner’s Office (ICO) suggests that the process of considering the public interest shouldn’t typically take more than an additional 20 working days, the law itself doesn’t set a time limit, which restricts the options available to requesters to challenge this practice.The situation is better if you have asked for information from a Scottish public authority, as these requests are handled under different legislation. The Freedom of Information (Scotland) Act does not allow for additional time to be taken when assessing the public interest in releasing information, which means that Scottish public authorities must do so within the original 20-working-day timeframe. The Independent Commission on Freedom of Information that the Public Interest Test extension be abolished, and replaced with a time limited extension that covered requests that were complex to handle, a call which we echoed.
If you find that an authority has sought to extend the response deadline for one of your requests multiple times, we would strongly recommend that you lodge a complaint with the ICO. This is quick and easy to do using their online complaint form. They will typically write to the authority, asking them to respond to your request within 10 working days. You can find examples on WhatDoTheyKnow where requesters have successfully done this. Although the ICO is continuing to work through a large casework backlog, our experience is that they usually handle complaints of this nature promptly.
We are continuing to collect examples of delays caused by authorities conducting public interest tests. If you know of any that we’ve missed, please contact us and let us know, especially if you have personal experiences to share.
Image: Lucian Alexe
Several Olympics team members have trained and competed on the athletics track in Tooting Bec, London, and it’s a valued facility for local schools and athletics clubs alike. But by 2021, runners were finding that the surface had degraded so much that it was becoming dangerously hard to run on and training in spikes was no longer possible. After rainfall, the uneven surface would be covered in puddles: no longer was it fit for serious training or holding athletics competitions.
In fact, as campaigners pointed out, the track hadn’t been resurfaced for 36 years. Mandy Brown was one of those who campaigned for its refurbishment, and told us how Freedom of Information requests made through WhatDoTheyKnow helped provide the evidence needed to persuade Wandsworth Council to make an upgrade.
As she explained, though, the battle wasn’t as simple as ‘one FOI request and done’. Putting in a request for what they’d thought of as a potential killer stat bore little fruit in the end — but as is so often the case, FOI supported the campaign in other, unforeseen ways.
What was that killer stat?
“Well, we wanted to know spend on the nearby Battersea Park track, which is in a more affluent ward, so we could compare it to spend on Tooting Bec track where the local population is less affluent and more diverse,” says Mandy, explaining the FOI request they submitted.
The thinking was that they might be able to use any disparity in funding to point out the lack of fairness in funding decisions — but the truth turned out to be more complex. There might not have been money for the Tooting track, but there had been funding for a gym in the borough; and England Athletics had been joint funders on the Battersea refurbishment, so it’s not as if the council had shelled out the full amount.
It’s never a waste of time asking for information, though: bringing the facts into the open means that public debate can be based on truth and not assumptions. Plus, the mere fact that citizens have an eye on council expenditure can have an effect, as Mandy says:
“I do think repeated requests for information on the basis of the rejection of the initial application for funding played their part,” says Mandy, who submitted requests such as this one for minutes of the meeting where funding was discussed.
“Now, I believe that what really made the difference in the end was the PR campaign. This led to coverage in both local and national press. But in fact, that was boosted by information from someone else’s FOI request!
“We spotted from another request on WhatDoTheyKnow that pre-covid, more than 80,000 people used the track every year. This turned out to be an incredibly useful statistic to include in our press releases and interviews, and was picked up in every piece of coverage including Wandsworth Council’s own announcement when they finally agreed to fund the resurfacing.
“So when all’s said and done, WhatDoTheyKnow played an important part in the overall campaign due to this publicly-available stat on track usage, and the general pressure applied by asking questions on the spend and decision-making process.”
Thanks to Mandy for sharing this story: we believe that Freedom of Information is always useful in campaigning, for the reasons she mentions, and because public debate always benefits from an injection of factual information.
And you never know: that FOI request you make today could be uncovering information that will inform someone else’s campaign in the future — a neat demonstration of why WhatDoTheyKnow publishes requests and responses in its permanent public archive.
Paul Bradshaw’s name is well known to those working around data and journalism in the UK. He has authored and contributed to several books on the topic, leads an MA in Data Journalism at Birmingham City University, and acts as a consultant in BBC England’s data unit.
In mySociety’s twentieth anniversary year, we’re looking to see where we’ve had impact, and in a recent conversation with Paul we were pleased when he noted that WhatDoTheyKnow was a something of a catalyst to his work around Freedom of Information for journalism.
In 2009, Paul secured funding from Channel 4 and Screen West Midlands to set up Help Me Investigate, a platform for collaborative journalism. As it happens, that year the same source of funding supported our time-mapping service Mapumental, and Will Perrin’s hyperlocal blog project Talk About Local. The three projects were often covered in the press as harbingers of a new, digital way of doing things.
The basic principle of Paul’s platform was that the internet permits collaboration between many people, each of whom can contribute a small piece towards the labour-intensive work of investigative journalism. It’s an approach we are all very familiar with these days: it is, of course, what we now call crowdsourcing — something mySociety has made use of in many of its own projects through the years, including our own WhatDoTheyKnow Projects.
A user, curious to get to the bottom of something, would share a central question and list out the tasks that needed to be completed in order to answer it. And of course, as often as not, some of these tasks would be the placing of FOI requests through our site, WhatDoTheyKnow.
“When I launched Help Me Investigate”, says Paul, “WhatDoTheyKnow was a major tool in our toolkit, allowing us to easily share FOI requests that others could clone or learn from.”
Even more than that, he reckons WhatDoTheyKnow was “probably responsible both for me getting started with FOI, and for teaching others to use the FOI Act.”
Since WhatDoTheyKnow’s beginnings, the aim has been to make FOI more accessible to everyone, so this was great to hear. We know that it’s a big leap to become ‘a person who submits FOI requests’, so what does that look like in practice?
“Firstly”, says Paul, “the site reduced the barriers considerably when it came to making an FOI request: knowing where to send that request is a big mental barrier when you lack confidence navigating faceless organisations; and having examples to look at also makes a big difference in being able to imagine what one looks like.”
Once someone has become adept with the Act, we can’t ask for much more than that they pass that knowledge onto others, creating a cascading effect of individuals who understand their rights and how to use them to uncover information. Paul is an example of exactly that:
“It made it possible for me to share that knowledge with others. I’ve used it with hundreds of journalism students to introduce them to FOI: ‘copy this request, find your organisation, paste, and send’ helps get them started, and empowers students who might be otherwise feeling disempowered.”
As proof of impact goes, ‘hundreds of journalism students learning how to use FOI’ certainly seems like a good one — it means that WhatDoTheyKnow has indirectly brought countless FOI-based stories to the public.
Paul listed some of the FOI-based investigations undertaken by users of his site — now no longer live, but visible through the Internet Archive. These include the uncovering of a £2.2 million overspend on Birmingham City Council’s website; police claims of sabotage against Climate Camp protesters; and the varying availability of hormonal contraceptives across different postcodes.
It’s been fascinating to explore Help Me Investigate‘s archived pages, and a real reminder of what people can do when they come together. We are glad that WhatDoTheyKnow has played such a key part in that, and in the training of so many future journalists.
Image: Ashkan Forouzani
Here at WhatDoTheyKnow, mySociety’s service which publishes Freedom of Information requests and responses online, we care a lot about how transparent public authorities are. That being so, it’s only right that we try to be transparent, too. Of course, we already strive to be as transparent as we can, but we’ve recently increased how much information we provide about requests that we’ve had to stop publishing.
It’s really rare that we stop publishing the entirety of a request, but since starting to produce annual transparency reports, we’ve noticed that in those exceptional cases, we weren’t always able to be clear about why we had to do it. Our records were kept in our support mail system, and couldn’t be published on WhatDoTheyKnow due to some technical limitations.
Since October 2022, if you visit a link to a request that has been removed, where the information exists, you’ll now see a brief message explaining why the request isn’t available. We were already doing something similar for instances where we’ve had to hide individual messages or attachments. While the information necessary to display these more detailed reasons isn’t always available, especially for older requests, we hope that this small improvement is yet another beneficial improvement to how we run the site.
Alaveteli, the software platform that powers WhatDoTheyKnow and many other Freedom of Information services globally, doesn’t just let us choose between something being visible or not;we can choose between several levels of visibility. These range from completely publicly visible, to visible only to those with a direct link, to visible only to the user who originally made the request, to visible only to the WhatDoTheyKnow administration team, which we only use in some of the most serious cases.
This feature allows us to minimise the potential negative impact of publication, while retaining as many of the benefits as possible, with varying levels depending on the results of our carefully considered, case-by-case assessments.
We recently ran a survey, asking the public to tell us how they’ve used our Freedom of Information site, WhatDoTheyKnow.
To get people thinking, we put a number of prompts on social media: we asked journalists to tell us what stories they’ve uncovered through the site; and campaigners and activists to tell us how they’ve used FOI to underpin their efforts.
But WhatDoTheyKnow isn’t just useful for grand causes. Our final prompts (example below) asked: Did you use WhatDoTheyKnow just because you wanted to know something?
For all that FOI can be used to uncover vitally important information — information that changes hearts and minds, that even potentially brings down governments — there’s no doubt that it is also a useful tool for the person who simply starts wondering about something, and knows that the answer is held by our public authorities.
Such a person is Bristolian Steve Woods, who has been using WhatDoTheyKnow since its launch in 2008 — and continues to lodge a request whenever something piques his interest. From the cost of the Mayor of Bristol’s trip to a sustainable cities and towns conference in Geneva, to more details about an art piece named the Black Cloud Pavilion, and noble enquiries about the local council’s commitment to open standards, Steve has made 60 requests through WhatDoTheyKnow at the time of writing.
We asked him to explain more about some of the more unusual ones.
The rubber bands request
“How many elastic bands – in terms of either numbers or weight – does the Royal Mail procure and/or consume per year?” was the main information that Steve asked for in this request.
He explains, “As a child in the 1950s and ’60s, I was taught not to litter, so I got really frustrated at seeing the large number of rubber bands dropped in the street by posties as they undid the bundles of letters on their rounds.”
Royal Mail obliged with a full response, and the answer may astound you: you can see the exact figures here.
Turned out that Steve wasn’t the only one interested in this figure. “The FoI request is cited on Wikipedia“, says Steve, “and resulted in a piece in the Daily Telegraph. My late brother-in-law used to take the paper and my nieces were delighted to see their uncle’s name in it.”
The Benin Bronze request
Earlier this year, Steve asked the council for updates on progress with repatriating an artefact acquired by dubious means and added to the city museum’s collection in 1935.
“Some years ago, the local media reported that this object was to be repatriated to Nigeria”, begins Steve’s request, adding: “There have been no subsequent reports of its repatriation, so I am assuming this has still to happen.”
We asked Steve to tell us more. “Bristol City Museum & Art Gallery has a single bronze as looted from the Oba’s palace in 1897”, he says. “I learned that discussions about repatriation are still continuing.”
This request is a great example of how FOI can be used to obtain fuller background detail than is discoverable from newspaper coverage, especially once the media has moved on from a story. The response from the council provides these insights:
“Representatives from NCMM [National Commission for Museums and Monuments] were due to come to Bristol in April 2022 but in the end their busy itineraries couldn’t allow for that. They have become even busier in recent months due to the increased amount of global returns, and pledges to return, that have taken up their time. They have concentrated their efforts with larger institutions but we await direction from them on what they would like to happen next in the Bristol case.”
Meanwhile, Steve points out, “The museum has contributed to the Digital Benin project“.
The city services requests
Steve has also found FOI to be a useful tool for bringing improvements to his local area. “I have made various requests over the years about litter, fly-tipping and enforcement around environmental crimes.
“In conjunction with working with local ward councillors and council officers, these have yielded positive results in the form of more enforcement officers – despite overall council job cuts – plus regular meetings at both ward and citywide level involving council officers, its waste management company, councillors, voluntary sector organisations and other interested parties.”
In 2009, Steve asked what proportion of his council tax was accounted for by weekly food waste collections, explaining his motivation thus: “My brown food waste recycling box was not emptied for one month while [waste management firm] SITA’s workforce was recently on strike. Via my council tax, I paid Bristol City Council for this service, which was not provided.”
The obliging response broke the figures down and came to the conclusion that each payer of council tax contributed 5p per week towards food waste collection.
The Colston request
The toppling of the Edward Colston statue and subsequent re-examination of Colston’s legacy in the city is something that is of interest to many Bristolians, and at the time of writing Steve has just received a response to his request asking what progress has been made by the council in renaming streets named after the slave trader.
“We require that all property owners on a given street provide their consent for its name to be changed, as such a move incurs administrative costs for those individuals/businesses to change legal documents etc. We have no plans to change this position,” says the council.
“We would therefore need to see overwhelming support from property owners on a given road in order to consider contacting all of them to confirm their unanimous consent to begin the process of changing their street’s name. This has not happened for any of the roads that your request mentioned.”
Thanks to Steve for letting us know about his long and interesting use of FOI over the last decade and a half – we hope that he will continue to be curious about the events that unfolds around him, and will continue to use WhatDoTheyKnow to satisfy that curiosity.
Support us in making information available to allDonate now
When Joanne Bartley saw a bus go by with an advert for a local private primary school on its side, she knew there was something amiss.
“It just happens that I’d recently made a Freedom of Information request through WhatDoTheyKnow, to check 11-plus pass rates at various primary schools,” she explained. “I was researching whether wealthier schools had more success than schools in poorer communities.”
Joanne lives in Kent, which unlike many counties, still assesses children with the 11-plus test. The bus advert claimed that 94% of the school’s pupils passed it, giving them access to grammar school.
“I had never seen a pass rate as high as 94%, not even in prep schools which can be mostly about 11-plus tuition,” says Joanne.
“So I found the school’s true pass rate, which was much lower, and reported the school advert to the Advertising Standards agency. They upheld the complaint and the advert had to be withdrawn.”
What do small victories like this achieve? Joanne puts is succinctly:
“My FOI request meant that no more parents were conned into paying expensive school fees because of some dodgy statistic! Because most of the country got rid of grammar schools decades ago, the government doesn’t scrutinise the 11-plus test processes very much, and there’s little analysis of entry to grammar schools either.”
Joanne now works for a group called Comprehensive Future, which campaigns for fair school admission and an end to the 11-plus. FOI is a tool they frequently turn to.
“We use WhatDoTheyKnow an awful lot to look at social inequality in access to grammar schools. I use FOI to check how many disadvantaged pupils access the schools, to find out about appeals to these schools and more.”
The campaign’s findings frequently inform media in the education sector, national news stories, and even debates in Parliament.
“FOI requests helped us see the numbers of disadvantaged pupils in the schools that were given £64 million by the government to expand to admit more children from poorer backgrounds.
“And we had a story in the I newspaper, on the large percentages of privately educated pupils who attend grammar schools, based on this FOI request.
“We also looked at how few looked after children (children in care) attend grammar schools compared to other secondary schools and this was highlighted by a Conservative peer in a Lords debate.
“Baroness Berridge said: At the census date last year, 68 of our grammar schools had no looked-after children at key stages 3 or 4. That is a product of not giving priority admissions and selecting on the basis of the entrance test only. If I think back to my school and remove all those children, it would have been a poorer education.
“She must have come across the figure about looked-after children on your website as we hadn’t given it to anyone at the time. So you see WhatDoTheyKnow is a useful resource for everyone!
“The FOI Act means that there’s transparency around school entry, and this is a very healthy thing. WhatDoTheyKnow is also super useful to see what other people are asking about schools and see their results.”
We were glad to hear this: Joanne’s examples make it clear that WhatDoTheyKnow helps campaigns get their causes into the national conversation. It’s free, and available to everyone, and as you can see from the examples given, it can be a very powerful tool.
Joanne agrees: “I just wanted to show the power of your site to make a difference to campaigns like ours. Freedom of Information creates openness around a problem in education that is not much talked about.
“I love WhatDoTheyKnow – it’s made a real difference to our campaign.”
Like what we do for transparency?Donate now
Journalist Lucas Amin was one of the first to try out our tool for professional users of Freedom of Information, WhatDoTheyKnow Pro.
Back in 2017, when Lucas put an early version through its paces, his feedback – together with that of his associate Jenna Corderoy – helped us shape the service to be as useful as possible for investigative journalists.
His comments were positive, but how do we know Lucas really found WhatDoTheyKnow Pro useful? Six years on, he’s still using the tool to help discover and inform his wide-ranging FOI-based scoops.
Lucas says, “I have made FOI requests for more than ten years. During that time I’ve made a few cool spreadsheets to help me track requests. But none of them provided anything like the convenience and power of WhatDoTheyKnow Pro – it has been a total gamechanger.”
Lucas, working for OpenDemocracy, has recently been uncovering information around river pollution and how airlines’ lobbying has impeded the UK’s progress in cutting carbon emissions. These requests were made under the Environmental Information Regulations (a similar regime to FOI, but specifically for access to information about the environment – and also handled by WhatDoTheyKnow).
The exposés broken on the platform are frequently picked up by mainstream media. “Requests made via WhatDoTheyKnow Pro have made it into the Times, Guardian, Observer and Mirror this year alone”, says Lucas, sharing a selection of stories to underline this point.
In this Guardian story from March, we learn that airlines’ submissions to government contested whether vapour trails contribute to the climate impact of flights – in contravention to the views of experts in the field.
A second Guardian story that month also reveals how airlines lobbied for the cut in Air Passenger Duty on domestic flights, as brought in by Sunak in the spring budget. This story was also picked up by the Mirror.
It’s easy to see the link between the requests Lucas has made, and facts that must be exposed in order for us to have a fully-informed public debate. Without the right to request such documentation, the public would be entirely unaware of the type of lobbying going on behind Whitehall doors.
We’re very glad that WhatDoTheyKnow Pro has made it easier for this to happen, and very pleased that Lucas is such a strong advocate!
“If you use FOI, WhatDoTheyKnow Pro is the only way to go,” he says, before making us blush with more praise: “I have nothing but respect, gratitude and admiration for the smart, hardworking team at mySociety! Congrats on 20 years; here’s to 200 more.”
Thanks Lucas, the admiration goes both ways. Long may you continue to bring vital facts into the public arena.
Image: Paul Berry
Like what we do for transparency?Donate now
The civic tech organisation ForSet runs AskGov, using our Alaveteli platform, and you may remember that we had a valuable exchange of views and experiences with their cofounder Teona Tomashvili in London last year.
In Copenhagen, fellows of the the Alliance of Democracies’ Democracy Tech Entrepreneurship Program, of which Teona is one, were invited to ‘pitch’ their project in a Dragon’s Den-like set-up. Teona gave an excellent explanation of the website — which would apply equally to any FOI site running on our Alaveteli platform — and you can watch it for yourself in this video:
Along with the glory of winning came a very useful prize in the shape of a cheque for $10,000 to be put towards the project, as you can see in the image below. This was presented by Anders Fogh Rasmussen, who founded the Alliance of Democracies.
Massive congratulations to Teona, whose pitching skills and determination were key to AskGov’s success in these awards.