Journalist Lucas Amin was one of the first to try out our tool for professional users of Freedom of Information, WhatDoTheyKnow Pro. 

Back in 2017, when Lucas put an early version through its paces, his feedback – together with that of his associate Jenna Corderoy – helped us shape the service to be as useful as possible for investigative journalists.

His comments were positive, but how do we know Lucas really found WhatDoTheyKnow Pro useful? Six years on, he’s still using the tool to help discover and inform his wide-ranging FOI-based scoops.

Lucas says, “I have made FOI requests for more than ten years. During that time I’ve made a few cool spreadsheets to help me track requests. But none of them provided anything like the convenience and power of WhatDoTheyKnow Pro – it has been a total gamechanger.”  

Lucas, working for OpenDemocracy, has recently been uncovering information around river pollution and how airlines’ lobbying has impeded the UK’s progress in cutting carbon emissions. The exposés broken on the platform are frequently picked up by mainstream media.

“Requests made via WhatDoTheyKnow Pro have made it into the Times, Guardian, Observer and Mirror this year alone”, says Lucas, sharing a selection of stories to underline this point.

In April, Freedom of Information requests revealed how water companies lobbied against their responsibilities to clean up rivers. The story broke on OpenDemocracy and was picked up by the Times. 

In this Guardian story from March, we learn that airlines’ submissions to government contested whether vapour trails contribute to the climate impact of flights – in contravention to the views of experts in the field.  

A second Guardian story that month also reveals how airlines lobbied for the cut in Air Passenger Duty on domestic flights, as brought in by Sunak in the spring budget. This story was also picked up by the Mirror.

It’s easy to see the link between the FOI requests Lucas has made, and facts that must be exposed in order for us to have a fully-informed public debate. Without the right to request such documentation, the public would be entirely unaware of the type of lobbying going on behind Whitehall doors.

We’re very glad that WhatDoTheyKnow Pro has made it easier for this to happen, and very pleased that Lucas is such a strong advocate!

“If you use FOI, WhatDoTheyKnow Pro is the only way to go,” he says, before making us blush with more praise: “I have nothing but respect, gratitude and admiration for the smart, hardworking team at mySociety! Congrats on 20 years; here’s to 200 more.”

Thanks Lucas, the admiration goes both ways. Long may you continue to bring vital facts into the public arena.

—

Image: Paul Berry

Great news from the Copenhagen Democracy Summit this week, where AskGov, the Access to Information website for Georgia, was recognised with an award for Democracy Tech.

The civic tech organisation ForSet runs AskGov, using our Alaveteli platform, and you may remember that we had a valuable exchange of views and experiences with their cofounder Teona Tomashvili in London last year.

In Copenhagen, fellows of the the Alliance of Democracies’ Democracy Tech Entrepreneurship Program, of which Teona is one, were invited to ‘pitch’ their project in a Dragon’s Den-like set-up. Teona gave an excellent explanation of the website — which would apply equally to any FOI site running on our Alaveteli platform — and you can watch it for yourself in this video:

Along with the glory of winning came a very useful prize in the shape of a cheque for $10,000 to be put towards the project, as you can see in the image below. This was presented by Anders Fogh Rasmussen, who founded the Alliance of Democracies.

Massive congratulations to Teona, whose pitching skills and determination were key to AskGov’s success in these awards.

Have you used our Freedom of Information website, WhatDoTheyKnow, for something noteworthy? Or do you have fond memories of it that you’d like to share? Please tell us all about it here.

Perhaps you’re:

• A journalist who used FOI requests sent via the site (or the Pro service) to inform news stories
• A campaigner who changed hearts and minds with facts you uncovered through FOI
• An activist who requested information that was of use to yourself or a wider community
• A request-maker who used the site independently to discover information you wanted (or needed) to know
• A developer, volunteer or supporter who remembers the launch of the site or some other milestone in its history
• An organisation that’s used WhatDoTheyKnow’s capabilities within your own app or site

Or maybe none of these labels apply, but you’ve got something to tell us about your use, or memories, of WhatDoTheyKnow. See yesterday’s post for a look back on its 15 years so far!

Also, perhaps you have a vision for where we should go next: new features we could consider adding; partners we could work with; or ways in which we can advocate more effectively for transparency from our public authorities.

We’re all ears

It’s mySociety’s twentieth anniversary year, and like any organisation celebrating a significant milestone, we want to reflect on what’s been achieved. Just as importantly, we’re also taking the opportunity to set our course for the future.

Over the course of the year, we’ll be inviting your memories, thoughts, ideas and stories around everything we do — and we’re starting with WhatDoTheyKnow.

If you have something to share, please go ahead and fill in our form here; or if you want more detail of the sort of things we’re looking for, read on.

Looking back

WhatDoTheyKnow was launched in 2008, after a 2006 call-out to our supporters and followers to suggest what project mySociety should work on next. The chosen suggestion, originally conceived as the Freedom of Information Filer and Archive, was a response to the UK’s rights under its then-fairly-new Freedom of Information Act.

The vision has changed surprisingly little since then: as Tom Steinberg said at the time:

We think that the best way to build a top quality archive is to simultaneously build the best possible “File an FOI request” tool, and then publish both the requests and the responses made through it in the archive. From the private desire to easily file FOI requests we hope that we can generate the public benefit of an easy to use archive.

Were you following mySociety’s blog at the time of that call-out? Were you one of the first people to use WhatDoTheyKnow when it was launched? Have you been a volunteer at any time in the site’s history, and if so, has that affected any other areas of your life for the better?

Great uses of WhatDoTheyKnow

Over the years, we’ve heard about some truly gratifying and impactful uses of the service: we’ve written up many of these as case studies.

There’s the campaign to find out how much asbestos is in our schools, another to turn empty shops into premises for start-ups, and not to mention a dogged – and successful – attempt to get the West Ham stadium contract released.

More recently, you may have seen the account of one organisation’s effective campaign against surveillance cameras which seemingly send worrying data back to China.

We’re always hungry to hear stories like these, and even more so in this anniversary year. Please do fill in the form to let us know if you’ve changed the world, in big ways or small, by using WhatDoTheyKnow.

To the future

As part of our thinking around #Democracy2043, we’d love to hear which way you think transparency is headed in the UK, and what mySociety’s place is in that future.

This one requires a bit more head-scratching, but there are no wrong answers, and we’d love to hear thoughts about how we as an organisation need to adapt, or what wider society needs to put in place to allow for healthier, more transparent public institutions.

And so, if you have thoughts about any or all of these areas, please go and fill in our form now. Thanks!

2023 is a momentous year for us, marking not only mySociety’s 20th anniversary but also the 15th birthday of our flagship Freedom of Information (FOI) service, WhatDoTheyKnow.

For over a decade and a half, we’ve been working to empower journalists, activists and campaigners, researchers, and tens of thousands of curious citizens to access information from UK public authorities.

Fighting your corner

It’s hard to imagine what the UK’s FOI landscape would look like without WhatDoTheyKnow, but in the early days, we faced many important battles to establish the right to have requests responded to via our platform at all. 

And they’re not over: even today, we face fresh challenges, such as from public authorities who are putting barriers in the way of our users by refusing to answer valid requests unless these are submitted using a particular form. We are determined to continue to highlight poor practice and defend users’ information rights.

Half a million pieces of information

One of the advantages to using WhatDoTheyKnow is that it serves as a permanent archive of requests and responses. Any information that you get released using WhatDoTheyKnow is accessible to others to share and build on. From humble beginnings, there have now been over 500,000 requests that have resulted in the release of at least some information, turning this into a valuable resource.

Given the depth and breadth of the information on the site, it’s hard to pick a few examples to illustrate the impact of requests made through the service but here are some notable releases:

A 2013 request revealed the existence of the Home Office’s Interventions and Sanctions Directorate (ISD), which was responsible for overseeing the controversial hostile environment policy. Working with public and private sector partners, the ISD restricted access to benefits and services for irregular migrants, ensuring that sanctions were enforced. Four years later, the Windrush scandal exposed the devastating human consequences of this policy.

A request to the Independent Parliamentary Standards Authority (IPSA) uncovered a dispute with HMRC. This made front page headlines, after it was revealed that certain MPs had sought to utilise public funds to employ experts to complete their expense claims for them.

A 2021 request to the Science Museum revealed that the museum had signed a sponsorship agreement with Shell, where it gave an undertaking not to do or say anything that could damage the reputation of the oil company. The existence of this ‘gagging clause’ was reported by Channel 4 News and the Times among others.

Empowering others

Whilst it would be tempting to try to measure the platform’s success by the remarkable volume of information that has been released, or the myriad of news stories that have been written as a result, for me, WhatDoTheyKnow’s true strength lies in its ability to empower individuals. By simplifying and demystifying the requesting process, WhatDoTheyKnow has made it more accessible to individuals who might have otherwise never considered submitting a request for information.

The impact of WhatDoTheyKnow has stretched far beyond the United Kingdom. WhatDoTheyKnow gave rise to Alaveteli, the open-source FOI software that’s helping to open up governments across six continents. In addition to our core platform, we’ve also developed WhatDoTheyKnow Pro. Specifically tailored to journalists, this service enables users to keep their requests private while they work on their stories, before sharing the source data with the world.

None of this would have been possible without the dedication of our volunteer team, who have worked tirelessly behind the scenes to offer guidance and support to our users, as well as managing the day-to-day running of the site. We’re immensely grateful to them, and all of our donors and funders over the years, whose continued backing has ensured the ongoing success and growth of the service. 

We are excited about the next 15 years and we look forward to building on what we have already achieved to help more people to access more information more easily than ever before.

—

Image: Cottonbro Studio

Links

• Online version of this document
• Application form
• Potential collaborators form
• Join the research commissioning mailing list/Research pool
• Download PDF

In one sentence

mySociety is looking for an individual, organisation or joint team to explore requests made through WhatDoTheyKnow.com to understand how requests for environmental information are being used in practice, highlighting success stories in releasing information at the local level, problems requesters encountered, and suggestions on how mySociety could better make use of the environmental information requests. 

About mySociety

mySociety is the charity behind UK civic services like TheyWorkForYou, WhatDoTheyKnow, FixMyStreet, and WriteToThem. We build open, digital solutions to help repower democracy, in the UK and around the world.

mySociety’s climate programme is funded by Quadrature Climate Foundation and the National Lottery Community Fund.

About mySociety’s existing work in this area 

mySociety’s Climate Programme is exploring how digital tools and approaches can work to reduce the third of UK emissions that local authorities have influence over. A key focus of this work is improving the quality and quantity of information that exists around local climate action. We make local authority plans more accessible and visible, campaign for better official data, and support pooling and crowdsourcing of data to improve knowledge and accountability for climate action. You can read more about mySociety’s Climate programme here.

WhatDoTheyKnow.com is a website built by mySociety, and administered by mySociety in partnership with volunteers, that helps people make Freedom of Information (FOI) requests and publishes the results in public. Over 900,000 FOI requests have been made through this platform. We have published research looking at the scale of FOI in local government, and in improving FOI through the UK, and Europe. 

About this project

Our ultimate goal in our climate work is to decrease UK carbon emissions that are either directly controlled or influenced by UK local government. Our main approach is by improving the information environment so that a variety of local institutions and actors are better able to understand the current situation, share knowledge on their approaches, and take more informed action. We have done this by making local authorities own plans and documents more discoverable and searchable, supporting crowdsourcing of information about plans and actions, pooling information held by different groups of campaigners, and arguing for improved publishing of official information. 

Another method open to us is to use (or facilitate the use of) Freedom of Information laws to get more official data and information into the public domain. Freedom of Information laws give a wide range of public access to information held by authorities. This is useful in releasing information about authorities’ own plans and actions, but is also useful in releasing underlying information which local authorities, so that other organisations (including companies, charities, and other public sector bodies) can use to inform their own approach and actions. 

The Environmental Information Regulations (EIR) substantially overlap with Freedom of Information. EIR requests make it easier to ask for and receive information specifically related to the environment, and covers a wider range of organisations than the general Freedom of Information law. We believe there is the potential to use improved support and guidance for Environmental Information Requests to facilitate more information relevant to local climate action being released, and being made discoverable by local authority actors, civil society and communities. This might take the form of modifications to WhatDoTheyKnow, or light-weight specialist services that sit on top of WhatDoTheyKnow’s FOI archive and infrastructure (similar to FragDenStaat’s Climate Helpdesk). 

But before we can do that, we want to understand more about Environmental Information Requests. WhatDoTheyKnow’s archive gives us access to a huge number of requests made for environmental information, but we understand very little about the contents of these requests. 

Based on a quick text search, we have a maximum of 70,000 requests that in some way mention environmental information regulations. As part of our wider pool of 1 million FOI requests, there may be other requests with environmental interests that have not explicitly mentioned EIR. We are looking for a contractor to explore the large WhatDoTheyKnow dataset, and highlight themes, patterns, and individual examples that show how EIRs are being used, obstacles users are encountering, and the implications of this for mySociety’s future work in this area. 

The available budget for this work is up to £8000-10,000 (inclusive of VAT), and the project would need to be completed by the end of July. There is a small amount of flexibility on this end date – but we are looking to make decisions in August about best ways to follow up the project in September. 

What we want to know more about

The broad goal of this project is to refine our understanding of how EIR works in practice. We’ve broken this into six areas we’d like to understand more about. 

1. EIRs have fewer exemptions, and a higher threshold for withholding material on public interest grounds, than FOI and. In principle, relevant information should be easier to get through EIR, but it is unclear how much this is significant in practice. We want to know more about the kinds of information that are successfully being released through EIR.
2. EIRs apply to more types of organisation – and organisations that have an important local footprint may be accessible to FOI. We want to know about the kinds of institutions and authorities who are or could be releasing information under EIR that is relevant to our local emissions goals.
3. Authorities should consider which regime a request falls under – and default to the more permissive EIR regime when a request is for environmental information. We want to know if there are cases where information is being withheld under FOI but should have been considered under EIR.
4. EIR might be more poorly understood by both requesters and authorities – information may be being incorrectly withheld that should be released. We want to understand any patterns of refusal that would inform advice tools. 
5. At the moment we don’t have an automatic way of identifying EIR responses (when authorities have judged a request under these rules), but it should be possible to create classification rules that separate them from FOI requests. We want to refine this approach so we can more consistently identify and analyse EIR responses, so they can be in current or future local climate services.
6. The big way in which EIRs are less accessible is that authorities can charge fees without a minimum amount of time taken, which is the case for FOI requests. Generally we think this doesn’t happen much (the ICO’s guidance is not to charge for a reasonable request), but we don’t know. We want a clearer sense of how often authorities ask that requesters pay a fee for environmental information.

What we are looking for in and from a partner

Expertise / skill set

We think there are a few different approaches to this project and are open to a range of approaches. In general we anticipate two main kinds of applicants: one leaning more on technical ability, and the other on specialist knowledge of FOI and EIR, or environmental data more broadly. For either kind of applicant, we can provide basic support in the other skillset. The ideal candidate would cover all of these bases (where a subject matter expert makes the technical search process sharper) – and we can help facilitate partnership applications between technical and specialist partners who might otherwise submit separately. If you are interested in this route, please fill out this collaboration form and we can help put you in contact. 

The key thing an applicant needs to have is an approach to searching the large quantity of information on WhatDoTheyKnow. At a minimum this requires some technical skills and ability to navigate the site – but beyond that might be accomplished by large scale text analysis, or a sampling and search method. WhatDoTheyKnow’s search allows searching for specific search terms – but the search count is not always accurate for larger queries. 

For technical strategies, we can provide data exports to enable searches off-platform. But for subject matter experts, we can also provide an Excel sheet of links to URLs that have triggered particular keywords, and authorities. 

Something that is important to us is being able to point to individual examples – and so approaches based on aggregate analysis (which might be the better approach to identify the scale of fees being charged, for instance, or automatically analyse themes of requests) need to also be able to drill down to individual requests. 

With this in mind, the following is a list of skills we would expect the candidate to have a couple but not necessarily all of the following:

• Technical skills
  • Experience with search and processing large amounts of text delivered in a mostly unstructured way.
  • Experience with Natural Language Processing (NLP) or automated topic extraction. 
• Subject matter skills
  • Knowledge and experience of Freedom of Information and/or Environmental Information Requests
    • Especially practical understanding of refusal and appeal grounds in both FOI and EIR.
  • Less important, but understanding of the wider European context of Environmental Information Regulations might provide additional understanding of potential approaches mySociety could take in the UK. 

Alignment with values and aims

Our Repowering Democracy strategy puts a special emphasis on embedding equity and inclusion in our work practices and services, and our work aims in general to fulfil values of equity/justice, openness and collaboration. 

Applicants should consider if this presents any obstacles to a working relationship, and think about how these values should be reflected in the project plan, either in terms of subject matter to investigate, or research approach.

Working practices

mySociety works flexibly and remotely, and there is no requirement to work from or visit an office. Applicants can distribute their work as appropriate over the time available, but we would expect regular check-ins on progress to be arranged over that period. A shared Slack channel and a specific contact person will be used to help coordinate and quickly share questions and information between mySociety and the researcher. 

Successful applicants would be expected to abide by the mySociety Code of Conduct in mySociety communications channels and events. 

Outputs and deliverables

The purpose of this project is to inform mySociety’s future projects, especially forming the base of knowledge around the use of EIRs for a prototyping week.

We are open to the form of outputs – this may take the form of one large report, briefings around our individual question areas, proposed amendments to guidance, etc. For technical submissions, analysis code under an appropriate licence would form part of the output.

The outputs should also work as a general contribution to knowledge of the current use of EIRs, and we might either publish or edit down and publish these briefings/reports.

Q&A and contact details

The application timeline includes a Q&A event, to which you can sign up at the link at the top of this document. The Q&A session will include an element to help individual researchers coordinate to form a joint submission (applications are also welcome from individual researchers). Answers will be made available in a video on this page for applicants who cannot take part. Questions can be emailed to the contact address below. 

Please send any queries or questions to research-commissioning+eir2023@mysociety.org and mention which project it is in regard to. Questions in advance are preferred and will be prioritised in the session. 

Your application

Applications can be submitted by individuals, organisations, or joint teams of individuals/organisations. These should be sent to research-commissioning+eir2023@mysociety.org by the closing date. 

You should submit a short application, of up to 4 pages of A4. A template for the response can be downloaded at the link at the top of this page, and covers: 

• Who you are (whether an individual, organisation, or joint team).
• A description of your previous experience/previous work and why you want to take on this project.
  • To the extent that this is possible, this should be anonymous and not include names of the org or members of the team (to help with anonymous stages of the recruitment process)
• How you would approach and deliver this project – a short project plan with approximate timings. 
  • This could include discussion of appropriate outputs for the project, and balancing technical and subject matter requirements. 
• The total value (£) of your proposal (including VAT), and high-level breakdown of costs (perhaps an indication of days per person, any other expenses). This does not need to include production costs of the report. 
  • Given the cost of the project, we will not be giving a great deal of weight to budget plans so please keep this short and high-level – we can dig into further details during interviews, if necessary.
• A short description of the individuals or team who will do the work, including biographies

There is a separate equalities monitoring form to fill out, which is processed separately from the main application (there is a link to the form in the application form). This is for understanding the reach of our method of distributing the call for proposals. 

If you are interested in joining a ‘researcher pool’ mailing list that we will contact with details of future projects, please see the link at the top of this document. 

Application timeline

If there are changes during this timeline, the table on the website version of this form will be updated. 

Stage	Date	Description
Call for proposals published	7 March 2023
Q&A Webinar	21 March 2023	An open, online public event for interested bidders to learn more about the project and ask questions. This will be recorded and available afterwards. You can submit questions in advance to research-commissioning@mysociety.org. Questions in advance are preferred.
Questions answered	23 March 2023	Video of the webinar to be made available to all potential bidders, in addition to answers to any other questions submitted via email
Deadline for applications	31 March 2023 (end of day)
Initial decisions	7th April 2023	Applicants to be informed whether they have made it through to a short panel interview (and may be asked for a sample of existing work). Applicants not progressing past this stage to be offered written feedback
Interviews	w/c 10 April 2023	Format to be decided, but this will likely be a one-hour panel interview with several people involved in the climate programme, towards the end of the week (14th, 15th April)
Final decision	w/c 18 April 2023	Remaining applicants to be informed of the final decision. Applicants not progressing to be offered feedback
Project briefing/kick-off meeting	End of w/c 18 April 2023	To include a brief introduction to mySociety, discussion of any onboarding required and approach to project management, communication and catch-ups
Project deadline	End of July 2023	End of project (Time range of project is a little flexible – we want it to inform decisions in August about any follow up work in September)

What happens after the project 

We intend to publish the report or briefs you produce, credited to you, on the mySociety website, licensed under a Creative Commons licence (see recent publications on research.mysociety.org for details). We may make some light edits (beyond proofreading) before we publish. You will be free to make publicly available your own version should you wish to, and any other material based on the research you conducted. 

We will convene a short ‘lessons learned’ session for the contractor and mySociety to discuss how the project went – what went well and anything that could have been improved. We will also discuss any future work based on the delivered project (e.g. if you are an academic and might want to co-author an article) and our ongoing relationship. We would also like to arrange a presentation on the project to mySociety staff, and there may also be an opportunity to promote the work in a public event held by mySociety (budgeting for this would be separate to the project above). 

Terms and conditions

Interested parties must be UK-based individuals or organisations.

 

Questions and Answers

 

For data analysis, what format is the data available in?

We create a regular research export of the database in CSV format. This doesn’t include the last few months of data and removes requester names where possible (this is generally available on the website, but we are trying to future proof against information being too available if redactions are needed in future).

There is a a data table of requests made by authority, and then a dataset of the individual messages (with full text) for each request. 

We currently don’t make the downloaded files themselves available in bulk (but are accessible through the site) – but the bulk export does include any cached conversion of a word document or pdf attached.

We can produce reduced data sets limited to specific authorities or keywords. For instance, to those flagged as potentially EIR projects at the start.

WhatDoTheyKnow data sharing policy for bulk data

What is the time scale for the research?

Broadly we’re looking for work to be completed by the End of July to help us inform how we spend our time in September. Depending on the nature of outputs, there may be some flexibility around this.

The retained EU law will ‘sunset’ EIR at the end of the year, unless explicitly retained. How does this work relate to that?

Our current default assumption is “everything will be fine” and this project can proceed as if EIR will continue past 2023. In the event it is looking like everything is not fine, this research helps us understand more about the impact of EIRs for campaigning purposes.

How have you handled projects like this before and how have they worked?

We’ve previously commissioned two pieces of research: one was about how we should commission research, the other was the role of local government in climate change. In both case we worked with a sole researcher, with regular check-in meetings and a shared slack channel. That said, we’re open to group applications (and this project may be appropriate for that). Neither of those projects was particularly data heavy, but we have worked to provide external researchers with data before. 

What support will you give the project?

The main support and contact for the project will be Alex Parsons (Senior researcher) from aa research and data perspective. There is limited available of the WhatDoTheyKnow on a day to day basis team – but I can either answer questions or get answers to questions we need to know.

—

If you have any other questions about this project or the application, please email : research-commissioning+eir2023@mysociety.org

Scrutiny of government ministers’ meetings with lobbyists has been boosted by the Open Access UK project from Transparency International UK, which integrates with our Freedom of Information service WhatDoTheyKnow. 

The fact that meetings between ministers, and people or organisations outside of Government have taken place is published, as required by the Ministerial Code, but typically, few details are proactively released. 

The Open Access UK service collates published declarations of meetings, and provides special links to WhatDoTheyKnow where users can find a pre-written Freedom of Information request for each meeting, asking for:

• The agenda
• The list of attendees, including details of any organisations they represented
• Copies of briefing notes and papers prepared in advance
• Notes or minutes recording  what was discussed
• Any correspondence associated with the attendees, including messages sent to follow up on the meeting

The service covers almost 90,000 ministerial meetings which have taken place since 2012, and it’s being actively maintained, with 1,500 more meetings added last month.  

To identify meetings of interest, searches can be carried out for the name of an organisation (such as a company, charity, union, or lobby group), minister, or by subject matter/policy area. Anyone interested in details of what happened at a meeting can request such information in public via WhatDoTheyKnow with a couple of clicks. You can also follow requests on any meeting, whether or not you are the person who submitted them.

It appears that meetings listed don’t only include physical ones, but online events and phone calls. Some detailed Government guidance on what should, and should not, be included in ministers’ transparency disclosures has been released via WhatDoTheyKnow (though the Government initially refused the request); more up-to date material also appears to be available, including a “pandemic-related update” which specifically covers remote meetings. 

The service does not currently cover meetings with ministers’ special advisers or civil servants where ministers are not present themselves, but Transparency International UK are inviting contact from anyone who would like to fund expanding the scope of the service to cover such meetings.  

 Responses to requests

The service has been running for some time, so everyone can see examples of how requests made through it have been responded to. 

There is a wide range of responses: in some cases the information sought has been substantively released promptly, while in other cases the responses have been less forthcoming.

As one would expect, the names of junior officials attending meetings, and involved in correspondence, are typically redacted. Often though, details of the substance of the matters discussed are also withheld. Exemptions commonly cited include those applying to “formulation of government policy” and “commercial interests”. Those exemptions are not absolute, but are subject to a public interest test: material should be released if the public interest in releasing it outweighs the interest in keeping it secret. 

Public bodies are permitted to delay a response while they consider whether the public interest lies in disclosure or not. We are concerned about the impact that such delays have on the speed of responses: we have noted examples of such delays both in responses to these requests and elsewhere on our service.  

We encourage requesters to ask for internal reviews if they are unhappy with the response to a request. 25% of internal reviews to UK central government departments result in the release of additional material, so asking for reconsideration is often worthwhile. We also provide advice on referring responses to the Information Commissioner, who is empowered to make decisions on whether information should be released or not. 

Ten things we’ve spotted in responses

(more…)

Big Brother Watch is a UK organisation that campaigns to defend civil liberties and privacy. As such, they keep a close eye on the ‘surveillance state’: the degree to which authorities are monitoring citizens, and the ever-more sophisticated technology that enables them to do so.  

They have recently been investigating the use of Chinese-manufactured CCTV cameras in the UK, submitting Freedom of Information (FOI) requests via WhatDoTheyKnow Pro as one part of their research.

In doing so they’ve uncovered the full extent to which this technology has permeated our schools and colleges, local and central government and beyond. Linked with information about both the capabilities of the tech, and the ways in which these companies support the human rights violations of the Chinese state, Big Brother Watch’s recent report makes for disturbing reading.

The campaign has caught the attention of government, and driven a change in policy over their internal usage of cameras. Hearing of this positive outcome, we were keen to talk to Big Brother Watch and find out more about their use of FOI, and their Head of Research & Investigations Jake Hurfurt filled us in.

What can they see?

Big Brother Watch opposes the rise of surveillance in this country, pointing out that the UK is one of the most surveilled countries in the world with an estimated 6 million CCTV cameras across the nation. 

As a society we may have become more accustomed to CCTV in public areas, but most people aren’t aware of recent rapid advances in surveillance functionality — such as facial, gender and even race recognition. And until there is widespread understanding of exactly what these cameras can do, there can’t be informed consent. 

Because of its affordability, Hikvision is the most common provider of CCTV equipment in this country, with Dahua in second place. Both companies are known to supply surveillance equipment that has been used to target ethnic Uyghur minorities in the Chinese province of Xinjiang.

Finally, Hikvision software has known vulnerabilities that could be exploited by hackers — and in countries like the US and Italy, breaches have been noted where cameras were found to be ‘communicating with China’.

That’s the background: so what prompted Big Brother Watch’s investigation?

Taking a closer look

Jake says, “We’d become aware of Hikvision’s dominance of the UK market. We had also seen the US and Australia move to restrict Hikvision and Dahua, given their links to the atrocities in Xinjiang and their cybersecurity flaws, and were concerned of the risks this posed to privacy and civil liberties in the UK, so decided to dig deeper.”

And, given the methods by which other countries had brought about change in this area, they knew they’d need to be looking at public authorities. That meant that FOI, which gives everyone the right to ask public bodies for the information they hold, was an obvious choice: 

“From the start the project was designed as a public-sector focused investigation. We could see that successful restrictions on state-owned firms in other countries had originated in the public sector. It was clear from the beginning that we would ask questions using FOI.”

As Big Brother Watch’s report states, “Over 5 months, [we] submitted more than 4,500 Freedom of Information requests to a range of public bodies to establish where Hikvision and Dahua equipment is in use and what advanced capabilities this equipment has. Some subsequent focused requests were submitted to public bodies who confirmed they do use Hikvision cameras.

“The public bodies included all secondary schools and FE colleges in England, all UK universities, police forces, NHS Trusts, all Oxbridge Colleges, all central government departments, the House of Commons, the three devolved administrations and the Greater London Assembly.”

A survey of surveillance

Our WhatDoTheyKnow Pro service is designed for people making large scale investigative FOI requests: it makes it simpler to send FOI requests to specific types of authority in a single batch, and keeps all correspondence private until you are ready to go public.

Jake says, “We used WhatDoTheyKnow Pro to send an initial wave of requests to police forces, government departments and the largest local authorities in the country, which acted as a scoping exercise to see how widespread Hikvision & Dahua are from a selection of public bodies. 

“Batch requesting and WhatDoTheyKnow’s built-in reminders of when it’s time to chase up a response have been useful. It was also useful in sending requests to public bodies who are hard to contact, as the contact address database is amazing. 

“The other helpful feature was the ability to look across requests others have made in the past, to compare responses and see what kind of wording worked and which did not – it’s something I always do when conducting large FOI projects as I can learn from other’s experience in the kind of questions that are successful, or start by knowing more than I would have done otherwise.”

Getting results

You can read Big Brother Watch’s full report for all the details, but their investigation revealed that three out of five schools, 60% of hospitals, 31% of police forces, 53% of universities, and 73% of councils use Hikvision and Dahua CCTV cameras, representing, in total, 60.8% of our public bodies.

Once the full extent of their usage had been quantified, it was time to get the word out to the people capable of bringing about change — the UK’s elected representatives.

Jake says, “We sent the report to MPs and held an event in Parliament to complement the launch of the report, and we had a good reception.

“Around publication we also got dozens of Parliamentarians to back a pledge to ban Hikvision, and working alongside other NGOs we have been heavily involved in advocacy around a number of bills to push for this. Big Brother Watch does a lot of advocacy in Parliament and the report has supported these efforts.”

Subsequently, with Oliver Dowden describing them as “current and future possible security risks”, the Chinese cameras were banned from being installed in or on government buildings.

Still campaigning

The government may have cracked down on their own use of these cameras, but that still leaves thousands of institutions and private companies who may not even be aware of the capabilities or implications of the cameras they’ve installed.

Big Brother Watch are pressing for further action and you can find their campaign page here. If you are concerned, you can add your name to a petition to request that Parliament pass a law to ban this type of CCTV; and you can add sightings of the distinctive cameras to a collaborative Google map.

Thanks very much to Jake for filling us in on the details of how WhatDoTheyKnow played a part in Big Brother Watch’s wider investigation.

You might also be interested to read our 2019 post on Privacy International’s investigation into police use of facial recognition.

At our Freedom of Information service WhatDoTheyKnow, when faced with requests to remove material, we operate on the principle of removing the minimum amount possible.

Alaveteli, the codebase which underlies WhatDoTheyKnow and a number of other FOI sites around the world, gives moderators a range of options for removing content – with the ability to surgically remove text ranging from individual words and phrases, to individual messages, or even entire request threads. This is useful when we spot misuse of our service, for example.

What we’ve been lacking, up until now, was a way to apply these types of removals to attachments.

Back in the early days of WhatDoTheyKnow, attachments were less common, but now we see many more: there can often be several attachments to one individual message.

Over the last few years, there have been occasions where we’ve had to remove an entire message, which may contain several useful attachments, just because of a small issue with one of them.

We’d then go through an annoying manual process to download the publishable ones, upload them to our file server, and then annotate the request with the links – here’s an example.

Back in 2013, when the original suggestion for enabling finer grained control was raised, the site contained around 400,000 attachments. There are now more than 3,500,000! We don’t remove content often, but at this scale it’s inevitable that we need to intervene now and then.

After a little code cleanup we were able to make individual attachment removal a reality. This allows us much more control over how we balance preserving a historic archive of information released under Access to Information laws, and running the site responsibly and meeting our legal obligations under GDPR.

As an example, let’s imagine that the FOI officer replying to our request inadvertently makes a data breach when releasing some organisation charts in `organisation chart b.pdf`.

Previously we’d have had to have hidden the whole response. Now, we can go into the admin interface and inspect each individual attachment.

We can then set our usual “prominence” value – offering a few options from fully visible to completely hidden – and include a reason for why the content has been hidden. We always seek to run the site transparently and explain any actions taken.

On saving the form, you can see that only the problematic attachment has been removed, with the remainder of the response intact. This saves us considerable time when reviewing and handling material with potential data issues, and keeps as much information published as possible while we do so.

As an extra bonus, since the main body text of emails is also treated as an “attachment” in Alaveteli, we’re now able to hide potentially problematic material there without affecting the attachments we present.

We’ve already used this feature several times to republish material where we’d previously had to hide the entire message due to the technical limitations at the time.

—

Image: Kenny Eliason

Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

…or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

—

Image: Keila Hötzel

On 21 December, we added the National Centre for Circus Arts as a public authority on WhatDoTheyKnow, making it easy for anyone to send them a Freedom of Information request. We could have left it off the site and let our users find the email address themselves, but we didn’t want them to have to jump through hoops.⭕

Joking aside, this is a valid addition to the site — see below for more details about why this college is subject to the FOI Act — and one where we can imagine some interesting requests being submitted.

For example, one could ask for copies of the risk assessments used for the various circus skills taught to students — we imagine these would contain quite a bit of detail.

A procurement list might throw up some items you wouldn’t see from other education institutions, like juggling equipment, unicycles or clown shoes.

Prospective students who want to understand more about their chosen course’s application procedure and admission statistics often submit requests to UCAS, but applications for the National Centre for Circus Arts are handled directly (which makes sense: it would be hard to evidence circus skills on an UCAS form!).

Instead, anyone looking for more information about how the application process works could submit an FOI request directly to the college. 

In addition, there is a wide range of requests that can be sent to almost any public authority on matters such as the use of public money, minutes of meetings, policies, letters, emails and contracts. All of these might have uniquely circusy angles when applied to this particular institution!

Some decisions are in-tents

It’s not always easy to work out whether a body is subject to FOI. For some bodies it’s really simple: they’re listed by name in Schedule 1 of the Freedom of Information Act 2000 (or the Scottish equivalent). 

The National Centre for Circus Arts isn’t named here, but Schedule 1 does include references to definitions contained in other laws.  

In the course of recent work to find more bodies to add to WhatDoTheyKnow, our volunteers have read and interpreted three pieces of legislation and an official register to work out which higher education institutions are subject to FOI. 

The upshot of our research is that we consider that all bodies listed on the Office for Students register as “Approved (fee cap)” are subject to FOI. This includes the National Centre for Circus Arts (registry entry). Helpfully, the Centre acknowledges the fact that it is subject to FOI on its website which not all higher education institutions do – kudos for that.

UK higher education institutions walk a tightrope between the public and private sectors. They are subject to rules such as competition law that are normally relevant to private business, whilst at the same time being subject to laws that apply to the public sector such as the equality duty and FOI. It’s quite a lot to juggle. 🤹

Interestingly, the debate about the role of circuses in public life goes back to Roman times where the provision of food and entertainment to the poor were used to gain political power.

WhatDoTheyKnow lists a number of obscure public authorities on the site ranging from the Crown Estate Paving Commission to the Treasure Valuation Committee but the National Centre for Circus Arts is certainly one of the more unusual higher education institutions we list. There can’t be many places where people jump through fire for their graduation (page 13 of the 2023-24 prospectus).

In conclusion, if anyone is planning to run away during 2023 then please be advised that with this addition, WhatDoTheyKnow has both the sea and the circus covered. 

We wish all our users a happy and healthy 2023.

—

Image: Jonny Gios