2023 is a momentous year for us, marking not only mySociety’s 20th anniversary but also the 15th birthday of our flagship Freedom of Information (FOI) service, WhatDoTheyKnow.

For over a decade and a half, we’ve been working to empower journalists, activists and campaigners, researchers, and tens of thousands of curious citizens to access information from UK public authorities.

Fighting your corner

It’s hard to imagine what the UK’s FOI landscape would look like without WhatDoTheyKnow, but in the early days, we faced many important battles to establish the right to have requests responded to via our platform at all. 

And they’re not over: even today, we face fresh challenges, such as from public authorities who are putting barriers in the way of our users by refusing to answer valid requests unless these are submitted using a particular form. We are determined to continue to highlight poor practice and defend users’ information rights.

Half a million pieces of information

One of the advantages to using WhatDoTheyKnow is that it serves as a permanent archive of requests and responses. Any information that you get released using WhatDoTheyKnow is accessible to others to share and build on. From humble beginnings, there have now been over 500,000 requests that have resulted in the release of at least some information, turning this into a valuable resource.

Given the depth and breadth of the information on the site, it’s hard to pick a few examples to illustrate the impact of requests made through the service but here are some notable releases:

A 2013 request revealed the existence of the Home Office’s Interventions and Sanctions Directorate (ISD), which was responsible for overseeing the controversial hostile environment policy. Working with public and private sector partners, the ISD restricted access to benefits and services for irregular migrants, ensuring that sanctions were enforced. Four years later, the Windrush scandal exposed the devastating human consequences of this policy.

A request to the Independent Parliamentary Standards Authority (IPSA) uncovered a dispute with HMRC. This made front page headlines, after it was revealed that certain MPs had sought to utilise public funds to employ experts to complete their expense claims for them.

A 2021 request to the Science Museum revealed that the museum had signed a sponsorship agreement with Shell, where it gave an undertaking not to do or say anything that could damage the reputation of the oil company. The existence of this ‘gagging clause’ was reported by Channel 4 News and the Times among others.

Empowering others

Whilst it would be tempting to try to measure the platform’s success by the remarkable volume of information that has been released, or the myriad of news stories that have been written as a result, for me, WhatDoTheyKnow’s true strength lies in its ability to empower individuals. By simplifying and demystifying the requesting process, WhatDoTheyKnow has made it more accessible to individuals who might have otherwise never considered submitting a request for information.

The impact of WhatDoTheyKnow has stretched far beyond the United Kingdom. WhatDoTheyKnow gave rise to Alaveteli, the open-source FOI software that’s helping to open up governments across six continents. In addition to our core platform, we’ve also developed WhatDoTheyKnow Pro. Specifically tailored to journalists, this service enables users to keep their requests private while they work on their stories, before sharing the source data with the world.

None of this would have been possible without the dedication of our volunteer team, who have worked tirelessly behind the scenes to offer guidance and support to our users, as well as managing the day-to-day running of the site. We’re immensely grateful to them, and all of our donors and funders over the years, whose continued backing has ensured the ongoing success and growth of the service. 

We are excited about the next 15 years and we look forward to building on what we have already achieved to help more people to access more information more easily than ever before.

—

Image: Cottonbro Studio

Links

• Online version of this document
• Application form
• Potential collaborators form
• Join the research commissioning mailing list/Research pool
• Download PDF

In one sentence

mySociety is looking for an individual, organisation or joint team to explore requests made through WhatDoTheyKnow.com to understand how requests for environmental information are being used in practice, highlighting success stories in releasing information at the local level, problems requesters encountered, and suggestions on how mySociety could better make use of the environmental information requests. 

About mySociety

mySociety is the charity behind UK civic services like TheyWorkForYou, WhatDoTheyKnow, FixMyStreet, and WriteToThem. We build open, digital solutions to help repower democracy, in the UK and around the world.

mySociety’s climate programme is funded by Quadrature Climate Foundation and the National Lottery Community Fund.

About mySociety’s existing work in this area 

mySociety’s Climate Programme is exploring how digital tools and approaches can work to reduce the third of UK emissions that local authorities have influence over. A key focus of this work is improving the quality and quantity of information that exists around local climate action. We make local authority plans more accessible and visible, campaign for better official data, and support pooling and crowdsourcing of data to improve knowledge and accountability for climate action. You can read more about mySociety’s Climate programme here.

WhatDoTheyKnow.com is a website built by mySociety, and administered by mySociety in partnership with volunteers, that helps people make Freedom of Information (FOI) requests and publishes the results in public. Over 900,000 FOI requests have been made through this platform. We have published research looking at the scale of FOI in local government, and in improving FOI through the UK, and Europe. 

About this project

Our ultimate goal in our climate work is to decrease UK carbon emissions that are either directly controlled or influenced by UK local government. Our main approach is by improving the information environment so that a variety of local institutions and actors are better able to understand the current situation, share knowledge on their approaches, and take more informed action. We have done this by making local authorities own plans and documents more discoverable and searchable, supporting crowdsourcing of information about plans and actions, pooling information held by different groups of campaigners, and arguing for improved publishing of official information. 

Another method open to us is to use (or facilitate the use of) Freedom of Information laws to get more official data and information into the public domain. Freedom of Information laws give a wide range of public access to information held by authorities. This is useful in releasing information about authorities’ own plans and actions, but is also useful in releasing underlying information which local authorities, so that other organisations (including companies, charities, and other public sector bodies) can use to inform their own approach and actions. 

The Environmental Information Regulations (EIR) substantially overlap with Freedom of Information. EIR requests make it easier to ask for and receive information specifically related to the environment, and covers a wider range of organisations than the general Freedom of Information law. We believe there is the potential to use improved support and guidance for Environmental Information Requests to facilitate more information relevant to local climate action being released, and being made discoverable by local authority actors, civil society and communities. This might take the form of modifications to WhatDoTheyKnow, or light-weight specialist services that sit on top of WhatDoTheyKnow’s FOI archive and infrastructure (similar to FragDenStaat’s Climate Helpdesk). 

But before we can do that, we want to understand more about Environmental Information Requests. WhatDoTheyKnow’s archive gives us access to a huge number of requests made for environmental information, but we understand very little about the contents of these requests. 

Based on a quick text search, we have a maximum of 70,000 requests that in some way mention environmental information regulations. As part of our wider pool of 1 million FOI requests, there may be other requests with environmental interests that have not explicitly mentioned EIR. We are looking for a contractor to explore the large WhatDoTheyKnow dataset, and highlight themes, patterns, and individual examples that show how EIRs are being used, obstacles users are encountering, and the implications of this for mySociety’s future work in this area. 

The available budget for this work is up to £8000-10,000 (inclusive of VAT), and the project would need to be completed by the end of July. There is a small amount of flexibility on this end date – but we are looking to make decisions in August about best ways to follow up the project in September. 

What we want to know more about

The broad goal of this project is to refine our understanding of how EIR works in practice. We’ve broken this into six areas we’d like to understand more about. 

1. EIRs have fewer exemptions, and a higher threshold for withholding material on public interest grounds, than FOI and. In principle, relevant information should be easier to get through EIR, but it is unclear how much this is significant in practice. We want to know more about the kinds of information that are successfully being released through EIR.
2. EIRs apply to more types of organisation – and organisations that have an important local footprint may be accessible to FOI. We want to know about the kinds of institutions and authorities who are or could be releasing information under EIR that is relevant to our local emissions goals.
3. Authorities should consider which regime a request falls under – and default to the more permissive EIR regime when a request is for environmental information. We want to know if there are cases where information is being withheld under FOI but should have been considered under EIR.
4. EIR might be more poorly understood by both requesters and authorities – information may be being incorrectly withheld that should be released. We want to understand any patterns of refusal that would inform advice tools. 
5. At the moment we don’t have an automatic way of identifying EIR responses (when authorities have judged a request under these rules), but it should be possible to create classification rules that separate them from FOI requests. We want to refine this approach so we can more consistently identify and analyse EIR responses, so they can be in current or future local climate services.
6. The big way in which EIRs are less accessible is that authorities can charge fees without a minimum amount of time taken, which is the case for FOI requests. Generally we think this doesn’t happen much (the ICO’s guidance is not to charge for a reasonable request), but we don’t know. We want a clearer sense of how often authorities ask that requesters pay a fee for environmental information.

What we are looking for in and from a partner

Expertise / skill set

We think there are a few different approaches to this project and are open to a range of approaches. In general we anticipate two main kinds of applicants: one leaning more on technical ability, and the other on specialist knowledge of FOI and EIR, or environmental data more broadly. For either kind of applicant, we can provide basic support in the other skillset. The ideal candidate would cover all of these bases (where a subject matter expert makes the technical search process sharper) – and we can help facilitate partnership applications between technical and specialist partners who might otherwise submit separately. If you are interested in this route, please fill out this collaboration form and we can help put you in contact. 

The key thing an applicant needs to have is an approach to searching the large quantity of information on WhatDoTheyKnow. At a minimum this requires some technical skills and ability to navigate the site – but beyond that might be accomplished by large scale text analysis, or a sampling and search method. WhatDoTheyKnow’s search allows searching for specific search terms – but the search count is not always accurate for larger queries. 

For technical strategies, we can provide data exports to enable searches off-platform. But for subject matter experts, we can also provide an Excel sheet of links to URLs that have triggered particular keywords, and authorities. 

Something that is important to us is being able to point to individual examples – and so approaches based on aggregate analysis (which might be the better approach to identify the scale of fees being charged, for instance, or automatically analyse themes of requests) need to also be able to drill down to individual requests. 

With this in mind, the following is a list of skills we would expect the candidate to have a couple but not necessarily all of the following:

• Technical skills
  • Experience with search and processing large amounts of text delivered in a mostly unstructured way.
  • Experience with Natural Language Processing (NLP) or automated topic extraction. 
• Subject matter skills
  • Knowledge and experience of Freedom of Information and/or Environmental Information Requests
    • Especially practical understanding of refusal and appeal grounds in both FOI and EIR.
  • Less important, but understanding of the wider European context of Environmental Information Regulations might provide additional understanding of potential approaches mySociety could take in the UK. 

Alignment with values and aims

Our Repowering Democracy strategy puts a special emphasis on embedding equity and inclusion in our work practices and services, and our work aims in general to fulfil values of equity/justice, openness and collaboration. 

Applicants should consider if this presents any obstacles to a working relationship, and think about how these values should be reflected in the project plan, either in terms of subject matter to investigate, or research approach.

Working practices

mySociety works flexibly and remotely, and there is no requirement to work from or visit an office. Applicants can distribute their work as appropriate over the time available, but we would expect regular check-ins on progress to be arranged over that period. A shared Slack channel and a specific contact person will be used to help coordinate and quickly share questions and information between mySociety and the researcher. 

Successful applicants would be expected to abide by the mySociety Code of Conduct in mySociety communications channels and events. 

Outputs and deliverables

The purpose of this project is to inform mySociety’s future projects, especially forming the base of knowledge around the use of EIRs for a prototyping week.

We are open to the form of outputs – this may take the form of one large report, briefings around our individual question areas, proposed amendments to guidance, etc. For technical submissions, analysis code under an appropriate licence would form part of the output.

The outputs should also work as a general contribution to knowledge of the current use of EIRs, and we might either publish or edit down and publish these briefings/reports.

Q&A and contact details

The application timeline includes a Q&A event, to which you can sign up at the link at the top of this document. The Q&A session will include an element to help individual researchers coordinate to form a joint submission (applications are also welcome from individual researchers). Answers will be made available in a video on this page for applicants who cannot take part. Questions can be emailed to the contact address below. 

Please send any queries or questions to research-commissioning+eir2023@mysociety.org and mention which project it is in regard to. Questions in advance are preferred and will be prioritised in the session. 

Your application

Applications can be submitted by individuals, organisations, or joint teams of individuals/organisations. These should be sent to research-commissioning+eir2023@mysociety.org by the closing date. 

You should submit a short application, of up to 4 pages of A4. A template for the response can be downloaded at the link at the top of this page, and covers: 

• Who you are (whether an individual, organisation, or joint team).
• A description of your previous experience/previous work and why you want to take on this project.
  • To the extent that this is possible, this should be anonymous and not include names of the org or members of the team (to help with anonymous stages of the recruitment process)
• How you would approach and deliver this project – a short project plan with approximate timings. 
  • This could include discussion of appropriate outputs for the project, and balancing technical and subject matter requirements. 
• The total value (£) of your proposal (including VAT), and high-level breakdown of costs (perhaps an indication of days per person, any other expenses). This does not need to include production costs of the report. 
  • Given the cost of the project, we will not be giving a great deal of weight to budget plans so please keep this short and high-level – we can dig into further details during interviews, if necessary.
• A short description of the individuals or team who will do the work, including biographies

There is a separate equalities monitoring form to fill out, which is processed separately from the main application (there is a link to the form in the application form). This is for understanding the reach of our method of distributing the call for proposals. 

If you are interested in joining a ‘researcher pool’ mailing list that we will contact with details of future projects, please see the link at the top of this document. 

Application timeline

If there are changes during this timeline, the table on the website version of this form will be updated. 

Stage	Date	Description
Call for proposals published	7 March 2023
Q&A Webinar	21 March 2023	An open, online public event for interested bidders to learn more about the project and ask questions. This will be recorded and available afterwards. You can submit questions in advance to research-commissioning@mysociety.org. Questions in advance are preferred.
Questions answered	23 March 2023	Video of the webinar to be made available to all potential bidders, in addition to answers to any other questions submitted via email
Deadline for applications	31 March 2023 (end of day)
Initial decisions	7th April 2023	Applicants to be informed whether they have made it through to a short panel interview (and may be asked for a sample of existing work). Applicants not progressing past this stage to be offered written feedback
Interviews	w/c 10 April 2023	Format to be decided, but this will likely be a one-hour panel interview with several people involved in the climate programme, towards the end of the week (14th, 15th April)
Final decision	w/c 18 April 2023	Remaining applicants to be informed of the final decision. Applicants not progressing to be offered feedback
Project briefing/kick-off meeting	End of w/c 18 April 2023	To include a brief introduction to mySociety, discussion of any onboarding required and approach to project management, communication and catch-ups
Project deadline	End of July 2023	End of project (Time range of project is a little flexible – we want it to inform decisions in August about any follow up work in September)

What happens after the project 

We intend to publish the report or briefs you produce, credited to you, on the mySociety website, licensed under a Creative Commons licence (see recent publications on research.mysociety.org for details). We may make some light edits (beyond proofreading) before we publish. You will be free to make publicly available your own version should you wish to, and any other material based on the research you conducted. 

We will convene a short ‘lessons learned’ session for the contractor and mySociety to discuss how the project went – what went well and anything that could have been improved. We will also discuss any future work based on the delivered project (e.g. if you are an academic and might want to co-author an article) and our ongoing relationship. We would also like to arrange a presentation on the project to mySociety staff, and there may also be an opportunity to promote the work in a public event held by mySociety (budgeting for this would be separate to the project above). 

Terms and conditions

Interested parties must be UK-based individuals or organisations.

 

Questions and Answers

 

For data analysis, what format is the data available in?

We create a regular research export of the database in CSV format. This doesn’t include the last few months of data and removes requester names where possible (this is generally available on the website, but we are trying to future proof against information being too available if redactions are needed in future).

There is a a data table of requests made by authority, and then a dataset of the individual messages (with full text) for each request. 

We currently don’t make the downloaded files themselves available in bulk (but are accessible through the site) – but the bulk export does include any cached conversion of a word document or pdf attached.

We can produce reduced data sets limited to specific authorities or keywords. For instance, to those flagged as potentially EIR projects at the start.

WhatDoTheyKnow data sharing policy for bulk data

What is the time scale for the research?

Broadly we’re looking for work to be completed by the End of July to help us inform how we spend our time in September. Depending on the nature of outputs, there may be some flexibility around this.

The retained EU law will ‘sunset’ EIR at the end of the year, unless explicitly retained. How does this work relate to that?

Our current default assumption is “everything will be fine” and this project can proceed as if EIR will continue past 2023. In the event it is looking like everything is not fine, this research helps us understand more about the impact of EIRs for campaigning purposes.

How have you handled projects like this before and how have they worked?

We’ve previously commissioned two pieces of research: one was about how we should commission research, the other was the role of local government in climate change. In both case we worked with a sole researcher, with regular check-in meetings and a shared slack channel. That said, we’re open to group applications (and this project may be appropriate for that). Neither of those projects was particularly data heavy, but we have worked to provide external researchers with data before. 

What support will you give the project?

The main support and contact for the project will be Alex Parsons (Senior researcher) from aa research and data perspective. There is limited available of the WhatDoTheyKnow on a day to day basis team – but I can either answer questions or get answers to questions we need to know.

—

If you have any other questions about this project or the application, please email : research-commissioning+eir2023@mysociety.org

Scrutiny of government ministers’ meetings with lobbyists has been boosted by the Open Access UK project from Transparency International UK, which integrates with our Freedom of Information service WhatDoTheyKnow. 

The fact that meetings between ministers, and people or organisations outside of Government have taken place is published, as required by the Ministerial Code, but typically, few details are proactively released. 

The Open Access UK service collates published declarations of meetings, and provides special links to WhatDoTheyKnow where users can find a pre-written Freedom of Information request for each meeting, asking for:

• The agenda
• The list of attendees, including details of any organisations they represented
• Copies of briefing notes and papers prepared in advance
• Notes or minutes recording  what was discussed
• Any correspondence associated with the attendees, including messages sent to follow up on the meeting

The service covers almost 90,000 ministerial meetings which have taken place since 2012, and it’s being actively maintained, with 1,500 more meetings added last month.  

To identify meetings of interest, searches can be carried out for the name of an organisation (such as a company, charity, union, or lobby group), minister, or by subject matter/policy area. Anyone interested in details of what happened at a meeting can request such information in public via WhatDoTheyKnow with a couple of clicks. You can also follow requests on any meeting, whether or not you are the person who submitted them.

It appears that meetings listed don’t only include physical ones, but online events and phone calls. Some detailed Government guidance on what should, and should not, be included in ministers’ transparency disclosures has been released via WhatDoTheyKnow (though the Government initially refused the request); more up-to date material also appears to be available, including a “pandemic-related update” which specifically covers remote meetings. 

The service does not currently cover meetings with ministers’ special advisers or civil servants where ministers are not present themselves, but Transparency International UK are inviting contact from anyone who would like to fund expanding the scope of the service to cover such meetings.  

 Responses to requests

The service has been running for some time, so everyone can see examples of how requests made through it have been responded to. 

There is a wide range of responses: in some cases the information sought has been substantively released promptly, while in other cases the responses have been less forthcoming.

As one would expect, the names of junior officials attending meetings, and involved in correspondence, are typically redacted. Often though, details of the substance of the matters discussed are also withheld. Exemptions commonly cited include those applying to “formulation of government policy” and “commercial interests”. Those exemptions are not absolute, but are subject to a public interest test: material should be released if the public interest in releasing it outweighs the interest in keeping it secret. 

Public bodies are permitted to delay a response while they consider whether the public interest lies in disclosure or not. We are concerned about the impact that such delays have on the speed of responses: we have noted examples of such delays both in responses to these requests and elsewhere on our service.  

We encourage requesters to ask for internal reviews if they are unhappy with the response to a request. 25% of internal reviews to UK central government departments result in the release of additional material, so asking for reconsideration is often worthwhile. We also provide advice on referring responses to the Information Commissioner, who is empowered to make decisions on whether information should be released or not. 

Ten things we’ve spotted in responses

(more…)

Big Brother Watch is a UK organisation that campaigns to defend civil liberties and privacy. As such, they keep a close eye on the ‘surveillance state’: the degree to which authorities are monitoring citizens, and the ever-more sophisticated technology that enables them to do so.  

They have recently been investigating the use of Chinese-manufactured CCTV cameras in the UK, submitting Freedom of Information (FOI) requests via WhatDoTheyKnow Pro as one part of their research.

In doing so they’ve uncovered the full extent to which this technology has permeated our schools and colleges, local and central government and beyond. Linked with information about both the capabilities of the tech, and the ways in which these companies support the human rights violations of the Chinese state, Big Brother Watch’s recent report makes for disturbing reading.

The campaign has caught the attention of government, and driven a change in policy over their internal usage of cameras. Hearing of this positive outcome, we were keen to talk to Big Brother Watch and find out more about their use of FOI, and their Head of Research & Investigations Jake Hurfurt filled us in.

What can they see?

Big Brother Watch opposes the rise of surveillance in this country, pointing out that the UK is one of the most surveilled countries in the world with an estimated 6 million CCTV cameras across the nation. 

As a society we may have become more accustomed to CCTV in public areas, but most people aren’t aware of recent rapid advances in surveillance functionality — such as facial, gender and even race recognition. And until there is widespread understanding of exactly what these cameras can do, there can’t be informed consent. 

Because of its affordability, Hikvision is the most common provider of CCTV equipment in this country, with Dahua in second place. Both companies are known to supply surveillance equipment that has been used to target ethnic Uyghur minorities in the Chinese province of Xinjiang.

Finally, Hikvision software has known vulnerabilities that could be exploited by hackers — and in countries like the US and Italy, breaches have been noted where cameras were found to be ‘communicating with China’.

That’s the background: so what prompted Big Brother Watch’s investigation?

Taking a closer look

Jake says, “We’d become aware of Hikvision’s dominance of the UK market. We had also seen the US and Australia move to restrict Hikvision and Dahua, given their links to the atrocities in Xinjiang and their cybersecurity flaws, and were concerned of the risks this posed to privacy and civil liberties in the UK, so decided to dig deeper.”

And, given the methods by which other countries had brought about change in this area, they knew they’d need to be looking at public authorities. That meant that FOI, which gives everyone the right to ask public bodies for the information they hold, was an obvious choice: 

“From the start the project was designed as a public-sector focused investigation. We could see that successful restrictions on state-owned firms in other countries had originated in the public sector. It was clear from the beginning that we would ask questions using FOI.”

As Big Brother Watch’s report states, “Over 5 months, [we] submitted more than 4,500 Freedom of Information requests to a range of public bodies to establish where Hikvision and Dahua equipment is in use and what advanced capabilities this equipment has. Some subsequent focused requests were submitted to public bodies who confirmed they do use Hikvision cameras.

“The public bodies included all secondary schools and FE colleges in England, all UK universities, police forces, NHS Trusts, all Oxbridge Colleges, all central government departments, the House of Commons, the three devolved administrations and the Greater London Assembly.”

A survey of surveillance

Our WhatDoTheyKnow Pro service is designed for people making large scale investigative FOI requests: it makes it simpler to send FOI requests to specific types of authority in a single batch, and keeps all correspondence private until you are ready to go public.

Jake says, “We used WhatDoTheyKnow Pro to send an initial wave of requests to police forces, government departments and the largest local authorities in the country, which acted as a scoping exercise to see how widespread Hikvision & Dahua are from a selection of public bodies. 

“Batch requesting and WhatDoTheyKnow’s built-in reminders of when it’s time to chase up a response have been useful. It was also useful in sending requests to public bodies who are hard to contact, as the contact address database is amazing. 

“The other helpful feature was the ability to look across requests others have made in the past, to compare responses and see what kind of wording worked and which did not – it’s something I always do when conducting large FOI projects as I can learn from other’s experience in the kind of questions that are successful, or start by knowing more than I would have done otherwise.”

Getting results

You can read Big Brother Watch’s full report for all the details, but their investigation revealed that three out of five schools, 60% of hospitals, 31% of police forces, 53% of universities, and 73% of councils use Hikvision and Dahua CCTV cameras, representing, in total, 60.8% of our public bodies.

Once the full extent of their usage had been quantified, it was time to get the word out to the people capable of bringing about change — the UK’s elected representatives.

Jake says, “We sent the report to MPs and held an event in Parliament to complement the launch of the report, and we had a good reception.

“Around publication we also got dozens of Parliamentarians to back a pledge to ban Hikvision, and working alongside other NGOs we have been heavily involved in advocacy around a number of bills to push for this. Big Brother Watch does a lot of advocacy in Parliament and the report has supported these efforts.”

Subsequently, with Oliver Dowden describing them as “current and future possible security risks”, the Chinese cameras were banned from being installed in or on government buildings.

Still campaigning

The government may have cracked down on their own use of these cameras, but that still leaves thousands of institutions and private companies who may not even be aware of the capabilities or implications of the cameras they’ve installed.

Big Brother Watch are pressing for further action and you can find their campaign page here. If you are concerned, you can add your name to a petition to request that Parliament pass a law to ban this type of CCTV; and you can add sightings of the distinctive cameras to a collaborative Google map.

Thanks very much to Jake for filling us in on the details of how WhatDoTheyKnow played a part in Big Brother Watch’s wider investigation.

You might also be interested to read our 2019 post on Privacy International’s investigation into police use of facial recognition.

At our Freedom of Information service WhatDoTheyKnow, when faced with requests to remove material, we operate on the principle of removing the minimum amount possible.

Alaveteli, the codebase which underlies WhatDoTheyKnow and a number of other FOI sites around the world, gives moderators a range of options for removing content – with the ability to surgically remove text ranging from individual words and phrases, to individual messages, or even entire request threads. This is useful when we spot misuse of our service, for example.

What we’ve been lacking, up until now, was a way to apply these types of removals to attachments.

Back in the early days of WhatDoTheyKnow, attachments were less common, but now we see many more: there can often be several attachments to one individual message.

Over the last few years, there have been occasions where we’ve had to remove an entire message, which may contain several useful attachments, just because of a small issue with one of them.

We’d then go through an annoying manual process to download the publishable ones, upload them to our file server, and then annotate the request with the links – here’s an example.

Back in 2013, when the original suggestion for enabling finer grained control was raised, the site contained around 400,000 attachments. There are now more than 3,500,000! We don’t remove content often, but at this scale it’s inevitable that we need to intervene now and then.

After a little code cleanup we were able to make individual attachment removal a reality. This allows us much more control over how we balance preserving a historic archive of information released under Access to Information laws, and running the site responsibly and meeting our legal obligations under GDPR.

As an example, let’s imagine that the FOI officer replying to our request inadvertently makes a data breach when releasing some organisation charts in `organisation chart b.pdf`.

Previously we’d have had to have hidden the whole response. Now, we can go into the admin interface and inspect each individual attachment.

We can then set our usual “prominence” value – offering a few options from fully visible to completely hidden – and include a reason for why the content has been hidden. We always seek to run the site transparently and explain any actions taken.

On saving the form, you can see that only the problematic attachment has been removed, with the remainder of the response intact. This saves us considerable time when reviewing and handling material with potential data issues, and keeps as much information published as possible while we do so.

As an extra bonus, since the main body text of emails is also treated as an “attachment” in Alaveteli, we’re now able to hide potentially problematic material there without affecting the attachments we present.

We’ve already used this feature several times to republish material where we’d previously had to hide the entire message due to the technical limitations at the time.

—

Image: Kenny Eliason

Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

…or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

—

Image: Keila Hötzel

Last year, mySociety provided technical support to Climate Emergency UK (CE UK) for their Council Climate Scorecards project, which marked every UK local authority’s climate action plan across 78 different areas. The resulting data made clear where plans were adequate, and where there was still work to do. It has informed campaigns, researchers, news stories and councils themselves, as well as feeding into government-level policy.

But plans are one thing, and putting them into action is quite another — not to mention, rather more crucial. So this year, CE UK have set themselves the task of scoring councils on the progress they’ve made on climate action.

To do so, they’ll be using many of the same methods they put to such good effect in the Action Plan Scorecards: they’re currently assembling teams of volunteers (want to get involved? See the end of this post) that they’ll train up with the research skills needed to scrutinise such a huge body of data accurately and with a good understanding of the issues at hand.

Scoring the plans may have seemed like a big task, but at least they are documents which were  — to a greater or lesser extent — possible to find online. Action, of course, happens in the real world, so some different methods are required. 

CE UK’s methodology for the Action Scorecards can be seen in detail here; it relies not just on the councils’ own reporting, but on a number of different documents and news reports. And where the information can’t easily be found in the public arena, they’ll be submitting Freedom of Information requests.

Of course, this is an area in which we at mySociety have long experience, so our Transparency team is helping out. CE UK will be using our WhatDoTheyKnow Pro service to send the large batches of FOI requests and manage the responses; once the Action Scorecards are launched, the data will, of course, be made public for everyone to access.

With our help, the requests have been refined to provide minimum disruption to busy council officers; at the same time, we hope that these requests, which are all for information that really should be available — energy standards for council-operated housing, for example, or numbers of staff members in climate-related roles — will encourage more proactive publication of data, so that it won’t need to be requested in future years.

We’ve also been able to advise CE UK on forming good FOI requests that will surface the required information.

Because of CE UK’s training strategy, we’re delighted that this knowledge will be passed on to their cohorts of volunteers, effectively informing a new tranche of citizens on how and why to use FOI responsibly.  They’ll be helping to classify the responses and compile useful datasets through our early-stage FOI collaboration tool.

We’re proud to be supporting this important work from a climate perspective, too: councils have a crucial role to play in cutting emissions, and there’s an obvious public interest in how they go about doing so — how they allocate public funds, how effective their interventions are, and whether they are on track to reach carbon zero by their self-set deadlines.

All in all, the small team at CE UK have embarked on a massive but vital task. Can it be done? Their approach, as always is: there’s only one way to find out, and that is to try it!

 —

If you’re interested in helping out, there’s still time to apply to be a volunteer — closing date is this Thursday though, so hurry! You’ll be working from home, trained up via online webinars and then helping to collect data as part of this huge effort. Sounds good? More details are here.

On 21 December, we added the National Centre for Circus Arts as a public authority on WhatDoTheyKnow, making it easy for anyone to send them a Freedom of Information request. We could have left it off the site and let our users find the email address themselves, but we didn’t want them to have to jump through hoops.⭕

Joking aside, this is a valid addition to the site — see below for more details about why this college is subject to the FOI Act — and one where we can imagine some interesting requests being submitted.

For example, one could ask for copies of the risk assessments used for the various circus skills taught to students — we imagine these would contain quite a bit of detail.

A procurement list might throw up some items you wouldn’t see from other education institutions, like juggling equipment, unicycles or clown shoes.

Prospective students who want to understand more about their chosen course’s application procedure and admission statistics often submit requests to UCAS, but applications for the National Centre for Circus Arts are handled directly (which makes sense: it would be hard to evidence circus skills on an UCAS form!).

Instead, anyone looking for more information about how the application process works could submit an FOI request directly to the college. 

In addition, there is a wide range of requests that can be sent to almost any public authority on matters such as the use of public money, minutes of meetings, policies, letters, emails and contracts. All of these might have uniquely circusy angles when applied to this particular institution!

Some decisions are in-tents

It’s not always easy to work out whether a body is subject to FOI. For some bodies it’s really simple: they’re listed by name in Schedule 1 of the Freedom of Information Act 2000 (or the Scottish equivalent). 

The National Centre for Circus Arts isn’t named here, but Schedule 1 does include references to definitions contained in other laws.  

In the course of recent work to find more bodies to add to WhatDoTheyKnow, our volunteers have read and interpreted three pieces of legislation and an official register to work out which higher education institutions are subject to FOI. 

The upshot of our research is that we consider that all bodies listed on the Office for Students register as “Approved (fee cap)” are subject to FOI. This includes the National Centre for Circus Arts (registry entry). Helpfully, the Centre acknowledges the fact that it is subject to FOI on its website which not all higher education institutions do – kudos for that.

UK higher education institutions walk a tightrope between the public and private sectors. They are subject to rules such as competition law that are normally relevant to private business, whilst at the same time being subject to laws that apply to the public sector such as the equality duty and FOI. It’s quite a lot to juggle. 🤹

Interestingly, the debate about the role of circuses in public life goes back to Roman times where the provision of food and entertainment to the poor were used to gain political power.

WhatDoTheyKnow lists a number of obscure public authorities on the site ranging from the Crown Estate Paving Commission to the Treasure Valuation Committee but the National Centre for Circus Arts is certainly one of the more unusual higher education institutions we list. There can’t be many places where people jump through fire for their graduation (page 13 of the 2023-24 prospectus).

In conclusion, if anyone is planning to run away during 2023 then please be advised that with this addition, WhatDoTheyKnow has both the sea and the circus covered. 

We wish all our users a happy and healthy 2023.

—

Image: Jonny Gios

We’ve had 109,653 Freedom of Information  and Environmental Information Regulations requests made on WhatDoTheyKnow this year. In the run up to the end of 2022 here’s a countdown of 12 of the more unusual ones that have caught our eye this year…

12: Bats

National Highways released 1.25 TB of bat survey data carried out for the Arundel bypass scheme. This was made up of over 115,000 files, that included 786 videos – that’s over 250 hours of footage – 54,570 audio files, 354 spreadsheets and 2,532 images.

We like this because we think it is the largest ever release of information, and as the climate crisis brings urgent challenges for our public institutions to address, access to environmental information will be increasingly valuable to businesses, campaign groups and the general public. You can read more about this release here.

11: Vaccines

The Medicines and Healthcare Products Regulatory Agency released the nucleotide sequences of the AstraZeneca & Pfizer/BioNTech Covid-19 vaccines used in the UK, after an initial refusal was overturned on review. The response says the companies involved consented to the release.

We liked this because it is a great use of FOI to get such important medical information released and available in the public domain.

10: Bins

Sheffield City Council released the location of every public bin in the city.

We liked this because not only is it really useful information; it is the sort of data that councils should be making freely available to citizens.

 

 

 

 

9: Underground

Edinburgh City Council released the sewer and cable plans for parts of the city centre.

We liked this because they are chaotically beautiful — not what you’d expect from an underground asset plan.

 

 

8: Textbook

The Open University released a full textbook in response to a request for the information held on the Early modern Europe: society and culture c.1500-1780 module(s).

We liked this because it’s not not often that you see full textbooks being released.

 

 

 

7: Tower Bridge

The City of London Corporation released a list of the past Tower Bridge lift dates, times, and vessel names from the start of 2022 until now.

We liked this because it generated a long and sometimes amusing conversation on Reddit.

 

 

6: War memorabilia

The Ministry of Defence released a WW2 medal card.

We liked this because we’ve never seen something like this obtained by FOI before.

 

 

 

 

5: Honours board

The Charter Trustees of the Town of Margate released their Freedom of the Town list in a more unusual format.

We liked this because the information released was a photo of a painted wooden board. This is, after all, still a form of recorded information — and a nice permanent one.

 

 

 

 

 

4: Seaside nuisance

Brighton & Hove City Council released a copy of all of the bye-laws that apply to the seafront.

We liked this because of the phrase: “no annoying gramophones on the beach”, which may be a slightly outdated view of the worst possible noise nuisance.

 

 

 

3: Big cats

North Wales police released the details of big cat sightings in 2021.

We liked this because the information disclosed in the request was used for a number of news articles in Wales.

 

 

 

2: Library books

One of our users has been doing some research into the top 25 books borrowed from libraries in 2021; here’s an example of one of them.

We liked this because it’s fascinating to see what books people are choosing to read, and how this varies between different areas of the country.

 

 

 

1: Trains galore

And finally, here at WhatDoTheyKnow the team are all big fans of trains. This means we tend to notice the more interesting disclosures on train related topics. Here are some of our favourites from this year:

 

 

 

a. Network Rail released the engineering drawings that were produced during the construction of the London & South Western Railway’s station at Branksome, near Bournemouth in the mid to late 1800. 

We liked this because the drawings are beautifully crafted and not something that we get to see very often.

b. Northern Trains Limited released the .wav file of the two jingles used for their station automated announcements system.

We liked this because it’s unusual for audio files to be released and we’ve found so many uses for this!

c. London North Eastern Railway Limited (LNER) released some information about the voiceover artists used for the automated announcements on the Class 800 and Class 801 fleet of trains

We like this because it shows that FOI responses don’t need to be formal or complicated; they just provide, where possible, the information that the request-maker has asked for. LNER is  particularly good at this.

d. Transport for London released 3D station layout drawings for the Elizabeth line.

We liked this because it’s really interesting to see how the new stations on the Elizabeth line have been designed, and how the layout works with their surroundings.

 

 

---

We hope you enjoyed 12(ish) of our favourite FOI requests from this year.

You can find out more information about the WhatDoTheyKnow service and  how to make requests on our help pages.

If you’d like to help WhatDoTheyKnow keep up their good work, you can help in two ways, either by volunteering us or by making a donation, any amount small or large is greatly appreciated.

—

Image credits:

Vaccines CDC; Bins Charlotte Harrison; Tower Bridge: R Spegel; Brighton seafront Tom Wheatley; Big cat: Mana5280; Library: Dollar Gill

WhatDoTheyKnow is a project of mySociety, run by a small team of staff and dedicated volunteers.

In 2022 WhatDoTheyKnow users made 109,653 Freedom of Information requests via WhatDoTheyKnow.

Those requests, and the responses they received, are public on the website for anyone to see.  

What’s not quite so visible is the work that the WhatDoTheyKnow team, which is largely made up of volunteers, do behind the scenes.  

Some of their most difficult calls arise around the removal of information. WhatDoTheyKnow’s guiding principle is that it is a permanent, public, archive of Freedom of Information requests and responses, open to all.

The team works incredibly hard to maintain the archive in the face of challenges, including the reduction of legal risks; dealing fairly and transparently when people ask for information to be removed from the site; answering users’ questions; supporting citizens to use their rights to FOI; dealing with misuse of the service which breaches our house rules inappropriate content and keeping everything ticking over.

Our default position is not to remove substantive public information requests and responses; however, we act quickly if problematic content is reported to us. And, to help everyone understand exactly what has been removed and why, where possible we record these details on the request page.

To allow for a full 12 months of data, the date range used throughout this report is 1 November 2021 to 31 October 2022.

Headline facts and figures

• 16,354,872 visits to WhatDoTheyKnow.com this year.
• 16,217 new WhatDoTheyKnow user accounts created this year, taking the total number of accounts to 239,540. This represents an increase of 7.6% in the total number of site users since last year.
• 8,912 total number of email threads in the support inbox in 2022… that’s an increase of 11.2%, making it all the more crucial that we continue to recruit volunteers to help spread the load.
• 1,381 requests hidden from WhatDoTheyKnow in 2022
  …in the context of 109,653 requests made in the year, and a total of 867,303 requests currently published on the site.
• 171 published requests where we redacted some material in 2022
  …usually due to the inappropriate inclusion of personal information, or defamation.

And in more detail

Requests made on WhatDoTheyKnow flagged for our attention

The table below shows the reasons that requests were reported by our users via the site for admin attention this year. 

Note that we also receive many reports directly by email, so while not comprehensive, this is indicative.

Reason for attention report	Total number
Vexatious	117
Not a valid request	109
Contains personal information	89
Request for personal information	85
Contains defamatory material	33
Other	642
Total*	1,075

*The number of requests flagged for attention this year is up 40% on last year. This is largely related to a single campaign of misuse.

Material removed from the site

The following tables show where members of the admin team have acted to remove or hide requests from WhatDoTheyKnow in the last year, and the reason why.

At WhatDoTheyKnow  we have a policy of removing as little material as possible, while seeking to run the site responsibly and take different viewpoints into account. Removing substantive FOI requests and responses is a last resort and something we do very rarely. However, we act quickly to remove problematic material.

Request visibility	Total number
Discoverable only to those who have the link to the request	2
Visible only to the request maker	1,282
Hidden from all site visitors	97

 

Reason for removing from public view	Total number**
Not a valid FOI request	1,117
Vexatious use of FOI	43
Other (reason not programmatically recorded*)	221

* Current processes do not create an easily retrievable list of reasons beyond the two above, however due to site improvements made in autumn of this year we expect to be able to provide more detailed information on this in the future.

** The number of requests hidden or removed from the site this year is up by 68% on last year. As above, this increase is largely related to a single campaign of misuse. 

Censor rules (targeted redactions to hide the problematic part/s of a request)	Total number
Number of censor rules applied	746
Number of requests with censor rules applied	171
Number of requests with censor rules applied which are still publicly visible, but with problematic material hidden	165

* Censor rules are used for many purposes, including redacting problematic content and removing personal data which should not be present

Cases relating to GDPR rights 

These are typically cases relating to requests to remove data published on the site as per the rights afforded under GDPR, the UK’s General Data Protection Regulations.

Right type	Total number of cases*
GDPR Right to Erasure	214
Data breaches by third parties	79
GDPR Right to Rectification	15
GDPR Right of Access	21
Data breach – internal**	6
GDPR Right to Object	<5
Total	340

* Not all issues raised resulted in material being removed from the site.

* “Data Breach – internal” refers to cases where WhatDoTheyKnow has identified that a data breach may have been caused due to our own staff actions. We take our obligations seriously, and use such instances as a learning opportunity, so these are recorded by us even if very minor, and often when they’re nothing more than a near miss.

High risk concerns escalated for review 

Our policies ensure that certain issues can be escalated for review by the wider team and, where more complex, by a review panel that includes mySociety’s Chief Executive. Escalation is typically prompted by threats of legal action, complaints, notifications of serious data breaches, potential defamation concerns, safeguarding, complex GDPR cases, or cases that raise significant policy questions.

Case type*	Total number
Defamation	49
Data breach	40
GDPR Right to Erasure	33
Complaints	19
Safeguarding / Public harm	13
Takedown	13
GDPR Right of Access	9
Police user data requests	7
Site misuse	7
Data breach – internal	5
Other	39

* Email threads may be either automatically categorised by the system, or manually categorised by the WhatDoTheyKnow admin team on the basis of the information given by the person reporting them. Some cases can relate to two types: for example a GDPR Right to Erasure request may also be a complaint. For the purposes of this table, such instances have been included in the counts for both concerns.

Users

User accounts	Total
WhatDoTheyKnow users with activated accounts	239,540
New user accounts activated in 2022	16,217

 

Reason for banning users in 2022	Total
Spam	2,160
Other site misuse	300
Total number of users banned in 2022	2,460

 

Anonymisation*	Total
Accounts anonymised in 2022	139

* Accounts are anonymised at the user’s request, generally to comply with GDPR Right to Erasure requests.

Users are banned and their accounts may be closed due to site misuse and breach of the House Rules. Anonymised and banned users are no longer able to make requests or use their accounts.

User data requests

The table below shows the number of requests that we received from third parties for the personal data that we hold on our users in 2022. Details of which types of data we hold can be found in our privacy policy. As stated in our privacy policy, we do not provide this information to anyone else unless we are obliged to by law, or the user asks us to.

Type of request	Total
Police/law enforcement requests for user data	7
Other requests for user data	6

 

Material released	Total
Number of requests, where court orders were produced and we provided the material as required	2

 

Thank you for reading

We produce this report as we demand transparency from public authorities and it’s only right that we also practise it ourselves. 

Additionally, we hope that the report goes some way to showing the type of work the team do behind the scenes, and that running a well-used site like WhatDoTheyKnow is not without challenges.

If there are specific statistics that you’d like to see in subsequent Transparency reports, or you’d like to know more about any of those above, do drop the team a line. 

If you’d like to help WhatDoTheyKnow keep up their good work, please consider volunteering or making a donation. Any help small or large is greatly appreciated.

—

Image: Meriç Dağlı

.