—

For a while now, there’s been a feature on WhatDoTheyKnow that lets you link your Freedom of Information requests to news articles, campaigning pieces or research papers.

We’ve recently made it easier to link your batch requests to these types of stories in the same way. 

To celebrate, we’ll be offering free credit for WhatDoTheyKnow Pro subscribers who add links from their batch requests to the stories or papers that the requests have fed into.

For each qualifying link added during August, we’ll credit your account with a coupon that gives you a 20% discount on one month of WhatDoTheyKnow Pro. 

Add several links, and you’ll get several coupons — so you could be enjoying that 20% discount for many months to come.

It’s easy to add them – go to the batch request via the dashboard and you’ll find the section in the right hand column. Just click on ‘Let us know’:

…and paste the URL in:

If there’s more than one story, you can click ‘New citation’ to add another one.

You’ll then see all the links to sources where the requests have been cited:

They’ll also be shown on the pages for individual requests in the batch:

If you’re a journalist, campaigner or researcher, we hope this is a useful way to give your stories some more readership (not to mention a nice inbound link from a high-ranked site for your search engine ratings).

More broadly, when you use this feature you’ll be helping us to understand what sort of impact the site is having, too. We’re always keen to spot news stories based on WhatDoTheyKnow requests, but papers don’t always cite a source or link back to the site, meaning that our monitoring is often dependent on a manual search where stories look like they might have originated with one of our users.

The way we’ve set this feature up, WhatDoTheyKnow users can add a citation to any of their own requests — but if you spot a news story that’s linked to a request that isn’t yours, please do contact the WhatDotheyKnow team.

They’ll assess it and input it if they find it to be valid. Our aim here is, of course, to prevent spammers from adding irrelevant links to the site.

Users of WhatDoTheyKnow Pro, on the other hand, have the ability to add citations to any request.

—

For a link to qualify for the discount, it needs to be a link to a specific story, report, paper or dataset where the information released in the FOI request has been used (ie not just a link to your organisation’s homepage, or a general overview of a campaign – though we’re always delighted to hear about these cases, too!). 

We’ll cap the number of months on which you can claim a discount at 24, but we really do appreciate these links so please do add them even if it’s above the cap. We’ll apply coupons to any qualifying links at the end of the month.

—

Image: Etienne Girardet

Alaveteli is our platform that anyone can use to run a Freedom of Information site in their own country or jurisdiction.

As the number of requests grows on an Alaveteli site, it can become increasingly difficult to find released information that you’re interested in.

You can search, but the more general the term, the more likely that you’ll pull in results where the term is mentioned incidentally rather than being directly related to the information released. Or you can browse by authority, but that’s more fiddly when you want the same information from a range of authorities.

There’s also the issue that people new to FOI might not have a clear idea of what to ask about. Freedom of Information is great because you can ask for anything, but as a newcomer that can feel overwhelming. You need some direction to know where to start.

Request categories allow us to curate related requests to bridge the gaps mentioned above.

Here’s an example of a category on WhatDoTheyKnow that compiles successful Freedom of Information releases related to the British Post Office scandal.

Categories can be created in the admin interface via the Requests > Categories menu item.

Request categories work in a similar way to the current public body categories – in fact, as part of this development we’ve revamped the underlying code so that it applies to both!

At their core, they’re composed of three things – the title, body (where we can add explanatory content), and requests grouped by a tag.

Notes can be added to call out key information, and categories can be added to headings to create a layer of hierarchy. As part of this development, we’ve also improved notes so that they can be more easily styled with some preset colours, and added rich text editing to improve the formatting of longer notes.

We’ve started building up some interesting categories of requests on WhatDoTheyKnow, but we’d love to hear which ones you’d like to see.

If you’re interested in how the development unfolded you can take a look at the related work on GitHub.

—

Banner image: Garmin B

CSV is a great format for releasing sets of structured data in response to Freedom of Information requests. Indeed, on WhatDoTheyKnow we’ve seen several thousands of CSVs released.

We’ve recently added the ability to explore CSV files via a Datasette instance. Here’s an example.

Opening the CSV in Datasette makes it easy to explore and analyse it in an interactive website.

If you’re not familiar, Datasette converts the CSV to an sqlite database, which means you can then query the data using SQL.

Alaveteli uses the publicly available lite.datasette.io instance by default, but you can host your own instance and configure it at theme level like we’ve done for WhatDoTheyKnow.

You can see the implementation details at mysociety/alaveteli/#7961.

—

Banner image: Joshua Fuller

Alaveteli is our platform that anyone can use to run a Freedom of Information site in their own country or jurisdiction.

We’ve added new functionality that allows Alaveteli sites to highlight blog posts on the homepage, so they’re more visible. In this way, Alaveteli sites can not only help users with the ‘how’ of making an FOI request, but also show, in a very tangible way, the ‘why’  — especially if the posts are highlighting impactful uses of the site.

Previously, Alaveteli had a basic way to pull in an RSS feed of blog posts. These were only available on the blog page (/blog), which often does a disservice to all the great work that gets written about.

We wanted to better signpost the blog from other pages in Alaveteli to celebrate great FOI use, and help users understand how a seemingly simple FOI can go on to have an outsized impact.

When the site runner configures a blog feed, posts are pulled into Alaveteli and cached in the database. This makes them available on the homepage. Here’s what that looks like on WhatDoTheyKnow:

They’re also visible in the admin interface via the Tools > Blog Posts menu item.

At present only the title, URL and publication date are cached in Alaveteli. These records are intended to be “pointers” to the canonical URL of the article hosted on the external blog service.

In the admin interface, blog posts can be tagged to indicate their subject matter.

As FOI requests and authorities can also be tagged, this allows the blog posts to be highlighted in the sidebar of appropriate pages where there’s a matching tag. So, if someone’s browsing requests or visiting an authority that deals with the climate emergency, for example, they’ll be shown relevant blog posts – hopefully making them more visible to people who have already displayed that they have an interest in the topic, and giving those people a bit more contextual knowledge.

In future we’d like to make these posts more visible, by importing header images and a short summary, and give the ability to display some posts when there isn’t a direct tag match.

You can read about the initial design and subsequent conversation and pull requests starting at mysociety/alaveteli#6589.

—

Banner image: Patrick Perkins

Supported by the Wellcome Data Prize in Mental Health, mySociety are providing technical support to the organisation Black Thrive, whose work addresses the inequalities that negatively impact the mental health and wellbeing of Black people. Their question: does disproportional use of police stop and search impact the mental health of young Black people in England and Wales? This project aims to find out.

Late last year mySociety provided some low-key support during the Discovery phase of the project as Black Thrive were developing their statistics package to extract and enhance data from data.police.uk to make the data more accessible and research-ready (for example, by making it easier to combine the data with other datasets such as Understanding Society, a longitudinal household panel study).

We’re pleased to say that the project made an impact, and we’ll now be more actively sharing our expertise in creating data-heavy, citizen-focused services by collaborating with Black Thrive, King’s College London, and UNJUST CIC in the Prototyping phase of the Data Prize.

Now that Black Thrive have built an automated way of gathering the data and creating the analysis, the next phase focuses on presenting these in an open dashboard that’s straightforward for anyone to use and understand. We’ll also be looking at mechanisms for automatically keeping the data up to date, and adding new datapoints when they become available.

Having built several dashboards into our own services including WhatDoTheyKnow and FixMyStreet, we’ve got some great experience to build off; plus bags of experience in sourcing and munging lots of disparate data from our EveryPolitician project. More recently we’ve been making local council Climate Action Plans explorable and accessible, and also creating an easy-to-use data hub for data about local MPs, constituencies, public opinion and the climate and nature movement so this project sits comfortably within our wheelhouse.

In fact, as it happens, we’re currently using very similar tech on another project, our Local Intelligence Hub prototype, and we’re going to use it as the basis for this new dashboard. Nothing like a bit of crafty repurposing where it helps save time and effort! 

Here’s a sneak peek of how it’s looking at this early stage:

Watch this space and we’ll be sure to keep the updates coming as we progress.

—

Image: Chris White (CC by-nc-nd/2.0)

At our Freedom of Information service WhatDoTheyKnow, when faced with requests to remove material, we operate on the principle of removing the minimum amount possible.

Alaveteli, the codebase which underlies WhatDoTheyKnow and a number of other FOI sites around the world, gives moderators a range of options for removing content – with the ability to surgically remove text ranging from individual words and phrases, to individual messages, or even entire request threads. This is useful when we spot misuse of our service, for example.

What we’ve been lacking, up until now, was a way to apply these types of removals to attachments.

Back in the early days of WhatDoTheyKnow, attachments were less common, but now we see many more: there can often be several attachments to one individual message.

Over the last few years, there have been occasions where we’ve had to remove an entire message, which may contain several useful attachments, just because of a small issue with one of them.

We’d then go through an annoying manual process to download the publishable ones, upload them to our file server, and then annotate the request with the links – here’s an example.

Back in 2013, when the original suggestion for enabling finer grained control was raised, the site contained around 400,000 attachments. There are now more than 3,500,000! We don’t remove content often, but at this scale it’s inevitable that we need to intervene now and then.

After a little code cleanup we were able to make individual attachment removal a reality. This allows us much more control over how we balance preserving a historic archive of information released under Access to Information laws, and running the site responsibly and meeting our legal obligations under GDPR.

As an example, let’s imagine that the FOI officer replying to our request inadvertently makes a data breach when releasing some organisation charts in `organisation chart b.pdf`.

Previously we’d have had to have hidden the whole response. Now, we can go into the admin interface and inspect each individual attachment.

We can then set our usual “prominence” value – offering a few options from fully visible to completely hidden – and include a reason for why the content has been hidden. We always seek to run the site transparently and explain any actions taken.

On saving the form, you can see that only the problematic attachment has been removed, with the remainder of the response intact. This saves us considerable time when reviewing and handling material with potential data issues, and keeps as much information published as possible while we do so.

As an extra bonus, since the main body text of emails is also treated as an “attachment” in Alaveteli, we’re now able to hide potentially problematic material there without affecting the attachments we present.

We’ve already used this feature several times to republish material where we’d previously had to hide the entire message due to the technical limitations at the time.

—

Image: Kenny Eliason

Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

…or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

—

Image: Keila Hötzel

WhatDoTheyKnow Pro is the paid-for, premium, version of our Freedom of Information service WhatDoTheyKnow.com, designed for journalists, academics, campaigners and others whose needs exceed what our free service provides.

Features available to Pro users include the ability to delay publication of requests and responses; and to make requests to multiple authorities at the same time via the batch request tool. 

We’ve just made it much easier for Pro users to add relevant bodies to a batch request via a list of authorities within specific categories.

Our database contains FOI contact addresses for more than 42,000 authorities. Using our service saves you from having to source appropriate contact details yourself, and we’ve now made it even quicker and easier to make batch FOI requests. 

Since WhatDoTheyKnow Pro’s launch, creating a batch request has involved searching for bodies and adding them individually to the batch. WhatDoTheyKnow’s fantastic volunteers curate over 200 categories to help users on the main site to explore and navigate the UK authorities subject to FOI, and we’ve now incorporated these listings into WhatDoTheyKnow Pro’s batch tool.

As a result, requests are more likely to be sent to the bodies that hold the information being requested, and the number of requests sent to inappropriate bodies is minimised.

We’ve been trialling this feature in a limited beta period for a while, and thanks to our funding from the Swedish Postcode Foundation we were able to work with handlingar.se to iron out some bugs and performance issues before making it available to all Pro users.

We hope the new feature will aid some great cross-authority research, while helping to ensure that requests are targeted to appropriate bodies. 

Let us know if there are additional categories you’d like us to add!

—

Image: Nick Youngson (CC BY-SA 3.0)

On 31 May 2022, Northern Trains Limited (Northern) wrote to us to demand that we stop publishing the salaries and job titles of the ten highest paid managers at the company. The Department for Transport had released this data in response to a request made via our Freedom of Information service, WhatDoTheyKnow. The request for removal was not only made on behalf of the company, but was also represented as being a request on behalf of the “director group”, which we have interpreted to mean those senior staff at the company whose salary data has been disclosed.

Having carefully considered our position we are continuing to publish this information.

Table: Salaries of the highest paid managers at Northern Rail Limited in £5k bands. 

Source: DfT Freedom of Information release – released on at 30/05/2022

There is a strong public interest in favour of the release of information that helps people to understand  how resources are apportioned within an organisation. As we understand it, the Department for Transport has dealt with the FOI request in line with current best practice for transparency surrounding senior officials and high earners in the public sector, and has acted in accordance with current guidance from the Information Commissioner.

Northern Trains Limited, which operates under the ‘Northern’ brand, is wholly owned by the Department for Transport. The Government proactively publishes the exact salaries of the highest paid public sector employees as part of their regular proactive transparency releases. It would seem reasonable that Northern would also be expected to make similar information available about the salaries of its most senior staff, particularly when the salaries of senior officials at similar and related companies are already public. This includes those working for Northern’s parent company, DfT OLR Holdings Limited, Network Rail, and High Speed 2 Limited. Northern’s sister company LNER publishes information on the salaries of its directors in £5k bands on p46 of its latest annual accounts. In respect of DfT OLR Holdings Limited, the Government proactively publishes the salaries of their Chief Executive (£235,000-239,999), Group Finance Director (£220,000-£224,999) and Chair (£150,000-£154,999).

Northern routinely publishes exact salary information for junior roles on their careers website, and the material released by the Department for Transport is very similar to this. Recently, Northern has advertised that they will pay a full-time train cleaner based at Wigan £18,500 and a grade B maintenance worker based at Newton Heath £33,035 a year. We believe that Northern should have no objections to us publishing that their Managing Director receives a salary of between £245,001 and £250,000 a year.

We don’t know why the Northern Managing Director’s salary has been omitted from the data proactively published by the Cabinet Office. Perhaps they’ve been confused by the complexity of corporate structures involved, and have not looked beyond companies directly wholly owned by the Government when seeking to identify highly paid and senior public servants who should be included. We asked the Cabinet Office to comment and they shirked responsibility for the data they publish saying: 

“Although Cabinet Office compile and publish the £150k list on GOV.UK, other departments provide us with the list of salaries to be included. DfT will have sent us their senior salaries list covering its departments, agencies and non departmental public bodies. You would be best to direct your query to them, and they should be able to advise why this salary fell out of scope.”

We contacted the Department for Transport for comment but as of the time of writing we had not received a substantive response.   

We don’t know if there is an issue with the criteria for proactive publication of salaries by the Cabinet Office or if the Department for Transport have not followed the existing criteria. 

We strongly believe in preserving and promoting transparency and openness, and the accountability of those in positions of power and in maintaining a public archive of Freedom of Information requests and responses. We carefully consider all requests to remove material from our website. We balance the interests of individuals and organisations asking us to take material down with the interests in favour of continued publication.

Northern’s attempt to keep the salaries of its senior executives secret came while the threat of strike action on the railways over pay was growing. On 7 June 2022, the RMT announced 3 days of national strike action in what it called “the biggest dispute on the network since 1989.” Northern is expected to be one of the companies whose services are affected. When assessing whether to keep publishing the information, we considered the journalistic value of the data released. We expect the senior staff salaries, and the attempt to keep those salaries hidden from the public, may well be considered especially newsworthy during this period. The material that was released will help to inform the ongoing debate around pay levels in this sector.

We list DfT OLR Holdings Limited, and the three rail companies it owns on behalf of the British public, on WhatDoTheyKnow so anyone can make FOI requests to them in public. All the bodies are subject to Freedom of Information law:

• https://www.whatdotheyknow.com/body/dohl
• https://www.whatdotheyknow.com/body/northern
• https://www.whatdotheyknow.com/body/southeastern
• https://www.whatdotheyknow.com/body/lner

We thank Northern Rail for drawing our attention to this release of their senior management salary data, which might otherwise have gone largely unnoticed.

For more information on how we deal with takedown requests like this, and our legal basis for processing personal information see: https://www.whatdotheyknow.com/help/privacy#legal_basis

—

Image: Ben Garratt (Unsplash licence)

We recently became aware of extensive misuse of our Freedom of Information site WhatDoTheyKnow, in connection with the academic status of Taiwanese politician Dr Tsai Ing-wen.

This activity became apparent through a very large quantity of correspondence being sent through the site, all focusing on the validity of Dr Ing-wen’s qualification from the London School of Economics and Political Science (LSE). 

The majority of this material was repeating the same or very similar FOI requests, and some were not valid requests at all. We also saw mass posting of annotations, some on completely unrelated requests, and new requests which copied the titles of unrelated existing requests in an apparent attempt to evade our attention.

Running the service responsibly

As an organisation, we positively and passionately support the citizens’ right to access information and to hold organisations accountable: this is the very foundation that WhatDoTheyKnow is built upon, and its reason for existing. 

Over time, we’ve formulated and consolidated policies to ensure that information on the site is preserved, as far as possible, as a permanent archive. We robustly contest unjustified requests to remove material from our service, and will only remove any substantive Freedom of Information requests and responses if we absolutely have to. 

We initially treated this misuse assuming good faith, putting significant effort into removing problematic material from correspondence while continuing to publish elements which could have amounted to a valid Freedom of Information request. 

Understanding the problem 

Several users took the time to report the misuse of our service to us, for which we are thankful. As a matter of course, we review all material reported to us and assess it before making a decision on what to do. It took our small team of staff and volunteers a significant amount of  time to respond to the number of reports made in this case.

Researching the topic more deeply, we discovered a statement from the Information Commissioner on requests they’ve also received on this subject, in which they say:

“The intent of these requests is clearly to try to add weight to theories around the falsification of President Tsai’s PHD, which have already been considered at length by the Commissioner and the Tribunal and found to be entirely lacking in substance.”

Further, both the LSE and the University of London have published their own statements, and a copy of the PhD thesis in question is now available online via LSE’s website. 

While rejecting one FOI request on this subject as vexatious, LSE raised the possibility that people in China could be making requests to benefit from the country’s citizen evaluation system, stating:

“We have been made aware that there is the possibility that the LSE has been added to a list of targets to gain social credits in China. As such we believe that your request and the others we received in this time period have not been made for just the purpose of receiving information but for personal gain.” 

With this information in hand, we were confident to treat the issue as mass misuse, more akin to spam or even a disinformation attack than to people making misguided requests.

Taking action

During the course of this situation, we have banned 108 user accounts, most of which have been created to circumnavigate previous bans and to post inappropriate material to our site. We removed more than 300 requests from the site and 1,640 comments from pages. 

To put this in context, we only banned 126 newly created user accounts in the whole of 2021, mainly for spamming (see more details in our 2021 Transparency Report). 

Current approach to the misuse of service

As a result of this misuse we are taking the following actions. 

While we will continue to adhere to our reactive moderation policy in most instances, we may occasionally review activity by new users while this incident is ongoing. When we are alerted to correspondence on the subject in question, we will not be taking our usual approach of trying to preserve any valid FOI request contained within broader correspondence. We will instead make a very quick assessment of whether it appears to be a genuine request for information or part of the concerted misuse campaign, in which case the request will be hidden.

The users making these requests will then be banned without warning or notification. The same will apply to any comments being made on existing requests. It will be up to any users that are banned in this process to make a case to us that they are making genuine FOI requests.

This approach is in line with that we have taken in other instances of misuse of our service.

We have also enabled enhanced anti-spam measures on the site, which will help us deal with other instances of misuse more efficiently.

We may never fully understand what exact circumstances instigated this wave of misuse, but it has been instructive, and has helped us formulate new ways to tackle the always surprising means by which our work – to help citizens make valid requests for information in public – can be temporarily derailed.

—
Image: Olga Safronova