WhatDoTheyKnow Pro is the paid-for, premium, version of our Freedom of Information service WhatDoTheyKnow.com, designed for journalists, academics, campaigners and others whose needs exceed what our free service provides.
Features available to Pro users include the ability to delay publication of requests and responses; and to make requests to multiple authorities at the same time via the batch request tool.
We’ve just made it much easier for Pro users to add relevant bodies to a batch request via a list of authorities within specific categories.
Our database contains FOI contact addresses for more than 42,000 authorities. Using our service saves you from having to source appropriate contact details yourself, and we’ve now made it even quicker and easier to make batch FOI requests.
Since WhatDoTheyKnow Pro’s launch, creating a batch request has involved searching for bodies and adding them individually to the batch. WhatDoTheyKnow’s fantastic volunteers curate over 200 categories to help users on the main site to explore and navigate the UK authorities subject to FOI, and we’ve now incorporated these listings into WhatDoTheyKnow Pro’s batch tool.
As a result, requests are more likely to be sent to the bodies that hold the information being requested, and the number of requests sent to inappropriate bodies is minimised.
We’ve been trialling this feature in a limited beta period for a while, and thanks to our funding from the Swedish Postcode Foundation we were able to work with handlingar.se to iron out some bugs and performance issues before making it available to all Pro users.
We hope the new feature will aid some great cross-authority research, while helping to ensure that requests are targeted to appropriate bodies.
Let us know if there are additional categories you’d like us to add!
Image: Nick Youngson (CC BY-SA 3.0)
Here are a few stories that were in the news recently. They have two things in common — see if you can you guess what they are:
- Money laundering fears as universities accept £52m in cash (Times)
- Almost 1,000 UK homeless deaths recorded in 2020 (ITV)
- Covid bike and walking schemes do not delay ambulances, trusts say (Guardian)
- Councils fail to pay £1.3bn of emergency Covid business grants (Times)
- Covid-19: NHS trusts deny restricting PPE during pandemic (BMJ)
- Half of London boroughs found using Chinese surveillance tech linked to Uighur abuses (Japan Times)
If you’ve been keeping up with mySociety’s posts, it’s probably no surprise that the first thing these stories have in common is that they are all based on Freedom of Information requests — in fact, multiple requests made across many bodies.
We often mention how useful Freedom of Information can be in helping campaigns, journalists or individuals to gather information from a variety of sources, to create a dataset that didn’t exist in one place before.
Naturally we are all in favour of such stories — but we think the organisations and media behind these requests are missing an extra trick, and that’s the second thing they have in common.
In every case, it seems the journalist or organisation has submitted their requests, and gathered the data, then written the story — and that’s the end of it. That data is hidden away, and no-one else can access it to verify the story, dig further or to find more interesting leads.
Journalists understandably gather information for their stories in private so that they aren’t ‘scooped’: this is one factor that led us to develop WhatDoTheyKnow Pro, which allows users to embargo requests and responses until their story has been published. But, once it has, the tool features strong encouragements to put the underlying data live, so that everyone can access it.
After all, at this stage there is often little benefit to the journalist from keeping the data all to themselves — and lots of potential public good from putting it out in the open. This is also a great way of providing extra credibility for a news item, showing that the facts back it up.
Here are those stories again, together with details of the requests that informed them:
- University money laundering fears: The Times surveyed multiple British universities to break this front-page story.
- Homeless deaths: The Museum of Homelessness put in over 300 FOI requests to gain one part of the information backing up their Dying Homeless project.
- Bike and walking schemes not delaying ambulances The charity Cycling UK asked 10 ambulance trusts for their data.
- Councils fail to pay grants 400 FOI requests were issued by the Event Supplier and Services Association to local authorities across England.
- NHS trusts deny restricting PPE: The BMJ sent Freedom of Information requests to 130 acute, community, integrated, and ambulance trusts.
- London boroughs using Chinese surveillance tech FOI requests were submitted to all 32 London councils and the next 20 largest UK city councils.
If you’re a journalist or campaigner yourself, we’d like to suggest that you consider making your data public next time you use FOI like this. Do it via WhatDoTheyKnow Pro, or, if you prefer, do it elsewhere: naturally, the choice is yours, though it’s worth noting that data on WhatDoTheyKnow is easy for people to find, thanks to our excellent search engine positions.
Pro also has other features that aid journalists in their investigations, including the ability to send batch requests to multiple authorities.
With our citations tool, you can even link directly to your story, giving it a boost in visibility that is also accelerated by our good standing with Google et al (or other users can link to it in an annotation).
On the other hand, if you’re just an interested citizen who would love to know more about one or more of those news stories, don’t forget that you could use WhatDoTheyKnow to request the same information, and it will then be public for all to see.
For example, if the homelessness or the PPE story is of interest to you, you could make an FOI request to your own council or NHS Trust to get the local picture. Once you have the facts, you might take informed action on them: perhaps lobbying your local representatives for change, or contacting the local media if there’s a story to be told.
And, to help us in our attempts to get more journalists thinking about opening up their data, you could keep your eyes open for stories like these in the future.
If you see one, perhaps give the writer a friendly nudge to publish their data. After all, they’re using transparency to get their scoop — why not also practice transparency for the good of all?
If you were putting in a claim for benefits, challenging an accusation in court or phoning in sick to your employer, would you expect your local authority to be checking your social media presence?
How do you think a stranger might assess you as a parent, were they to skim over any public posts on your Facebook page? If you’ve been on a protest recently, would you be comfortable knowing that your local council was combing through any photos you’ve shared?
A Freedom of Information investigation by Privacy International, using WhatDoTheyKnow Pro, has discovered that a significant number of local authorities — 62.5% of those responding to their FOI requests — habitually monitor citizens’ Facebook or other social media profiles to gather intelligence.
What’s more, the majority have no policy in place or measurement of how often and to what extent these investigations occur.
If this concerns you, the first thing you should do is check that your social media privacy controls are up to date. Then you might like to go and read Privacy International’s full report, as well as checking how (or whether) your own local authority has responded to their requests for information.
And finally, you can join Privacy International’s call for stronger guidelines from the Investigatory Powers Commissioner.
Just… maybe think twice about putting it in a public Facebook post?
We’re only joking, of course. Or half joking.
Issues like this need to be shared far and wide. But as Privacy International point out, there are already sobering instances from abroad of threats to those following anti-government accounts. With so many completely unexpected changes to the status quo recently, can we say for certain that it could never happen here?
Image: John Schnobrich
In a strike for transparency, journalist Jenna Corderoy has secured the release of documents from the European Research Group (ERG), the pro Brexit lobby of which Jacob Rees-Mogg is a prominent member.
For more than a year, Jenna has been striving to ensure that the facts around Brexit — and the funding that drives it — reach the public domain: she also broke the now-infamous revelations about Vote Leave’s campaign overspending.
The release of material such as this into the public domain is beneficial to all, as it means that public debate is based on facts rather than conjecture. FOI can be a vital tool in ensuring that the documents shaping our society’s future direction are available for scrutiny.
On this latest release, a piece by Jenna and Peter Geoghegan reports:
“The ERG is part-funded by subscriptions paid out of MPs’ parliamentary expenses. As a consequence the group has to supply samples of its research for scrutiny to the Independent Parliamentary Standards Authority [IPSA] to ensure public money is being properly spent and not used for party political campaigning.”
Using WhatDoTheyKnow Pro — our service for professional users of FOI, which among other features, allows users to hold off from putting request correspondence in public until a story has been published — in January 2018 a request was made to IPSA to see these materials.
The request’s WhatDoTheyKnow page is now public. IPSA initially argued that the release of these materials was exempted under section 43 of the FOI Act as it would prejudice the commercial interests of the ERG, whose research is ordinarily available only to those paying a £2,000 annual subscription.
Subsequently Jenna referred this refusal to the Information Commissioner, who upheld the decision. Determined that the public has the right to see the research, Jenna and Peter did not leave the matter there, taking it to an information tribunal.
The tribunal made the final decision that the material must indeed be released, vindicating the effort and determination Jenna put into pursuing this request and stating that to make the documents available would:
“further transparency, accountability and public trust with respect to the working of Parliament”.
As a result, the documents will be made available on 11 July — keep an eye on OpenDemocracy for news of their release — and we’ll make sure we update the annotations on the original request as further details unfold. Meanwhile, you can see the full tribunal decision here.
Image: Udur Akdemir
Using WhatdoTheyKnow Pro, this project pieced together a nationwide dataset, and generated important stories at both national and local levels.
Sold from Under You, a project from the Bureau of Investigative Journalism, revealed how much publicly-owned property has been sold off across England, as a response to austerity measures. In all, TBIJ discovered that over 12,000 buildings and pieces of land have been disposed of, bringing councils revenue of £9.1 billion — some of which has been spent on staff redundancies.
In collaboration with HuffPost, the findings were presented in the form of an interactive map which allows users to explore sales in their own area.
The investigation required a significant amount of data collection via FOI requests to 353 councils, work which was aided by WhatDoTheyKnow Pro. More than 150 people across the UK, including local journalists, took part in the collaborative investigation. As well as HuffPost’s coverage, stories were run in regional news outlets across the country. The project has now been shortlisted for the Data Journalism awards.
We spoke to Gareth Davies from TBIJ to understand how the organisation approached this ambitious project, and what part WhatDoTheyKnow Pro played in it. Here’s what he told us:
“The Bureau has been investigating the local government funding crisis in the UK for the last 18 months. The initial part of this particular investigation focused on the overall financial health of local authorities and used data to determine which were under the most pressure. We then wanted to look at the impact of the funding crisis so teamed up with Hazel Sheffield and her Far Nearer project to look at the public spaces that were being lost as a result.
“At the start of the investigation we undertook a research period to determine what local authorities are required to publish about the buildings and land they own, and how many of them were adhering to those rules.
“We discovered that while councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid.
“Also, two thirds of councils update the same spreadsheet each year, meaning change over time is lost. As a result it became apparent that FOI would be required to obtain the information we were interested in. FOI is a tool we have used for a number of stories, particularly those produced by our Bureau Local team.
“The information we wanted could be divided into two groups: what assets councils were buying and selling, and what they were doing with the money raised when an asset is sold. The research period showed we would need FOI to obtain this data.”
More than 700 FOI requests
“To reduce the risk of requests being refused for exceeding the cost/time limit, we needed to submit two separate requests to each of the 353 local authorities in England.
“Previously I had submitted and managed bulk FOI requests via email. However, staying on top of more than 700 requests would have proven very challenging. I was aware of the WhatDoTheyKnow Pro platform but hadn’t used it before, so thought this would be the ideal opportunity to test it out.
I don’t think I would have achieved that without WhatDoTheyKnow Pro
“It was useful to have up-to-date contact details for each authority and to be able to send the FOI requests in one go. But probably the most useful feature was the way in which WhatDoTheyKnow Pro tracks the status of each request and shows you when the public body in question has exceeded the statutory time limit. This made it a lot easier to stay on top of which councils needed to be chased and when I needed to do it.
“Managing so many FOI requests was still challenging and very time consuming but it would have been much harder by email. The first batch of requests had a success rate of more than 95% and the other (which was more detailed) was around 85%.
“I don’t think I would have achieved that without WhatDoTheyKnow Pro and, as a result, the investigation and interactive map we created would not have been as comprehensive.”
Refining the requests
While councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid
“I sent requests to one of each type of local authority (London borough, metropolitan borough, unitary, county and district) to test what, if any, information councils would provide. The fact that all of those requests were successful meant I had confidence when submitting the batch requests.
“It also allowed me to include additional information in the bulk requests, because some of the test councils erroneously withheld, under Section 40, the identities of companies. As a result I added a note to the request highlighting that this would not be a correct application of that exemption.
“As each response came in I recorded them in two separate spreadsheets — one showing what assets had been bought/sold and another containing information about how the money raised from asset sales had been used. Gradually we built a comprehensive picture of what was happening with public spaces, and that was crucial for our story.”
Bringing about change
There have been tangible results from this investigation.
“The government launched an investigation into the sale of assets by Peterborough Council as a result of this particular story, focusing on that area.
“We submitted our findings to an inquiry currently being held by the Communities and Local Government select committee and were mentioned by name during the first day of oral hearings.
“And last month the Public Accounts Committee announced it would hold a similar inquiry into the sale of public land. Several councils halted their property investment policies after our coverage revealed how much they had borrowed to fund the purchases.”
Thank you very much to Gareth Davies for talking to us about the Sold From Under You project.
Find out more about WhatDoTheyKnow Pro.
Image: Daniel von Appen
WhatDoTheyKnow Pro, mySociety’s subscription service offering extra tools for journalists and other professional users of FOI, has been running in the UK for just about two years.
During that time we’ve launched, worked closely with users to refine the service, and — happily — watched it play a vital part in the making of several important data-driven news stories, on topics as diverse as Brexit campaign funding and the results of austerity cuts on councils. Journalists, in particular, have appreciated tools such as the ability to send and manage bulk requests to multiple authorities; and the embargo tool that keeps requests and responses hidden until the story has been published.
Now, thanks to support from Adessium Foundation, we are able to bring the same benefits to countries across Europe, and — we hope — some additional synergies that will be borne of organisations working across boundaries. The same functionality that extends WhatDoTheyKnow into the Pro version will be available to FOI sites run on the Alaveteli platform, under the name Alaveteli Pro.
The ultimate aim is to enable journalists, campaigners and citizens in Europe to make greater and more effective use of their right to access information; and in particular to generate public interest stories and campaigns that will hold power to account.
We’ll be focusing on three areas in order to achieve this aim:
- We’ll give selected existing Alaveteli sites in Europe the technical help they need to upgrade to the Pro version;
- We’ll be helping organisations in three new European jurisdictions to launch brand new Alaveteli sites, making access to information easier for citizens in these countries. The first site will be launched by VVOJ from the Netherlands.
- We’ll encourage cross-border collaborations between journalists and organisations using the sites (both the existing ones and the new ones) to investigate stories that span more than one EU country.
So watch this space: we’ll be sure to keep you posted as the work progresses. The planned start date is next month, and the project is set to run for three years.
We’re looking forward to sharing stories resulting from this initiative once they start rolling out, and supporting the incredible work that journalists do in putting them together.
Image: Emiliano Vittoriosi
‘Sold From Under You’ project used WhatDoTheyKnow Pro
Not long ago, we let you know about the Bureau of Investigative Journalism’s project to map and quantify the scale of properties being sold by councils up and down the country as they try to manage with reduced budgets under austerity.
The investigation, which made use of our WhatDoTheyKnow Pro service to send and manage hundreds of Freedom of Information requests, has now been shortlisted for a Data Journalism Award in the Open Data category.
We’re delighted that our platform for professional users of FOI could be of help; this is just the sort of broad data-driven investigation, requiring FOI requests to multiple authorities, that it was conceived for.
You can read BIJ’s interesting account of their methodology and the impact that the project has had here. We wish them the very best of luck for the award finals next month.
Back in March, we flagged up the ‘batch request’ feature we’d been working on for the WhatDoTheyKnow Pro service. Batch requests are now switched on for every WhatDoTheyKnow Pro subscriber, by default.
Batch enables users to send the same Freedom of Information request to several bodies at once, and we spent a substantial amount of time building and testing it because we wanted to be confident that the feature wouldn’t be abused — or if it was, that we could catch irregular behaviour.
Part of that testing has involved making the feature available to a limited number of WhatDoTheyKnow Pro subscribers, and loosely monitoring how it was used. We’re glad to say that during this four-month period, the activity was all acceptable.
However, we also realised that we should tighten up our terms and conditions to reflect our expectations around usage of Batch, and add some advice to our Help pages about making responsible and effective requests, both of which we’ve now done. We’ve also added some automatic notifications that will alert the team when multiple batch requests are made, so that we can check that everything is in order.
If you think Batch might be useful in your own work or campaigning, and you’d like to find out more about WhatDoTheyKnow Pro, you can do that here.
Image: Ankush Minda
When we started building WhatDoTheyKnow Pro, our toolkit for professional users of FOI, we knew that there was one feature which would be a game-changer for such users: the ability to send a request to multiple authorities at once.
In this blog post, we examine what we wanted the tool to do, how we are guarding against abuse, and finally we’ll give a step by step walkthrough of the interface.
Investigative news stories or in-depth pieces of research often require information from a multitude of different sources. By gathering statistics or information from multiple authorities, journalists, activists and researchers can build up a previously-unseen picture, for example of how widespread a particular problem is, or where there are inequities in medical provision across the country.
It’s something that many professional users of FOI are doing already, usually with the aid of their own homemade spreadsheets on which they keep track of requests made, dates by which replies should be expected, which bodies have responded, which need chasing, and of course the information held in the responses themselves.
The standard WhatDoTheyKnow website already provides several helpful features that you just don’t get with a DIY system: it has all the right email addresses for authorities, for example; it guides you through the FOI process; and it will send you an email reminder when the deadline for response arrives — even taking bank holidays into account.
But we knew that in order for our batch request feature to woo people away from their spreadsheets, it needed to do more than those homebuilt systems, some of which have been refined over several years and work well, even if a bit clunkily, for their owners.
Power and responsibility
One important consideration was uppermost in our minds when it came to batch requests: it costs authorities time and money to respond to each request, and of course that multiplies with batch requests. We are keen to promote responsible use of FOI, so we want to fold appropriate safeguards and guidance into whatever system we build.
As mentioned, with WhatDoTheyKnow Pro we’re focusing on features that are genuinely useful for professional users of FOI, but we also want to help those users make better, more focused requests — ones that are more likely to get useful responses and see the light of day as news stories. So it was important that, in making it simple to send multiple requests, we also help users find the most suitable authorities to send their requests to.
With that in mind, here are some balances we’ve put in place:
- Users are limited in how many batch requests they can send within any one month — so there’s no chance to go too wild.
- There’s a limit to the number of authorities that can be added to a single batch: we set this to be the number of local authorities in the UK, which is a logical sector to survey in this way.
- Before users do a batch mailout, we encourage the sending of an initial request that goes to just a few authorities. This safeguard can reveal where a request is flawed, so for example, if the data you get back is not what you need or in the wrong format, you don’t have to send to the full list all over again.
- We provide advice on cost limits to encourage succinct batch requests.
- Authorities have the facility to report a request which is unsuitable for review by our administrators.
- We’re rolling out the batch request functionality gradually to vetted WhatDoTheyKnow Pro users so that we can gradually learn how people use it in practice, and course-correct as necessary..
Testing and improvements
So far, the batch feature is only available to a select group of test users, who are giving us feedback on how they’re finding it. There’s certainly nothing like having your code being used by real people to help you see where improvements might be made!
That said, it’s been a very gratifying process. With the help of our test users, we’ve seen that the batch request functionality has the potential to be immensely helpful to professional users of FOI; even genuinely game-changing. We are certain that with the sending tools, we’ve created a service that really adds value for this sector.
We’re now in the next phase, and turning our attention to improving the functionality that helps users deal with incoming responses when they come in. This already exists in a basic form, and thanks to our testers, we’ve identified which improvements we need to make. We’re already working on incorporating them. But that is definitely material for the next update — for now, let’s take a look at just how the batch request function works.
Using batch request
There are three parts to making any request, whether you’re doing it yourself or using WhatDoTheyKnow Pro:
- Creating the request
- Managing the responses
- Analysing the results
The batch request functionality builds on our super-simple FOI workflow tools for WhatDoTheyKnow Pro, extending them to make larger investigations much easier.
Creating the Request
The first step is compiling a list of authorities to send the request to. From the compose screen, you might search on a keyword (for example, ‘hospital’, ‘Birmingham’, or ‘Birmingham hospitals’) and then add the authorities you’re interested in.
Each authority is added to a recipient list and WhatDoTheyKnow Pro creates a ‘mail merge’ setup. You’ll see how many authorities you’re writing to in the compose interface.
You can then draft your request. The special `Dear [Authority name]` salutation gets automatically replaced with each of the selected authorities when you send your batch.
Finally, before sending you can choose a privacy duration.
At this point you can either go straight ahead and send your request, or save the draft and come back to it later.
Once you’ve sent your batch request, you’re going to receive a lot of replies from authorities. This is where WhatDoTheyKnow Pro’s functionality really comes into its own, keeping all that clutter out of your email inbox.
Here’s what it looks like: the first thing you’ll see is a high-level progress bar showing you the overall progress of your batch. There are three main states that help you manage the requests in the batch:
- In progress (yellow): This means that there’s no action needed by you – you’re waiting on the authority to respond with an acknowledgement or the information you’ve requested.
- Action needed (red): When a request in the batch receives a response from the authority, you’ll need to check it out. We mark the response as “action needed” for you to review and decide what to do next.
- Complete (green): Once there’s no further action needed – either you’ve got the information you asked for, the authority didn’t have the information, or they’ve refused and you don’t want to challenge them – the request moves to the ‘complete’ state, so you know you don’t need to think about it until you come to analyse the data.
Clicking the title of the batch reveals the individual requests and their progress status. From there, you can click through, read the response and update the status.
Now you’ve got all your data, it’s time to compare the results from different authorities.
Sometimes authorities will reply in the main correspondence.
Other authorities respond with one or more attachments. You can view these inline or download them to your computer.
If you’re dealing with a batch sent to lots of authorities, sometimes it’s easier to just download everything. You can download a Zip file containing all the correspondence and attachments for each request via the “Actions” menu. From there you can pull out the attachments that contain the raw data and plug the numbers or answers in to your spreadsheet so that you can compare across authorities.
If your FOI requests have made the news, let us know! Send us links to your published stories and we’ll throw in an extra month of WhatDoTheyKnow Pro for free. Your stories help us improve WhatDoTheyKnow Pro.
Image: Peretz Partensky (CC by-sa/2.0)
If you happen to have visited the WhatDoTheyKnowPro page recently, you might have noticed that we’ve changed from offering free beta testers’ accounts to rolling out the payment interface.
Yes, we’ve launched! But not with a bang: having quietly introduced the payment option, we wanted to wait for a while and check that there were no issues before making a public announcement. Consider this that public announcement.
Just a reminder at this stage, in case you haven’t followed along with previous blog posts: WhatDoTheyKnowPro is a Freedom of Information toolkit which provides extra help with sending and organising requests. It’s designed to provide extra functionality for journalists and other people who use FOI in their jobs; if you’re a standard user of WhatDoTheyKnow, there’s nothing to pay and there never will be. We’ll always keep the site, and all its current functionality, free for everyone.
The question of price
Launching WhatDoTheyKnowPro provoked an interesting debate on where to position it, pricewise.
It’s not that we’ve never put a price on any of our services before: mySociety is a social enterprise, and we charge for some use of our APIs; our council clients pay us for FixMyStreet Pro; you can buy transit-time maps on Mapumental. These sources of income are just part of what help us to provide our core citizen-facing services for free.
And we’re hardly trailblazers in that respect: in fact, it was fascinating to read the Knight Foundation’s recent report Scaling Civic Tech and see how common it is in our sector to rely on a variety of revenue streams, from user donations to philanthropic grants, to paid-for services.
But while we may have experience in charging for our services, it’s definitely the first time we’ve had to price up a Freedom of Information toolkit for journalists and professionals!
What’s it worth?
We were effectively in the position of many an enterprise startup: with a market proposition that doesn’t exist in this exact form anywhere else. How do you know how to price something in those circumstances? Set it too low and you could miss out on important revenue; too high and you’ll alienate potential customers, many of whom are freelance journalists paying for their work tools out of their own pockets.
Still, this project has been a process of tackling problems and questions thoughtfully — from deciding which features to include in this initial version, to debating how to encourage journalists to link back to the news stories they’ve created once they’re live. Perhaps we could bring the same approach to pricing.
Ask the experts
Fortunately, with over 100 beta testers, we had a pool of users who knew the service well enough to be able to give an expert opinion on how much value it was bringing them. Sending out a survey brought some very useful responses, not to mention feedback about what our beta users liked and what they’d appreciate in future roll-outs.
That said, we’re well aware of research indicating that people are not always experts on what they are actually willing to pay.
And of course, it makes sense that different people will attach different values to a service, depending not only on their own finances but, in this case all sorts of other factors such as how frequently they use FOI and how accustomed they are to paying for technology.
Faced with responses to the survey that ranged between a suggested price of £1.00 a month to £50.00, that was, in some strange sense, reassuring to know.
To be fair, those were the extremes. There was a good consensus in the middle and that helped us decide on an introductory price of £10 a month. We’ll assess this after a few months to see whether it’s bringing the number of sign-ups we expect.
Open for business
If you’re a journalist or someone who uses FOI in your work, you can now go and give WhatDoTheyKnowPro a go! We hope you’ll let us know how you find it.
Image: Sven Scheuermeier (Unsplash)