With the aim of making large scale Freedom of Information investigations easier for community newsrooms and campaigning organisations, we’ve spent the first half of 2020 developing collaboration tools for WhatDoTheyKnow to speed up and bring others into the FOI management process.
In an initial pilot, 17 contributors saved a journalist 6.5 hours by taking on half of the work of managing responses to requests.
We’re actively looking to partner with membership-driven news organisations or impactful campaign groups to run further pilot projects to help refine the features. If that’s you, please get in touch.
FOI can be hard without dedicated tools
We know FOI can be hard work, especially when you make large batch requests that return a huge amount of data.
While our Pro tools make life easier, much of the work simply involves triaging whether you got a response or just an automated acknowledgement, and whether the authority actually released the information you requested.
After that, you then need to sift through various different formats of data, different understandings of the questions, and follow up with clarifications.
All this comes before you can start analysing the data to build up a narrative for a story.
A compelling membership proposition
News organisations are increasingly looking for sustainability by offering memberships – where you pay a monthly fee to support the organisation – instead of relying on advertising revenue to support themselves.
Memberships are still a relatively unproven and unexplored area, and organisations are still in the process of discovery over what makes someone want to pay for their news output. Is it just being able to read the stories, or do people want more involvement?
There’s evidence to suggest that members do want to get more involved.
Crowdsourcing some of the work of the FOI process from the membership presents an opportunity to help take some of the load off journalists, while also bringing members into the reporting process so that they value the final output more.
Many hands make light work
With this new functionality, once you’ve made your requests – either individually or as part of a batch – they can be added to a Project. Contributors can then be invited to the project where they are briefed on what the project is about and the tasks they can help with.
Helping to classifying responses
When you’re making FOI requests, each response to each request needs to be read to establish whether the authority has provided the information asked for – a process that is difficult to automate, given the huge variety of language that can be deployed by authorities. With large batch requests this can be a time-consuming process.
Projects creates a pool of responses that need classifying that contributors can work through to take some of the onus off the project owner.
Contributors read the original FOI request and latest response, and then classify its current status appropriately. This doesn’t take much specialist understanding of FOI, so it’s a really easy way to get lots of people to help out.
Helping to extract data
In larger FOI investigations requesters are usually looking to build up a dataset so that they can compare responses from different authorities.
This usually involves lots of spreadsheets, copy & paste, and hours of hard work.
Projects provides dedicated tools to help build this dataset by creating a pool of requests that contributors can extract data points from using structured forms.
Allowing contributors to help build up a dataset that will be used for real-life reporting and research helps them feel more directly involved and connected to the organisation, hopefully adding value to the membership proposition.
Project owners are then able to download the crowdsourced dataset to investigate, using their analysis tools of choice.
What we learned from our pilot
In our pilot project contributors took on 50% of the classification tasks, accounting for 57% of the 14.8 hours overall spent classifying, saving the journalist around 6.5 hours of the administrative work required before she could start reviewing the data releases. This is a clear indication that crowdsourcing key parts of the FOI investigation process can save a significant amount of time.
The journalist we worked with was enthusiastic about using the Projects interface again in the future, even if she wouldn’t be inviting external contributors. She expressed that it would be ideal to collaborate with interns to help sift through classifications and responses.
With an 82% conversion rate from joining to taking action and nearly 40% of contributors returning for more than one session there’s clearly an appetite from contributors to get involved and help out. The contributors we interviewed understood that by helping with menial tasks, they were allowing the journalist more time to focus on work which required specialist expertise.
A potential for global benefit
Through the Nesta Future News Fund we worked with openDemocracy to design and develop WhatDoTheyKnow Projects to support this collaboration, and ran a pilot collaborative project made up from a batch of over 800 FOI requests.
Projects is of course built into Alaveteli – the platform that powers WhatDoTheyKnow and many other FOI sites around the world, so it’s not just going to be of use in the UK, but for every jurisdiction where an Alaveteli site is utilising the Pro add-on.
Image: Duy Pham
Using WhatdoTheyKnow Pro, this project pieced together a nationwide dataset, and generated important stories at both national and local levels.
Sold from Under You, a project from the Bureau of Investigative Journalism, revealed how much publicly-owned property has been sold off across England, as a response to austerity measures. In all, TBIJ discovered that over 12,000 buildings and pieces of land have been disposed of, bringing councils revenue of £9.1 billion — some of which has been spent on staff redundancies.
In collaboration with HuffPost, the findings were presented in the form of an interactive map which allows users to explore sales in their own area.
The investigation required a significant amount of data collection via FOI requests to 353 councils, work which was aided by WhatDoTheyKnow Pro. More than 150 people across the UK, including local journalists, took part in the collaborative investigation. As well as HuffPost’s coverage, stories were run in regional news outlets across the country. The project has now been shortlisted for the Data Journalism awards.
We spoke to Gareth Davies from TBIJ to understand how the organisation approached this ambitious project, and what part WhatDoTheyKnow Pro played in it. Here’s what he told us:
“The Bureau has been investigating the local government funding crisis in the UK for the last 18 months. The initial part of this particular investigation focused on the overall financial health of local authorities and used data to determine which were under the most pressure. We then wanted to look at the impact of the funding crisis so teamed up with Hazel Sheffield and her Far Nearer project to look at the public spaces that were being lost as a result.
“At the start of the investigation we undertook a research period to determine what local authorities are required to publish about the buildings and land they own, and how many of them were adhering to those rules.
“We discovered that while councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid.
“Also, two thirds of councils update the same spreadsheet each year, meaning change over time is lost. As a result it became apparent that FOI would be required to obtain the information we were interested in. FOI is a tool we have used for a number of stories, particularly those produced by our Bureau Local team.
“The information we wanted could be divided into two groups: what assets councils were buying and selling, and what they were doing with the money raised when an asset is sold. The research period showed we would need FOI to obtain this data.”
More than 700 FOI requests
“To reduce the risk of requests being refused for exceeding the cost/time limit, we needed to submit two separate requests to each of the 353 local authorities in England.
“Previously I had submitted and managed bulk FOI requests via email. However, staying on top of more than 700 requests would have proven very challenging. I was aware of the WhatDoTheyKnow Pro platform but hadn’t used it before, so thought this would be the ideal opportunity to test it out.
I don’t think I would have achieved that without WhatDoTheyKnow Pro
“It was useful to have up-to-date contact details for each authority and to be able to send the FOI requests in one go. But probably the most useful feature was the way in which WhatDoTheyKnow Pro tracks the status of each request and shows you when the public body in question has exceeded the statutory time limit. This made it a lot easier to stay on top of which councils needed to be chased and when I needed to do it.
“Managing so many FOI requests was still challenging and very time consuming but it would have been much harder by email. The first batch of requests had a success rate of more than 95% and the other (which was more detailed) was around 85%.
“I don’t think I would have achieved that without WhatDoTheyKnow Pro and, as a result, the investigation and interactive map we created would not have been as comprehensive.”
Refining the requests
While councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid
“I sent requests to one of each type of local authority (London borough, metropolitan borough, unitary, county and district) to test what, if any, information councils would provide. The fact that all of those requests were successful meant I had confidence when submitting the batch requests.
“It also allowed me to include additional information in the bulk requests, because some of the test councils erroneously withheld, under Section 40, the identities of companies. As a result I added a note to the request highlighting that this would not be a correct application of that exemption.
“As each response came in I recorded them in two separate spreadsheets — one showing what assets had been bought/sold and another containing information about how the money raised from asset sales had been used. Gradually we built a comprehensive picture of what was happening with public spaces, and that was crucial for our story.”
Bringing about change
There have been tangible results from this investigation.
“The government launched an investigation into the sale of assets by Peterborough Council as a result of this particular story, focusing on that area.
“We submitted our findings to an inquiry currently being held by the Communities and Local Government select committee and were mentioned by name during the first day of oral hearings.
“And last month the Public Accounts Committee announced it would hold a similar inquiry into the sale of public land. Several councils halted their property investment policies after our coverage revealed how much they had borrowed to fund the purchases.”
Thank you very much to Gareth Davies for talking to us about the Sold From Under You project.
Image: Daniel von Appen
‘Sold From Under You’ project used WhatDoTheyKnow Pro
Not long ago, we let you know about the Bureau of Investigative Journalism’s project to map and quantify the scale of properties being sold by councils up and down the country as they try to manage with reduced budgets under austerity.
The investigation, which made use of our WhatDoTheyKnow Pro service to send and manage hundreds of Freedom of Information requests, has now been shortlisted for a Data Journalism Award in the Open Data category.
We’re delighted that our platform for professional users of FOI could be of help; this is just the sort of broad data-driven investigation, requiring FOI requests to multiple authorities, that it was conceived for.
You can read BIJ’s interesting account of their methodology and the impact that the project has had here. We wish them the very best of luck for the award finals next month.