1. Data: sharing is caring

    Here are a few stories that were in the news recently. They have two things in common — see if you can you guess what they are:

    If you’ve been keeping up with mySociety’s posts, it’s probably no surprise that the first thing these stories have in common is that they are all based on Freedom of Information requests — in fact, multiple requests made across many bodies.

    We often mention how useful Freedom of Information can be in helping campaigns, journalists or individuals to gather information from a variety of sources, to create a dataset that didn’t exist in one place before.

    Naturally we are all in favour of such stories — but we think the organisations and media behind these requests are missing an extra trick, and that’s the second thing they have in common.

    In every case, it seems the journalist or organisation has submitted their requests, and gathered the data, then written the story — and that’s the end of it. That data is hidden away, and no-one else can access it to verify the story, dig further or to find more interesting leads.

    Journalists understandably gather information for their stories in private so that they aren’t ‘scooped’: this is one factor that led us to develop WhatDoTheyKnow Pro, which allows users to embargo requests and responses until their story has been published. But, once it has, the tool features strong encouragements to put the underlying data live, so that everyone can access it.

    After all, at this stage there is often little benefit to the journalist from keeping the data all to themselves — and lots of potential public good from putting it out in the open. This is also a great way of providing extra credibility for a news item, showing that the facts back it up.

    Here are those stories again, together with details of the requests that informed them:

    • University money laundering fears: The Times surveyed multiple British universities to break this front-page story.
    • Homeless deaths: The Museum of Homelessness put in over 300 FOI requests to gain one part of the information backing up their Dying Homeless project.
    • Bike and walking schemes not delaying ambulances The charity Cycling UK asked 10 ambulance trusts for their data.
    • Councils fail to pay grants  400 FOI requests were issued by the Event Supplier and Services Association to local authorities across England.
    • NHS trusts deny restricting PPE: The BMJ sent Freedom of Information requests to 130 acute, community, integrated, and ambulance trusts.
    • London boroughs using Chinese surveillance tech FOI requests were submitted to all 32 London councils and the next 20 largest UK city councils.

    If you’re a journalist or campaigner yourself, we’d like to suggest that you consider making your data public next time you use FOI like this. Do it via WhatDoTheyKnow Pro, or, if you prefer, do it elsewhere: naturally, the choice is yours, though it’s worth noting that data on WhatDoTheyKnow is easy for people to find, thanks to our excellent search engine positions.

    Pro also has other features that aid journalists in their investigations, including the ability to send batch requests to multiple authorities.

    With our citations tool, you can even link directly to your story, giving it a boost in visibility that is also accelerated by our good standing with Google et al (or other users can link to it in an annotation).

    On the other hand, if you’re just an interested citizen who would love to know more about one or more of those news stories, don’t forget that you could use WhatDoTheyKnow to request the same information, and it will then be public for all to see.

    For example, if the homelessness or the PPE story is of interest to you, you could make an FOI request to your own council or NHS Trust to get the local picture. Once you have the facts, you might take informed action on them: perhaps lobbying your local representatives for change, or contacting the local media if there’s a story to be told.

    And, to help us in our attempts to get more journalists thinking about opening up their data, you could keep your eyes open for stories like these in the future.

    If you see one, perhaps give the writer a friendly nudge to publish their data. After all, they’re using transparency to get their scoop — why not also practice transparency for the good of all?

  2. Introducing WhatDoTheyKnow Projects

    With the aim of making large scale Freedom of Information investigations easier for community newsrooms and campaigning organisations, we’ve spent the first half of 2020 developing collaboration tools for WhatDoTheyKnow to speed up and bring others into the FOI management process.

    In an initial pilot, 17 contributors saved a journalist 6.5 hours by taking on half of the work of managing responses to requests.

    We’re actively looking to partner with membership-driven news organisations or impactful campaign groups to run further pilot projects to help refine the features. If that’s you, please get in touch.

    FOI can be hard without dedicated tools

    We know FOI can be hard work, especially when you make large batch requests that return a huge amount of data.

    While our Pro tools make life easier, much of the work simply involves triaging whether you got a response or just an automated acknowledgement, and whether the authority actually released the information you requested.

    After that, you then need to sift through various different formats of data, different understandings of the questions, and follow up with clarifications.

    All this comes before you can start analysing the data to build up a narrative for a story.

    A compelling membership proposition

    News organisations are increasingly looking for sustainability by offering memberships – where you pay a monthly fee to support the organisation – instead of relying on advertising revenue to support themselves.

    Memberships are still a relatively unproven and unexplored area, and organisations are still in the process of discovery over what makes someone want to pay for their news output. Is it just being able to read the stories, or do people want more involvement?

    There’s evidence to suggest that members do want to get more involved.

    Crowdsourcing some of the work of the FOI process from the membership presents an opportunity to help take some of the load off journalists, while also bringing members into the reporting process so that they value the final output more.

    Many hands make light work

    With this new functionality, once you’ve made your requests – either individually or as part of a batch – they can be added to a Project. Contributors can then be invited to the project where they are briefed on what the project is about and the tasks they can help with.

    Screenshot of Project Homepage

    Helping to classifying responses

    When you’re making FOI requests, each response to each request needs to be read to establish whether the authority has provided the information asked for – a process that is difficult to automate, given the huge variety of language that can be deployed by authorities. With large batch requests this can be a time-consuming process.

    Projects creates a pool of responses that need classifying that contributors can work through to take some of the onus off the project owner.

    2up of Project Classify page

    Contributors read the original FOI request and latest response, and then classify its current status appropriately. This doesn’t take much specialist understanding of FOI, so it’s a really easy way to get lots of people to help out.

    Helping to extract data

    In larger FOI investigations requesters are usually looking to build up a dataset so that they can compare responses from different authorities.

    This usually involves lots of spreadsheets, copy & paste, and hours of hard work.

    Projects provides dedicated tools to help build this dataset by creating a pool of requests that contributors can extract data points from using structured forms.

    Allowing contributors to help build up a dataset that will be used for real-life reporting and research helps them feel more directly involved and connected to the organisation, hopefully adding value to the membership proposition.

    Screenshot of Project Extract page

    Project owners are then able to download the crowdsourced dataset to investigate, using their analysis tools of choice.

    Screenshot of downloaded Dataset

    What we learned from our pilot

    In our pilot project contributors took on 50% of the classification tasks, accounting for 57% of the 14.8 hours overall spent classifying, saving the journalist around 6.5 hours of the administrative work required before she could start reviewing the data releases. This is a clear indication that crowdsourcing key parts of the FOI investigation process can save a significant amount of time.

    The journalist we worked with was enthusiastic about using the Projects interface again in the future, even if she wouldn’t be inviting external contributors. She expressed that it would be ideal to collaborate with interns to help sift through classifications and responses.

    With an 82% conversion rate from joining to taking action and nearly 40% of contributors returning for more than one session there’s clearly an appetite from contributors to get involved and help out. The contributors we interviewed understood that by helping with menial tasks, they were allowing the journalist more time to focus on work which required specialist expertise.

    A potential for global benefit

    Through the Nesta Future News Fund we worked with openDemocracy to design and develop WhatDoTheyKnow Projects to support this collaboration, and ran a pilot collaborative project made up from a batch of over 800 FOI requests.

    Projects is of course built into Alaveteli – the platform that powers WhatDoTheyKnow and many other FOI sites around the world, so it’s not just going to be of use in the UK, but for every jurisdiction where an Alaveteli site is utilising the Pro add-on.

    Image: Duy Pham

  3. Case study: Sold From Under You by the Bureau of Investigative Journalism

    Using WhatdoTheyKnow Pro, this project pieced together a nationwide dataset, and generated important stories at both national and local levels.

    Sold from Under You, a project from the Bureau of Investigative Journalism, revealed how much publicly-owned property has been sold off across England, as a response to austerity measures. In all, TBIJ discovered that over 12,000 buildings and pieces of land have been disposed of, bringing councils revenue of £9.1 billion — some of which has been spent on staff redundancies.

    In collaboration with HuffPost, the findings were presented in the form of an interactive map which allows users to explore sales in their own area.

    The investigation required a significant amount of data collection via FOI requests to 353 councils, work which was aided by WhatDoTheyKnow Pro. More than 150 people across the UK, including local journalists, took part in the collaborative investigation. As well as HuffPost’s coverage, stories were run in regional news outlets across the country. The project has now been shortlisted for the Data Journalism awards.

    We spoke to Gareth Davies from TBIJ to understand how the organisation approached this ambitious project, and what part WhatDoTheyKnow Pro played in it. Here’s what he told us:


    “The Bureau has been investigating the local government funding crisis in the UK for the last 18 months. The initial part of this particular investigation focused on the overall financial health of local authorities and used data to determine which were under the most pressure. We then wanted to look at the impact of the funding crisis so teamed up with Hazel Sheffield and her Far Nearer project to look at the public spaces that were being lost as a result.

    “At the start of the investigation we undertook a research period to determine what local authorities are required to publish about the buildings and land they own, and how many of them were adhering to those rules.

    “We discovered that while councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid.

    “Also, two thirds of councils update the same spreadsheet each year, meaning change over time is lost. As a result it became apparent that FOI would be required to obtain the information we were interested in. FOI is a tool we have used for a number of stories, particularly those produced by our Bureau Local team.

    “The information we wanted could be divided into two groups: what assets councils were buying and selling, and what they were doing with the money raised when an asset is sold. The research period showed we would need FOI to obtain this data.”

    More than 700 FOI requests

    “To reduce the risk of requests being refused for exceeding the cost/time limit, we needed to submit two separate requests to each of the 353 local authorities in England.

    “Previously I had submitted and managed bulk FOI requests via email. However, staying on top of more than 700 requests would have proven very challenging. I was aware of the WhatDoTheyKnow Pro platform but hadn’t used it before, so thought this would be the ideal opportunity to test it out.

    I don’t think I would have achieved that without WhatDoTheyKnow Pro

    “It was useful to have up-to-date contact details for each authority and to be able to send the FOI requests in one go. But probably the most useful feature was the way in which WhatDoTheyKnow Pro tracks the status of each request and shows you when the public body in question has exceeded the statutory time limit. This made it a lot easier to stay on top of which councils needed to be chased and when I needed to do it.

    “Managing so many FOI requests was still challenging and very time consuming but it would have been much harder by email. The first batch of requests had a success rate of more than 95% and the other (which was more detailed)  was around 85%.

    “I don’t think I would have achieved that without WhatDoTheyKnow Pro and, as a result, the investigation and interactive map we created would not have been as comprehensive.”

    Refining the requests

    While councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid

    “I sent requests to one of each type of local authority (London borough, metropolitan borough, unitary, county and district) to test what, if any, information councils would provide. The fact that all of those requests were successful meant I had confidence when submitting the batch requests.

    “It also allowed me to include additional information in the bulk requests, because some of the test councils erroneously withheld, under Section 40, the identities of companies. As a result I added a note to the request highlighting that this would not be a correct application of that exemption.

    As each response came in I recorded them in two separate spreadsheets — one showing what assets had been bought/sold and another containing information about how the money raised from asset sales had been used. Gradually we built a comprehensive picture of what was happening with public spaces, and that was crucial for our story.”

    Bringing about change

    There have been tangible results from this investigation.

    The government launched an investigation into the sale of assets by Peterborough Council as a result of this particular story, focusing on that area.

    “We submitted our findings to an inquiry currently being held by the Communities and Local Government select committee and were mentioned by name during the first day of oral hearings.

    “And last month the Public Accounts Committee announced it would hold a similar inquiry into the sale of public land. Several councils halted their property investment policies after our coverage revealed how much they had borrowed to fund the purchases.”

    Thank you very much to Gareth Davies for talking to us about the Sold From Under You project.

    Find out more about WhatDoTheyKnow Pro.

    Image: Daniel von Appen

  4. Bureau of Investigative Journalists shortlisted for Open Data award

    ‘Sold From Under You’ project used WhatDoTheyKnow Pro

    Not long ago, we let you know about the Bureau of Investigative Journalism’s project to map and quantify the scale of properties being sold by councils up and down the country as they try to manage with reduced budgets under austerity.

    The investigation, which made use of our WhatDoTheyKnow Pro service to send and manage hundreds of Freedom of Information requests, has now been shortlisted for a Data Journalism Award in the Open Data category.

    We’re delighted that our platform for professional users of FOI could be of help; this is just the sort of broad data-driven investigation, requiring FOI requests to multiple authorities, that it was conceived for.

    You can read BIJ’s interesting account of their methodology and the impact that the project has had here. We wish them the very best of luck for the award finals next month.