With the aim of making large scale Freedom of Information investigations easier for community newsrooms and campaigning organisations, we’ve spent the first half of 2020 developing collaboration tools for WhatDoTheyKnow to speed up and bring others into the FOI management process.

In an initial pilot, 17 contributors saved a journalist 6.5 hours by taking on half of the work of managing responses to requests.

We’re actively looking to partner with membership-driven news organisations or impactful campaign groups to run further pilot projects to help refine the features. If that’s you, please get in touch.

FOI can be hard without dedicated tools

We know FOI can be hard work, especially when you make large batch requests that return a huge amount of data.

While our Pro tools make life easier, much of the work simply involves triaging whether you got a response or just an automated acknowledgement, and whether the authority actually released the information you requested.

After that, you then need to sift through various different formats of data, different understandings of the questions, and follow up with clarifications.

All this comes before you can start analysing the data to build up a narrative for a story.

A compelling membership proposition

News organisations are increasingly looking for sustainability by offering memberships – where you pay a monthly fee to support the organisation – instead of relying on advertising revenue to support themselves.

Memberships are still a relatively unproven and unexplored area, and organisations are still in the process of discovery over what makes someone want to pay for their news output. Is it just being able to read the stories, or do people want more involvement?

There’s evidence to suggest that members do want to get more involved.

Crowdsourcing some of the work of the FOI process from the membership presents an opportunity to help take some of the load off journalists, while also bringing members into the reporting process so that they value the final output more.

Many hands make light work

With this new functionality, once you’ve made your requests – either individually or as part of a batch – they can be added to a Project. Contributors can then be invited to the project where they are briefed on what the project is about and the tasks they can help with.

Helping to classifying responses

When you’re making FOI requests, each response to each request needs to be read to establish whether the authority has provided the information asked for – a process that is difficult to automate, given the huge variety of language that can be deployed by authorities. With large batch requests this can be a time-consuming process.

Projects creates a pool of responses that need classifying that contributors can work through to take some of the onus off the project owner.

Contributors read the original FOI request and latest response, and then classify its current status appropriately. This doesn’t take much specialist understanding of FOI, so it’s a really easy way to get lots of people to help out.

Helping to extract data

In larger FOI investigations requesters are usually looking to build up a dataset so that they can compare responses from different authorities.

This usually involves lots of spreadsheets, copy & paste, and hours of hard work.

Projects provides dedicated tools to help build this dataset by creating a pool of requests that contributors can extract data points from using structured forms.

Allowing contributors to help build up a dataset that will be used for real-life reporting and research helps them feel more directly involved and connected to the organisation, hopefully adding value to the membership proposition.

Project owners are then able to download the crowdsourced dataset to investigate, using their analysis tools of choice.

What we learned from our pilot

In our pilot project contributors took on 50% of the classification tasks, accounting for 57% of the 14.8 hours overall spent classifying, saving the journalist around 6.5 hours of the administrative work required before she could start reviewing the data releases. This is a clear indication that crowdsourcing key parts of the FOI investigation process can save a significant amount of time.

The journalist we worked with was enthusiastic about using the Projects interface again in the future, even if she wouldn’t be inviting external contributors. She expressed that it would be ideal to collaborate with interns to help sift through classifications and responses.

With an 82% conversion rate from joining to taking action and nearly 40% of contributors returning for more than one session there’s clearly an appetite from contributors to get involved and help out. The contributors we interviewed understood that by helping with menial tasks, they were allowing the journalist more time to focus on work which required specialist expertise.

A potential for global benefit

Through the Nesta Future News Fund we worked with openDemocracy to design and develop WhatDoTheyKnow Projects to support this collaboration, and ran a pilot collaborative project made up from a batch of over 800 FOI requests.

Projects is of course built into Alaveteli – the platform that powers WhatDoTheyKnow and many other FOI sites around the world, so it’s not just going to be of use in the UK, but for every jurisdiction where an Alaveteli site is utilising the Pro add-on.

—

Image: Duy Pham

Over the last few months, we’ve been working with Hackney Council to design and make a Freedom of Information management system, imaginatively named FOI For Councils — and last time we left you with our pre-development thoughts. Well, now it’s up and running.

With this project, we had two main aims:

• first, to make the process really slick and easy to use for citizens;
• second, to reduce the quantity of FOI requests submitted, relieving some pressure on the Information Officers at the receiving end.

The solution we came up with achieves both those aims, and there’s one feature in particular that we’re super-excited about.

Case Management Integration

One of the development decisions taken early on was for the system to be a very lightweight layer, largely powered by the new Infreemation case management system that Hackney were in the process of commissioning.

Infreemation is targeted primarily at Information Officers, so there was no use in reinventing the wheel and building a heavy backend for our own FOI for Councils software.

Instead we built the FOI request process, using our experience in designing for citizens, and submitted the data directly to Infreemation using their API. This means that every request goes straight in to the case management system used by Information Officers, with no need for double entry; a set-up we’re very familiar with from our work integrating council systems with FixMyStreet.

Information Officers respond to the FOI request through Infreemation, and when they publish the response to Infreemation’s disclosure log, FOI for Councils can pull that response into its innovative suggestions engine, which we’ll discuss shortly.

All this means that Information Officers get to use the tools that are designed directly with them in mind, but citizens get the best experience possible for the process at hand, rather than trying to battle the typical generic forms offered by one-size-fits-all solutions.

Citizens

On the user side of things we managed to reduce the process to a maximum of 6 screens for the entire process.

Throughout the whole user journey we ask for only three details: name; email address and then the actual request for information.

Each screen provides contextual help along the way, maximising the chances that the FOI request will be well-formed by the time it gets submitted.

Making the process intuitive for the people using it is a key factor in building citizens’ trust in an authority. Too often we see complex forms with terrible usability that almost seem designed to put people off exercising their rights.

Information Officers

So far, so good. But for us, the most interesting piece of the process is the suggestions step.

Before the citizen submits their request to the authority, we scan the text for keywords to see if anything matches the pool of already-published information.

If we find any matches, we show the top three to the citizen to hopefully answer their question before they submit it to the authority. This helps the citizen avoid a 20-day wait for information that they might be able to access immediately. If the suggestions don’t answer their question, the citizen can easily continue with their request.

Suggestions also benefit the authority, by reducing workload when requests can be answered by existing public information.

We’ve tried to make this suggestions step as unobtrusive as possible, while still adding value for the citizen and the authority.

The suggestions system is driven by two sources:

• Manually curated links to existing information
• The published answers to previous FOI requests

The curated links can be added to the suggestions pool by Information Officers where they spot patterns in the information most commonly requested, or perhaps in response to current events.

The intelligent part of the system though, is the automated suggestions.

As FOI for Councils integrates with the Infreemation case management system, we can feed the suggestion pool with the anonymised responses to previous requests where the authority has published them to the disclosure log.

By doing this the authority is making each FOI response work a little harder for them. Over time this automatic suggestion pool should help to reduce duplicate FOI requests.

FOI for Councils also analyses the number of times each suggestion is shown, clicked, and even whether the suggestion has prevented any additional FOI requests being made.

This allows Information Officers to see which information is being asked for, but where existing resources aren’t providing the information necessary to the citizen.

Future Plans

We’ll be keeping a keen eye on how this works out for Hackney, and we’ll be sure to report back with any insights.

As you’ll know if you read our first blog post from this project, we did originally envision a platform that would process Subject Access Requests as well as FOI. In the end this proved beyond the resources we had available for this phase of work.

For us, this has been a really instructive piece of work in showing how authorities can commission process-specific services that connect together to give everyone a better user experience.

As with most mySociety projects, FOI for Councils is open source which improves its transparency, flexibility and accountability.

If you’re responsible for managing FOI requests or data protection in your own public sector body and you’d like to talk about project in more detail, please get in touch at hello@mysociety.org.

We’ve just released Alaveteli 0.26! Here are some of the highlights.

Request page design update

After some research in to where people enter the site we decided to revamp the request pages to give a better first impression.

We’ve used the “action bar” pattern from the authority pages to move the request actions to a neater drop-down menu. We’ve also promoted the “follow” button to help other types of users interact with the site.

Since lots of users are entering an Alaveteli on the request pages, it might not be obvious that they too can ask for information. We’ve now made an obvious link to the new request flow from the sidebar of the request pages to emphasise this.

The correspondence bubbles have had a bit of a makeover too. Its now a lot more obvious how to link to a particular piece of correspondence, and we’ve tidied the header so that its a little clearer who’s saying what.

The listing of similar requests in the request page sidebar has been improved after observing they were useful to users.

Also in design-world we’ve added the more modern request status icons, made the search interfaces more consistent and helped prevent blank searches on the “Find an authority” page.

Admin UI Improvements

As an Alaveteli grows it can get trickier to keep an eye on everything that’s happening on the site.

We’ve now added a new comments list so that admins can catch offensive or spam comments sooner.

For the same reasons, we’ve added sorting to the users list and made banned users more obvious.

The CSV import page layout and inline documentation has also been updated.

More stats!

The wonderful team at the OpenAustralia Foundation contributed an epic pull request to revamp the statistics page.

The new statistics page adds contributor leaderboards to help admins identify users as potential volunteers, as well as a graph showing when site admins hide things to improve the transparency of the site.

Extra search powers

As if that wasn’t enough, the the OpenAustralia Foundation team also added a new advanced search term to be able to find requests to public authorities with a given tag.

Conversion tracking improvements

We’re constantly asking how we can improve Alaveteli. In order to answer that question meaningfully, we need good tracking of what happens when people make requests on the site. When we tried to track down whether people coming from social media sites were more likely to make a request than those coming from other sources, we found a problem with the way we tracked ‘conversions’ – the process of getting all the way through making a request. We were using a particular URL to mark the end of that process in Google Analytics. The issue was that sometimes, requesters would share that URL with other people, causing us to record the same request being made multiple times. We’re now using a little bit of javascript to make sure we only record the conversion when a new request is actually being made.

The full list of highlights and upgrade notes for this release is in the changelog.

Thanks again to everyone who’s contributed!

We’ve just released Alaveteli 0.24! Here are some of the highlights.

Admin Tools

We’ve added better management for censor rules in the admin interface. Previously, only request and user censor rules could be managed; now you can manage rules for authorities and global rules that get applied to everything.

We’ve added support deleting incoming messages in bulk on a request page. This is useful if you’re experiencing spam to the holding pen. You can zap them all in a couple of clicks.

Design

Facebook is a big driver of traffic to Alavetelis. We’ve added support for a specific opengraph image for pages when shared on Facebook. You’ll need to add a custom version for your site in your theme. Take a look at the upgrade notes for more info.

As Zarino has already mentioned, we’ve improved handling of long translations in the banner.

Long Term Support

Alaveteli 0.24 adds support for the most current versions of Debian and Ruby. Jessie has security support until May 2018 and Long Term Support until May 2020.

The full list of highlights and upgrade notes for this release is in the changelog.

Thanks again to everyone who’s contributed!

We’ve just released Alaveteli 0.22!

Its a big one, with over 400 commits and lots of general improvements.

Accessibility

Luke Bacon improved the design and accessibility of the search form.

Code Quality

Code quality came top of the list at AlaveteliCon 2015. This release includes contributions from James McKinney, Henare Degan, Caleb Tutty, Petter Reinholdtsen and Gorm Eriksen – all helping to clean up code, make Alaveteli work even better and make it easier to translate.

Performance

WhatDoTheyKnow’s Public Authority pages were suffering, so we took a dive in to the code around this area. Improving it had a huge impact on the page we were looking at and should have benefits across the application.

Maintenance & Security

We’ve added fixes for CVE-2015-3225, CVE-2015-3227 and CVE-2015-1840, and updated xapian-full-alaveteli for Ruby 2.1 compatibility.

HTML Widgets

We’re always trying to think of more ways to promote an Alaveteli site, so this release includes new feature developed by Jody McIntyre for the AskTheEU Alaveteli site.

When enabled, each request has a “Create a widget for this request” action available in the sidebar.

Any visitor can copy the iframe embed code to paste on their own website.

It’s just a flick of the switch to enable widgets for all requests on your site!

Waving Goodbye to the Past

Alaveteli 0.22 is the last release to support Ruby 1.8.

We’ve got an upgrading guide available, and we’re always here to help on the alaveteli-dev mailing list.

You can see the full list of highlights and upgrade notes in the changelog.

Thanks again to everyone who’s contributed.