When we started building WhatDoTheyKnow Pro, our toolkit for professional users of FOI, we knew that there was one feature which would be a game-changer for such users: the ability to send a request to multiple authorities at once.
In this blog post, we examine what we wanted the tool to do, how we are guarding against abuse, and finally we’ll give a step by step walkthrough of the interface.
Investigative news stories or in-depth pieces of research often require information from a multitude of different sources. By gathering statistics or information from multiple authorities, journalists, activists and researchers can build up a previously-unseen picture, for example of how widespread a particular problem is, or where there are inequities in medical provision across the country.
It’s something that many professional users of FOI are doing already, usually with the aid of their own homemade spreadsheets on which they keep track of requests made, dates by which replies should be expected, which bodies have responded, which need chasing, and of course the information held in the responses themselves.
The standard WhatDoTheyKnow website already provides several helpful features that you just don’t get with a DIY system: it has all the right email addresses for authorities, for example; it guides you through the FOI process; and it will send you an email reminder when the deadline for response arrives — even taking bank holidays into account.
But we knew that in order for our batch request feature to woo people away from their spreadsheets, it needed to do more than those homebuilt systems, some of which have been refined over several years and work well, even if a bit clunkily, for their owners.
Power and responsibility
One important consideration was uppermost in our minds when it came to batch requests: it costs authorities time and money to respond to each request, and of course that multiplies with batch requests. We are keen to promote responsible use of FOI, so we want to fold appropriate safeguards and guidance into whatever system we build.
As mentioned, with WhatDoTheyKnow Pro we’re focusing on features that are genuinely useful for professional users of FOI, but we also want to help those users make better, more focused requests — ones that are more likely to get useful responses and see the light of day as news stories. So it was important that, in making it simple to send multiple requests, we also help users find the most suitable authorities to send their requests to.
With that in mind, here are some balances we’ve put in place:
- Users are limited in how many batch requests they can send within any one month — so there’s no chance to go too wild.
- There’s a limit to the number of authorities that can be added to a single batch: we set this to be the number of local authorities in the UK, which is a logical sector to survey in this way.
- Before users do a batch mailout, we encourage the sending of an initial request that goes to just a few authorities. This safeguard can reveal where a request is flawed, so for example, if the data you get back is not what you need or in the wrong format, you don’t have to send to the full list all over again.
- We provide advice on cost limits to encourage succinct batch requests.
- Authorities have the facility to report a request which is unsuitable for review by our administrators.
- We’re rolling out the batch request functionality gradually to vetted WhatDoTheyKnow Pro users so that we can gradually learn how people use it in practice, and course-correct as necessary..
Testing and improvements
So far, the batch feature is only available to a select group of test users, who are giving us feedback on how they’re finding it. There’s certainly nothing like having your code being used by real people to help you see where improvements might be made!
That said, it’s been a very gratifying process. With the help of our test users, we’ve seen that the batch request functionality has the potential to be immensely helpful to professional users of FOI; even genuinely game-changing. We are certain that with the sending tools, we’ve created a service that really adds value for this sector.
We’re now in the next phase, and turning our attention to improving the functionality that helps users deal with incoming responses when they come in. This already exists in a basic form, and thanks to our testers, we’ve identified which improvements we need to make. We’re already working on incorporating them. But that is definitely material for the next update — for now, let’s take a look at just how the batch request function works.
Using batch request
There are three parts to making any request, whether you’re doing it yourself or using WhatDoTheyKnow Pro:
- Creating the request
- Managing the responses
- Analysing the results
The batch request functionality builds on our super-simple FOI workflow tools for WhatDoTheyKnow Pro, extending them to make larger investigations much easier.
Creating the Request
The first step is compiling a list of authorities to send the request to. From the compose screen, you might search on a keyword (for example, ‘hospital’, ‘Birmingham’, or ‘Birmingham hospitals’) and then add the authorities you’re interested in.
Each authority is added to a recipient list and WhatDoTheyKnow Pro creates a ‘mail merge’ setup. You’ll see how many authorities you’re writing to in the compose interface.
You can then draft your request. The special `Dear [Authority name]` salutation gets automatically replaced with each of the selected authorities when you send your batch.
Finally, before sending you can choose a privacy duration.
At this point you can either go straight ahead and send your request, or save the draft and come back to it later.
Once you’ve sent your batch request, you’re going to receive a lot of replies from authorities. This is where WhatDoTheyKnow Pro’s functionality really comes into its own, keeping all that clutter out of your email inbox.
Here’s what it looks like: the first thing you’ll see is a high-level progress bar showing you the overall progress of your batch. There are three main states that help you manage the requests in the batch:
- In progress (yellow): This means that there’s no action needed by you – you’re waiting on the authority to respond with an acknowledgement or the information you’ve requested.
- Action needed (red): When a request in the batch receives a response from the authority, you’ll need to check it out. We mark the response as “action needed” for you to review and decide what to do next.
- Complete (green): Once there’s no further action needed – either you’ve got the information you asked for, the authority didn’t have the information, or they’ve refused and you don’t want to challenge them – the request moves to the ‘complete’ state, so you know you don’t need to think about it until you come to analyse the data.
Clicking the title of the batch reveals the individual requests and their progress status. From there, you can click through, read the response and update the status.
Now you’ve got all your data, it’s time to compare the results from different authorities.
Sometimes authorities will reply in the main correspondence.
Other authorities respond with one or more attachments. You can view these inline or download them to your computer.
If you’re dealing with a batch sent to lots of authorities, sometimes it’s easier to just download everything. You can download a Zip file containing all the correspondence and attachments for each request via the “Actions” menu. From there you can pull out the attachments that contain the raw data and plug the numbers or answers in to your spreadsheet so that you can compare across authorities.
You can sign up to WhatDoTheyKnow Pro today and receive 1 month free with the voucher code BLOGMARCH18. Make some requests to try out the FOI workflow tools for professionals, and get in touch to request to join the waiting list for batch access.
If your FOI requests have made the news, let us know! Send us links to your published stories and we’ll throw in an extra month of WhatDoTheyKnow Pro for free. Your stories help us improve WhatDoTheyKnow Pro.
On 22 February 2008, we posted an announcement on this very blog: “the new mySociety Freedom of Information site is now live”.
More than 450,000 requests later, WhatDoTheyKnow is marking its tenth anniversary: as part of the celebrations, we’ve put together a timeline showing how WhatDoTheyKnow has intersected with the history of FOI in the UK since we first gained our right to information in 2005. If nothing else, you may enjoy looking at the site’s rather more primitive design back in its early days.
The past decade has seen legal challenges, contributions to Parliamentary inquiries, and the development of our code for use in other countries (26 and counting). It has proved that an ambitious project can be kept going thanks to the efforts of unpaid but skilled and dedicated volunteers.
Most of all, though, it’s seen you, the general public, submitting requests for information, and sharing the responses you receive. That was always the idea, and, it turns out, it’s a pretty sound one.
The latest installation of our Alaveteli software, OPRAmachine, is an interesting new use of the platform. Rather than covering a whole country, as most of the other Alaveteli installations around the world do, it services just a single US state.
OPRAmachine, which launched in October, allows citizens to request information from state and local governmental agencies in New Jersey, under the Open Public Records Act (OPRA).
We asked Gavin Rozzi, the local journalist who has built and runs OPRAmachine, about the site and its impacts so far:
Why did you decide to set up OPRAmachine?
I developed an interest in New Jersey’s Freedom of Information law in the course of my work as an independent journalist. I created OPRAmachine because there is a void in our state for a statewide Freedom of Information portal.
Historically, New Jersey has gained a reputation as a state with excessive spending on state and local government, along with an enduring “culture” of political corruption, as defined by The New York Times.
I have found that in all too many cases, a lack of transparency and compliance with OPRA disclosure requirements has gone hand in hand with instances of government mismanagement and corruption at the state and local level, some of which have been publicised over the years.
While working in my capacity as an independent journalist, I began making extensive use of the OPRA law in order to study the activities of local governments in New Jersey. I became very familiar with the process and the how the law is effective at bringing about vitally needed transparency through the right it gives citizens to obtain public records.
A year ago, we helped Belgian group Anticor launch the Alaveteli site for Belgium, Transparencia.be, and so far it’s been an incredible success. Since launch, the site’s had almost 60,000 visitors and over 375,000 page views— that’s unusually high for an FOI site in its infancy — and they’ve even brought about a change in local FOI law. So how exactly have Anticor achieved so much in such a short time?
We chatted to Claude, a key member of the Transparencia team, to learn more, and the first thing we discovered was that the stellar visitor numbers were actually news to him. That might give us a clue as to one factor in their success: they’ve been far too busy to check their site analytics!
Fortunately we keep track of traffic on all our partners’ sites (where we have permission to do so), and we were glad to be able to pass on the good news. That done, we were keen to ask what exactly brought so many people to the site — and to see what tips other Alaveteli sites might be able to pick up so that they can enjoy the same level of success. (more…)
Code for Croatia are one of many groups around the world who have used our software Alaveteli to set up a Freedom of Information site — ImamoPravoZnati (“We have the right to know”) was launched in 2015 and has processed more than 4,000 requests.
Many organisations might count that a success and leave it there, but Code for Croatia are clearly a little more ambitious. We’ve been interested to hear about their two latest projects.
A platform for consumer complaints
The Alaveteli code was written to send FOI requests to public authorities. But in essence, it’s little more than a system for sending emails to a predetermined list of recipients, and publishing the whole thread of correspondence online.
Change that list of recipients, and you can create a whole new type of site. Reklamacije (“Complaints”) puts the process of making consumer complaints online. It’s early days as yet — the site’s still in the beta phase, during which testers are putting it through its paces. There have been messages about bank closures, insurance policies… and even the inconsistent quality of the quesadillas at a Mexican food chain.
As we’ve often mentioned here on this blog, our FixMyStreet codebase has been put to many different purposes that require map-based reporting, but as far as we’re aware this is the first non-FOI use of Alaveteli so we’ll be watching with interest. Perhaps it might give you ideas about setting up a similar service elsewhere?
Probing travel expenses
Code for Croatia have also launched a campaign asking users to request details of ministers’ travel expenses.
If that sounds familiar, you’ll be remembering that back in January, AccessInfo did much the same with EU Commissioners and their expenses on the European Union FOI site AskTheEU. We can tentatively say that they were successful, too: it’s been announced that the EU expenses will be proactively published every two months. AskTheEU say they welcome the move ‘cautiously’, so let’s see how it all pans out.
The key to both these campaigns is pre-filled requests that make it really simple for supporters to make a request to a specific politician, while ensuring that the requests aren’t duplicated.
That’s something that Gemma explained how to do in this blog post — it’s a massive benefit of the friendly global Alaveteli community that we can all share insights like this, and especially that other groups can try out initiatives that have proved successful.
Access to information is a particularly powerful tool in countries like Colombia, where corruption is high and vital peace treaties are underway.
To make accessing information easier for citizens and public authorities alike, a group of journalists in Bogotá including DataSketch, have recently set up the Freedom of Information request platform QueremosDatos (the name of which translates as “We want data/information”).
The platform uses our Alaveteli software, and we thoroughly enjoyed working with the Colombian team to set the site up with them.
We asked María Isabel Magaña, who is coordinating the QueremosDatos project, about the site and its impacts so far:
Why did you decide to set up QueremosDatos?
I first learned about Alaveteli in Spain while I was doing my Masters in Investigative Journalism. There I was introduced to the platform TuDerechoASaber.es and to the power that FOIA and transparency had. I just knew Colombia needed something like that, especially since the Congress had just approved the first law regarding this matter.
What made you choose to use Alaveteli software for your platform?
What I love about Alaveteli is how easy it is to use for both users and admins. Designing the platform and making it useful for any type of person was the most attractive feature Alaveteli had. But also, because of the people behind it. Gemma, Gareth, and so many more people were ready to help me achieve this goal despite the different time zones and how much time it took to get it up and running.
What impact do you hope the site has?
It has been almost six months since we launched the site. The impact has been great! We have helped people make 274 requests to more than 6,000 authorities. The Government has been interested in the project and has helped us get in touch with different authorities to help them learn about FOIA and the Colombian law and how to work with people through the platform. Users love it, especially journalists.
Which responses on the site have you been most excited about seeing?
My favourite response so far has been one regarding victims of the Colombian conflict. It was very exciting to get the information because of what it meant for the person who was requesting it, and because of the historical context my country is going through. I also enjoyed seeing the transformation the police had when giving their answers: at first they always sent a response asking the user to call them. After a few explanations, they’re now sending complete answers to the requests via the site.
Do you know of examples where information obtained through the site has been used?
Yes! Journalists have used it mostly in ongoing investigations regarding medicines, drug trafficking, and abortion. Students have used it for journalism classes and homework too.
What are your future plans for QueremosDatos?
We are confirming an alliance with the government to promote the site in public offices and to teach public servants about what the Right to Know is, and their responsibilities with it. This pedagogy will be replicated in universities to teach different users about their power to request information.
Many thanks to María for answering our questions. It’s been great to see the impact the site has already had on authorities and citizens alike, especially the change in behaviour by certain public authorities.
We’re really looking forward to following the project’s continuing work, and wish the team the utmost success in their quest to make Colombia a much more transparent society!
We recently explained how to use pre-written Freedom of Information requests for a campaign. We’re glad to see this being used by AskTheEU, the Alaveteli site for Europe.
Today, AskTheEU launches a campaign to request the travel expenses of EU Commissioners — and they are calling on the public to help submit a total of 168 requests.
No matter what your feeling are towards the EU (let’s not even go there), we hope that everyone is in favour of transparency. AskTheEU’s campaign follows the discovery from a request that Commission President Jean-Claude Juncker spent €63,000 on an air taxi to Turkey for the G20 summit. Naturally, they were keen to know whether this level of spending is replicated across the organisation.
After a two-year battle, AskTheEU’s parent organisation Access Info has established that the European Commission will provide information on Commissioner’s travel expenses, but only in two-month bundles.
They’ve already made a start: after submitting legal appeals and new requests, Access Info won access to a handful of documents about the travel expenses of five Commissioners: these can be seen here.
But there’s plenty more to discover, and that’s where the general public comes in. Thanks to the pre-written requests function, all the hard work is already done: it’s just a matter of picking one or two time periods and submitting the already-composed request.
Anyone can participate by going to the campaign website from today. All requests and responses will be made public on AsktheEU.
Well, we’re delighted to say we’ve been shortlisted for a grant. innovateAFRICA judges will take a few weeks to consider shortlisted applications, and winners will be announced on 30th January.
In the meantime, we thought we’d ask the project’s coordinators, Henry Maina from ARTICLE 19 East Africa and Louise Crow from mySociety, to describe the project in a bit more detail and explain why they think it’s so important.
What is the Alaveteli Professional project?
Louise: Alaveteli Professional is a new toolset that we are currently building as a companion service to our existing Alaveteli software. Alaveteli is mySociety’s open-source platform for making public freedom of information (FOI) requests to public bodies.
Alaveteli Professional will provide journalists and those who use FOI in their work with extra functionality and training to ease the process of raising, managing and interpreting FOI requests, which can be a very time consuming and overwhelming task. This is so that they can spend their valuable time on creating more high-impact journalism and research that holds public authorities to account.
Why bring the Alaveteli Professional project to Kenya?
Henry: The project will enable more Kenyan journalists to utilise one critical tool in their armoury: namely the Freedom of Information law enacted on 31st August 2016. It will also complement our earlier training of 25 journalists on the FOI law.
Louise: innovateAFRICA funding will allow us to bring our newly developed toolset to the Kenyan context. The toolset will have already been tried and tested by journalists in the UK and Czech Republic, so we’ll use examples of how these European journalists have successfully used the platform to generate stories in our trainings with Kenyan media. Simply building these tools is not, on its own, enough. For this reason, the Alaveteli Professional project in Kenya will also involve refining the tools for the Kenyan context, the training of journalists, the creation of support materials and the provision of direct assistance in making and analysing requests.
From ARTICLE 19’s experience of training Kenyan journalists on the new FOI law, how will the Alaveteli Professional project help them with their work?
Henry: ARTICLE 19 has trained journalists on the Freedom of Information laws in Rwanda, Kenya, Uganda and South Sudan. In all our past training, we created manual request protocols and follow-up required making telephone calls. The Alaveteli Professional project will help most journalists to easily file, track and share information about information requests in an easy to engage, review platform.
Why is it so important for journalists and citizens alike to hold authorities to account in Kenya?
Henry: First, journalists and citizens are keen to understand why and how their public servants and officials take decisions. Second, citizens have a right to participate in the management of public affairs and effective engagement is only possible if the citizens are well informed.
Will the project also benefit Kenyan citizens who aren’t journalists?
Louise: Yes. Providing journalists with the extra toolset requires us to first install a standard version of Alaveteli. Therefore, alongside citizens in 25 other countries in the world, Kenyan citizens will be able to use the platform to easily send requests to public authorities, or, as all responses to requests are published on the site, browse already-released information.
Citizens will also benefit even if they don’t use the site at all: they’ll benefit from news stories that expose corruption and mismanagement or missing funds and so on, and thus hold those in power to account.
What impact will the project have on Kenyan information officers/civil servants?
Henry: The project is likely to have great impact on Kenyan information officers and public officials. First, it will offer an objective platform to recognise and reward civil servants that enhance access to information as they will be able to manage requests more efficiently. Second, given the trend in questions, officers will be aware of the information that they can and should proactively disclose to lessen individual requests. Third, it will bolster ARTICLE 19’s ongoing work of training information officers that seeks to help them better understand the law and their obligations under it. Four, most of the government decisions will gain traction with citizens as there will be publicly available information on why and how such decisions were arrived at.
What lasting impact do you hope the project will achieve?
Henry: The Kenyan government will be more transparent and accountable, journalists will be more professional and their stories more credible and factual, allowing the country to entrench democratic values.
Louise: As with all our Alaveteli projects, we hope the project will amplify the power of Freedom of Information and open government, by giving a broad swathe of citizens the information they need to hold those in power to account, and to improve their own lives.
How you can help
So there you are — a little more detail on why we hope to bring Alaveteli Professional to Kenya. We hope you can see the value as much as we can! If so, and you’d like to help support the project, please do tweet with the hashtag #innovateAFRICA: every such public show of support brings us a little closer to winning the grant.
If you are using Freedom of Information for a campaign, and you need to request the same information from several different bodies, or a variety of information from one body, it can be useful to put your supporters to work for you.
We recently profiled the Detention Logs project, which is using Freedom of Information requests to uncover conditions in Australia’s detention centres. Anyone can use the information already uncovered to request further documents or clarify ambiguous facts.
One aspect we didn’t mention is that, in order to make this process as quick and simple as possible, Detention Logs provides users with a pre-written FOI request which they can tweak as necessary before sending off to the relevant authority. This is linked to from a button on the Detention Logs website
This nifty bit of functionality could be useful for all kinds of campaigns. If yours is one of them, read on to discover how to set it up.
As you can see, this unwieldy web address contains all the information that RightToKnow, Australia’s Freedom of Information site, needs in order to create a pre-filled request.The URL tells it who the request should go to, what the title of the request is, and what should go in the main body.
It’s quite simple to create these yourself. Just build the URL up in steps:
- Begin by telling the site that this is a new request: https://www.righttoknow.org.au/new/
- Add a forward slash (/) and then the body you want the request to be sent to (exactly as it is written in the url of the body’s page of the website): https://www.righttoknow.org.au/new/nsw_police_force
- Add a question mark: This tells the website that we are going to introduce a ‘parameter string’. Now our URL looks like this: https://www.righttoknow.org.au/new/nsw_police_force?
- Input a title: we need to indicate that the next part should go into the ‘title’ field, like this: https://www.righttoknow.org.au/new/nsw_police_force?title= and then dictate what the title should be: https://www.righttoknow.org.au/new/nsw_police_force?title=Police%20brutality Notice that if there is a space between words, it should be shown as %20. To make the process of encoding the URLs easier, you can use an encoder tool like this one: http://meyerweb.com/eric/tools/dencoder/
- Input the body of the request, again using ‘%20’ between each word. This is where your URL can become very long! We use the parameter default_letter and the salutation (Dear…) and signoff (Yours…) are automatically wrapped around this by the site, so there’s no need to include them:
So, there you have it. A customised URL that you can set up if you need supporters to send a pre-written request to a specified body or bodies.
As mentioned above, the Detention Logs project used this method to help their supporters request detention centre incident reports, attaching a different URL to each report so that the title would contain the relevant report number. To see the technical details of how they set this up, visit their GitHub page.
Here are some other parameters that can be used in addition to the ones above:
- body – This is an alternative to default_letter which lets you specify the entire body of the request including the salutation and signoff.
- tags – This allows you to add a space-separated list of tags, so for example you can identify any requests made through your campaign or which refer to the same topic. For example, the Detention Logs project used tags like this: &tags=detentionlogs%20incident-number%3A1-2PQQH5
A tag can have a ‘name’ and an optional ‘value’ (created in the form “name:value”). The first tag in the above example is ‘detentionlogs’ (‘name’) and the second tag is ‘incident-number:A1-2PQQH5’ (‘name:value’). The encoder tool above changes the colon to ‘%3’.
If you use this pre-written request tool we’d love to hear about it, so please get in touch if you do.
We’ve just released Alaveteli 0.26! Here are some of the highlights.
Request page design update
After some research in to where people enter the site we decided to revamp the request pages to give a better first impression.
We’ve used the “action bar” pattern from the authority pages to move the request actions to a neater drop-down menu. We’ve also promoted the “follow” button to help other types of users interact with the site.
Since lots of users are entering an Alaveteli on the request pages, it might not be obvious that they too can ask for information. We’ve now made an obvious link to the new request flow from the sidebar of the request pages to emphasise this.
The correspondence bubbles have had a bit of a makeover too. Its now a lot more obvious how to link to a particular piece of correspondence, and we’ve tidied the header so that its a little clearer who’s saying what.
The listing of similar requests in the request page sidebar has been improved after observing they were useful to users.
Also in design-world we’ve added the more modern request status icons, made the search interfaces more consistent and helped prevent blank searches on the “Find an authority” page.
Admin UI Improvements
As an Alaveteli grows it can get trickier to keep an eye on everything that’s happening on the site.
We’ve now added a new comments list so that admins can catch offensive or spam comments sooner.
For the same reasons, we’ve added sorting to the users list and made banned users more obvious.
The CSV import page layout and inline documentation has also been updated.
The new statistics page adds contributor leaderboards to help admins identify users as potential volunteers, as well as a graph showing when site admins hide things to improve the transparency of the site.
Extra search powers
Conversion tracking improvements
The full list of highlights and upgrade notes for this release is in the changelog.
Thanks again to everyone who’s contributed!