We’ve just released version 0.32 of Alaveteli, our open source platform for running Freedom Of Information sites. Here are some of the highlights.
Making correspondence threads easier to navigate
Thanks to our designers, it’s now possible to collapse individual messages in a correspondence thread in order to focus on just the parts you’re trying to read. Plus you can quickly collapse (or expand) all the messages in the thread using the “Collapse all” and “Expand all” links from the “Actions” menu.
Alaveteli Pro users gain the additional benefit of a redesigned sidebar which allows for easier navigation of lengthy correspondence and avoids having to scroll to the top of the request thread to update its status. See Martin’s full explanation here.
Better password security
We’ve started enforcing stricter password length constraints wherever a password is set or updated to help users keep their accounts secure. And we’re also using a stronger encryption method for storing password data, using bcrypt rather than the older SHA1 algorithm to obscure the actual password. (Be sure to run the rake task documented in the release upgrade notes to upgrade secure password storage for all existing users.)
You can read more about what this does and why it’s important if you’re interested in the technical details behind this upgrade.
Authorities not subject to FOI law
We’ve adopted WhatDoTheyKnow’s
foi_notag for authorities to indicate that although the authority is listed on the site, it is not legally subject to FOI law. This could be for advocacy purposes – if it’s felt an authority should be covered by legislation – or where the authority has agreed to respond on a voluntary basis.
foi_notag now causes an extra message to appear under the authority’s name on their page and on all related requests, and removes language about legal responsibilities to reply from the messages sent to users.
To improve the UI, we’ve made a similar change for authorities with the
eir_onlytag to make it clearer that such authorities are only accepting requests about the environment.
(Don’t worry admins, you don’t need to remember all this – we’ve updated the documentation on the edit page to reflect the new functionality!)
Improvements for site admins
We’ve made it easier for admins to ban users who sign up to post spam links in their profile. There’s now a “Ban for spamming” button which is available on the user edit page or as soon as you expand the user’s details in the listing rather than having to manually edit user metadata.
We’ve also made it harder to leave requests flagged as vexatious (or “not_foi”) in an inconsistent state. Previously the site just assumed that vexatious requests would always be hidden. Now the admin interface enforces the hiding of vexatious requests by showing warnings when a request is set as vexatious while it’s visible on the site, and prevents the updated request from being saved until a valid state is selected.
And last but not least – introducing the new Announcements feature!
Easier popup banner management
Site admins will be relieved to hear that they can now update the popup banner message on the site without needing to schedule developer time.
This feature supports multi-language sites so if you set the announcement for your main (default) language, it will appear across all language versions that you have not added a specific translation for.
You can set announcements that will only be seen by fellow administrators when they visit the summary page. (If you’re running a Pro site, you can also have announcements that will only be seen by your Pro admins.)
Announcements for Pro users appear as a carousel at the top of their dashboard. So far we’ve used it on WhatDoTheyKnow Pro to publicise new features, offer discount codes, and encourage people to share their published stories with us.
The full list of highlights and upgrade notes for this release is in the changelog.
Thanks again to everyone who’s contributed!
Last year, we launched WhatDoTheyKnow Pro, our service for journalists and other professional users of Freedom of Information.
As it’s a new venture, we’re keen to track whether it’s achieving everything we’d hoped for when first planning the service. One of the targets which we set, as a measure of success, is the number of impactful press stories generated by its use.
What might count as impactful? Well, that’s obviously up for debate, but loosely we’d say that we’re looking for news stories that have a wide readership, and uncover previously unknown facts, offer new insights, or bring about change.
Stories in the news
A couple of recent stories, generated through WhatDoTheyKnow Pro, have ticked at least a few of those boxes. At mySociety, we keep a position of political neutrality — our services are available to everyone of any persuasion, and we don’t campaign on any political issue — so we present these stories not to comment on their substance, but to note that they certainly fit the criteria above.
Brexit is clearly one of the most vital stories of our day, here in the UK, and while many might feel that we’ve had a surfeit of commentary on the issue, we can only benefit from understanding the facts.
One of WhatDoTheyKnow Pro’s earliest users, Jenna Corderoy, broke two important stories on that topic. First, that UK parliamentary standards watchdog IPSA is investigating Jacob Rees Mogg’s hard Brexit European Research Group over their second bank account and ‘informal governance structure’. This also ran in the Daily Mail.
Secondly (and this progresses a previous story about expenses – also uncovered thanks to Jenna’s use of WhatDoTheyKnow Pro) there is the widely-reported story that the Electoral Commission had misinterpreted laws around campaigning expenses, allowing Vote Leave to overspend. This was picked up by the BBC and Guardian, among many others.
No matter which side of the Brexit debate you support, hopefully you will agree that it benefits society as a whole to have the facts out in the open.
From FOI request to the national news
As a last thought: it’s interesting to us to see how a story grows from one or more FOI requests into something that hits the national news platforms. We think these stories were broken using a tried and true method that goes like this:
- As a journalist, campaigner or researcher, you might be investigating a topic. Perhaps you’ve heard a rumour that you’re hoping to substantiate, or perhaps you’re inquiring more deeply into a story that’s already in the air. Using FOI, you can retrieve facts and figures to bolster your investigation.
- Once you have a story, you can publish it in a smaller publication like Open Democracy, testing the water to see if it gains any traction.
- If the story is well-received there, it’s easy to approach larger outlets with the proof (in the form of FOI responses) underpinning it.
If you’re a journalist or you use FOI in your professional life and you’d like to try WhatDoTheyKnow Professional for yourself, then head over to whatdotheyknow.com/pro. Put in the code WELCOME18 when you sign up, and you’ll get 25% off your first month’s subscription.
Image: Roman Kraft
WhatDoTheyKnow Pro is our Freedom of Information service for journalists, and campaigners, and we’ve recently rolled out some major changes to the request sidebar to make reading, navigating, and classifying Pro requests a lot easier.
Since the very first Alpha version of WhatDoTheyKnow Pro we’ve been receiving feedback from our users, which we have been feeding directly into our future development plans. The sidebar changes are the first round of changes that have come as result of direct Pro user feedback, and there will be more to follow.
In WhatDoTheyKnow a member of the public can send a Freedom of Information request to an authority, which they receive in the form of an email. The request, as well as any replies or follow up from the requester, or the authority, are published on WhatDoTheyKnow. If the request is made by a Pro user, they have the added option of making a request private for a limited time.
In the request process we observed the following:
- A new response from an authority goes to the bottom of the thread (the bottom of the page)
- The user interface for updating the status of a request is located at the top of the page (a request’s’ status is a way of keeping track of where it is in the request process – for example ‘awaiting response’, ‘needs clarification’, or ‘refused’)
- A longstanding or complicated request will often consist of many, many messages. So scrolling to the bottom of the page to read the most recent response, then back to the top to update its status involves a lot of interaction that we can remove.
We can’t say for sure that a user will always be at the bottom of the request thread when updating the status of a request, but we can safely assume they are sometimes.
For these changes we set ourselves the following goals:
- Speed up the process of updating the status of requests
- Improve the experience of navigating requests with a long history.
In previous research we established that a typical workflow for dealing with request responses is:
Get reply → Read reply → Take action (typically reply, or update the status of the request)
It’s a short, three step process, but a busy user catching up with a backlog may do this hundreds of times a day, so if we can optimise this workflow we can save a lot of time and frustration.
So what have we done?
For desktop users we’ve made the sidebar controls (where the ‘update status’ button is) “sticky”, so it will follow you as you scroll up and down the page, meaning you can update the request status from any position on the page. This really helps as requests get longer, as you no longer need to scroll back to the top to classify the latest response.
We’ve added new message navigation buttons. This is to enable you to move through a request thread message-by-message by clicking the up and down arrow buttons, or using the arrow keys on your keyboard. We’ve also added a counter so that it’s easier to see where you are in the list, and to go back and forth to specific messages.
We’ve also taken this opportunity to make some key information about the privacy of your request visible at all times (this was previously hidden behind a click), and to tweak the design of the sidebar – making it easier to read and removing some visual noise.
More to do
We’re looking at a way to add similar functionality to mobiles and other small screen devices. As screen space is limited it will require a separate design process.
We’re aware that the problems we’re trying to solve aren’t unique to our Pro customers, so if the features work, and are well received, we’ll be making a similar feature available to all WhatDoTheyKnow users in the future.
Keeping in mind that as our public users have different needs to our Pro users there are some design challenges to overcome beforehand. For example – the public request page has more features to help less-frequent users, because we’re keen to ensure that everyone can participate in the FOI process, not just experts. Conversely, Pro users are by their very nature more likely to require less guidance. We’re going to need to do more research on this shortly.
Got some feedback?
Whether you’re a WhatDoTheyKnow Pro customer or not, we’d love feedback on this feature — or any other. Drop us an email to firstname.lastname@example.org.
Back in March, we flagged up the ‘batch request’ feature we’d been working on for the WhatDoTheyKnow Pro service. Batch requests are now switched on for every WhatDoTheyKnow Pro subscriber, by default.
Batch enables users to send the same Freedom of Information request to several bodies at once, and we spent a substantial amount of time building and testing it because we wanted to be confident that the feature wouldn’t be abused — or if it was, that we could catch irregular behaviour.
Part of that testing has involved making the feature available to a limited number of WhatDoTheyKnow Pro subscribers, and loosely monitoring how it was used. We’re glad to say that during this four-month period, the activity was all acceptable.
However, we also realised that we should tighten up our terms and conditions to reflect our expectations around usage of Batch, and add some advice to our Help pages about making responsible and effective requests, both of which we’ve now done. We’ve also added some automatic notifications that will alert the team when multiple batch requests are made, so that we can check that everything is in order.
If you think Batch might be useful in your own work or campaigning, and you’d like to find out more about WhatDoTheyKnow Pro, you can do that here.
Image: Ankush Minda
Back in February, we postponed celebrations for the tenth anniversary of our Freedom of Information site WhatDoTheyKnow, because of extreme weather conditions. Gales and snow had shut down public transport; guests from further afield were unsure they’d make it to our London venue.
Little did we know that our rescheduled event would face its own exceptional circumstances. Not only did we find ourselves at the other end of the thermometer, with the hottest temperatures of the year thus far, but we were also competing with England playing a World Cup match.
All this being so, we were glad to see so many people turn out to help us celebrate — though it was pointed out that the Venn diagram between FOI enthusiasts and football fans might have a fairly small overlap. We’ll get our Research department on to that, at some point.
We’d decked the room with some rather unique — but meaningful — decorations: a selection of information uncovered by WhatDoTheyKnow’s users over the past decade (see photo, above), and screenshots of the many FOI sites running on our Alaveteli software around the world.
Talking of Alaveteli sites, we were delighted to welcome among our guests Andreas Pavlou who previously worked with AccessInfo, the organisation who run Europe FOI site AskTheEU, and Claude Archer from Anticor, who run Belgium’s Transparencia.be.
Claude actually drove, without incident, all the way from Brussels — only to scrape against the kerb right outside Newspeak House and get a flat tyre. But mySociety is not just a collection of weedy developers, you know. Well, ok, fair enough, until recently we were just that — but since Georgie joined our ranks a few weeks ago, it turns out that we now have a highly practical colleague who can change a wheel. And that’s just what she did.
That drama aside, the party went smoothly.
There were cakes, of course.
Then some mingling. It was great to meet many WhatDoTheyKnow users, and especially those who employ the site for their campaigns.
And on to the presentations. WhatDoTheyKnow’s Richard Taylor spoke about what it is like to be a volunteer on the site, and the kind of tasks they deal with in keeping the service available for everyone, not to mention free from litigation. You can read his talk here.
We interviewed Francis Irving, who was one of two people to suggest that mySociety build an FOI site when we had an open call for ideas — and who then went on to help build it. Much as we enjoy mySociety’s current status as an established organisation, Francis’ descriptions of our early days and ‘punk’ attitude were rather beguiling.
Finally, investigative journalist Jenna Corderoy shared her experiences of being one of the first people to try WhatDoTheyKnow Pro, our toolkit for FOI professionals and activists. In a stroke of incredible timing, she mentioned a story which she’d been working on, saying that she knew it would break soon, but it might be weeks or even a year before it did.
We woke up the next morning to hear that this very story was the BBC’s main headline for the day. Watch this space, because we’ll be asking Jenna to fill us in with some more background, and we’ll be sure to share it all here on the blog.
Oh, and in case you’re wondering… we did eventually switch the big screen over to the football, and all those Civic Tech geeks did actually get caught up in watching the penalty shoot-out decider.
I guess the Venn diagram stretched a little bit that night.
Thank you so much to everyone who came along: we hope you had as much fun as we did.
I’m Richard Taylor, a member of the volunteer team which administers WhatDoTheyKnow.com on a day to day basis, and I spoke at the event highlighting the broad range of people who have collaborated to make WhatDoTheyKnow a success, and sharing some ideas for the future. Here’s what I said:
I’m someone who wants to see our representative democracy working; that’s why I support what mySociety does; I support giving tools to people to help people engage with our society, how we make decisions about running our society, how we run our public services, our health service, policing, how we organise our cities, how we plan development of new homes and design, or evolve, our transport systems.
I joined WhatDoTheyKnow as a user on the 22nd of July 2008, so almost exactly ten years ago. My first Freedom of Information requests were on policing, for the local Stop and Account policy – as you can see from those kinds of requests I’m keen on transparency and accountability of those we give powers over us. I looked up my early FOI requests and I was rapidly onto my local councillor allowances, details of which weren’t online, and as I’m from Cambridge and there were some very Cambridge requests in there too – on the running of the river – on the regulation of punting – a perennial local issue, and for the terms and charges for grazing on the city’s commons. One of the things I do is campaign for proportional police use of TASERs, I made requests on that subject too.
Within just a few days of joining the site I was sending in lists of public bodies to add to the system; and shortly after that I was invited on to the administration team so I didn’t have to bug developer Francis Irving, or the volunteers who’d already started to help running the site, including John Cross, Alex Skene, Tony Bowden to do things like add new bodies, but could make changes myself.
The volunteer team
Mine is the same route many of our volunteers took to joining the team running the site in the early years; those making lots of good proposals for bodies to add, or making other suggestions were invited to help out. The way we’ve found new volunteers has changed a little over time, and we have had to keep topping up the pool of volunteers as people have moved on. We started to approach users of the site who were making helpful annotations assisting other users, and who were making great use of the site themselves. We found Ganesh Sittampalam and Doug Paulley that way, both of whom have put huge amounts of time into developing the site, the service. Latterly we’ve moved to advertising for new team members and seeking applications from those who want to join us, and that’s brought us some of our current active volunteers, Michael Bimmler, and Kieran Casey-McEvoy for example.
Volunteers have put in an enormous amount of time into running the site. If you put a cash value on that time I’m sure the volunteers would by far be the biggest donor to the site. The site probably wouldn’t exist, and certainly wouldn’t exist in its current form without volunteer input; so many good ideas for websites get built, often with funding to kick them off, but they don’t do what WhatDoTheyKnow has done, and survive, grow, and thrive. Volunteer input has enabled that.
The site certainly has grown and thrived, we now have around seven million users viewing the site per year; according to Google analytics, and 162,000 signed up users. There are approaching half a million request threads on the site now. An interesting aspect of those statistics is the viewing is not focused on a small handful of requests, but rather visitors are spread broadly across the long-tail of requests and released information. In 2016 17% of requests to central government monitored bodies went via our service; but the vast majority of requests, 88%, go to bodies where central government don’t track FOI request statististics.
The volunteers I’ve mentioned already, plus Helen Cross and Alastair Sloan, have put substantial chunks of time into running the site. There are many others too including Rob McDowell, Ben Harris, Gavin Chait and Peter Williams. The volunteers supporting the service have not just come from the volunteer team, the trustees who’re ultimately responsible for the site are volunteers too, ten years ago mySociety was more of a volunteer based organisation than it is now, trustee Amandeep Rehlon was dealing with the finances on a volunteer basis, we’ve had great moral and policy guidance from Manar Hussain and Owen Blacker, and the chair of the trustees, another volunteer, James Cronin.
We have been amazingly lucky with the volunteers we’ve attracted to the administration team. Doug Paulley is an incredible activist and campaigner on disability rights, and so many of the others are legal and information rights experts, activists and campaigners in their own rights.
Volunteers are only part of the story, we wouldn’t be able to do what we do, and what we want to do without the institutional support of mySociety, and the organisation’s brilliant staff. When the initial developer and project manager Francis Irving moved on he was succeeded by a series of great lead developers, Robin Houston, Seb Bacon, and now Louise Crow and other staff team members, currently Gareth Rees, Graeme Porteous, Liz Colnan …(See Github for the full list of contributors to the code!) the site is supported by the whole mySociety team, including designers Zarino Zappier and Martin Wright, Abi Broom, who runs the show, Gemma Moulder – events organiser from our perspective, who also works on spreading services based on WhatDoTheyKnow around the world, and mySociety’s communications person Myf Nixon. Thanks are also due to ten years’ of mySociety sysadmins including Sam Pearson,Ian Chard, and in the early days volunteers who’d keep things running, Adam McGreggor, and Alex Smith.
A key WhatDoTheyKnow volunteer was Francis Davey who was our volunteer legal advisor for many years. Francis Davey’s top piece of advice which I recall was to avoid court. We’ve pretty much succeeded to date-with that. One of the key roles of the volunteer team is to run what is a relatively legally risky site without getting sued and consequently, probably, taking down not just WhatDoTheyKnow but the rest of mySociety too.
We deal with a lot of defamation claims, personal information takedown requests, and an array of more obscure legal challenges.
As well as trying to avoid getting annihilated via legal processes a key aspect of our approach to running the site is we try our utmost to run it responsibly. What those involved didn’t do is find a legally friendly jurisdiction and anonymously just let the system loose to run unmanaged and unchecked. We’re real accountable people who respond to concerns from all comers, individuals, public bodies, our own users, about what’s published on our site.
What are we doing by running our site?
We’re doing a lot more than just helping users make a request for information to a public body. We’re activists, we’re promoting running our society in a transparent, inclusive, accountable, way, not just by lobbying, making speeches, writing articles, but by doing something, by running our site.
Running our service promotes Freedom of Information and other access to information laws; people come across our site when searching for information they’re seeking; we show what can be obtained by publishing requests and responses; others might find the information they’re seeking directly, or see that they can make a similar request, perhaps adapting a request that’s been made elsewhere to their local public bodies..
Anyone can make a Freedom of Information request by private email to a public body. I’d find that potentially a bit of a selfish action, incurring cost to the public for a response only I might see, but making a request via WhatDoTheyKnow to obtain information which should be accessible to the public automatically makes that information accessible to anyone who searches for it, anyone who Googles for the information. Even if a requester doesn’t themselves do something with the information released by making a request via WhatDoTheyKnow.com they’ve enabled others to do so. You’re often doing public good just by making a request via WhatDoTheyKnow.com (though do see our advice on making responsible and effective requests).
WhatDoTheyKnow makes something which would otherwise be quite challenging for many people – getting a FOI request and response online – easy. I’m sure only a fraction of users of our site would have taken the time to write a blog about their request, and update it with the response, if they had to do that manually.
A big benefit of making a request on WhatDoTheyKnow.com is many people are already using our site and watching for responses; if you make a request to a local council on WhatDoTheyKnow.com the chances are your local journalists are tracking requests to the local council and they’ll be alerted to any response.
At WhatDoTheyKnow we’re an independent third party, we’re not the requestor and we’re not the public body. This can be useful when there’s a dispute about a response to a request, if a public body denies receiving it for example. We’d love to work more closely with the regulator, the Information Commissioner’s office, we’d love them to use our service more to help them in their role in enforcing the law. Often just having a request on our site can help people get a response, good public bodies really care about the impression those visiting their pages on our site get. Lots of public bodies will get in touch with us if they don’t like the way a request has been classified by one of our users for example.
A really big advantage having information released via our service is people can cite it when they take action based on it, be that action a blog post, an article in the media, an academic publication, or a letter to an MP. You can show, again using WhatDoTheyKnow.com as an independent third party, where the information you are relying on has come from, giving more weight, more credibility, to whatever it is you’re doing, your lobbying, your journalism, your research. WhatDoTheyKnow, and mySociety more broadly, has been in the business of enabling better informed debate and higher quality journalism well before “fake news” entered our lexicon.
We’re always looking for new bodies to add to our site, the database of public bodies which is behind the site keeps growing, we’re now at over 23,000 public bodies. That compares to about 450 public bodies listed on the Gov.uk website, and just 305 in the latest “Public Bodies” report by the Cabinet Office. The big difference is made up by schools, GP surgeries and NHS dentists, all of which are subject to FOI; we also list groups of organisations like companies owned by local government – public bodies in terms of the Freedom of Information Act but all but invisible to central government.
I said we were in the business of activism; changing society by doing things. One big part of or Freedom of Information law related activism is listing bodies on our site which are not, or not yet, subject to access to information laws. We’ve listed many bodies before they became subject to the Freedom of Information Act, showing the demand for information, and showing the kind of information people want, but couldn’t access. One example was Network Rail which we listed before it became subject to FOI in March 2015, another was the Association of Chief Police Officers .. however that’s now become the National Police Chief’s Council and MPs failed to make that successor body subject to FOI – in that case it’s not a huge problem as they realise they need to be transparent and they voluntarily comply, but, significantly, the Information Commissioner can’t enforce a law which a body is not technically subject to.
There are always more public bodies to add, we list Housing Associations for example, they’re a another class of body which are not subject to FOI, even coroners aren’t subject to FOI which you might find surprising given their important public role in ensuring our society is safe, and more people don’t die in the future for the same, preventable, reasons people have died in the past. We list some coroners, and volunteer Kieran is working on making our coverage comprehensive. Local medical committees; committees of GPs are another set we’re hoping to add soon.
Maintaining the body database is a constant task. Government is constantly reorganising, we try to keep up with changes recently for example, recently, in research councils, and keeping track of NHS reorganisations is a challenge on its own. There have already been 17 requests to London North Eastern Railway Limited, the Government rail operator of last resort which we listed when it took over running trains on the East Coast mainline about ten days ago.
Seeking improvements to laws which impact our service, its users, and the accessibility of public information
As well as our activism we have a record of more traditional lobbying; sharing the experience running our service has given us experience of the operation of access to information law. We took part in the Post-Legislative Scrutiny of the Freedom of Information Act 2000 in 2012 for example, and just a few months ago we responded to a consultation by the Cabinet Office on the Code of Practice which bodies responding to FOI requests have take into account.
In terms of what we’re calling for, we’re not FOI fans specifically, we’d actually rather people didn’t have to make FOI requests, we’re in favour of proactively releasing information and running public services transparently, though that said FOI requests are requests for information people want to know; rather than information which public bodies want to publish so they will probably always have their place.
Why not make public bodies consider proactive publication of information of the sort requested, when dealing with a FOI request? That’s a provision which is in the specialised law on access to datasets but doesn’t apply to access to information requests more broadly.
Timeliness of responses, and timeliness of enforcement action from the Information Commissioner are other key things we campaign on. If you want a copy of a FOI response that’s been made to particular union, lobby group, or journalist and is the information behind the day’s news, surely you should be able to get a copy of it pretty much straight away, and there can be no excuse for a body dithering until the 20 working day deadline. The law requires a prompt response; that aspect of the law needs following and enforcing.
We also want to close loopholes in FOI; one terrible one, is if a public body can think of a class of information and list it on its website with a price for it, it becomes exempt from disclosure for free under FOI. This is clearly open to abuse, fortunately few bodies have misused it too-date, but there are examples – just look at your local council’s list of information they make available for a fee.
Running the Site
Some might be interested to know administration has changed as the site has grown. There’s been a constant improvement of the site’s software to make it easier to run, but that needs to continue so we can cope with it getting bigger without having to increase the volunteer effort exponentially in-line with the site’s growth. We’ve outgrown the team@ mailing list system the site started with; we now separate the support mail from discussions among volunteers, and on top of that there’s a separate discussion of legal matters; so people aren’t overwhelmed.
One challenge we have is the workload, and volunteer input, are both variable. Sometimes there’s a week where you really need someone full time running the site. Sometimes you could firefight the incoming issues in maybe an hour a day, or day a week.
Something we’d like to do is encourage past volunteers to join our monthly calls; join the legal discussion list, volunteers list, drop into the support mailbox and help out on occasion, every little helps, following what we’re doing for a week a couple of times a year might provide some outside, detached, input; help keep us on-track, challenge us, and assist us in spotting drifts in policy / practice.
Ideas for the future
We’re always keen to hear any ideas for what we could be doing better, or differently we welcome input from anyone and everyone who cares about the service in some way. Some of the things we could do improve:
- We could do even better at transparently running the site. We already try to run the site as transparently as we can; if we hide a message, or redact content from correspondence, we make clear where we’ve done so and explain why. We don’t though have a transparency report like Google and Reddit do, reporting on takedown requests, how many there have been, who they’ve come from – individuals, requesters, public bodies, public officials, regulated professionals, and how we responded. Requests for user data. One challenge is sometimes the moral thing to do is not shout about and draw attention to something we’ve taken down too quickly; don’t want to draw attention to taking down something that’s still in Google’s cache for example – if we really believe it shouldn’t be online any more.
- We should do more to highlight excellent, interesting and influential, uses of the site. It would be great to have ways within the system to note when responses have been used by others, cited in Parliament, resulted in a news story, or if someone has analysed responses from a range of public bodies around the country for example.
- We have volunteers, but there is no real community of users around the site, or around our lobbying activities, or, to the extent there has been in the past, a community – around mySociety any more. There’s an opportunity there..
- I think we have a duty to be careful with the way the WhatDoTheyKnow pro-service is used. Anyone can sign up for a Gmail account and make requests; but we are doing more than Gmail to encourage and enable FOI requests, and not least the pro system is built on a largely volunteer built and maintained database. Use to-date has apparently been good, and we have a general principle of not spending time discussing hypothetical situations, but, in running the free site as volunteers we’ve always been mindful of the impact of our actions on our reputation, and the reputation of Freedom of Information law itself. For example we ask those considering bulk requests if they’ve carefully selected the set of bodies to make their request to, if the request could be made to a central body rather than lots of local bodies, if a sampling exercise would suffice instead of asking perhaps hundreds of bodies, and we advise on making clear requests in the first instance to reduce the need for clarifications – saving public bodies and requesters time and effort. [Update: following the event we agreed to update our House Rules to include a reference to our advice on making responsible and effective requests|]).
- Lastly, on sustainable funding for the site, ideally I think this would be though a handful of media organisations, campaign groups, or other bodies paying for a pro-service; which would hopefully give them great value in terms of organising FOI requests, prompting them to chase up late requests, saving time finding contact details and easily making bulk requests. Perhaps as the number of individual users of the Pro service grows organisations will see the value of providing access to all their staff.
When we started building WhatDoTheyKnow Pro, our toolkit for professional users of FOI, we knew that there was one feature which would be a game-changer for such users: the ability to send a request to multiple authorities at once.
In this blog post, we examine what we wanted the tool to do, how we are guarding against abuse, and finally we’ll give a step by step walkthrough of the interface.
Investigative news stories or in-depth pieces of research often require information from a multitude of different sources. By gathering statistics or information from multiple authorities, journalists, activists and researchers can build up a previously-unseen picture, for example of how widespread a particular problem is, or where there are inequities in medical provision across the country.
It’s something that many professional users of FOI are doing already, usually with the aid of their own homemade spreadsheets on which they keep track of requests made, dates by which replies should be expected, which bodies have responded, which need chasing, and of course the information held in the responses themselves.
The standard WhatDoTheyKnow website already provides several helpful features that you just don’t get with a DIY system: it has all the right email addresses for authorities, for example; it guides you through the FOI process; and it will send you an email reminder when the deadline for response arrives — even taking bank holidays into account.
But we knew that in order for our batch request feature to woo people away from their spreadsheets, it needed to do more than those homebuilt systems, some of which have been refined over several years and work well, even if a bit clunkily, for their owners.
Power and responsibility
One important consideration was uppermost in our minds when it came to batch requests: it costs authorities time and money to respond to each request, and of course that multiplies with batch requests. We are keen to promote responsible use of FOI, so we want to fold appropriate safeguards and guidance into whatever system we build.
As mentioned, with WhatDoTheyKnow Pro we’re focusing on features that are genuinely useful for professional users of FOI, but we also want to help those users make better, more focused requests — ones that are more likely to get useful responses and see the light of day as news stories. So it was important that, in making it simple to send multiple requests, we also help users find the most suitable authorities to send their requests to.
With that in mind, here are some balances we’ve put in place:
- Users are limited in how many batch requests they can send within any one month — so there’s no chance to go too wild.
- There’s a limit to the number of authorities that can be added to a single batch: we set this to be the number of local authorities in the UK, which is a logical sector to survey in this way.
- Before users do a batch mailout, we encourage the sending of an initial request that goes to just a few authorities. This safeguard can reveal where a request is flawed, so for example, if the data you get back is not what you need or in the wrong format, you don’t have to send to the full list all over again.
- We provide advice on cost limits to encourage succinct batch requests.
- Authorities have the facility to report a request which is unsuitable for review by our administrators.
- We’re rolling out the batch request functionality gradually to vetted WhatDoTheyKnow Pro users so that we can gradually learn how people use it in practice, and course-correct as necessary..
Testing and improvements
So far, the batch feature is only available to a select group of test users, who are giving us feedback on how they’re finding it. There’s certainly nothing like having your code being used by real people to help you see where improvements might be made!
That said, it’s been a very gratifying process. With the help of our test users, we’ve seen that the batch request functionality has the potential to be immensely helpful to professional users of FOI; even genuinely game-changing. We are certain that with the sending tools, we’ve created a service that really adds value for this sector.
We’re now in the next phase, and turning our attention to improving the functionality that helps users deal with incoming responses when they come in. This already exists in a basic form, and thanks to our testers, we’ve identified which improvements we need to make. We’re already working on incorporating them. But that is definitely material for the next update — for now, let’s take a look at just how the batch request function works.
Using batch request
There are three parts to making any request, whether you’re doing it yourself or using WhatDoTheyKnow Pro:
- Creating the request
- Managing the responses
- Analysing the results
The batch request functionality builds on our super-simple FOI workflow tools for WhatDoTheyKnow Pro, extending them to make larger investigations much easier.
Creating the Request
The first step is compiling a list of authorities to send the request to. From the compose screen, you might search on a keyword (for example, ‘hospital’, ‘Birmingham’, or ‘Birmingham hospitals’) and then add the authorities you’re interested in.
Each authority is added to a recipient list and WhatDoTheyKnow Pro creates a ‘mail merge’ setup. You’ll see how many authorities you’re writing to in the compose interface.
You can then draft your request. The special `Dear [Authority name]` salutation gets automatically replaced with each of the selected authorities when you send your batch.
Finally, before sending you can choose a privacy duration.
At this point you can either go straight ahead and send your request, or save the draft and come back to it later.
Once you’ve sent your batch request, you’re going to receive a lot of replies from authorities. This is where WhatDoTheyKnow Pro’s functionality really comes into its own, keeping all that clutter out of your email inbox.
Here’s what it looks like: the first thing you’ll see is a high-level progress bar showing you the overall progress of your batch. There are three main states that help you manage the requests in the batch:
- In progress (yellow): This means that there’s no action needed by you – you’re waiting on the authority to respond with an acknowledgement or the information you’ve requested.
- Action needed (red): When a request in the batch receives a response from the authority, you’ll need to check it out. We mark the response as “action needed” for you to review and decide what to do next.
- Complete (green): Once there’s no further action needed – either you’ve got the information you asked for, the authority didn’t have the information, or they’ve refused and you don’t want to challenge them – the request moves to the ‘complete’ state, so you know you don’t need to think about it until you come to analyse the data.
Clicking the title of the batch reveals the individual requests and their progress status. From there, you can click through, read the response and update the status.
Now you’ve got all your data, it’s time to compare the results from different authorities.
Sometimes authorities will reply in the main correspondence.
Other authorities respond with one or more attachments. You can view these inline or download them to your computer.
If you’re dealing with a batch sent to lots of authorities, sometimes it’s easier to just download everything. You can download a Zip file containing all the correspondence and attachments for each request via the “Actions” menu. From there you can pull out the attachments that contain the raw data and plug the numbers or answers in to your spreadsheet so that you can compare across authorities.
You can sign up to WhatDoTheyKnow Pro today and receive 1 month free with the voucher code BLOGMARCH18. Make some requests to try out the FOI workflow tools for professionals, and get in touch to request to join the waiting list for batch access.
If your FOI requests have made the news, let us know! Send us links to your published stories and we’ll throw in an extra month of WhatDoTheyKnow Pro for free. Your stories help us improve WhatDoTheyKnow Pro.
The latest installation of our Alaveteli software, OPRAmachine, is an interesting new use of the platform. Rather than covering a whole country, as most of the other Alaveteli installations around the world do, it services just a single US state.
OPRAmachine, which launched in October, allows citizens to request information from state and local governmental agencies in New Jersey, under the Open Public Records Act (OPRA).
We asked Gavin Rozzi, the local journalist who has built and runs OPRAmachine, about the site and its impacts so far:
Why did you decide to set up OPRAmachine?
I developed an interest in New Jersey’s Freedom of Information law in the course of my work as an independent journalist. I created OPRAmachine because there is a void in our state for a statewide Freedom of Information portal.
Historically, New Jersey has gained a reputation as a state with excessive spending on state and local government, along with an enduring “culture” of political corruption, as defined by The New York Times.
I have found that in all too many cases, a lack of transparency and compliance with OPRA disclosure requirements has gone hand in hand with instances of government mismanagement and corruption at the state and local level, some of which have been publicised over the years.
While working in my capacity as an independent journalist, I began making extensive use of the OPRA law in order to study the activities of local governments in New Jersey. I became very familiar with the process and the how the law is effective at bringing about vitally needed transparency through the right it gives citizens to obtain public records.
If you happen to have visited the WhatDoTheyKnowPro page recently, you might have noticed that we’ve changed from offering free beta testers’ accounts to rolling out the payment interface.
Yes, we’ve launched! But not with a bang: having quietly introduced the payment option, we wanted to wait for a while and check that there were no issues before making a public announcement. Consider this that public announcement.
Just a reminder at this stage, in case you haven’t followed along with previous blog posts: WhatDoTheyKnowPro is a Freedom of Information toolkit which provides extra help with sending and organising requests. It’s designed to provide extra functionality for journalists and other people who use FOI in their jobs; if you’re a standard user of WhatDoTheyKnow, there’s nothing to pay and there never will be. We’ll always keep the site, and all its current functionality, free for everyone.
The question of price
Launching WhatDoTheyKnowPro provoked an interesting debate on where to position it, pricewise.
It’s not that we’ve never put a price on any of our services before: mySociety is a social enterprise, and we charge for some use of our APIs; our council clients pay us for FixMyStreet Pro; you can buy transit-time maps on Mapumental. These sources of income are just part of what help us to provide our core citizen-facing services for free.
And we’re hardly trailblazers in that respect: in fact, it was fascinating to read the Knight Foundation’s recent report Scaling Civic Tech and see how common it is in our sector to rely on a variety of revenue streams, from user donations to philanthropic grants, to paid-for services.
But while we may have experience in charging for our services, it’s definitely the first time we’ve had to price up a Freedom of Information toolkit for journalists and professionals!
What’s it worth?
We were effectively in the position of many an enterprise startup: with a market proposition that doesn’t exist in this exact form anywhere else. How do you know how to price something in those circumstances? Set it too low and you could miss out on important revenue; too high and you’ll alienate potential customers, many of whom are freelance journalists paying for their work tools out of their own pockets.
Still, this project has been a process of tackling problems and questions thoughtfully — from deciding which features to include in this initial version, to debating how to encourage journalists to link back to the news stories they’ve created once they’re live. Perhaps we could bring the same approach to pricing.
Ask the experts
Fortunately, with over 100 beta testers, we had a pool of users who knew the service well enough to be able to give an expert opinion on how much value it was bringing them. Sending out a survey brought some very useful responses, not to mention feedback about what our beta users liked and what they’d appreciate in future roll-outs.
That said, we’re well aware of research indicating that people are not always experts on what they are actually willing to pay.
And of course, it makes sense that different people will attach different values to a service, depending not only on their own finances but, in this case all sorts of other factors such as how frequently they use FOI and how accustomed they are to paying for technology.
Faced with responses to the survey that ranged between a suggested price of £1.00 a month to £50.00, that was, in some strange sense, reassuring to know.
To be fair, those were the extremes. There was a good consensus in the middle and that helped us decide on an introductory price of £10 a month. We’ll assess this after a few months to see whether it’s bringing the number of sign-ups we expect.
Open for business
If you’re a journalist or someone who uses FOI in your work, you can now go and give WhatDoTheyKnowPro a go! We hope you’ll let us know how you find it.
Image: Sven Scheuermeier (Unsplash)
A year ago, we helped Belgian group Anticor launch the Alaveteli site for Belgium, Transparencia.be, and so far it’s been an incredible success. Since launch, the site’s had almost 60,000 visitors and over 375,000 page views— that’s unusually high for an FOI site in its infancy — and they’ve even brought about a change in local FOI law. So how exactly have Anticor achieved so much in such a short time?
We chatted to Claude, a key member of the Transparencia team, to learn more, and the first thing we discovered was that the stellar visitor numbers were actually news to him. That might give us a clue as to one factor in their success: they’ve been far too busy to check their site analytics!
Fortunately we keep track of traffic on all our partners’ sites (where we have permission to do so), and we were glad to be able to pass on the good news. That done, we were keen to ask what exactly brought so many people to the site — and to see what tips other Alaveteli sites might be able to pick up so that they can enjoy the same level of success. (more…)