WhatDoTheyKnow is kept up and running by a dedicated team of volunteers. Do you have the time or skills required to help? If you think you might like to lend a hand, read on to see what they do on a daily basis, as well as some examples of desired site improvements.
One of the volunteers’ many tasks is to maintain what we believe to be the largest existing database of public bodies in the UK (38,362 of them…and counting).
This requires quite a bit of time and effort to keep up to date: email addresses change; bodies merge, get new names or just cease to exist.
The turnover of the financial year always brings an extra slew of required changes; presumably many bodies like to use this date for a nice neat cut-off in their records. So, to give a snapshot of the sort of admin work the volunteers undertake, let’s take a look at every task April 1 brought the team this year.
Thirteen new authorities were added. Some of them are so new that they haven’t yet had any FOI requests made through the site. Perhaps you’ll be the first?
- The Hampshire and Isle of Wight Fire and Rescue Service was formed through the merger of two existing services.
- 39 NHS Clinical Commissioning Groups became defunct, and nine new bodies were added:
- NHS North West London Clinical Commissioning Group
- NHS Kirklees Clinical Commissioning Group
- NHS Coventry and Warwickshire Clinical Commissioning Group
- NHS Black Country and West Birmingham Clinical Commissioning Group
- NHS Shropshire, Telford and Wrekin Clinical Commissioning Group
- NHS North East London Clinical Commissioning Group
- NHS Frimley Clinical Commissioning Group
- NHS Hampshire, Southampton and Isle of Wight Clinical Commissioning Group
- NHS Bedfordshire, Luton and Milton Keynes Clinical Commissioning Group
- We also marked 2 NHS Trusts as defunct and added one successor: the University Hospitals Sussex NHS Foundation Trust.
- We’ve added the new UK Health Security Agency, which has been set up to work on public health threats, combining elements of Public Health England with NHS Test and Trace and the Joint Biosecurity Centre.
- All district, borough and county councils in Northamptonshire (eight in total) were abolished on 1 April to be replaced by two new unitary authorities:
When we add a new body that replaces an existing one, we also make sure that no-one can make requests to the now-defunct authority — while at the same time, requests made to it in the past, along with any responses, are still available to view, and requests in progress can still be followed up.
We also set up page redirects to the new body, and replicate all of the metadata that helps WhatDoTheyKnow’s system work behind the scenes. It might be a bit of a faff but it’s worth the effort to keep things running smoothly.
Many thanks to volunteer Martyn for completing the lion’s share of the work listed above.
How you can help
If you know of any other changes that haven’t been reflected on the site, please do let us know.
If this post has reminded you how much you enjoy admin, consider joining the team! We always need more volunteers to help us run the site, keep the database up to date, deal with requests to remove material, and support our users. Find out more here.
There are some specific tasks that are top of our wish-list, too:
- We’d love to do some intensive work on our list of parish level councils to make it comprehensive — this could mean a few people working systematically through a list, or several checking how well their local area is represented on WhatDoTheyKnow. Local democracy matters, more so than ever, and transparency is important for bringing happenings to light (as events in Handforth have recently reminded us!).
- We have ambitions to organise our bodies geographically, showing bodies which operate in particular areas, or showing maps of the areas covered by bodies. See this ticket for a discussion of some of the possibilities which we haven’t had the resource to completely finesse.
mySociety has experience in mapping UK governmental areas, but we’re yet to integrate that expertise into WhatDoTheyKnow — do you have the required coding skills to make it happen?
- We’d like to do more organising of the bodies by their function too, helping guide users to the appropriate body fo their request.
If you have skills in web-scraping, spreadsheet wrangling, database maintenance or other relevant areas and think you can help us — please let us know!
Subscribe to our newsletter.
Image: Anastasia Zhenina
In our previous post, we identified WhatDoTheyKnow’s current need for sources of funding.
But WhatDoTheyKnow also needs more volunteers to join the team. Since the site’s launch, it’s always depended on a highly-motivated, active group of administrators who work to keep it running.
At mySociety, we’re very grateful for the work the volunteers do; for their part, they tell us that they find the work rewarding and interesting — but we’re always aware that we can’t, and shouldn’t, demand too much from them. The more volunteers we can recruit, of course, the less the workload will be for everyone.
We’ve identified three general areas in which volunteer help would be very welcome, and if you think you’d fit in to any of these, we’d love to hear from you.
- interested in FOI and transparency
- happy to work remotely but as part of a team, communicating mainly via email
- able to dedicate a minimum of a few hours per week to helping run the site
Each of our volunteer administrators give their time freely and are the only reason we can run the service day to day at all.
Being a volunteer is both rewarding but also challenging, as each juggles their day jobs and home lives. So the more volunteers we have, the more we can spread the workload between them.
If you have a specific interest in FOI or transparency, or indeed you’d just like to help support a well used civic tech service then we’d love to hear from you. There is always a diverse range of jobs and tasks needing to be done, even if you can only help a couple of hours a week. We all work from home and communicate via email and other online tools.
If you can help us a volunteer the first thing to do is to write to the team introducing yourself and letting us know about your relevant skills, experience and interests.
- a law student or professional who can offer expertise in the day-to-day running of the site; or
- a legal firm or chambers who could offer legal advice on an ad hoc, pro bono basis
Volunteers with legal backgrounds We take our legal and moral responsibilities in running WhatDoTheyKnow very seriously and we always welcome volunteers with experience of legal matters. Some of the legal aspects of running the site are handled routinely on a day to day basis by the admin team.
They may, for example, remove correspondence which could give rise to claims of defamation, or where personal data is disclosed by an authority mistakenly and they consider continued publication to be unwarranted.
The legal challenges thrown up by operating our service are varied and interesting. Joining us could be an opportunity for someone to get some hands on experience of modern media law, or for a more experienced individual, to provide some occasional advice and guidance on more challenging matters.
We often find ourselves balancing claims that material published on our site could aid criminals or terrorists, or could cause harm in other ways, and we do our best to weigh, and balance, such claims against the public interest in making the material available.
As material published on our website may have been used to support news articles, academic research, questions from elected representatives, and actions by campaign groups or individuals it’s important we don’t remove correspondence lightly and that we’re in a position to stand up, where necessary, to powerful people and institutions.
Legal firms that can offer advice As from time to time there are cases which are more complicated, we would like to build a relationship with a legal firm or chambers that can advise us on an ad hoc basis on defamation, privacy (misuse of private information) and data protection.
The ability to advise on copyright law and harassment law would also be an advantage. And we also on very rare occasions may need help as to how to respond to the threat of litigation.
Could you offer help in this area? Please do get in touch to discuss getting involved.
- a committed, organised, empathetic person who could volunteer a few hours (working from home) a week
In our previous post we mentioned that we’d ideally secure funding for an administrator who could handle our user support mail and deal with routine but potentially complex and time-sensitive tasks such as GDPR-based requests.
While we seek funding for this role, would you be willing to fill it on a voluntary basis? Please get in touch.
Lots to help with
So in summary, what we need to keep WhatDoTheyKnow running is money, volunteer help, and legal support. If you can help with any of these, or have some ideas of leads we might be able to follow, please do get in touch. It also helps to share this post with your networks!
Alternatively, you can help out with a donation large or small — every little helps.Donate now
Image: CC0 Public Domain
We’ve just released version 0.32 of Alaveteli, our open source platform for running Freedom Of Information sites. Here are some of the highlights.
Making correspondence threads easier to navigate
Thanks to our designers, it’s now possible to collapse individual messages in a correspondence thread in order to focus on just the parts you’re trying to read. Plus you can quickly collapse (or expand) all the messages in the thread using the “Collapse all” and “Expand all” links from the “Actions” menu.
Alaveteli Pro users gain the additional benefit of a redesigned sidebar which allows for easier navigation of lengthy correspondence and avoids having to scroll to the top of the request thread to update its status. See Martin’s full explanation here.
Better password security
We’ve started enforcing stricter password length constraints wherever a password is set or updated to help users keep their accounts secure. And we’re also using a stronger encryption method for storing password data, using bcrypt rather than the older SHA1 algorithm to obscure the actual password. (Be sure to run the rake task documented in the release upgrade notes to upgrade secure password storage for all existing users.)
You can read more about what this does and why it’s important if you’re interested in the technical details behind this upgrade.
Authorities not subject to FOI law
We’ve adopted WhatDoTheyKnow’s
foi_notag for authorities to indicate that although the authority is listed on the site, it is not legally subject to FOI law. This could be for advocacy purposes – if it’s felt an authority should be covered by legislation – or where the authority has agreed to respond on a voluntary basis.
foi_notag now causes an extra message to appear under the authority’s name on their page and on all related requests, and removes language about legal responsibilities to reply from the messages sent to users.
To improve the UI, we’ve made a similar change for authorities with the
eir_onlytag to make it clearer that such authorities are only accepting requests about the environment.
(Don’t worry admins, you don’t need to remember all this – we’ve updated the documentation on the edit page to reflect the new functionality!)
Improvements for site admins
We’ve made it easier for admins to ban users who sign up to post spam links in their profile. There’s now a “Ban for spamming” button which is available on the user edit page or as soon as you expand the user’s details in the listing rather than having to manually edit user metadata.
We’ve also made it harder to leave requests flagged as vexatious (or “not_foi”) in an inconsistent state. Previously the site just assumed that vexatious requests would always be hidden. Now the admin interface enforces the hiding of vexatious requests by showing warnings when a request is set as vexatious while it’s visible on the site, and prevents the updated request from being saved until a valid state is selected.
And last but not least – introducing the new Announcements feature!
Easier popup banner management
Site admins will be relieved to hear that they can now update the popup banner message on the site without needing to schedule developer time.
This feature supports multi-language sites so if you set the announcement for your main (default) language, it will appear across all language versions that you have not added a specific translation for.
You can set announcements that will only be seen by fellow administrators when they visit the summary page. (If you’re running a Pro site, you can also have announcements that will only be seen by your Pro admins.)
Announcements for Pro users appear as a carousel at the top of their dashboard. So far we’ve used it on WhatDoTheyKnow Pro to publicise new features, offer discount codes, and encourage people to share their published stories with us.
The full list of highlights and upgrade notes for this release is in the changelog.
Thanks again to everyone who’s contributed!
I’m Richard Taylor, a member of the volunteer team which administers WhatDoTheyKnow.com on a day to day basis, and I spoke at the event highlighting the broad range of people who have collaborated to make WhatDoTheyKnow a success, and sharing some ideas for the future. Here’s what I said:
I’m someone who wants to see our representative democracy working; that’s why I support what mySociety does; I support giving tools to people to help people engage with our society, how we make decisions about running our society, how we run our public services, our health service, policing, how we organise our cities, how we plan development of new homes and design, or evolve, our transport systems.
I joined WhatDoTheyKnow as a user on the 22nd of July 2008, so almost exactly ten years ago. My first Freedom of Information requests were on policing, for the local Stop and Account policy – as you can see from those kinds of requests I’m keen on transparency and accountability of those we give powers over us. I looked up my early FOI requests and I was rapidly onto my local councillor allowances, details of which weren’t online, and as I’m from Cambridge and there were some very Cambridge requests in there too – on the running of the river – on the regulation of punting – a perennial local issue, and for the terms and charges for grazing on the city’s commons. One of the things I do is campaign for proportional police use of TASERs, I made requests on that subject too.
Within just a few days of joining the site I was sending in lists of public bodies to add to the system; and shortly after that I was invited on to the administration team so I didn’t have to bug developer Francis Irving, or the volunteers who’d already started to help running the site, including John Cross, Alex Skene, Tony Bowden to do things like add new bodies, but could make changes myself.
The volunteer team
Mine is the same route many of our volunteers took to joining the team running the site in the early years; those making lots of good proposals for bodies to add, or making other suggestions were invited to help out. The way we’ve found new volunteers has changed a little over time, and we have had to keep topping up the pool of volunteers as people have moved on. We started to approach users of the site who were making helpful annotations assisting other users, and who were making great use of the site themselves. We found Ganesh Sittampalam and Doug Paulley that way, both of whom have put huge amounts of time into developing the site, the service. Latterly we’ve moved to advertising for new team members and seeking applications from those who want to join us, and that’s brought us some of our current active volunteers, Michael Bimmler for example.
Volunteers have put in an enormous amount of time into running the site. If you put a cash value on that time I’m sure the volunteers would by far be the biggest donor to the site. The site probably wouldn’t exist, and certainly wouldn’t exist in its current form without volunteer input; so many good ideas for websites get built, often with funding to kick them off, but they don’t do what WhatDoTheyKnow has done, and survive, grow, and thrive. Volunteer input has enabled that.
The site certainly has grown and thrived, we now have around seven million users viewing the site per year; according to Google analytics, and 162,000 signed up users. There are approaching half a million request threads on the site now. An interesting aspect of those statistics is the viewing is not focused on a small handful of requests, but rather visitors are spread broadly across the long-tail of requests and released information. In 2016 17% of requests to central government monitored bodies went via our service; but the vast majority of requests, 88%, go to bodies where central government don’t track FOI request statististics.
The volunteers I’ve mentioned already, plus Helen Cross and Alastair Sloan, have put substantial chunks of time into running the site. There are many others too including Rob McDowell, Ben Harris, Gavin Chait and Peter Williams. The volunteers supporting the service have not just come from the volunteer team, the trustees who’re ultimately responsible for the site are volunteers too, ten years ago mySociety was more of a volunteer based organisation than it is now, trustee Amandeep Rehlon was dealing with the finances on a volunteer basis, we’ve had great moral and policy guidance from Manar Hussain and Owen Blacker, and the chair of the trustees, another volunteer, James Cronin.
We have been amazingly lucky with the volunteers we’ve attracted to the administration team. Doug Paulley is an incredible activist and campaigner on disability rights, and so many of the others are legal and information rights experts, activists and campaigners in their own rights.
Volunteers are only part of the story, we wouldn’t be able to do what we do, and what we want to do without the institutional support of mySociety, and the organisation’s brilliant staff. When the initial developer and project manager Francis Irving moved on he was succeeded by a series of great lead developers, Robin Houston, Seb Bacon, and now Louise Crow and other staff team members, currently Gareth Rees, Graeme Porteous, Liz Conlan …(See Github for the full list of contributors to the code!) the site is supported by the whole mySociety team, including designers Zarino Zappia and Martin Wright, Abi Broom, who runs the show, Gemma Moulder – events organiser from our perspective, who also works on spreading services based on WhatDoTheyKnow around the world, and mySociety’s communications person Myf Nixon. Thanks are also due to ten years’ of mySociety sysadmins including Sam Pearson,Ian Chard, and in the early days volunteers who’d keep things running, Adam McGreggor, and Alex Smith.
A key WhatDoTheyKnow volunteer was Francis Davey who was our volunteer legal advisor for many years. Francis Davey’s top piece of advice which I recall was to avoid court. We’ve pretty much succeeded to date-with that. One of the key roles of the volunteer team is to run what is a relatively legally risky site without getting sued and consequently, probably, taking down not just WhatDoTheyKnow but the rest of mySociety too.
We deal with a lot of defamation claims, personal information takedown requests, and an array of more obscure legal challenges.
As well as trying to avoid getting annihilated via legal processes a key aspect of our approach to running the site is we try our utmost to run it responsibly. What those involved didn’t do is find a legally friendly jurisdiction and anonymously just let the system loose to run unmanaged and unchecked. We’re real accountable people who respond to concerns from all comers, individuals, public bodies, our own users, about what’s published on our site.
What are we doing by running our site?
We’re doing a lot more than just helping users make a request for information to a public body. We’re activists, we’re promoting running our society in a transparent, inclusive, accountable, way, not just by lobbying, making speeches, writing articles, but by doing something, by running our site.
Running our service promotes Freedom of Information and other access to information laws; people come across our site when searching for information they’re seeking; we show what can be obtained by publishing requests and responses; others might find the information they’re seeking directly, or see that they can make a similar request, perhaps adapting a request that’s been made elsewhere to their local public bodies..
Anyone can make a Freedom of Information request by private email to a public body. I’d find that potentially a bit of a selfish action, incurring cost to the public for a response only I might see, but making a request via WhatDoTheyKnow to obtain information which should be accessible to the public automatically makes that information accessible to anyone who searches for it, anyone who Googles for the information. Even if a requester doesn’t themselves do something with the information released by making a request via WhatDoTheyKnow.com they’ve enabled others to do so. You’re often doing public good just by making a request via WhatDoTheyKnow.com (though do see our advice on making responsible and effective requests).
WhatDoTheyKnow makes something which would otherwise be quite challenging for many people – getting a FOI request and response online – easy. I’m sure only a fraction of users of our site would have taken the time to write a blog about their request, and update it with the response, if they had to do that manually.
A big benefit of making a request on WhatDoTheyKnow.com is many people are already using our site and watching for responses; if you make a request to a local council on WhatDoTheyKnow.com the chances are your local journalists are tracking requests to the local council and they’ll be alerted to any response.
At WhatDoTheyKnow we’re an independent third party, we’re not the requestor and we’re not the public body. This can be useful when there’s a dispute about a response to a request, if a public body denies receiving it for example. We’d love to work more closely with the regulator, the Information Commissioner’s office, we’d love them to use our service more to help them in their role in enforcing the law. Often just having a request on our site can help people get a response, good public bodies really care about the impression those visiting their pages on our site get. Lots of public bodies will get in touch with us if they don’t like the way a request has been classified by one of our users for example.
A really big advantage having information released via our service is people can cite it when they take action based on it, be that action a blog post, an article in the media, an academic publication, or a letter to an MP. You can show, again using WhatDoTheyKnow.com as an independent third party, where the information you are relying on has come from, giving more weight, more credibility, to whatever it is you’re doing, your lobbying, your journalism, your research. WhatDoTheyKnow, and mySociety more broadly, has been in the business of enabling better informed debate and higher quality journalism well before “fake news” entered our lexicon.
We’re always looking for new bodies to add to our site, the database of public bodies which is behind the site keeps growing, we’re now at over 23,000 public bodies. That compares to about 450 public bodies listed on the Gov.uk website, and just 305 in the latest “Public Bodies” report by the Cabinet Office. The big difference is made up by schools, GP surgeries and NHS dentists, all of which are subject to FOI; we also list groups of organisations like companies owned by local government – public bodies in terms of the Freedom of Information Act but all but invisible to central government.
I said we were in the business of activism; changing society by doing things. One big part of or Freedom of Information law related activism is listing bodies on our site which are not, or not yet, subject to access to information laws. We’ve listed many bodies before they became subject to the Freedom of Information Act, showing the demand for information, and showing the kind of information people want, but couldn’t access. One example was Network Rail which we listed before it became subject to FOI in March 2015, another was the Association of Chief Police Officers .. however that’s now become the National Police Chief’s Council and MPs failed to make that successor body subject to FOI – in that case it’s not a huge problem as they realise they need to be transparent and they voluntarily comply, but, significantly, the Information Commissioner can’t enforce a law which a body is not technically subject to.
There are always more public bodies to add, we list Housing Associations for example, they’re a another class of body which are not subject to FOI, even coroners aren’t subject to FOI which you might find surprising given their important public role in ensuring our society is safe, and more people don’t die in the future for the same, preventable, reasons people have died in the past. We list some coroners, and volunteer Kieran is working on making our coverage comprehensive. Local medical committees; committees of GPs are another set we’re hoping to add soon.
Maintaining the body database is a constant task. Government is constantly reorganising, we try to keep up with changes recently for example, recently, in research councils, and keeping track of NHS reorganisations is a challenge on its own. There have already been 17 requests to London North Eastern Railway Limited, the Government rail operator of last resort which we listed when it took over running trains on the East Coast mainline about ten days ago.
Seeking improvements to laws which impact our service, its users, and the accessibility of public information
As well as our activism we have a record of more traditional lobbying; sharing the experience running our service has given us experience of the operation of access to information law. We took part in the Post-Legislative Scrutiny of the Freedom of Information Act 2000 in 2012 for example, and just a few months ago we responded to a consultation by the Cabinet Office on the Code of Practice which bodies responding to FOI requests have take into account.
In terms of what we’re calling for, we’re not FOI fans specifically, we’d actually rather people didn’t have to make FOI requests, we’re in favour of proactively releasing information and running public services transparently, though that said FOI requests are requests for information people want to know; rather than information which public bodies want to publish so they will probably always have their place.
Why not make public bodies consider proactive publication of information of the sort requested, when dealing with a FOI request? That’s a provision which is in the specialised law on access to datasets but doesn’t apply to access to information requests more broadly.
Timeliness of responses, and timeliness of enforcement action from the Information Commissioner are other key things we campaign on. If you want a copy of a FOI response that’s been made to particular union, lobby group, or journalist and is the information behind the day’s news, surely you should be able to get a copy of it pretty much straight away, and there can be no excuse for a body dithering until the 20 working day deadline. The law requires a prompt response; that aspect of the law needs following and enforcing.
We also want to close loopholes in FOI; one terrible one, is if a public body can think of a class of information and list it on its website with a price for it, it becomes exempt from disclosure for free under FOI. This is clearly open to abuse, fortunately few bodies have misused it too-date, but there are examples – just look at your local council’s list of information they make available for a fee.
Running the Site
Some might be interested to know administration has changed as the site has grown. There’s been a constant improvement of the site’s software to make it easier to run, but that needs to continue so we can cope with it getting bigger without having to increase the volunteer effort exponentially in-line with the site’s growth. We’ve outgrown the team@ mailing list system the site started with; we now separate the support mail from discussions among volunteers, and on top of that there’s a separate discussion of legal matters; so people aren’t overwhelmed.
One challenge we have is the workload, and volunteer input, are both variable. Sometimes there’s a week where you really need someone full time running the site. Sometimes you could firefight the incoming issues in maybe an hour a day, or day a week.
Something we’d like to do is encourage past volunteers to join our monthly calls; join the legal discussion list, volunteers list, drop into the support mailbox and help out on occasion, every little helps, following what we’re doing for a week a couple of times a year might provide some outside, detached, input; help keep us on-track, challenge us, and assist us in spotting drifts in policy / practice.
Ideas for the future
We’re always keen to hear any ideas for what we could be doing better, or differently we welcome input from anyone and everyone who cares about the service in some way. Some of the things we could do improve:
- We could do even better at transparently running the site. We already try to run the site as transparently as we can; if we hide a message, or redact content from correspondence, we make clear where we’ve done so and explain why. We don’t though have a transparency report like Google and Reddit do, reporting on takedown requests, how many there have been, who they’ve come from – individuals, requesters, public bodies, public officials, regulated professionals, and how we responded. Requests for user data. One challenge is sometimes the moral thing to do is not shout about and draw attention to something we’ve taken down too quickly; don’t want to draw attention to taking down something that’s still in Google’s cache for example – if we really believe it shouldn’t be online any more.
- We should do more to highlight excellent, interesting and influential, uses of the site. It would be great to have ways within the system to note when responses have been used by others, cited in Parliament, resulted in a news story, or if someone has analysed responses from a range of public bodies around the country for example.
- We have volunteers, but there is no real community of users around the site, or around our lobbying activities, or, to the extent there has been in the past, a community – around mySociety any more. There’s an opportunity there..
- I think we have a duty to be careful with the way the WhatDoTheyKnow pro-service is used. Anyone can sign up for a Gmail account and make requests; but we are doing more than Gmail to encourage and enable FOI requests, and not least the pro system is built on a largely volunteer built and maintained database. Use to-date has apparently been good, and we have a general principle of not spending time discussing hypothetical situations, but, in running the free site as volunteers we’ve always been mindful of the impact of our actions on our reputation, and the reputation of Freedom of Information law itself. For example we ask those considering bulk requests if they’ve carefully selected the set of bodies to make their request to, if the request could be made to a central body rather than lots of local bodies, if a sampling exercise would suffice instead of asking perhaps hundreds of bodies, and we advise on making clear requests in the first instance to reduce the need for clarifications – saving public bodies and requesters time and effort. [Update: following the event we agreed to update our House Rules to include a reference to our advice on making responsible and effective requests|]).
- Lastly, on sustainable funding for the site, ideally I think this would be though a handful of media organisations, campaign groups, or other bodies paying for a pro-service; which would hopefully give them great value in terms of organising FOI requests, prompting them to chase up late requests, saving time finding contact details and easily making bulk requests. Perhaps as the number of individual users of the Pro service grows organisations will see the value of providing access to all their staff.
What happens when your site is the target of a major spam attack? That wasn’t something we were particularly keen to find out — but it’s a scenario we’re now fully acquainted with. That’s all thanks to a recent concerted assault on our Freedom of Information site WhatDoTheyKnow.
All is calm again now, and hopefully, as a user of the site, you’ll have noticed very little. Yes, you’ll now have to complete a recaptcha when creating a new request*, and you might have discovered that the site was inaccessible for a couple of hours. Beyond that, everything is pretty much as it was.
From our point of view, though, it was an emergency situation that meant that several of us had to put down what we were doing and join in with some quick decision-making.
It was around 12:30 on a Wednesday afternoon when Richard, one of the volunteers who helps to run WhatDoTheyKnow, noticed unusual activity on the site.
WhatDoTheyKnow was created to help people send requests for information to public authorities — a service for the wider good. Unfortunately, at this point, it was also doing something quite the opposite of good: it was providing the means for unknown sources to send those same authorities hundreds of spam messages.
We’d like to apologise to those who were on the receiving end: clearly, spam is a nuisance for everyone who receives it and we’re unhappy to have played any part in its perpetuation.
We also had a secondary concern. It seemed likely that recipients would mark these incoming emails as spam. When enough people had done that, email providers would see us as an insecure source, and block all our messages, valid or otherwise, potentially preventing the WhatDoTheyKnow system from running efficiently.
A little fire-fighting? That’s actually situation normal
Spam is an obvious example of the site being abused, but it’s perhaps worth mentioning that we work hard on many levels to ensure that WhatDoTheyKnow is only used for its core purpose: the requesting of information under the FOI Act.
And note that we’ve always been careful to protect against abuse. WhatDoTheyKnow does already have several measures in place as standard: we only allow one account per email address; we verify that email addresses are genuine; and we cap the number of requests that users can make each day (a restriction that we only override for users who are demonstrably making acceptable use of our service). We reckon that these measures very much helped to reduce the impact of the attacks.
After a quick discussion between the volunteer team, trustees and mySociety staff, we took the site offline to give us time to work on a solution while stopping any more spam from being sent.
Of course, we then removed all the spam requests and comments from the site and banned the accounts that had made them. We also contacted the affected bodies to let them know what had happened and to assure them that we were taking steps to deal with it.
When we brought the site back up, a couple of hours later, we did so cautiously and with new restrictions and safeguards in place.
Spam ‘requests’ had been sent over a period of about 13 hours. There were in the region of 800 made, though only about 500 actually got sent to authorities. Additionally, around 368 spam comments were left on existing requests. These relatively small numbers lead us to believe that they were being made manually.
Time to breathe… or nearly
Once we’d discovered the issue, dealing with it and getting the site back up and running took us 2.5 hours.
Job done — so now we could sit back and relax, eh? But no: the next day we discovered that a couple of other sites running on the Alaveteli platform, AskTheEu and New Zealand’s FYI, were being subjected to the same attacks.
So we rolled out the changes we’d made on WhatDoTheyKnow to make them available to all Alaveteli users. And then, finally, we could get back to the everyday work we’d been doing before — making our sites better for you, and the other nice non-spamming people who use them.
* We’ll be looking at removing it as soon as we can, though, as recaptcha doesn’t offer a very accessible experience for many disabled people. Meanwhile, we can manually remove the recaptcha for specific accounts, so if you’re struggling with it, contact the team to implement this exemption.
Once a country has a Freedom of Information act in place, the battle for citizens’ Right To Know is pretty much over, right?
Er… that would be nice, wouldn’t it? But in fact, as those who have read our previous blog posts will know, all sorts of factors can stand between citizens and information about their public authorities — here in the UK, and all around the world. Factors like complex legislation, reluctant officialdom, bureaucracy… and a host of other impediments.
In Uganda, FOI has made a tangible difference to the level of corruption from officials, but a lack of resources and their politicians’ reluctance to perform the duties requested of them by the act mean that access to information is still a struggle.
Find out more about the people running Uganda’s Alaveteli site, Ask Your Government, and how they’re tackling these issues, in our latest case study.