This work will support users in taking the next steps, if appropriate, when their requests for information are denied.
A bit of background
In the last few years, there has been a significant and sustained decline in FOI requests being granted by the UK government.
According to the Institute for Government, the proportion of refused FOI requests reached a record level in the third quarter of 2019, with departments refusing to comply in full with more than half of all FOI requests that they received. This compares to around 40% in 2010 and around 30% in 2005.
And yet, our research found that, when challenged, a large proportion of refusals were overturned, suggesting that the fault did not lie with the type of request being made. 22% of internal reviews resulted in the full or partial release of information, and a further 22% of appeals to the ICO led to all or some of the information being released.
For local authorities, up to half of internal reviews – and just over half of all ICO appeals – led to the release of all or some of the information requested. In Scotland, with its own FOI regime, 64% of appeals to the Information Commissioner resulted in the full or partial release of information.
And so, while acknowledging that some refusals are certainly legitimate, there is a clear case for challenging such responses. But to do so is daunting, especially for novice requesters who can understandably be discouraged by an official response citing exemptions in legalese.
This new funding will allow us to approach the issue from four different, but interlinked directions, each intended to inform and support users in challenging government refusals of FOI requests.
- When a WhatDoTheyKnow user confirms that they’ve received a refusal, we’ll be integrating context-sensitive advice. This will inform the user of their right to appeal, give clear guidance on how to assess whether the authority has complied with the law, and also advise on other channels, beside FOI, by which information may be obtained.
- We’ll automatically identify which exemption has been cited in the refusal, giving us the ability to help users better understand why their request has been turned down.
- Based on this finding, we’ll offer context-specific advice for the exemption identified. For example, if the request has been turned down because of cost, we’ll show how to reframe it to fall below the ‘appropriate limit’.
- Finally, once the user has been fully informed, we’ll offer the support they need to escalate the request to an appeal.
Ultimately we hope that this work will help reset the balance on the public’s right to access information, better enabling citizens, journalists and civil society to effectively scrutinise and hold authorities to account.
As always, we’ll also be thinking hard about how to make all of this apply more universally, across the various legislatures that apply in jurisdictions where people are running sites on the Alaveteli platform.
If this interests you, watch this space. We’ll be sure to update when we’ve made some progress on the project.
Image: Tim Mossholder
We’ve added a new functionality to the Alaveteli Pro codebase, allowing you to download a zip file containing all correspondence and attachments from a batch, and a spreadsheet (csv) to show the progress status of every request.
Alaveteli Pro is our tool for professional users of Freedom of Information. If you’re UK-based, you’re probably most familiar with our local iteration WhatDoTheyKnow Pro — but don’t worry: when we talk about improvements to Alaveteli Pro, you can be sure they’re also part of the WhatDoTheyKnow toolkit.
How to export
You’ll find these tools at the foot of the batch container in the requests list.
Why data exports?
Of course, we like to think Alaveteli Pro is a useful tool in its own right: there’s a lot you can do within the Pro interface, and it was built specifically to help you keep track of all your FOI activity in one place.
But sometimes users want to use external tools – either because they’re just more familiar with them, or because they want to do something beyond the functionality we offer.
Now there’s a simple way to get data out of Alaveteli, allowing you to analyse it with the tools of your choice, or perhaps send a progress report to a supervisor or editor.
It’s part of a programme of work to support cross border journalism between European organisations, supported by Adessium Foundation, allowing us to refine and improve the codebase for the benefit of all Pro users.
The technical bit
Those with a bit of coding knowledge may be interested to hear how we approached the zip download functionality. mySociety developer Graeme explains:
“With batch requests potentially going to as many as 500 different authorities, each request can receive several responses and attachments in return.
“All these emails and files mean that compiling the zip for download could be a lengthy job and would normally cause the request to time out. So for this new feature we’re utilising file streaming to send chunks of the zip as they become available.
“This means that the zip starts downloading immediately and you don’t have to sit watching and wondering whether anything is happening – you can see more and more data being transmitted.”
We hope you find this new feature useful. Please do let us know how you’re using it and any feedback you may have.
Image: Startup Stock Photos
This brings some substantial improvements to the code. The update is available to anyone running a site on the FixMyStreet platform, which includes our own fixmystreet.com; the installations we provide for councils and authorities; and the FixMyStreet instances run by others, in places from Australia to Uruguay.
If you run a site on the FixMyStreet platform yourself, or are just interested in the technical details, you can read the release notes here.
Meanwhile, here’s a rundown of the new front-end features you might notice if you’re a user of FixMyStreet.
Run the site as an app
FixMyStreet can now be added to phones (and desktops for that matter) as a ‘progressive app’. Here’s what to look for when you visit fixmystreet.com:
On Chrome for Android:
Access from the bar at the bottom of the screen.
Click the share icon at the foot of the screen.
Then select ‘add to home screen’.
On Firefox for Android:
Look for the pop up notification or tap the home icon with a plus sign in it in the URL bar.
Any of these methods will install a version of FixMyStreet that will behave like an app, placing an icon on your desktop, browser start page or home screen.
This way there is no need to download or update from the app store, and changes to the main website (which are invariably released sooner than on the app) will be immediately available to you.
Cobrands (for example the councils that use FixMyStreet as part of their own websites, and people running FixMyStreet in their own countries) can provide their own logo and colourscheme as well.
Mobile browser improvements
Whether you install the progressive web app or just visit fixmystreet.com on your mobile browser, you may notice some nice new features.
- If you use the geolocation function (‘use my location’), your position will be displayed on the map:
- When viewing an area, you can access the filters to narrow the reports displayed down by their status (fixed/open etc) and category:
- If you’re about to report something that looks like a duplicate, you’ll not only be shown the report/s that have already been made, but you’ll also see a small inline map without having to scroll back to the main map to check where they are.
- The site recognises that when you’re on a mobile, the message about uploading a photo shouldn’t invite you to ‘drag and drop’, but rather to either take a new one or select a photo from your phone.
- If you’ve placed the pin incorrectly, the ‘try again’ process is clearer.
If a picture paints a thousand words, then your Twitter character count just went stratospheric. Now, when you share a report on places like Twitter or Facebook, if there’s a photo included in the report, that will also be pulled through.
Previously, the ‘open graph image’ that was shown by default was the same for every report — which could get a bit boring in aggregate, and certainly missed some of the impact that people might want to share when they’re posting about their own, or others’ reports.
Social media isn’t the only place that FixMyStreet reports can be piped to, though — the site also has several RSS capabilities that have been baked in since its early days.
For those not totally up to speed with RSS and what it can do, we’re now no longer displaying them as raw XML but as a nice simple web page that explains its purpose.
To see this in action, click ‘Local Alerts’ in the top menu of any page. Here’s a before and after:
What benefits one, benefits all
Much of this work is thanks to NDI, the National Democratic Institute.
NDI offer the FixMyStreet codebase as one of their DemTools, installing it in countries around the world as an innovation which empowers citizens to keep their neighbourhoods clean and safe.
Thanks to this partnership, NDI funded the addition of new features which they had identified as desirable — and which, thanks to the open codebase, will benefit users of every FixMyStreet site worldwide.
There are some other significant additions in this release, including integration, back end and security improvements, all of which will be of most interest to developers and site admins — so if you’d like to see them, head over to the full write up on the FixMyStreet platform blog.
Image: Max Fuchs
It’s obviously good citizen behaviour to report something that needs fixing to your council, whether it’s a pothole that could cause an accident, or a broken streetlight that has plunged the area into darkness.
But there’s one type of report that isn’t very useful to councils, and in fact brings unnecessary costs and inconvenience: when you tell the council about an issue that’s already been flagged up by someone else.
FixMyStreet has always been helpful in this regard. It was groundbreaking in displaying all reports in public, unlike most council systems when we were first developing it. A user who goes to make a report can see right away if there’s already a pin in that spot, and check whether the existing issue is the same one they were going to add.
Now we’ve taken that concept a step further in some work which we’re trialling on Bath & NE Somerset’s implementation of FixMyStreet Pro.
When a user starts to make a report, the system checks to see if there are any other open reports in the same category within a small radius. If it finds any, you’ll see a prompt, like this:
All similar reports will appear here. If you think one might be identical, but aren’t sure, you can click ‘read more’ to see the full text along with any photos attached to the report:
And if you recognise it as the issue you were about to report, you click the green button and will be given the option to subscribe to it, so you know when it’s being seen to, effectively being kept just as up to date as you would be if you’d made the original report:
If it’s not the same issue, no worries: just click ‘report a new problem’ and you can do just that:
Bath & NE Somerset will run this feature as a trial over the next month; then once they’ve got feedback from their users, we’ll hopefully offer it to every other council on the Avenue tier of FixMyStreet Pro.
If you come across this feature while making a report in Bath or environs, do let us know how it works for you.
Image: Kevin Grieve
If you’re reporting an issue on Buckinghamshire Council’s FixMyStreet installation, you might have seen yellow dots appearing on the map. These represent items such as streetlights, bins or drains, and we blogged about it when we first added the feature.
When it comes to assets like streetlights, it can save the council considerable time and effort if your report tells them precisely which light needs fixing: it’s far quicker to find an identified light than it is to follow well-meaning but perhaps vague descriptions like ‘opposite the school’!
But even when the assets are marked on a map, it’s not always easy for a user to identify exactly which one they want to report, especially if they’ve gone home to make the report and they’re no longer standing right in front of it.
After the system had been in place for a few weeks, the team at Buckinghamshire told us that users often weren’t pinpointing quite the right streetlight. So we thought a bit more about what could be done to encourage more accurate reports.
As you might have noticed, streetlights are usually branded with an ID number, like this:Buckinghamshire, as you’d expect, holds these ID numbers as data, which means that we were able to add it to FixMyStreet. Now when you click on one of the dots, you’ll see the number displayed, like this:
The same functionality works for signs, Belisha beacons, bollards and traffic signals, as well as streetlights. Each of them has their own unique identifier.
So, if you’re in Bucks and you want to make a report about any of these things, note down the ID number and compare it when you click on the asset. This means the correct information is sent through the first time — which, in turn, makes for a quicker fix. Win/win!
This type of functionality is available to any council using FixMyStreet Pro: find out more here.
Header image: Luca Florio
When you consider that FixMyStreet has been running for over a decade, it’s not really surprising that the maps in some areas are a little over-crowded with pins.
That can be a problem for anyone trying to make a new report — even when you zoom right in, we were beginning to find that in some very congested areas, it was difficult to place a new pin without clicking on an existing one.
We’ve tried to remedy this in various ways in the past. For a while we only displayed newer reports by default, a decision which we discarded when we brought in pagination, allowing users to click through batches of reports rather than seeing them all in one long list on a single page.
For some time now we’ve also provided the option to hide the pins completely, via this button both on the desktop and app versions:
And there’s also a ‘hide pins’ option at the foot of the map:
But even so, arriving at a map absolutely covered in pins and having to look around for that button doesn’t exactly seem like a nice, smooth user journey, so we’ve revisited the matter.
Why not just delete the old reports?
We’ve always had a policy of keeping every report live on FixMyStreet (unless it’s reported to us as abusive, or its maker contacts us to ask us to remove it — and even in this latter case we’d prefer to retain the content of the report while anonymising it).
This is because the reports made to councils build up to create an invaluable archive of the issues that various regions of the country face, through time.
The historic collection of reports allows planners to understand recurring or seasonal problems; and researchers use this data as well, to get insights into all sorts of issues. For examples, see Réka Solymosi’s presentation at TICTeC on using FixMyStreet data to understand what counts as ‘disorder’ in the environment, or mySociety’s own research on why some areas of the country report on FixMyStreet more than others.
And so here’s what we’ve done
- When you visit a map page on the main FixMyStreet site, by default, you’ll again only see reports that are less than six months old, and that are still open.
A report remains ‘open’ until the council marks it as ‘closed’, or a user or the council marks it as ‘fixed’. ‘Closed’ means that the council doesn’t intend to do further work on the issue, which can be for reasons such as the issue not falling within their responsibilities or because it is part of their regular maintenance schedule and will be seen to in time.
- You can still opt to see closed and fixed reports by selecting from the dropdown at the top of the list:
- And you can also still see reports older than six months by clicking the checkbox:
- The two filters work together, giving you the options of displaying:
- Open reports less than six months old (the default)
- Open reports of any age
- All reports less than six months old
- All reports of any age
- Any combination of open/closed/fixed reports less than six months old
- Any combination of open/closed/fixed reports of any age
To keep things simpler for app users, the display there is set to only show newer, open reports, so if you want the full range of options, you’ll need to switch to viewing the site on a desktop.
Additionally, reports that have been closed for six months without any update being made will now no longer allow updates. If you need to update an issue that falls into this category, we recommend starting a new report (possibly linking to the old one for reference if it provides useful information for the council).
But you might not see this everywhere
Some councils use FixMyStreet Pro as their own fault-reporting software. These councils can opt whether or not to adopt these defaults, so your experience may be slightly different when visiting FixMyStreet via your local council’s own site.
We think that we’ve arrived at a more intuitive solution than those we tried before — and we hope that these options will suit everyone, whether you’re a user in a hurry coming to make a quick report, or someone who’d like to see a more in-depth history of the area. Give it a go, and then let us know your thoughts.
Jenna Corderoy, Alaveteli Professional Advocate, brings us an update on the project.
Since our last blog post on Alaveteli Professional — our Freedom of Information toolkit for journalists, campaigners and activists — there have been a few exciting developments.
The batch request feature is coming along nicely: this will allow users of the service to send one Freedom of Information request to multiple authorities and help them to easily manage large volumes of responses.
We’re going to be working with a small group of our beta testers to develop this feature and make sure we release it in a useful and responsible form (click here to apply as a beta tester and get a year’s free access to WhatDoTheyKnowPro, the UK version of the service).
We’ve been pleased to see the first news story to emerge as the result of a request made through WhatDoTheyKnowPro: a response to the Foreign and Commonwealth Office showed which are the countries where UK holidaymakers are most likely to get arrested. The full list was covered in the Birmingham Mail.
But we also have plans for this Freedom of Information toolkit to go international: Alaveteli Professional will be a bolt-on option for anyone already running an FOI site on our software platform Alaveteli.
In April, mySociety team members traveled to roll out the first such project, with Info Pro Všechny, the Czech Republic’s Alaveteli site.
We were able to introduce beta users to the features we’ve been developing, such as the ability to keep requests private until the story has been published.
While in the Czech Republic, we held a roundtable discussion with journalists and campaigners, swapping Freedom of Information battle stories and sharing tips and tricks for getting the best results when submitting requests for information, as well as experiences of filing requests to European Union institutions.
mySociety was also invited to give a talk to student journalists based in Olomouc about Info Pro Všechny and Alaveteli Professional in general, discussing success stories generated from our Freedom of Information sites from around the world.
We’re currently working on subscription options, which will allow us to officially launch WhatDoTheyKnowPro as a paid-for service in the UK, and later in the year, we plan to introduce the Pro toolkit to the Belgian Alaveteli site Transparencia.be, which has been making a splash in Belgian politics.
You may know Dr Ben Goldacre from his ‘Bad Science’ and ‘Bad Pharma’ campaigns, which fight misinformation around medicine. Ben has just launched his latest project, the AllTrials Transparency Index — and mySociety helped with the website side of things.
The AllTrials campaign focuses around the fact that a shockingly large proportion of clinical trials do not have their results publicly published.
Not only does this devalue the time, goodwill and even potential risk put in by participants, but there are also issues around bias. Those trials published tend to be the ones which show positive results: if that doesn’t sound like such a terrible situation to you, try playing this game from the Economist magazine, which graphically depicts the problems with skewed coverage. At worst, such selective publication can be dangerous, or lead to poor choices from bodies making medical purchasing decisions.
Transparency and data visualisation are two areas where mySociety has a long history, and so it came to be that we fashioned the deceptively simple AllTrials Transparency Index site, on which anyone can browse the transparency index of the world’s major drug companies, and dive in deeper to the data to see how it was compiled. The source data is free for others to download too, so anyone can integrate it into other projects.
AllTrials are also tracking whether companies register new trials:
That’s the part that anyone can understand — and now, notes for the more technically-inclined who may be wondering how we took the data on each drugs company and presented it in a way that can be quickly and easily taken in.
This is an ongoing campaign with a commitment to future audits, so we wanted to make it easy for the AllTrials team to update the site and republish the source data each time they do.
It’s a static Jekyll site. We wrote a custom plug-in to parse a CSV and produce a page for each company within that CSV, as well as creating some summary data that feeds into the graphs on the front page.
This data is then pulled from the CSV, and D3 is employed to build the graphs and insert them into the generated pages.
The end result is a site that looks good and which can automatically update whenever the underlying data changes. We hope we’ll have played a small part in helping to ensure that it does — and for the better.
If you’ve been watching the progress of our Freedom of Information toolkit for journalists, campaigners and activists, you might be interested to know that we are now accepting applications for pre-launch access.
Successful applicants will have an early opportunity to put the service through its paces. WhatDoTheyKnowPro will launch as a paid-for service, but as a beta tester you’ll have up to a year’s access for free, because we’re keen to see how you’ll use it — and to hear your feedback on which features are useful.
WhatDoTheyKnowPro is our first launch of Alaveteli Professional, accessed via the WhatDoTheyKnow website and specifically for UK users who utilise FOI in their work or campaigning.
It’s the first instance of the service we plan to make available to other Freedom of Information sites running around the world on our Alaveteli platform.
What you’ll get
Since we last caught up with Alaveteli Professional, we’ve made really concrete progress with several of the features that, at the time of that blog post, were just entries on our long to-do list.
Here’s what users will access:
- The ability to keep requests private until your story has been published
- A powerful private dashboard that helps you track and manage your FOI projects
- A super-smart to-do list that makes it easier to follow the progress of your requests
- Action alerts that nudge you when it’s time to take the next step in a request
And very soon, we’ll also be carefully rolling out the batch request features, which will allow you to:
- Make one request to multiple authorities
- Manage large volumes of responses and easily keep track of the status of each request
- Get regular updates as the responses come in, without overwhelming your inbox
We’re excited about the batch feature in particular, and we know that many of our prospective users are, too. At the same time, we’ve heard some concerns that it might encourage a scattergun approach that wastes authorities’ time.
Our planned development will ensure that people use this feature responsibly, and, consequently, get the best returns from it. This will include a prompt to send a smaller batch initially, so that the remainder of the requests can be refined based on the quality of information that is returned — there’s nothing worse that asking every council in the country for information and then realising that you’ve worded your question in a way that means you can’t use the resulting data!
At the moment, batch requests can’t be made on WhatDoTheyKnow without help from the site administrators. We’re aware that many journalists and activists already make many batch requests outside WhatDoTheyKnow for this very reason. We’d like more of these requests to be released in public (we estimate that around 15% of UK FOI requests are made via our site): so by including this capability in WhatDoTheyKnowPro, we hope we’ll not only be steering people to use those powers sensibly, but that much more information will also end up in the public domain — maximising its usefulness.
Apply for free access
If that all sounds exciting, then apply here. We’d love to hear how you plan to use WhatDoTheyKnowPro.
It’s something we wanted to build, and something we believed there was a need for: but wanting and believing do not make a sound business case, and that’s why we spent the first few weeks of the project in a ‘discovery’ phase.
Our plan was to find out as much as we could about journalists, our prospective users — and particularly just how they go about using FOI in their work. Ultimately, though, we were seeking to understand whether journalists really would want, or need, the product as we were imagining it.
So we went and talked to people at both ends of the FOI process: on the one hand, journalists who make requests, and on the other, the information officers who respond to them.
Since we’re planning on making Alaveteli Professional available to partners around the world, it also made sense to conduct similar interviews outside the UK. Thanks to links with our Czech partner, running Informace Pro Všechny on Alaveteli, that was a simple matter. A recent event at the Times building in London also allowed us to present and discuss our findings, and listen to a couple of interesting expert presentations: Matt Burgess of Buzzfeed talked about some brilliant use of FOI to expose criminal landlords, and listed FOI officers’ biggest complaints about journalists. Josh Boswell of the Sunday Times was equally insightful as he ran through the ways that he uses FOI when developing stories.
These conversations have all helped.
The life of an investigative journalist is never simple
The insights our interviewees gave us were turned by Mike Thompson (formerly of mySociety, and brought back in for this phase) into a simple process model showing how journalists work when they’re pursuing an investigation using FOI.
After conceiving of a story that requires input from one or more FOI request, every journalist will go through three broad phases: research; request and response; and the final data analysis and writing. The more complicated cases can also involve refused requests and the appeals process.
For a busy working journalist, there are challenges at every step. Each of these adds time and complexity to the process of writing a story, which is an anathema to the normal daily news cycle. FOI-based stories can be slow, and timing unpredictable — editors do not particularly like being told that you’re working on a story, but can’t say when it will be ready, or how much value it will have.
During the research phase diligent journalists will make a time-consuming trawl through resources like authorities’ own disclosure logs and our own site WhatDoTheyKnow (or its equivalents in other countries), to see if the data they need has already been released.
Where a ‘round robin’ request is planned, asking for information from multiple authorities — sometimes hundreds — for information, further research is needed to ensure that only relevant bodies are included. In our two-tired council system, where different levels of authority deal with different responsibilities, and not always according to a consistent pattern, that can be a real challenge.
Wording a request also takes some expertise: get that wrong and the authorities will be coming back for clarification, which adds even more time to the process.
Once the request has been made it’s hard to keep on top of correspondence, especially for a large round robin request. Imagine sending a request to every council in the country, as might well be done for a UK-wide story, and then dealing with each body’s acknowledgements, requests for clarifications and refusals.
When the responses are in journalists often find that interpretation is a challenge. Different authorities might store data or measure metrics differently from one another; and pulling out a meaningful story means having the insight to, for example, adjust figures to account for the fact that different authorities are different sizes and cater for differently-dispersed populations.
Sadly, it’s often at this stage that journalists realise that they’ve asked the wrong question to start with, or wish that they’d included an additional dimension to the data they’ve requested.
What journalists need
As we talked through all these difficulties with journalists, we gained a pretty good understanding of their needs. Some of these had been on our list from the start, and others were a surprise, showing the value of this kind of exploration before you sit down to write a single line of code.
Here’s what our final list of the most desirable features looks like:
An embargo We already knew, anecdotally, that journalists tend not to use WhatDoTheyKnow to make requests, because of its public nature. It was slightly sobering to have this confirmed via first person accounts from journalists who had had their stories ‘stolen’… and those who admitted to having appropriated stories themselves! Every journalist we spoke to agreed that any FOI tool for their profession would need to include a way of keeping requests hidden until after publication of their story.
However, this adds a slight dilemma. Using Alaveteli Professional and going through the embargo-setting process introduces an extra hurdle into the journalist’s process, when our aim is, of course, to make the FOI procedure quicker and smoother. Can we ensure that everything else is so beneficial that this one additional task is worthwhile for the user?
Talking to journalists, we discovered that almost all are keen to share their data once their story has gone live. Not only does it give concrete corroboration of the piece, but it was felt that an active profile on an Alaveteli site, bursting with successful investigations, could add to a journalist’s reputation — a very important consideration in the industry.
Request management tools Any service that could put order into the myriad responses that can quickly descend into chaos would be welcome for journalists who typically have several FOI requests on the go at any one time.
Alaveteli Professional’s dashboard interface would allow for a snapshot view of request statuses. Related requests could be bundled together, and there would be the ability to quickly tag and classify new correspondence.
Round-robin tools Rather than send a notification every time a body responds (often with no more than an acknowledgement), the system could hold back, alerting you only when a request appears to need attention, or send you status updates for the entire project at predefined intervals.
Refusal advice Many journalists abandon a request once it’s been refused, whether from a lack of time or a lack of knowledge about the appeals process. WhatDoTheyKnow Professional would be able to offer in-context advice on refusals, helping journalists take the next step.
Insight tools Can we save journalists’ time in the research phase, by giving an easy representation of what sort of information is already available on Alaveteli sites, and by breaking down what kind of information each authority holds? That could help with terminology, too: if a request refers to data in the same language that is used internally within the council, then their understanding of the request and their response is likely to be quicker and easier.
Onwards to Alpha
We’re currently working on the next part of the build — the alpha phase.
In this, we’re building quick, minimally-functional prototypes that will clearly show how Alaveteli Professional will work, but without investing time into a fully-refined product. After all, what we discover may mean that we change our plans, and it’s better not to have gone too far down the line at that point.
If you are a journalist and you would like to get involved with testing during this stage and the next — beta — then please do get in touch at firstname.lastname@example.org.