We’ve added a new functionality to the Alaveteli Pro codebase, allowing you to download a zip file containing all correspondence and attachments from a batch, and a spreadsheet (csv) to show the progress status of every request.
Alaveteli Pro is our tool for professional users of Freedom of Information. If you’re UK-based, you’re probably most familiar with our local iteration WhatDoTheyKnow Pro — but don’t worry: when we talk about improvements to Alaveteli Pro, you can be sure they’re also part of the WhatDoTheyKnow toolkit.
How to export
You’ll find these tools at the foot of the batch container in the requests list.
Why data exports?
Of course, we like to think Alaveteli Pro is a useful tool in its own right: there’s a lot you can do within the Pro interface, and it was built specifically to help you keep track of all your FOI activity in one place.
But sometimes users want to use external tools – either because they’re just more familiar with them, or because they want to do something beyond the functionality we offer.
Now there’s a simple way to get data out of Alaveteli, allowing you to analyse it with the tools of your choice, or perhaps send a progress report to a supervisor or editor.
It’s part of a programme of work to support cross border journalism between European organisations, supported by Adessium Foundation, allowing us to refine and improve the codebase for the benefit of all Pro users.
The technical bit
Those with a bit of coding knowledge may be interested to hear how we approached the zip download functionality. mySociety developer Graeme explains:
“With batch requests potentially going to as many as 500 different authorities, each request can receive several responses and attachments in return.
“All these emails and files mean that compiling the zip for download could be a lengthy job and would normally cause the request to time out. So for this new feature we’re utilising file streaming to send chunks of the zip as they become available.
“This means that the zip starts downloading immediately and you don’t have to sit watching and wondering whether anything is happening – you can see more and more data being transmitted.”
We hope you find this new feature useful. Please do let us know how you’re using it and any feedback you may have.
Image: Startup Stock Photos
We’ve just come back from a heady couple of days in Oslo, where our AlaveteliCon event brought together those with a shared interest in the technology around Freedom of Information — in all, around 50 journalists, researchers, technologists and activists from 18 different countries.
As our Head of Development Louise announced in her opening words, AlaveteliCon has always been a slight misnomer, given that we’re keen to share knowledge not just with those who use Alaveteli, but with all the FOI platforms in our small but growing community — including MuckRock in the US and Frag Den Staat in Germany, both of whom were in attendance.
It was a timely event for us, as we embark on work to introduce our Alaveteli Pro functionality to newsrooms, researchers and campaigners across Europe, with an emphasis on encouraging cross-border collaboration in campaigns, research and journalistic investigations.
As well as picking up practical tips, we heard a variety of inspiring and instructive stories from FOI practitioners around the world; brainstormed ways forward in increasingly difficult political times; and shared knowledge on funding, publicity, site maintenance, and how to keep good relations with FOI officers.
Some of the most inspiring sessions came when delegates shared how they had used FOI in campaigns and investigations, from Vouliwatch’s Stefanos Loukopoulos explaining how they had taken their own government to court, to Beryl Lipton of MuckRock explaining why the government use of algorithms can have effects that are unforeseen, and indeed petrifying.
There was an affecting story from freelance journalist Mago Torres, who told us about a long campaign to map clandestine graves of those caught up in the war against drugs in Mexico; and from Camilla Graham Wood of Privacy International, on that organisation’s work to uncover some of the rather sinister but not widely known technologies being put into use by police services in the UK.
So much knowledge came out of these two days. We don’t want to lose it, so we’ll be making sure to update the conference page with photos, videos and the speakers’ slides as soon as they’re available. Meanwhile, you can follow the links from the agenda on that page to find the collaborative documents where we took notes for each session.
We’ve just released two new versions of Alaveteli, our open source platform for running Freedom Of Information sites — one which we’ve packed all of bug fixes, new features and performance improvements into, and one with some important technical updates. Here are some of the highlights.
Making your first request easier
mySociety’s designer Zarino has subtly redesigned the authority search page – which is the first thing most users will see when making a request – to make it easier to find the help links at a glance, making the process feel much more approachable.
Encouraging better quality requests
Part of our advice to users of WhatDoTheyKnow is to keep requests concise and focused. This new feature adds a visual reminder when the request text is approaching the point of being too long for an FOI officer to answer it efficiently:
With a second, stronger warning if the request becomes longer still:
We’ve also added an experimental feature on WhatDoTheyKnow to discourage users from making requests for personal information. (As well as not being valid FOI requests, publishing the responses would result in revealing confidential information about the requester.) By asking the user to clarify what they want to ask before sending a request to authorities which attract a high level of personal information requests, we can try to redirect them to the correct official channel instead:
One-click unsubscribe from notifications
An improvement for site owners and end users: it is now possible to unsubscribe from email notifications about requests that you’re following right from your inbox! (Previously – and still the case with other sorts of email that the site sends – you would have to visit the site and log in to change your email preferences.)
Allowing people to unsubscribe with ease – if they’ve forgotten that they signed up for notifications or have created a new track by mistake – should help to cut down on the number of messages being marked as spam which should in turn help improve the site’s mail sender reputation. (It also allows admins – if they receive feedback loop notification emails from email providers – to unsubscribe users who’ve marked these emails as spam, preventing further unwelcome emails being sent.)
In version 0.33, as well as our our features and fixes, we’ve added support for Ubuntu releases Xenial and Bionic and withdrawn support for Debian Jessie. The full list of highlights and upgrade notes for this release is in the changelog.
Version 0.34 contains our Rails 5.0 upgrade work which we’ve released separately to allow reusers time to adapt to new minimum requirements for operating systems and Ruby versions.
We’re no longer supporting Ubuntu Trusty and have also dropped Ruby versions older than 2.3 as that’s the minimum requirement for Rails 5. Upgrade notes are available in the changelog and, as ever, we recommend updating one version at a time to make sure everything’s working smoothly and reduce the risk of missing essential upgrade steps.
Moving to Rails 5.0 will allow us to retain support for major security issues when Rails 6 is released and dropping older Ruby versions removes some key technical barriers to modernising the Alaveteli codebase and allows us to focus on improving Alaveteli Pro so that it can be reused more widely.
Thanks again to everyone who’s contributed! Special thanks to Nigel Jones and Laurent Savaëte who contributed bug fixes for version 0.33!
Next week Gareth and I will be heading to Tunis to attend the 8th edition of RightsCon. RightsCon is the world’s leading summit on human rights in the digital age, so we’re thrilled to be hosting a session at the conference about digital Access to Information platforms with our awesome friends at MuckRock.
If you’ll also be there we’d love to talk to you about your campaigns or investigations and how using access to information platforms could help.
As Jen said in her recent blog post, we’ll be spending time this year developing our software platform Alaveteli Pro so more people across the world have access to its digital tools that help with the sending and management of information requests.
We’d love to get feedback on this work and would love to meet organisations who are interested in setting up Alaveteli Pro instances, in order to make access to information easier for citizens in their countries. We’re also very keen to talk to individuals and organisations who are interested in collaborating on cross-border public-interest investigations and campaigns using FOI-generated data.
We’d also love to talk to RightsCon attendees who might be interested in attending our AlaveteliCon event in Oslo on 23 and 24 September, where activists, journalists, technologists and campaigners from across the world will come together to discuss Freedom of Information technologies for creating public-interest investigations and campaigns.
And of course, our Call for Proposals is currently open for our TICTeC Local conference so it’d be great to chat to people interested in presenting their work using digital innovations to help local communities and/or public authorities to foster citizen engagement, drive efficiency, and combat social and environmental problems.
Using WhatdoTheyKnow Pro, this project pieced together a nationwide dataset, and generated important stories at both national and local levels.
Sold from Under You, a project from the Bureau of Investigative Journalism, revealed how much publicly-owned property has been sold off across England, as a response to austerity measures. In all, TBIJ discovered that over 12,000 buildings and pieces of land have been disposed of, bringing councils revenue of £9.1 billion — some of which has been spent on staff redundancies.
In collaboration with HuffPost, the findings were presented in the form of an interactive map which allows users to explore sales in their own area.
The investigation required a significant amount of data collection via FOI requests to 353 councils, work which was aided by WhatDoTheyKnow Pro. More than 150 people across the UK, including local journalists, took part in the collaborative investigation. As well as HuffPost’s coverage, stories were run in regional news outlets across the country. The project has now been shortlisted for the Data Journalism awards.
We spoke to Gareth Davies from TBIJ to understand how the organisation approached this ambitious project, and what part WhatDoTheyKnow Pro played in it. Here’s what he told us:
“The Bureau has been investigating the local government funding crisis in the UK for the last 18 months. The initial part of this particular investigation focused on the overall financial health of local authorities and used data to determine which were under the most pressure. We then wanted to look at the impact of the funding crisis so teamed up with Hazel Sheffield and her Far Nearer project to look at the public spaces that were being lost as a result.
“At the start of the investigation we undertook a research period to determine what local authorities are required to publish about the buildings and land they own, and how many of them were adhering to those rules.
“We discovered that while councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid.
“Also, two thirds of councils update the same spreadsheet each year, meaning change over time is lost. As a result it became apparent that FOI would be required to obtain the information we were interested in. FOI is a tool we have used for a number of stories, particularly those produced by our Bureau Local team.
“The information we wanted could be divided into two groups: what assets councils were buying and selling, and what they were doing with the money raised when an asset is sold. The research period showed we would need FOI to obtain this data.”
More than 700 FOI requests
“To reduce the risk of requests being refused for exceeding the cost/time limit, we needed to submit two separate requests to each of the 353 local authorities in England.
“Previously I had submitted and managed bulk FOI requests via email. However, staying on top of more than 700 requests would have proven very challenging. I was aware of the WhatDoTheyKnow Pro platform but hadn’t used it before, so thought this would be the ideal opportunity to test it out.
I don’t think I would have achieved that without WhatDoTheyKnow Pro
“It was useful to have up-to-date contact details for each authority and to be able to send the FOI requests in one go. But probably the most useful feature was the way in which WhatDoTheyKnow Pro tracks the status of each request and shows you when the public body in question has exceeded the statutory time limit. This made it a lot easier to stay on top of which councils needed to be chased and when I needed to do it.
“Managing so many FOI requests was still challenging and very time consuming but it would have been much harder by email. The first batch of requests had a success rate of more than 95% and the other (which was more detailed) was around 85%.
“I don’t think I would have achieved that without WhatDoTheyKnow Pro and, as a result, the investigation and interactive map we created would not have been as comprehensive.”
Refining the requests
While councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid
“I sent requests to one of each type of local authority (London borough, metropolitan borough, unitary, county and district) to test what, if any, information councils would provide. The fact that all of those requests were successful meant I had confidence when submitting the batch requests.
“It also allowed me to include additional information in the bulk requests, because some of the test councils erroneously withheld, under Section 40, the identities of companies. As a result I added a note to the request highlighting that this would not be a correct application of that exemption.
“As each response came in I recorded them in two separate spreadsheets — one showing what assets had been bought/sold and another containing information about how the money raised from asset sales had been used. Gradually we built a comprehensive picture of what was happening with public spaces, and that was crucial for our story.”
Bringing about change
There have been tangible results from this investigation.
“The government launched an investigation into the sale of assets by Peterborough Council as a result of this particular story, focusing on that area.
“We submitted our findings to an inquiry currently being held by the Communities and Local Government select committee and were mentioned by name during the first day of oral hearings.
“And last month the Public Accounts Committee announced it would hold a similar inquiry into the sale of public land. Several councils halted their property investment policies after our coverage revealed how much they had borrowed to fund the purchases.”
Thank you very much to Gareth Davies for talking to us about the Sold From Under You project.
Image: Daniel von Appen
WhatDoTheyKnow Pro, mySociety’s subscription service offering extra tools for journalists and other professional users of FOI, has been running in the UK for just about two years.
During that time we’ve launched, worked closely with users to refine the service, and — happily — watched it play a vital part in the making of several important data-driven news stories, on topics as diverse as Brexit campaign funding and the results of austerity cuts on councils. Journalists, in particular, have appreciated tools such as the ability to send and manage bulk requests to multiple authorities; and the embargo tool that keeps requests and responses hidden until the story has been published.
Now, thanks to support from Adessium Foundation, we are able to bring the same benefits to countries across Europe, and — we hope — some additional synergies that will be borne of organisations working across boundaries. The same functionality that extends WhatDoTheyKnow into the Pro version will be available to FOI sites run on the Alaveteli platform, under the name Alaveteli Pro.
The ultimate aim is to enable journalists, campaigners and citizens in Europe to make greater and more effective use of their right to access information; and in particular to generate public interest stories and campaigns that will hold power to account.
We’ll be focusing on three areas in order to achieve this aim:
- We’ll give selected existing Alaveteli sites in Europe the technical help they need to upgrade to the Pro version;
- We’ll be helping organisations in three new European jurisdictions to launch brand new Alaveteli sites, making access to information easier for citizens in these countries. The first site will be launched by VVOJ from the Netherlands.
- We’ll encourage cross-border collaborations between journalists and organisations using the sites (both the existing ones and the new ones) to investigate stories that span more than one EU country.
So watch this space: we’ll be sure to keep you posted as the work progresses. The planned start date is next month, and the project is set to run for three years.
We’re looking forward to sharing stories resulting from this initiative once they start rolling out, and supporting the incredible work that journalists do in putting them together.
Image: Emiliano Vittoriosi
We’re delighted to be hosting the third AlaveteliCon, our Freedom of Information (FOI) technologies conference, on 23 and 24 September 2019 in Oslo, Norway.
A few days ahead of International Right to Know Day (28 September), AlaveteliCon will bring together activists, journalists, technologists and campaigners from across the world who use Freedom of Information laws, data and technologies to create public-interest investigations and campaigns.
This time we’ll be including a focus on catalysing collaborations on cross-border European public interest investigations and campaigns, in order to strengthen the use of FOI across the region. This focus is in part due to our latest Transparency project, funded by Adessium Foundation. This Transparency project seeks to enable journalists, campaigners and citizens in Europe to make greater and more effective use of their right to access information, in particular to generate public interest stories and campaigns that hold power to account.
Here are some of the other things AlaveteliCon attendees will get up to this time:
- Exchanging insights and ideas on how to run and improve open-source Freedom of Information platforms such as WhatDoTheyKnow, MuckRock and FragDenStaat
- Hearing how journalists and campaigners around the world have used FOI to power high-profile investigations and campaigns
- Making connections with journalists and campaigners who would like to collaborate on cross-border investigations that hold governments to account
- Learning about data sources available on FOI platforms around the world, waiting to be analysed and turned into public-interest stories
- Hearing tips from FOI activists on getting governments to release information in their countries including going to court
- Contributing to the further development of mySociety’s open-source FOI toolkit for journalists, Alaveteli Pro, which helps journalists and campaigners to manage their FOI investigations
- Connecting to a global network of FOI experts and advocates
- Working together to plan potential joint activities to celebrate International Right to Know Day on 28th September 2019
Apply to attend
As spaces at AlaveteliCon are limited, we’re asking those wishing to come to fill out this application form to tell us what unique perspective they can bring that’s relevant to the above conference themes/topics.
Are you a journalist or campaigner who uses FOI for investigations/campaigns? Do you want to collaborate on FOI-generated investigations? Or interested in setting up an FOI platform in your country? Or want to learn how to use FOI to its fullest potential? Or interested in any of the above topics? If so, then please submit your application by 2nd September at the very latest.
We’ll let you know if you’ve been allocated a place by Friday 6th September at the latest. However, we will endeavour to reply to you sooner than this.
AlaveteliCon is free to attend, but delegates will be required to arrange their own travel, accommodation and costs in Oslo.
We’ve just released version 0.32 of Alaveteli, our open source platform for running Freedom Of Information sites. Here are some of the highlights.
Making correspondence threads easier to navigate
Thanks to our designers, it’s now possible to collapse individual messages in a correspondence thread in order to focus on just the parts you’re trying to read. Plus you can quickly collapse (or expand) all the messages in the thread using the “Collapse all” and “Expand all” links from the “Actions” menu.
Alaveteli Pro users gain the additional benefit of a redesigned sidebar which allows for easier navigation of lengthy correspondence and avoids having to scroll to the top of the request thread to update its status. See Martin’s full explanation here.
Better password security
We’ve started enforcing stricter password length constraints wherever a password is set or updated to help users keep their accounts secure. And we’re also using a stronger encryption method for storing password data, using bcrypt rather than the older SHA1 algorithm to obscure the actual password. (Be sure to run the rake task documented in the release upgrade notes to upgrade secure password storage for all existing users.)
You can read more about what this does and why it’s important if you’re interested in the technical details behind this upgrade.
Authorities not subject to FOI law
We’ve adopted WhatDoTheyKnow’s
foi_notag for authorities to indicate that although the authority is listed on the site, it is not legally subject to FOI law. This could be for advocacy purposes – if it’s felt an authority should be covered by legislation – or where the authority has agreed to respond on a voluntary basis.
foi_notag now causes an extra message to appear under the authority’s name on their page and on all related requests, and removes language about legal responsibilities to reply from the messages sent to users.
To improve the UI, we’ve made a similar change for authorities with the
eir_onlytag to make it clearer that such authorities are only accepting requests about the environment.
(Don’t worry admins, you don’t need to remember all this – we’ve updated the documentation on the edit page to reflect the new functionality!)
Improvements for site admins
We’ve made it easier for admins to ban users who sign up to post spam links in their profile. There’s now a “Ban for spamming” button which is available on the user edit page or as soon as you expand the user’s details in the listing rather than having to manually edit user metadata.
We’ve also made it harder to leave requests flagged as vexatious (or “not_foi”) in an inconsistent state. Previously the site just assumed that vexatious requests would always be hidden. Now the admin interface enforces the hiding of vexatious requests by showing warnings when a request is set as vexatious while it’s visible on the site, and prevents the updated request from being saved until a valid state is selected.
And last but not least – introducing the new Announcements feature!
Easier popup banner management
Site admins will be relieved to hear that they can now update the popup banner message on the site without needing to schedule developer time.
This feature supports multi-language sites so if you set the announcement for your main (default) language, it will appear across all language versions that you have not added a specific translation for.
You can set announcements that will only be seen by fellow administrators when they visit the summary page. (If you’re running a Pro site, you can also have announcements that will only be seen by your Pro admins.)
Announcements for Pro users appear as a carousel at the top of their dashboard. So far we’ve used it on WhatDoTheyKnow Pro to publicise new features, offer discount codes, and encourage people to share their published stories with us.
The full list of highlights and upgrade notes for this release is in the changelog.
Thanks again to everyone who’s contributed!
Last year, we launched WhatDoTheyKnow Pro, our service for journalists and other professional users of Freedom of Information.
As it’s a new venture, we’re keen to track whether it’s achieving everything we’d hoped for when first planning the service. One of the targets which we set, as a measure of success, is the number of impactful press stories generated by its use.
What might count as impactful? Well, that’s obviously up for debate, but loosely we’d say that we’re looking for news stories that have a wide readership, and uncover previously unknown facts, offer new insights, or bring about change.
Stories in the news
A couple of recent stories, generated through WhatDoTheyKnow Pro, have ticked at least a few of those boxes. At mySociety, we keep a position of political neutrality — our services are available to everyone of any persuasion, and we don’t campaign on any political issue — so we present these stories not to comment on their substance, but to note that they certainly fit the criteria above.
Brexit is clearly one of the most vital stories of our day, here in the UK, and while many might feel that we’ve had a surfeit of commentary on the issue, we can only benefit from understanding the facts.
One of WhatDoTheyKnow Pro’s earliest users, Jenna Corderoy, broke two important stories on that topic. First, that UK parliamentary standards watchdog IPSA is investigating Jacob Rees Mogg’s hard Brexit European Research Group over their second bank account and ‘informal governance structure’. This also ran in the Daily Mail.
Secondly (and this progresses a previous story about expenses – also uncovered thanks to Jenna’s use of WhatDoTheyKnow Pro) there is the widely-reported story that the Electoral Commission had misinterpreted laws around campaigning expenses, allowing Vote Leave to overspend. This was picked up by the BBC and Guardian, among many others.
No matter which side of the Brexit debate you support, hopefully you will agree that it benefits society as a whole to have the facts out in the open.
From FOI request to the national news
As a last thought: it’s interesting to us to see how a story grows from one or more FOI requests into something that hits the national news platforms. We think these stories were broken using a tried and true method that goes like this:
- As a journalist, campaigner or researcher, you might be investigating a topic. Perhaps you’ve heard a rumour that you’re hoping to substantiate, or perhaps you’re inquiring more deeply into a story that’s already in the air. Using FOI, you can retrieve facts and figures to bolster your investigation.
- Once you have a story, you can publish it in a smaller publication like Open Democracy, testing the water to see if it gains any traction.
- If the story is well-received there, it’s easy to approach larger outlets with the proof (in the form of FOI responses) underpinning it.
If you’re a journalist or you use FOI in your professional life and you’d like to try WhatDoTheyKnow Professional for yourself, then head over to whatdotheyknow.com/pro. Put in the code WELCOME18 when you sign up, and you’ll get 25% off your first month’s subscription.
Image: Roman Kraft
WhatDoTheyKnow Pro is our Freedom of Information service for journalists, and campaigners, and we’ve recently rolled out some major changes to the request sidebar to make reading, navigating, and classifying Pro requests a lot easier.
Since the very first Alpha version of WhatDoTheyKnow Pro we’ve been receiving feedback from our users, which we have been feeding directly into our future development plans. The sidebar changes are the first round of changes that have come as result of direct Pro user feedback, and there will be more to follow.
In WhatDoTheyKnow a member of the public can send a Freedom of Information request to an authority, which they receive in the form of an email. The request, as well as any replies or follow up from the requester, or the authority, are published on WhatDoTheyKnow. If the request is made by a Pro user, they have the added option of making a request private for a limited time.
In the request process we observed the following:
- A new response from an authority goes to the bottom of the thread (the bottom of the page)
- The user interface for updating the status of a request is located at the top of the page (a request’s’ status is a way of keeping track of where it is in the request process – for example ‘awaiting response’, ‘needs clarification’, or ‘refused’)
- A longstanding or complicated request will often consist of many, many messages. So scrolling to the bottom of the page to read the most recent response, then back to the top to update its status involves a lot of interaction that we can remove.
We can’t say for sure that a user will always be at the bottom of the request thread when updating the status of a request, but we can safely assume they are sometimes.
For these changes we set ourselves the following goals:
- Speed up the process of updating the status of requests
- Improve the experience of navigating requests with a long history.
In previous research we established that a typical workflow for dealing with request responses is:
Get reply → Read reply → Take action (typically reply, or update the status of the request)
It’s a short, three step process, but a busy user catching up with a backlog may do this hundreds of times a day, so if we can optimise this workflow we can save a lot of time and frustration.
So what have we done?
For desktop users we’ve made the sidebar controls (where the ‘update status’ button is) “sticky”, so it will follow you as you scroll up and down the page, meaning you can update the request status from any position on the page. This really helps as requests get longer, as you no longer need to scroll back to the top to classify the latest response.
We’ve added new message navigation buttons. This is to enable you to move through a request thread message-by-message by clicking the up and down arrow buttons, or using the arrow keys on your keyboard. We’ve also added a counter so that it’s easier to see where you are in the list, and to go back and forth to specific messages.
We’ve also taken this opportunity to make some key information about the privacy of your request visible at all times (this was previously hidden behind a click), and to tweak the design of the sidebar – making it easier to read and removing some visual noise.
More to do
We’re looking at a way to add similar functionality to mobiles and other small screen devices. As screen space is limited it will require a separate design process.
We’re aware that the problems we’re trying to solve aren’t unique to our Pro customers, so if the features work, and are well received, we’ll be making a similar feature available to all WhatDoTheyKnow users in the future.
Keeping in mind that as our public users have different needs to our Pro users there are some design challenges to overcome beforehand. For example – the public request page has more features to help less-frequent users, because we’re keen to ensure that everyone can participate in the FOI process, not just experts. Conversely, Pro users are by their very nature more likely to require less guidance. We’re going to need to do more research on this shortly.
Got some feedback?
Whether you’re a WhatDoTheyKnow Pro customer or not, we’d love feedback on this feature — or any other. Drop us an email to firstname.lastname@example.org.