We’re delighted to announce that we’ve received funding from the Swedish Postcode Foundation that will help us extend our work on Freedom of Information in Europe.
The Foundation uses proceeds from the country’s lottery sales to help fund projects that support democracy and freedom of speech, as one of three areas where they believe they can help bring about long term positive change to the world.
The connection is particularly apt, as it was in Sweden that the world’s first FOI law was passed in 1766. From that beginning grew a worldwide good: since then, access to information has been recognised as a fundamental right by the European Court of Human Rights, and has been adopted in countries around the globe.
In May 2019 we received funding from Adessium Foundation for a three-year project to increase access to online FOI tools across Europe. The ultimate aim is to enable journalists, campaigners and citizens in Europe to make greater and more effective use of their right to access information; and in particular to generate public interest stories and campaigns that will hold power to account.
Now this new match funding will allow us to dig further and build better within the main elements of the project, which are:
- To help partners to launch new FOI sites in the Netherlands, France (already completed) and another jurisdiction (coming soon).
- To upgrade existing sites to include the Alaveteli Pro functionality: AskTheEU already has this and five others will gain it shortly. By 2022 there’ll be 13 Alaveteli sites in Europe, 10 of which will have Pro.
- To improve the Alaveteli Pro software with new features that’ll make it a more powerful tool for investigations and campaigns (so far we’ve worked on exporting data from batch requests and enabling users to add links to news stories).
- To support journalist and campaigning organisations to use Alaveteli tools as part of their investigations (such as Privacy International’s use of FOI in their investigation into surveillance technologies used by police in the UK).
- To monitor government compliance with FOI, especially in the wake of the coronavirus pandemic.
Now we can spread the goodness even further, so we’re planning to run some online training/learning activities around using Alaveteli tools as part of an investigation or campaign. If your work would benefit from this, and you live in an EU country with an Alaveteli Pro site, do get in touch.
We’re also keen to partner with membership-based news or campaign organisations to run more pilot projects using our new Projects feature. If you have a project that could benefit from contributors helping to extract and analyse data from FOI responses, let us know.
And finally: we’ll soon be starting to gather data about FOI compliance in different EU countries. If this is something that could benefit your work, register your interest and we’ll keep you posted.
Image: Jonathan Brinkhorst
Investigative journalism platform The Ferret has just launched an online training course on using Freedom of Information — and all trainees get a free subscription to our WhatDoTheyKnow Pro service for professional users of FOI.
Based in Edinburgh, the Ferret is a community journalism initiative that describes itself as ‘for Scotland and beyond’. Since 2012 its members’ investigations have rooted out the truth around local, national and international issues including coronavirus, Brexit, dark money — and much more. They’re a co-operative, so supporters become part-owners. If they want to, they can also access the resources and training to pursue their own stories.
And now, the Ferret’s online Freedom of Information course shares everything the founders know about the use of FOI for tracking down facts. This resource would be useful for anyone wanting to know the ins and outs of the act and how to use it, not just for journalism but potentially for campaigning or research purposes too. And it’s not just restricted to the use of FOI in Scotland: you’ll learn everything you need to know to use FOI across the UK… and beyond.
The course costs £30, but six months’ WhatDoTheyKnow Pro usage is bundled in. Since that’s worth £60 on its own, you’re ahead before you even begin.
We’re big fans of the Ferret at mySociety, and we have every confidence that this course will be a springboard for a new generation of great investigative journalists. If you think you might like to be one of them, then why not give it a try? More details here, and in this Twitter thread.
We’ve added a new functionality to the Alaveteli Pro codebase, allowing you to download a zip file containing all correspondence and attachments from a batch, and a spreadsheet (csv) to show the progress status of every request.
Alaveteli Pro is our tool for professional users of Freedom of Information. If you’re UK-based, you’re probably most familiar with our local iteration WhatDoTheyKnow Pro — but don’t worry: when we talk about improvements to Alaveteli Pro, you can be sure they’re also part of the WhatDoTheyKnow toolkit.
How to export
You’ll find these tools at the foot of the batch container in the requests list.
Why data exports?
Of course, we like to think Alaveteli Pro is a useful tool in its own right: there’s a lot you can do within the Pro interface, and it was built specifically to help you keep track of all your FOI activity in one place.
But sometimes users want to use external tools – either because they’re just more familiar with them, or because they want to do something beyond the functionality we offer.
Now there’s a simple way to get data out of Alaveteli, allowing you to analyse it with the tools of your choice, or perhaps send a progress report to a supervisor or editor.
It’s part of a programme of work to support cross border journalism between European organisations, supported by Adessium Foundation, allowing us to refine and improve the codebase for the benefit of all Pro users.
The technical bit
Those with a bit of coding knowledge may be interested to hear how we approached the zip download functionality. mySociety developer Graeme explains:
“With batch requests potentially going to as many as 500 different authorities, each request can receive several responses and attachments in return.
“All these emails and files mean that compiling the zip for download could be a lengthy job and would normally cause the request to time out. So for this new feature we’re utilising file streaming to send chunks of the zip as they become available.
“This means that the zip starts downloading immediately and you don’t have to sit watching and wondering whether anything is happening – you can see more and more data being transmitted.”
We hope you find this new feature useful. Please do let us know how you’re using it and any feedback you may have.
Image: Startup Stock Photos
We’ve just come back from a heady couple of days in Oslo, where our AlaveteliCon event brought together those with a shared interest in the technology around Freedom of Information — in all, around 50 journalists, researchers, technologists and activists from 18 different countries.
As our Head of Development Louise announced in her opening words, AlaveteliCon has always been a slight misnomer, given that we’re keen to share knowledge not just with those who use Alaveteli, but with all the FOI platforms in our small but growing community — including MuckRock in the US and Frag Den Staat in Germany, both of whom were in attendance.
It was a timely event for us, as we embark on work to introduce our Alaveteli Pro functionality to newsrooms, researchers and campaigners across Europe, with an emphasis on encouraging cross-border collaboration in campaigns, research and journalistic investigations.
As well as picking up practical tips, we heard a variety of inspiring and instructive stories from FOI practitioners around the world; brainstormed ways forward in increasingly difficult political times; and shared knowledge on funding, publicity, site maintenance, and how to keep good relations with FOI officers.
Some of the most inspiring sessions came when delegates shared how they had used FOI in campaigns and investigations, from Vouliwatch’s Stefanos Loukopoulos explaining how they had taken their own government to court, to Beryl Lipton of MuckRock explaining why the government use of algorithms can have effects that are unforeseen, and indeed petrifying.
There was an affecting story from freelance journalist Mago Torres, who told us about a long campaign to map clandestine graves of those caught up in the war against drugs in Mexico; and from Camilla Graham Wood of Privacy International, on that organisation’s work to uncover some of the rather sinister but not widely known technologies being put into use by police services in the UK.
So much knowledge came out of these two days. We don’t want to lose it, so we’ll be making sure to update the conference page with photos, videos and the speakers’ slides as soon as they’re available. Meanwhile, you can follow the links from the agenda on that page to find the collaborative documents where we took notes for each session.
We’ve just released two new versions of Alaveteli, our open source platform for running Freedom Of Information sites — one which we’ve packed all of bug fixes, new features and performance improvements into, and one with some important technical updates. Here are some of the highlights.
Making your first request easier
mySociety’s designer Zarino has subtly redesigned the authority search page – which is the first thing most users will see when making a request – to make it easier to find the help links at a glance, making the process feel much more approachable.
Encouraging better quality requests
Part of our advice to users of WhatDoTheyKnow is to keep requests concise and focused. This new feature adds a visual reminder when the request text is approaching the point of being too long for an FOI officer to answer it efficiently:
With a second, stronger warning if the request becomes longer still:
We’ve also added an experimental feature on WhatDoTheyKnow to discourage users from making requests for personal information. (As well as not being valid FOI requests, publishing the responses would result in revealing confidential information about the requester.) By asking the user to clarify what they want to ask before sending a request to authorities which attract a high level of personal information requests, we can try to redirect them to the correct official channel instead:
One-click unsubscribe from notifications
An improvement for site owners and end users: it is now possible to unsubscribe from email notifications about requests that you’re following right from your inbox! (Previously – and still the case with other sorts of email that the site sends – you would have to visit the site and log in to change your email preferences.)
Allowing people to unsubscribe with ease – if they’ve forgotten that they signed up for notifications or have created a new track by mistake – should help to cut down on the number of messages being marked as spam which should in turn help improve the site’s mail sender reputation. (It also allows admins – if they receive feedback loop notification emails from email providers – to unsubscribe users who’ve marked these emails as spam, preventing further unwelcome emails being sent.)
In version 0.33, as well as our our features and fixes, we’ve added support for Ubuntu releases Xenial and Bionic and withdrawn support for Debian Jessie. The full list of highlights and upgrade notes for this release is in the changelog.
Version 0.34 contains our Rails 5.0 upgrade work which we’ve released separately to allow reusers time to adapt to new minimum requirements for operating systems and Ruby versions.
We’re no longer supporting Ubuntu Trusty and have also dropped Ruby versions older than 2.3 as that’s the minimum requirement for Rails 5. Upgrade notes are available in the changelog and, as ever, we recommend updating one version at a time to make sure everything’s working smoothly and reduce the risk of missing essential upgrade steps.
Moving to Rails 5.0 will allow us to retain support for major security issues when Rails 6 is released and dropping older Ruby versions removes some key technical barriers to modernising the Alaveteli codebase and allows us to focus on improving Alaveteli Pro so that it can be reused more widely.
Thanks again to everyone who’s contributed! Special thanks to Nigel Jones and Laurent Savaëte who contributed bug fixes for version 0.33!
Next week Gareth and I will be heading to Tunis to attend the 8th edition of RightsCon. RightsCon is the world’s leading summit on human rights in the digital age, so we’re thrilled to be hosting a session at the conference about digital Access to Information platforms with our awesome friends at MuckRock.
If you’ll also be there we’d love to talk to you about your campaigns or investigations and how using access to information platforms could help.
As Jen said in her recent blog post, we’ll be spending time this year developing our software platform Alaveteli Pro so more people across the world have access to its digital tools that help with the sending and management of information requests.
We’d love to get feedback on this work and would love to meet organisations who are interested in setting up Alaveteli Pro instances, in order to make access to information easier for citizens in their countries. We’re also very keen to talk to individuals and organisations who are interested in collaborating on cross-border public-interest investigations and campaigns using FOI-generated data.
We’d also love to talk to RightsCon attendees who might be interested in attending our AlaveteliCon event in Oslo on 23 and 24 September, where activists, journalists, technologists and campaigners from across the world will come together to discuss Freedom of Information technologies for creating public-interest investigations and campaigns.
And of course, our Call for Proposals is currently open for our TICTeC Local conference so it’d be great to chat to people interested in presenting their work using digital innovations to help local communities and/or public authorities to foster citizen engagement, drive efficiency, and combat social and environmental problems.
Using WhatdoTheyKnow Pro, this project pieced together a nationwide dataset, and generated important stories at both national and local levels.
Sold from Under You, a project from the Bureau of Investigative Journalism, revealed how much publicly-owned property has been sold off across England, as a response to austerity measures. In all, TBIJ discovered that over 12,000 buildings and pieces of land have been disposed of, bringing councils revenue of £9.1 billion — some of which has been spent on staff redundancies.
In collaboration with HuffPost, the findings were presented in the form of an interactive map which allows users to explore sales in their own area.
The investigation required a significant amount of data collection via FOI requests to 353 councils, work which was aided by WhatDoTheyKnow Pro. More than 150 people across the UK, including local journalists, took part in the collaborative investigation. As well as HuffPost’s coverage, stories were run in regional news outlets across the country. The project has now been shortlisted for the Data Journalism awards.
We spoke to Gareth Davies from TBIJ to understand how the organisation approached this ambitious project, and what part WhatDoTheyKnow Pro played in it. Here’s what he told us:
“The Bureau has been investigating the local government funding crisis in the UK for the last 18 months. The initial part of this particular investigation focused on the overall financial health of local authorities and used data to determine which were under the most pressure. We then wanted to look at the impact of the funding crisis so teamed up with Hazel Sheffield and her Far Nearer project to look at the public spaces that were being lost as a result.
“At the start of the investigation we undertook a research period to determine what local authorities are required to publish about the buildings and land they own, and how many of them were adhering to those rules.
“We discovered that while councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid.
“Also, two thirds of councils update the same spreadsheet each year, meaning change over time is lost. As a result it became apparent that FOI would be required to obtain the information we were interested in. FOI is a tool we have used for a number of stories, particularly those produced by our Bureau Local team.
“The information we wanted could be divided into two groups: what assets councils were buying and selling, and what they were doing with the money raised when an asset is sold. The research period showed we would need FOI to obtain this data.”
More than 700 FOI requests
“To reduce the risk of requests being refused for exceeding the cost/time limit, we needed to submit two separate requests to each of the 353 local authorities in England.
“Previously I had submitted and managed bulk FOI requests via email. However, staying on top of more than 700 requests would have proven very challenging. I was aware of the WhatDoTheyKnow Pro platform but hadn’t used it before, so thought this would be the ideal opportunity to test it out.
I don’t think I would have achieved that without WhatDoTheyKnow Pro
“It was useful to have up-to-date contact details for each authority and to be able to send the FOI requests in one go. But probably the most useful feature was the way in which WhatDoTheyKnow Pro tracks the status of each request and shows you when the public body in question has exceeded the statutory time limit. This made it a lot easier to stay on top of which councils needed to be chased and when I needed to do it.
“Managing so many FOI requests was still challenging and very time consuming but it would have been much harder by email. The first batch of requests had a success rate of more than 95% and the other (which was more detailed) was around 85%.
“I don’t think I would have achieved that without WhatDoTheyKnow Pro and, as a result, the investigation and interactive map we created would not have been as comprehensive.”
Refining the requests
While councils have to publish annual lists of the assets they own, this does not include vital information such as who assets were bought from or sold to and the prices paid
“I sent requests to one of each type of local authority (London borough, metropolitan borough, unitary, county and district) to test what, if any, information councils would provide. The fact that all of those requests were successful meant I had confidence when submitting the batch requests.
“It also allowed me to include additional information in the bulk requests, because some of the test councils erroneously withheld, under Section 40, the identities of companies. As a result I added a note to the request highlighting that this would not be a correct application of that exemption.
“As each response came in I recorded them in two separate spreadsheets — one showing what assets had been bought/sold and another containing information about how the money raised from asset sales had been used. Gradually we built a comprehensive picture of what was happening with public spaces, and that was crucial for our story.”
Bringing about change
There have been tangible results from this investigation.
“The government launched an investigation into the sale of assets by Peterborough Council as a result of this particular story, focusing on that area.
“We submitted our findings to an inquiry currently being held by the Communities and Local Government select committee and were mentioned by name during the first day of oral hearings.
“And last month the Public Accounts Committee announced it would hold a similar inquiry into the sale of public land. Several councils halted their property investment policies after our coverage revealed how much they had borrowed to fund the purchases.”
Thank you very much to Gareth Davies for talking to us about the Sold From Under You project.
Image: Daniel von Appen
WhatDoTheyKnow Pro, mySociety’s subscription service offering extra tools for journalists and other professional users of FOI, has been running in the UK for just about two years.
During that time we’ve launched, worked closely with users to refine the service, and — happily — watched it play a vital part in the making of several important data-driven news stories, on topics as diverse as Brexit campaign funding and the results of austerity cuts on councils. Journalists, in particular, have appreciated tools such as the ability to send and manage bulk requests to multiple authorities; and the embargo tool that keeps requests and responses hidden until the story has been published.
Now, thanks to support from Adessium Foundation, we are able to bring the same benefits to countries across Europe, and — we hope — some additional synergies that will be borne of organisations working across boundaries. The same functionality that extends WhatDoTheyKnow into the Pro version will be available to FOI sites run on the Alaveteli platform, under the name Alaveteli Pro.
The ultimate aim is to enable journalists, campaigners and citizens in Europe to make greater and more effective use of their right to access information; and in particular to generate public interest stories and campaigns that will hold power to account.
We’ll be focusing on three areas in order to achieve this aim:
- We’ll give selected existing Alaveteli sites in Europe the technical help they need to upgrade to the Pro version;
- We’ll be helping organisations in three new European jurisdictions to launch brand new Alaveteli sites, making access to information easier for citizens in these countries. The first site will be launched by VVOJ from the Netherlands.
- We’ll encourage cross-border collaborations between journalists and organisations using the sites (both the existing ones and the new ones) to investigate stories that span more than one EU country.
So watch this space: we’ll be sure to keep you posted as the work progresses. The planned start date is next month, and the project is set to run for three years.
We’re looking forward to sharing stories resulting from this initiative once they start rolling out, and supporting the incredible work that journalists do in putting them together.
Image: Emiliano Vittoriosi
We’re delighted to be hosting the third AlaveteliCon, our Freedom of Information (FOI) technologies conference, on 23 and 24 September 2019 in Oslo, Norway.
A few days ahead of International Right to Know Day (28 September), AlaveteliCon will bring together activists, journalists, technologists and campaigners from across the world who use Freedom of Information laws, data and technologies to create public-interest investigations and campaigns.
This time we’ll be including a focus on catalysing collaborations on cross-border European public interest investigations and campaigns, in order to strengthen the use of FOI across the region. This focus is in part due to our latest Transparency project, funded by Adessium Foundation. This Transparency project seeks to enable journalists, campaigners and citizens in Europe to make greater and more effective use of their right to access information, in particular to generate public interest stories and campaigns that hold power to account.
Here are some of the other things AlaveteliCon attendees will get up to this time:
- Exchanging insights and ideas on how to run and improve open-source Freedom of Information platforms such as WhatDoTheyKnow, MuckRock and FragDenStaat
- Hearing how journalists and campaigners around the world have used FOI to power high-profile investigations and campaigns
- Making connections with journalists and campaigners who would like to collaborate on cross-border investigations that hold governments to account
- Learning about data sources available on FOI platforms around the world, waiting to be analysed and turned into public-interest stories
- Hearing tips from FOI activists on getting governments to release information in their countries including going to court
- Contributing to the further development of mySociety’s open-source FOI toolkit for journalists, Alaveteli Pro, which helps journalists and campaigners to manage their FOI investigations
- Connecting to a global network of FOI experts and advocates
- Working together to plan potential joint activities to celebrate International Right to Know Day on 28th September 2019
Apply to attend
As spaces at AlaveteliCon are limited, we’re asking those wishing to come to fill out this application form to tell us what unique perspective they can bring that’s relevant to the above conference themes/topics.
Are you a journalist or campaigner who uses FOI for investigations/campaigns? Do you want to collaborate on FOI-generated investigations? Or interested in setting up an FOI platform in your country? Or want to learn how to use FOI to its fullest potential? Or interested in any of the above topics? If so, then please submit your application by 2nd September at the very latest.
We’ll let you know if you’ve been allocated a place by Friday 6th September at the latest. However, we will endeavour to reply to you sooner than this.
AlaveteliCon is free to attend, but delegates will be required to arrange their own travel, accommodation and costs in Oslo.
We’ve just released version 0.32 of Alaveteli, our open source platform for running Freedom Of Information sites. Here are some of the highlights.
Making correspondence threads easier to navigate
Thanks to our designers, it’s now possible to collapse individual messages in a correspondence thread in order to focus on just the parts you’re trying to read. Plus you can quickly collapse (or expand) all the messages in the thread using the “Collapse all” and “Expand all” links from the “Actions” menu.
Alaveteli Pro users gain the additional benefit of a redesigned sidebar which allows for easier navigation of lengthy correspondence and avoids having to scroll to the top of the request thread to update its status. See Martin’s full explanation here.
Better password security
We’ve started enforcing stricter password length constraints wherever a password is set or updated to help users keep their accounts secure. And we’re also using a stronger encryption method for storing password data, using bcrypt rather than the older SHA1 algorithm to obscure the actual password. (Be sure to run the rake task documented in the release upgrade notes to upgrade secure password storage for all existing users.)
You can read more about what this does and why it’s important if you’re interested in the technical details behind this upgrade.
Authorities not subject to FOI law
We’ve adopted WhatDoTheyKnow’s
foi_notag for authorities to indicate that although the authority is listed on the site, it is not legally subject to FOI law. This could be for advocacy purposes – if it’s felt an authority should be covered by legislation – or where the authority has agreed to respond on a voluntary basis.
foi_notag now causes an extra message to appear under the authority’s name on their page and on all related requests, and removes language about legal responsibilities to reply from the messages sent to users.
To improve the UI, we’ve made a similar change for authorities with the
eir_onlytag to make it clearer that such authorities are only accepting requests about the environment.
(Don’t worry admins, you don’t need to remember all this – we’ve updated the documentation on the edit page to reflect the new functionality!)
Improvements for site admins
We’ve made it easier for admins to ban users who sign up to post spam links in their profile. There’s now a “Ban for spamming” button which is available on the user edit page or as soon as you expand the user’s details in the listing rather than having to manually edit user metadata.
We’ve also made it harder to leave requests flagged as vexatious (or “not_foi”) in an inconsistent state. Previously the site just assumed that vexatious requests would always be hidden. Now the admin interface enforces the hiding of vexatious requests by showing warnings when a request is set as vexatious while it’s visible on the site, and prevents the updated request from being saved until a valid state is selected.
And last but not least – introducing the new Announcements feature!
Easier popup banner management
Site admins will be relieved to hear that they can now update the popup banner message on the site without needing to schedule developer time.
This feature supports multi-language sites so if you set the announcement for your main (default) language, it will appear across all language versions that you have not added a specific translation for.
You can set announcements that will only be seen by fellow administrators when they visit the summary page. (If you’re running a Pro site, you can also have announcements that will only be seen by your Pro admins.)
Announcements for Pro users appear as a carousel at the top of their dashboard. So far we’ve used it on WhatDoTheyKnow Pro to publicise new features, offer discount codes, and encourage people to share their published stories with us.
The full list of highlights and upgrade notes for this release is in the changelog.
Thanks again to everyone who’s contributed!