Alaveteli is our platform that anyone can use to run a Freedom of Information site in their own country or jurisdiction.

As the number of requests grows on an Alaveteli site, it can become increasingly difficult to find released information that you’re interested in.

You can search, but the more general the term, the more likely that you’ll pull in results where the term is mentioned incidentally rather than being directly related to the information released. Or you can browse by authority, but that’s more fiddly when you want the same information from a range of authorities.

There’s also the issue that people new to FOI might not have a clear idea of what to ask about. Freedom of Information is great because you can ask for anything, but as a newcomer that can feel overwhelming. You need some direction to know where to start.

Request categories allow us to curate related requests to bridge the gaps mentioned above.

Here’s an example of a category on WhatDoTheyKnow that compiles successful Freedom of Information releases related to the British Post Office scandal.

Categories can be created in the admin interface via the Requests > Categories menu item.

Request categories work in a similar way to the current public body categories – in fact, as part of this development we’ve revamped the underlying code so that it applies to both!

At their core, they’re composed of three things – the title, body (where we can add explanatory content), and requests grouped by a tag.

Notes can be added to call out key information, and categories can be added to headings to create a layer of hierarchy. As part of this development, we’ve also improved notes so that they can be more easily styled with some preset colours, and added rich text editing to improve the formatting of longer notes.

We’ve started building up some interesting categories of requests on WhatDoTheyKnow, but we’d love to hear which ones you’d like to see.

If you’re interested in how the development unfolded you can take a look at the related work on GitHub.

—

Banner image: Garmin B

CSV is a great format for releasing sets of structured data in response to Freedom of Information requests. Indeed, on WhatDoTheyKnow we’ve seen several thousands of CSVs released.

We’ve recently added the ability to explore CSV files via a Datasette instance. Here’s an example.

Opening the CSV in Datasette makes it easy to explore and analyse it in an interactive website.

If you’re not familiar, Datasette converts the CSV to an sqlite database, which means you can then query the data using SQL.

Alaveteli uses the publicly available lite.datasette.io instance by default, but you can host your own instance and configure it at theme level like we’ve done for WhatDoTheyKnow.

You can see the implementation details at mysociety/alaveteli/#7961.

—

Banner image: Joshua Fuller

Alaveteli is our platform that anyone can use to run a Freedom of Information site in their own country or jurisdiction.

We’ve added new functionality that allows Alaveteli sites to highlight blog posts on the homepage, so they’re more visible. In this way, Alaveteli sites can not only help users with the ‘how’ of making an FOI request, but also show, in a very tangible way, the ‘why’  — especially if the posts are highlighting impactful uses of the site.

Previously, Alaveteli had a basic way to pull in an RSS feed of blog posts. These were only available on the blog page (/blog), which often does a disservice to all the great work that gets written about.

We wanted to better signpost the blog from other pages in Alaveteli to celebrate great FOI use, and help users understand how a seemingly simple FOI can go on to have an outsized impact.

When the site runner configures a blog feed, posts are pulled into Alaveteli and cached in the database. This makes them available on the homepage. Here’s what that looks like on WhatDoTheyKnow:

They’re also visible in the admin interface via the Tools > Blog Posts menu item.

At present only the title, URL and publication date are cached in Alaveteli. These records are intended to be “pointers” to the canonical URL of the article hosted on the external blog service.

In the admin interface, blog posts can be tagged to indicate their subject matter.

As FOI requests and authorities can also be tagged, this allows the blog posts to be highlighted in the sidebar of appropriate pages where there’s a matching tag. So, if someone’s browsing requests or visiting an authority that deals with the climate emergency, for example, they’ll be shown relevant blog posts – hopefully making them more visible to people who have already displayed that they have an interest in the topic, and giving those people a bit more contextual knowledge.

In future we’d like to make these posts more visible, by importing header images and a short summary, and give the ability to display some posts when there isn’t a direct tag match.

You can read about the initial design and subsequent conversation and pull requests starting at mysociety/alaveteli#6589.

—

Banner image: Patrick Perkins

CAPE is our data-packed website that makes it easier for everyone to track and improve local action on climate change, built in collaboration with Climate Emergency UK.

When CAPE launched, it was basically a list of councils’ climate action plans — but we’ve continued to make regular improvements and additions, and it’s better, and more useful, than ever.

If you’re a council officer with climate duties; a researcher looking for climate data; an activist needing facts to back up your climate campaign; or simply an interested citizen, you’ll find some truly invaluable facts and figures on CAPE.

And if you haven’t visited for a while, you’re in for a surprise. Here’s what we’ve added this year:

Improved search

Now you don’t need to think of all the different ways that councils might have described the concept you’re looking for. We’ve integrated machine learning into the search function, to help surface your search term even in documents that use different words for the same concept. Interested to know more? We wrote it up in detail here. 

Updated for new councils

In April, a set of UK local authorities was abolished and merged into new unitary authorities.  When this happens, we do work to update the data that CAPE depends on. You can make use of it too!

We’ve updated:

• Our dataset of local authorities names and codes
• Our datasets of ‘nearest neighbour’ councils. 
• Our composite UK-wide multiple deprivation dataset for councils. 
• Our composite Rural Urban Classification dataset for councils. 

And this has all fed back into CAPE to make changes of local government structure transparent in the site. 

Local polling

We wanted to make it easier to see where there is support for climate action, so we took a few sets of MRP polling on people’s attitudes to net zero and energy sources and converted it from per-constituency to per-local council.

For each local authority (except in Northern Ireland) – see Croydon, for example – you can now see estimates for support for net zero (even when it is described as expensive), and for different kinds of renewable energy project.

Not sure what MRP polling is? We wrote something up about that. 

Climate Assemblies

In 2019, we were part of the UK-wide Climate Assembly, and we’ve kept an interest in citizens assemblies about climate change. There’s been one held for the whole of Scotland, and more widely we’ve found 17 (so far) held by local authorities across the UK.

We’ve uploaded the final reports of those assemblies into CAPE. It is now easy to see which authorities have held climate assemblies, and to search the results of those assemblies. Find all this at cape.mysociety.org/assemblies.

Hundreds of new documents have a fresh coat of paint

Thanks to people telling us about new plans, and us conducting an extensive search for new documents across council domains, we’ve updated and added hundreds of new documents to councils’ pages. 

With all that content, we’ve gone back and tidied up the design of our council page layout, giving us more space to add explainers and summaries to each section. Why not take a look for your area?

Council Climate Action Scorecards

Let’s not forget that while all this activity has taken place on CAPE, we were also helping Climate Emergency UK with the Council Climate Action Scorecards. At the time of their launch in October, we wrote about the part we played in building the site, wrangling the data, advising on the methodology and building a scoring interface.

The result of all this is, of course, that whatever your climate interests – be you council staff, researcher, journalist, campaigner or interested member of the public – you can dive in to any council’s page to see how they’re doing against several markers of climate action.

Since launch there has also been the significant addition of Question Pages, allowing a never before seen view of how each council scores on every individual action point. Julia wrote about that in detail, in this post.

Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

…or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

—

Image: Keila Hötzel

Are you investigating, researching or gathering large quantities of data through Freedom of Information requests? Perhaps you’re a journalist, academic or NGO. We’re looking people based in the UK who’d like to try out our new ‘Projects’ feature for WhatDoTheyKnow Pro.

Projects allows you to crowdsource the extraction of data from multiple (or batch) FOI requests made to multiple authorities. You can set up a project with a brief description of what it is and what you are hoping to achieve, and some tasks that volunteers can complete to help you with this aim (like categorising responses, or answering questions about the data released).

Once that’s done, you can set it up to invite volunteers, who can help you to extract all the information you need from the released responses.

You’ll be able to download your volunteers’ input as a spreadsheet, meaning analysis of the data is much quicker and easier — so you can get on with the task of forming conclusions and writing up your findings.

What we’ll need from you

Projects is still in its nascent stage, so we need feedback from our testers. This will help us improve the service and tailor it to users’ needs, based on real life use cases.

Right now, we handle the setup and importing of the requests you want to work on manually (that is, our developers have to do it) — but we’re working on improving this aspect, and your feedback will be crucial in shaping the direction our development takes. We’re also looking for general comments, once you’ve used the service, on what’s useful and what’s missing; what you tried to do but couldn’t, and what made things easier for you.

If this sounds interesting, please get in touch at pro_team@whatdotheyknow.com. We look forward to hearing from you!

—

Image: Jessica Lee

We’ve added a new functionality to the Alaveteli Pro codebase, allowing you to download a zip file containing all correspondence and attachments from a batch, and a spreadsheet (csv) to show the progress status of every request.

Alaveteli Pro is our tool for professional users of Freedom of Information. If you’re UK-based, you’re probably most familiar with our local iteration WhatDoTheyKnow Pro — but don’t worry: when we talk about improvements to Alaveteli Pro, you can be sure they’re also part of the WhatDoTheyKnow toolkit.

How to export

You’ll find these tools at the foot of the batch container in the requests list.

Why data exports?

Of course, we like to think Alaveteli Pro is a useful tool in its own right: there’s a lot you can do within the Pro interface, and it was built specifically to help you keep track of all your FOI activity in one place.

But sometimes users want to use external tools – either because they’re just more familiar with them, or because they want to do something beyond the functionality we offer.

Now there’s a simple way to get data out of Alaveteli, allowing you to analyse it with the tools of your choice, or perhaps send a progress report to a supervisor or editor.

It’s part of a programme of work to support cross border journalism between European organisations, supported by Adessium Foundation, allowing us to refine and improve the codebase for the benefit of all Pro users.

The technical bit

Those with a bit of coding knowledge may be interested to hear how we approached the zip download functionality. mySociety developer Graeme explains:

“With batch requests potentially going to as many as 500 different authorities, each request can receive several responses and attachments in return.

“All these emails and files mean that compiling the zip for download could be a lengthy job and would normally cause the request to time out. So for this new feature we’re utilising file streaming to send chunks of the zip as they become available.

“This means that the zip starts downloading immediately and you don’t have to sit watching and wondering whether anything is happening – you can see more and more data being transmitted.”

We hope you find this new feature useful. Please do let us know how you’re using it and any feedback you may have.

—

Image: Startup Stock Photos

We’ve recently made a small change to WhatDoTheyKnow. Now, if your request has resulted in a news story, you can add the link as a ‘citation’.

If you’re a journalist or a campaigner, we hope this is a useful way to give your stories some more readership (not to mention a nice inbound link from a high-ranked site for your search engine ratings).

And if you’re simply a citizen whose request was picked up by the press, there’s now a way to share how the information you’ve obtained fits into the news agenda.

There’s a benefit for the wider transparency community, too. We think that, in aggregate, these links will serve to show others just what a simple FOI request can do.

Requests often start off as nothing more than an inkling or a nagging question, but there’s always the chance that one of them will reveal important or interesting information, hitting the news, reaching a wider readership and — who knows?  — maybe even changing hearts and minds. For big stories, it will be good to be able to create a permanent record of where it all began.

More broadly, when you use this feature you’ll be helping us to understand what sort of impact the site is having, too. We’re always keen to spot news stories based on WhatDoTheyKnow requests, but papers don’t always cite a source or link back to the site, meaning that our monitoring is often dependent on a manual search where stories look like they might have originated with one of our users.

How to add a citation

You’ll find the feature in the right hand column of your request page. Just click on ‘Let us know’:

…and paste the URL in:

Here’s how it will show up on the request page, as seen on the first request to gain a citation, which informed a story about electoral letters sent in error:

If there’s more than one story, you can click ‘New Citation’ to add another one.

The way we’ve set this feature up, WhatDoTheyKnow users can add a citation to any of their own requests — but if you spot a news story that’s linked to a request that isn’t yours, please do contact the WhatDotheyKnow team.

They’ll assess it and input it if they find it to be valid. Our aim here is, of course, to prevent spammers from adding irrelevant links to the site.

Users of WhatDoTheyKnow Pro, on the other hand, have the ability to add citations to any request.

This work is one of the ongoing improvements that we’re working on thanks to a grant from Adessium.

—

Image: Kaboom Pics

Today we’ve added twelve new topics to the voting pages for every MP on TheyWorkForYou. Covering important areas such as membership of the EU, mass surveillance of data, and military action against Daesh/ISIL, these voting lines will help give an even better at-a-glance picture of what your MP stands for, and how that was reflected in Parliament.

Check your own MP’s votes on these topics by inputting your postcode on the TheyWorkForYou homepage:

and then clicking on the ‘voting record’ tab:

You’ll see all your MP’s voting lines laid out in several topic areas on a single page:

From there, it’s easier to explore further by clicking ‘show votes’, which will take you to a page listing every vote taken into consideration when calculating the MP’s stance.

For a full explanation on the methodology behind these pages — a combination of manual and automated inputs — see our previous blog post.

How do we choose which topics to include? It’s all driven by what MPs have voted on in Parliament, and consequently how much data there is to draw from in any specific subject area (there are also a few ‘topics in waiting’: areas we know we want to cover, but where there need to be some more votes before we have sufficient data to present meaningful stances).

New topics in full

The new voting lines we’ve added are:

• In Constitutional Reform
  • Reducing the voting age
• In Home Affairs
  • Mass surveillance of people’s communications and activities
  • Merging police and fire services under Police and Crime Commissioners
• In Transport (a new topic area)
  • High speed rail network
  • Greater public control of bus services
  • Publicly-owned railways
• In Foreign Policy & Defence
  • UK membership of the EU
  • The right to remain for EU nationals
  • Military action against the group known as ISIS, ISIL or Daesh
• In Taxation & Employment
  • The reduction of Capital Gains Tax
• In Housing (a new topic area)
  • Secure tenancies for life
  • Charging a market rent to high earners renting a council home

We hope this helps you see, even more clearly, the effect your MP has had on a broad range of national and international issues.

Don’t forget that we also offer the chance to discuss these votes (or any other issue) with your MP. Just click the ‘send a message’ button at the top of the page to go to WriteToThem.com, where you can compose a message to your representatives quickly and easily.

—

Image: © UK Parliament/Jessica Taylor

When you send a Freedom of Information request, a clock starts ticking. Here in the UK, public authorities are bound by law to answer a request “promptly, and in any case within 20 working days” — but of course they can only respond if they’ve received the request.

And, while email is generally reliable, we’re all familiar with the occasional mishaps it can bring: mailboxes that are full, accounts that have been closed down, or messages being returned because they look too much like spam.

As Liz’s post yesterday mentioned, our Freedom of Information platform Alaveteli now includes a new feature which verifies that your message actually has been delivered.

Sign here please

Email works a bit like signed-for physical mail. When a letter is delivered to a recipient they either sign to say they’ve received the letter, or the mail company records that there was no-one available to accept the delivery. This lets the mail company keep the sender up to date with where their letter is. Mail servers do the same — the recipient server sends a confirmation that a particular email has been received, or an error code is reported by your mail server if there’s a problem delivering the email.

Like physical mail, we can only verify that the message has been accepted at the destination address. It’s then under the recipient’s control to get it to the right person at that address, a bit like a receptionist receiving a letter for an employee 10 floors above. We think that if an authority’s mail server confirms that one of our emails has been delivered, it’s their responsibility to ensure it reaches the correct people to be able to answer your FOI request.

Proof of receipt

Look at the header of any request on WhatDoTheyKnow, and within 24 hours, in most cases you’ll now see a small green ‘delivered’ confirmation:

Most users can click on this to see further confirmation:

But if you’re the owner of the request, when you click on the green ‘delivered’ link, you’ll see information from the mail logs as the message passed through our server. If there’s ever a query about whether or not a message was delivered, you can hand these on to the authority to help them analyse any issues.

On the rare occasions that something goes wrong, here’s what users will see instead:

– and if it’s your own request, again, you’ll have access to the mail logs.

Small but mighty

This feature might look small, but there’s a lot of thinking behind it — just check the length of the trail on Github, our ticketing system. Anyone will be able to understand the amount of discussion and problem-solving that went into the addition of this small green tick, while the more technically-minded may also find it interesting to see the coding solutions as they unfolded.

This small green tick also gives users something rather powerful: proof that their request was received by the authority’s mail server at a specific time, should that be disputed.

The suggestion for this feature came initially from one of the WhatDoTheyKnow volunteers. It took some time to implement, but we’re pleased to say that it has now been made available for all Alaveteli sites in release 0.25.0.0.

—
Image: Pacific Northwest Agricultural Safety and Health Center (CC)