---

Freedom of Information is a right that gives people power over public authorities. The case for FOI is often made on the high profile investigations and the public disclosure of scandals. But the value can be also seen in the quiet success of ordinary citizens being able to access information from the public authorities that hold power over their lives.

We ran a poll with Opinium to find out how widespread the use of Freedom of Information is. We found that 10% of UK adults have used FOI to try and get information they thought would be useful to themselves, their community or wider society. When including people who made a request as part of their work, this figure goes up to 14%. 

Giving evidence to Parliament, the former Information Commissioner said that “one in 1,000 citizens in the UK will file a Freedom of Information Act request, but journalists are standing in their shoes. It is through journalists that the public can understand or get to know why decisions are being made on their behalf. Journalists, public interest researchers and advocacy groups are important requesters”.  This is an important point about the value of Freedom of Information even if it is only used by a few, but based not just on our polling but the ICO’s own polling, this figure is 100 times too small. Freedom of Information is not a niche right, or mainly used by journalists,  but has been used by millions of people. 

Alongside a team of volunteers, mySociety runs WhatDoTheyKnow, a website that helps people make freedom of information requests (so far over 850,000 requests), and displays the results of those requests in public, so more people have access to the results. 

Through running this website, we’re very aware of the many different ways Freedom of Information is used by people, in ways that are not captured by official FOI statistics. We want more and more people to be able to use their information rights, and we want more and more of them to be using WhatDoTheyKnow so that what they find out is available to all.

We want to be doing more to support and advocate for people using their right to know, and see it as our role to be a voice for our users, and this large group of ordinary FOI users, in arguments about the future of Freedom of Information.

We’re proud of what we’re able to do with a small budget, but we want to do more. If you want to help us do that, there’s a number of things you can do:

• If you’ve benefited from Freedom of Information, or support our mission to make information more accessible, please consider making a regular or one-off donation to support our work. 
• If you’re not able to donate now, please join our newsletter so we can keep you up to date with our work and campaigns. 
• If you’re a journalist, researcher or campaigner, have a look at our pro service, which for £10 a month provides a wealth of features to help make and manage requests. 
• If you’re interested in volunteering time, WhatDoTheyKnow as a day-to-day service is run by a dedicated group of volunteers. If you’re interested in getting involved, you can learn more on the volunteering page. 

To read more about the numbers behind the polling, and how we validated it, please see this companion blog post. 

---

Header image: Photo by Samuel Regan-Asante on Unsplash

---

This blog post is a companion post to a shorter blog post explaining the significance of this polling to mySociety’s FOI work.

We know very little about the real picture of Freedom of Information use because there are not comprehensive statistics. Information on users of Freedom of Information is very hard to come by.  We have some information through a survey we run on WhatDoTheyKnow, but we know this only covers the minority of requesters who use our service. 

Knowing about this picture is important to us for several reasons.

The first reason is one of the big benefits to society of WhatDoTheyKnow is that we make public information easier to discover without explicitly asking for it. If we can know more about how many FOIs are being made in total, we can have a better sense of what proportion of this information we’re publishing (based on some of the maths below, It’s probably somewhere from 5-10%). 

The second reason is that conversations about the pros and cons of freedom of information can be dominated by the problems journalists experience in requesting information from the central government. This is a big and important problem, but it shifts the general understanding of the impact freedom of information has had on our society. Through WhatDoTheyKnow, we get a glimpse of a bigger world when citizens are making requests that affect them and their communities – but we don’t see everything, and getting more information about this is vital in informing how we approach our policy and campaigning work. 

As part of a “Giving Tuesday”, Opinium gave five survey questions (for a national representative survey panel)  to a number of charities, including mySociety. We used one of these questions to find out how many people had made a freedom of information request. The rest of this blog post explains the results of that survey. 

The question we asked

Our data comes from an Opinium survey of a representative selection of UK adults that ran between 30th November – 3rd December 2021. Respondents were asked:

The Freedom of Information Act gives you the right to request a large range of information from public authorities (government departments, local authorities, NHS trusts, schools, etc). These are called Freedom of information requests. Have you ever made a Freedom of Information request?

Respondents had the option of responding:

• No I haven’t made a request, and I am not aware of Freedom of Information
• No, I am aware of freedom of information but haven’t used it
• Yes, as part of my job
• Yes, to find out something that might be useful for me personally
• Yes, to find out something that might be useful to my community/society in general
• I’m not sure / NA

Results

mySociety/Opinium polling in 2021 found that 10% of UK adults have used FOI to try and get information they thought would be useful to themselves, their community or society. When including people who made a request as part of their work, this figure goes up to 14%. When looking just at personal use, the figure is 6%. Overall, a majority of people (62%)  had either used FOI or were otherwise aware of it. 

There was a small gender difference in both awareness and use of FOI, with men having higher awareness than women (68% to 57%), and greater use (16% to 11%). Our polling found that the 18-34 age group were the least aware of Freedom of Information (55%), but were also the age group most likely to have made an FOI request (25%). This is possibly partially explained by a much higher rate of using it as part of employment in younger demographics (12% compared to 6% overall), but the number using it for other reasons is still notably higher (some more discussion of this further down). Looking at respondents by nation/region, there was a less than expected proportion of people who made a request in Wales (6%) and Northern Ireland (2%), but a greater number who made a request in London (28%).

Validating these figures

When I first saw some of these figures, I was a little surprised and wanted to explore some different ways to validate the number.

Digging into it, I found that other polls asking different versions of the question show a similar figure, and back of the envelope calculations based on known statistics suggest the basic ballpark is right – there are millions, rather than hundreds of thousands, of people who have used the Freedom of Information Act. 

Part of the reason this figure might be surprising is that our statistical picture of Freedom of Information is so poor, we have  very little idea of the scale of it – and what we do know is misleading as to that scale.  For instance, a recent Financial Times article, when highlighting the (bad) trend of how central governments are withholding more and more information requested, falls into the trap of assuming that this picture represents all freedom of information requests. But departments and ministries are not the only public bodies that receive Freedom of Information requests. In fact there’s good reason to believe they receive only a small percentage of the overall total. 

Most FOI requests in the UK are not covered by official statistics. In 2017, we did a meta-FOI to ask local authorities about the number of FOIs they received. We calculated around 467k were made that year, compared to 46k made in that same year to the central government. From running WhatDoTheyKnow, we know that only 10% of requests made through the site go to the central government departments that are covered in the statistics. 

As the number of Freedom of Information requests is much higher than the official statistics show, this helps explain why the number of requesters can be far higher than expected. Not only are there many more public bodies outside central government, but these bodies are closer to people’s day to day lives, and so a broader range of people might want information, and find it through the Freedom of Information Act. 

Polling by the UK’s Information Commissioners

The clearest reassurance of the 10%ish figure is that a similar poll found a very similar number. Polling by the Information Commissioner’s Office in 2021 found 49% were aware of “the right to request information held by public organisations”. This is lower than our equivalent question, where 62% had either made use of FOI, or were aware of it and not used it. However in the same 2021 survey, 10% of respondents to the ICO’s survey said they had already made use of “the right to request information held by public organisations”. In the previous year this figure was 12%. This figure is very close to our figure of 10-14% making use of FOI, and it is reassuring to see something in this ball park come from a different survey company.

Both these surveys might be wrong of course, but polling by the Scottish Information Commission in 2022 found an even higher number. This poll found 36% of a weighted sample of Scottish respondents had at some time “asked for information from a Scottish public body by letter, email or online form”. 18% said they did this annually or more frequently. This is a much higher number than the other survey. There are several possible reasons why.

1. There is a genuine difference in awareness and use of rights between Scotland and the rest of the UK. 
2. The way this question is phrased should also include requests for personal information (subject access requests) as well as freedom of information requests.
3. This version of the question does not ask about a right, just if someone did something that might have engaged the Freedom of Information Act. This might catch people who get through the process unaware they may have benefited or made use of information rights.

How should we interpret this? There is no strong reason to believe the use of rights is significantly different in Scotland. The Scottish figure was 63% awareness of freedom of Information, which is higher than the ICO UK-wide, but in the same general area as our UK-wide polling, which did not show a significant difference for Scotland. Similarly, our survey found a statistically significant difference in use of FOI by respondents in London and did not find this for Scotland. 

As for subject access requests, we actually know from the (really good) statistics recorded in Scotland that around one-quarter of information requests are subject access requests. So even applying this correction, this question is still suggesting around double the figure from our survey. This is likely to be part of the explanation, but is not all of that. 

This leaves the possibility that by not prompting about rights or freedom of information, this is capturing a set of people who are coming into contact with information rights issues without noticing it. It is possible to exercise your freedom of information rights without being aware you are doing so. My first freedom of information request was made this way as a student, asking OFCOM if they held some information from the Broadcasting Complaints Commissions’s archive. If you email your local council wanting to know something, it should be processed under the Freedom of Information Act, even if you were unaware of it. This is likely to include some interactions with authorities that will have existed before FOI (and information may have been made available) but which is now covered formally through the Freedom of Information process. For example, a request for library opening times could be processed as an FOI request, but may well have been answered before the FOI act existed.

If this is an explanation for a higher number in response to the OSIC survey, it might also explain the higher proportion of 18-34 respondents in our poll who had used Freedom of Information for personal reasons. Contact with public bodies for information is more likely now to be by email, and trigger the formal FOI process. There is more to explore here around possible shifting patterns of first contact with FOI. 

Back of the envelope calculation based on WhatDoTheyKnow statistics

The relevant polling we have is supportive of our poll not being outrageously high. The other approach is to try a very back of the envelope approach based on known statistics to see if this is a reasonable amount of FOIs to have been made. 

Based on previous research by mySociety and the Constitution Unit, we have estimates for the number of FOIs made to local authorities in 2005-2010 and 2017. Filling in the extra years between those dates, extending forward, and doubling the number (roughly 48% of requests made through WhatDoTheyKnow are to local authorities – but we don’t know if this applies more generally or not), this gives roughly 11 million FOI requests all time. On WhatDoTheyKnow there are an average of six requests per user (again, don’t know if this applies more generally) – so applying that ratio gives roughly 2 million requesters all time. A figure of 10% of UK adults would expect roughly 4.3 million requesters all time. 

To get the two numbers more  into sync some combination of the following could be true:

• More local government requests have been made all time than this assumes.
  • Not impossible given this is based on three data points (all of which are incomplete surveys and require some amount of extrapolation).
• A greater proportion of requests being made to non-local government bodies than happens in WhatDoTheyKnow
  • No way of knowing this without a complete statistical picture.
  • OSIC statistics show a higher 60% statistic in Scotland being made to local government.
  • Given there are many  more non-local government public authorities in the rest of the UK, it is reasonable to guess it’s closer to the WhatDoTheyKnow statistic of 50%, but could it be lower than that?
• The ratio between requesters and requests is different outside of WhatDoTheyKnow.
  • Arguments both ways, WhatDoTheyKnow is missing all the ‘not intentionally using FOI’  one-offs, but also some of the bulk requesters who don’t want the results to be public on WhatDoTheyKnow. 

Given so many of these numbers are made-up or trying to generalise from WhatDoTheyKnow to all uses of FOI, there is no real reason not to prefer the figure two separate polls agree on. That said, it is reassuring it is in the right order of magnitude (still talking millions rather than hundreds of thousands of FOI users). This question would be helped by a complete statistical picture of FOI in the UK, and to be honest, that would be so useful, it’d be fine if it proved our current numbers wrong. 

---

Header image: Photo by Mitchell Luo on Unsplash

Demographic difference graphs

Download the polling tables.

The following graphs show the demographic split on aspects of the FOI polling. Where the percentage for a category is higher than would be expected statistically if there was no difference between groups, it is highlighted in blue. If it is smaller than would be expected, it is highlighted in red. For non-highlighted categories there is insufficient data to say the category differs from the general average.

Summary:

• We have updated our UK Index of Multiple Deprivation dataset with deciles and quintiles for local authorities and Westminster constituencies.
• We have updated our UK Rural Urban Classification dataset to sort local authorities and Westminster constituencies into four categories. 
• We want to help you use our data for climate work – see our previous blog post for more details. 

–

Back in 2020 we released a UK-wide version of the index of multiple deprivation (see original blog post). This is a dataset that uses multiple metrics of deprivation to rank all small neighbourhood sized chunks of the UK from most to least deprived.

This data is produced for each nation, but our dataset allows areas to be roughly compared across the whole UK (with a separate file for comparing just Great Britain, without Northern Ireland).

This is useful if you have postcode data you want to add information about deprivation to, but sometimes you want to be comparing the bigger areas like local councils and Westminster constituencies.

In the course of some of mySociety’s recent work,  we’ve added new sheets to the deprivation dataset that show the relative deprivation of UK councils and Westminster constituencies.

This works by using a population weighted average – where each neighbourhood’s raw score is multiplied by its population, added together for the authority/constituency and then divided by the total population. This new score can then be ranked and put into deciles.

Because local authorities and (to a lesser extent) constituencies, have different sizes at a national level, the deciles are based on the percentage of the population rather than number of councils or constituencies. So the 1st decile contains the councils with the highest deprivation, that collectively account for 10% of the population. 

If working with data that is entirely from Scotland, Wales or Northern Ireland, it is better to use one of the official datasets that are derived from the national index.

If you want to use our data for climate related purposes, I run drop-in hours on Thursdays and Friday to talk about our data, or just email me! For more information on our climate data, see our previous blog post. 

You can sign up to our data newsletter to keep up to date with future updates.

–

We’ve also updated the UK-wide Rural-Urban-Classification (RUC) dataset to include local authorities and Westminster constituencies. This is a dataset that combines the different measurements of whether a neighbourhood is urban or rural into a single UK wide dataset. 

Here my approach was slightly different. Our RUC dataset recognises three classifications (“Urban”, “Rural”, “Highly Rural”) as this was the best way of combining the different approaches from different nations. 

For each authority/constituency, we calculated the percentage of the population who lives in areas that fit these three criteria. Then, using a clustering approach, authorities/constituencies were split into four loose categories. 

• Urban – All, or the overwhelming majority of the population live in an urban area.
• Urban with rural areas – places with significant rural areas by volume, but generally where the population is concentrated in an urban area. 
• Rural – Less of the population is concentrated in urban areas.
• Sparse and rural – large rural areas with very dispersed populations.

All of these make sense on a spectrum, so at the margins some authorities will be more similar to ones in other classifications, than to the mean of that classification – but in broad terms these categories reflect different kinds of areas. The original population breakdowns are included if further processing work is useful.

If you want to use our data for climate change related purposes, I run drop-in hours on Thursdays and Friday to talk about our data, or just email me! For more information on our climate data, see our previous blog post. 

You can sign up to our data newsletter to keep up to date with future updates.

Header image: Photo by Héctor J. Rivas on Unsplash

Data users in the UK often encounter fragmented public data, where  public authorities are each spending money to publish data independently, but their outputs are difficult to find and join together. This means that a lot of effort is going into creating data which cannot be used to its full potential.

As a joint project between mySociety and the Centre for Public Data, we are writing what we hope is a simple approach to help address this problem. 

Our current thinking is that some level of central coordination is required to accompany a central mandate to publish. Public bodies need support and coordination to publish data to a lightweight common standard and in a common location.

We want to kick the wheels of this conclusion, identify existing attempts to fix the problem, and talk to people who produce and use the data to see what the obstacles are. If you’re interested, get in touch at research-publicdata@mysociety.org, or drop your views in this survey. 

What is fragmented public data?

Fragmented public data is when many public authorities are required to publish the same data, but not to a common standard (structure) or in a single location – so data becomes fragmented across multiple locations and multiple formats. 

For example, every English local authority is currently supposed to publish all spending over £500 each month. From Adur to Wyre Forest, council officers are working hard to publish monthly spending. 

In theory, this data should be being used by companies, researchers and journalists to provide insight into spending, spot fraud, and find opportunities to sell to councils.

But in practice, to use the data, you’ll need to search all 333 council websites each month, then import each spreadsheet into a central database – and you’ll spend a lot of time pulling your hair out, because the spreadsheets don’t use a consistent format. 

As a result, not much has actually been done with all this data. And the grand promises that spending data would unleash an ‘army of armchair auditors’ have largely failed to materialise. 

Why does this matter?

This is a problem because most of the effort is already being spent to do the job ineffectually. Councils do a lot of work to produce this data, and companies and analysts waste time fixing import scripts or crowdsourcing data, rather than creating new products or insights — and for many organisations, the skills and resources required to create national level datasets are beyond them.

This is not an isolated problem – there are many other examples of fragmented data. 

From assets of community value to election information to council land and property assets, data is often published in a fragmented and hard to reassemble way. 

For many datasets, while individual disclosures are useful, the combined data is much more than the sum of its parts because it allows real understanding of the picture across the whole country, and makes it easier to draw comparisons between different areas.

Across all these datasets the potential loss is huge – and just a bit of extra work could unlock huge amounts of the overall value of the data. We want to fix this for data that is already being published, and make sure that datasets in future are published in the best possible way. 

So what can we do?

The big problem is one of coordination. We think the UK’s central data teams just need to help public authorities do two things:

1. Use a lightweight common standard
2. Report the location of the data in a central register.

By taking things that individual authorities are doing anyway, but getting an agreed format and location, all the individual datasets become far more useful. But the details of the standard are less important than the fact that the government and legislators should be as interested in this side of the problem as they are in requiring the data to be published in the first place. 

We’re keen to learn lessons from previous attempts to do this, and reviewing old publication guides, our initial conclusion is that these over-complicated the idea of what open data is (with a fixation on file formats and linked standards), rather than simple interventions that help both publishers and users (where generally, the best approach is probably an Excel template with common headers). Common standards need to be a compromise between technical requirements and the people who work in local authorities who produce this data.

We’re encouraged by an approach taken by the Scottish Parliament, where publication of compliance with climate change duties was mandated in a particular format – and a consultation on this process found most organisations involved thought standard reporting was an improvement. We’re interested in any other examples of this kind of approach. 

What are we doing now?

We’re trying to explore this problem a bit more, to understand the scale of the problem, and the viability of our approach.

We are interested in talking to:

• People who have run into this issue, what they were trying to do, if they were able to overcome it, or if they had to give up.
• People who publish information in public bodies to understand what their restrictions are
• People or organisations with experience in trying to coordinate a single standard or location for public data releases.

Get in touch at research-publicdata@mysociety.org, or drop your views in this survey. 

—

Header image: Photo by Olav Ahrens Røtne on Unsplash

This is a more technical blog post in companion to our recent blog about local climate data. Read on if you’re interested in the tools and approaches we’re using in the Climate team to analyse and publish data. 

How we’re handling common data analysis and data publishing tasks.

Generally we do all our data analysis in Python and Jupyter notebooks. While we have some analysis using R, we have more Python developers and projects, so this makes it easier for analysis code to be shared and understood between analysis and production projects. 

Following the same basic ideas as (and stealing some folder structure from) the cookiecutter data science approach that each small project should live in a separate repository, we have a standard repository template for working with data processing and analysis. 

The template defines a folder structure, and standard config files for development in Docker and VS Code. A shared data_common library builds a base Docker image (for faster access to new repos), and common tools and utilities that are shared between projects for dataset management. This includes helpers for managing dataset releases, and for working with our charting theme. The use of Docker means that the development environment and the GitHub Actions environment can be kept in sync – and so processes can easily be shifted to a scheduled task as a GitHub Action. 

The advantage of this common library approach is that it is easy to update the set of common tools from each new project, but because each project is pegged to a commit of the common library, new projects get the benefit of advances, while old projects do not need to be updated all the time to keep working. 

This process can run end-to-end in GitHub – where the repository is created in GitHub, Codespaces can be used for development, automated testing and building happens with GitHub Actions and the data is published through GitHub Pages. The use of GitHub Actions especially means testing and validation of the data can live on Github’s infrastructure, rather than requiring additional work for each small project on our servers.

Dataset management

One of the goals of this data management process is to make it easy to take a dataset we’ve built for our purposes, and make it easily accessible for re-use by others. 

The data_common library contains a dataset command line tool – which automates the creation of various config files, publishing, and validation of our data. 

Rather than reinventing the wheel, we use the frictionless data standard as a way of describing the data. A repo will hold one or more data packages, which are a collection of data resources (generally a CSV table). The dataset tool detects changes to the data resources, and updates the config files. Changes between config files can then be used for automated version changes. 

Data integrity

Leaning on the frictionless standard for basic validation that the structure is right, we use pytest to run additional tests on the data itself. This means we define a set of rules that the dataset should pass (eg ‘all cells in this column contain a value’), and if it doesn’t, the dataset will not validate and will fail to build. 

This is especially important because we have datasets that are fed by automated processes, read external Google Sheets, or accept input from other organisations. The local authority codes dataset has a number of tests to check authorities haven’t been unexpectedly deleted, that the start date and end dates make sense, and that only certain kinds of authorities can be designated as the county council or combined authority overlapping with a different authority. This means that when someone submits a change to the source dataset, we can have a certain amount of faith that the dataset is being improved because the automated testing is checking that nothing is obviously broken. 

The automated versioning approach means the defined structure of a resource is also a form of automated testing. Generally following the semver rules for frictionless data (exception that adding a new column after the last column is not a major change), the dataset tool will try and determine if a change from the previous version is a MAJOR (backward compatibility breaking), MINOR (new resource, row or column), or PATCH (correcting errors) change. Generally, we want to avoid major changes, and the automated action will throw an error if this happens. If a major change is required, this can be done manually. The fact that external users of the file can peg their usage to a particular major version means that changes can be made knowing nothing is immediately going to break (even if data may become more stale in the long run).

Data publishing and accessibility

The frictionless standard allows an optional description for each data column. We make this required, so that each column needs to have been given a human readable description for the dataset to validate successfully. Internally, this is useful as enforcing documentation (and making sure you really understand what units a column is in), and means that it is much easier for external users to understand what is going on. 

Previously, we were uploading the CSVs to GitHub repositories and leaving it as that – but GitHub isn’t friendly to non-developers, and clicking a CSV file opens it up in the browser rather than downloading it. 

To help make data more accessible, we now publish a small GitHub Pages site for each repo, which allows small static sites to be built from the contents of a repository (the EveryPolitician project also used this approach). This means we can have fuller documentation of the data, better analytics on access, sign-posting to surveys, and better sign-posted links to downloading multiple versions of the data. 

The automated deployment means we can also very easily create Excel files that packages together all resources in a package into the same file, and include the meta-data information about the dataset, as well as information about how they can tell us about how they’re using it. 

Publishing in an Excel format acknowledges a practical reality that lots of people work in Excel. CSVs don’t always load nicely in Excel, and since Excel files can contain multiple sheets, we can add a cover page that makes it easier to use and understand our data by packaging all the explanations inside the file. We still produce both CSVs and XLSX files – and can now do so with very little work.

For developers who are interested in making automated use of the data, we also provide a small package that can be used in Python or as a CLI tool to fetch the data, and instructions on the download page on how to use it. 

At mySociety Towers, we’re fans of Datasette, a tool for exploring datasets. Simon Willison recently released Datasette Lite, a version that runs entirely in the browser. That means that just by publishing our data as a SQLite file, we can add a link so that people can explore a dataset without leaving the browser. You can even create shareable links for queries: for example, all current local authorities in Scotland, or local authorities in the most deprived quintile. This lets us do some very rapid prototyping of what a data service might look like, just by packaging up some of the data using our new approach.

Data analysis

Something in use in a few of our repos is the ability to automatically deploy analysis of the dataset when it is updated. 

Analysis of the dataset can be designed in a Jupyter notebook (including tables and charts) – and this can be re-run and published on the same GitHub Pages deploy as the data itself. For instance, the UK Composite Rural Urban Classification produces this analysis. For the moment, this is just replacing previous automatic README creation – but in principle makes it easy for us to create simple, self-updating public charts and analysis of whatever we like. 

Bringing it all back together and keeping people to up to date with changes

The one downside of all these datasets living in different repositories is making them easy to discover. To help out with this, we add all data packages to our data.mysociety.org catalogue (itself a Jekyll site that updates via GitHub Actions) and have started a lightweight data announcement email list. If you have got this far, and want to see more of our data in future – sign up!

—

Image: Sigmund

One of the things we want to do as part of our Climate programme is help build an ecosystem of data around local authorities and climate data. 

We have a goal of reducing the carbon emissions that are within the control of local authorities, and we want to help people build tools and services that further that ambition. 

We want to do more to actively encourage people to use our data, and to understand if there are any data gaps we can help fill to make everyone’s work easier. 

So, have we already built something you think might be useful? We can help you use it. 

Also, if there’s a dataset that would help you, but you don’t have the data skills required to take it further, we might be able to help build it! Does MapIt almost meet your needs but not quite? Let’s talk about it!

You can email us, or we are experimenting with running some drop-in hours where you can talk through a data problem with one of the team. 

You can also sign up to our Climate newsletter to find up more about any future work we do to help grow this ecosystem. 

Making our existing data more accessible

Through our previous expertise in local authority data, and in building the Climate Action Plan Explorer, we have gathered a lot of data that can overcome common challenges in new projects.

These include:

• A swiss-army knife/skeleton key/useful spreadsheet that lists all current local authorities, and helps transform data between different lookups. 
• Mapit An API that can take postcodes and tell you which local authority they’re in (and much more!) Free for low traffic charitable projects.
• Datasets of which authorities have published climate action plans.
• Datasets of which authorities have published net zero dates, and their scopes. 
• A massive 1GB zip of all the climate plans we know about.
• Measure of local deprivation across the whole UK.
• A simplified version of the BEIS local authority emissions data. 
• Measures of similarity between all local authorities (emissions, deprivation, distance, rural/urban and then all of those things together). 

All of this data (plus more) can be found on our data portal. 

We’ve also been working to make our data more accessible and explorable (example):

• Datasets now have good descriptions of what is in each column.
• Datasets can be downloaded as Excel files
• Datasets can be previewed online using Datasette lite.
• Providing basic instructions on how to automatically download updated versions of the data.

If you think you can build something new out of this data, we can help you out! 

Building more data

There’s a lot of datasets we think we can make more of — for example, as part of our prototyping research we did some basic analysis of how we might use Energy Performance Certificate data (for home energy in general, and specific renting analysis). 

But before we just started making data, we want to make sure we’re making data that is useful to people and that can help people tell stories, and build websites and tools. If there’s a dataset you need, where you think the raw elements already exist, get in touch. We might be able to help you out. 

If you are using our data, please tell us you’re using our data

We really believe in the benefit of making our work open so that others can find and build on it. The big drawback is that the easier we make our data to access, the less we know about who is using it.

This is a problem, because ultimately our climate work is funded by organisations who would like to know what is happening because of our work. The more we know about what is useful about the data, and what you’re using it for, the better we can make the case to continue producing it. 

Each download page has a survey that you can fill out to tell us about how you use the data. We’re also always happy to receive emails!

Stay updated about everything

Our work growing the ecosystem also includes events and campaigning activity. If you want to stay up to date with everything we do around climate, you can sign up to our newsletter.

—

Image: Emma Gossett

The goal of mySociety’s Climate programme is to reduce the carbon emissions that are either directly controlled or influenced by local government in the UK. 

From 5-8 July, we will run a prototyping week to understand what mySociety could bring to the problem of improving energy efficiency in the private rented sector.

If you’re interested in being involved with discussions, brainstorming, testing out whatever we build — or all of the above, please fill in this short application form. 

We’ve already run one prototyping week exploring conditional commitment and home energy, but the private rental sector has different challenges, and a different role for local authorities. 

Far more than for owner-occupiers, there is a strong opportunity for local authorities to enforce energy efficiency standards in the private rental sector. The strength of the required standard and the effort needed to enforce it will go up in the next few years. 

Currently there are major problems in enforcement, that unaddressed will mean failing the 2030 deadlines for substantial improvements to the private rented sector stock of houses. 

We want to think about how we could build a service that helps local authorities enforce standards, and/or helps tenants understand and use their existing rights. 

Our initial scoping research can be read online. It summarises existing research into the problems in enforcement, and identifies some potential areas where there might be a mySociety-style approach to the problem. 

If you think you have something to contribute to our discussions and would like to join us as we co-create and test a prototype, for as much or as little time as you can spare, you are very welcome. 

If you are renting from a private landlord right now, we’d really welcome your input. 

Of course, we’re also keen to hear from anyone who can bring lived experience or sector-based knowledge from every angle around the topic. Please tell us all about your areas of knowledge in our application form.

—

Image:  Tracey Whitefoot: pilot net zero retrofit, Melius Homes, Nottingham (CC/by/2.0)

Today the House of Commons’  Public Administration and Constitutional Affairs Committee (PACAC) released its report on the Cabinet Office Freedom of Information Clearing House. 

We submitted evidence to the inquiry. It is also highly worth reading journalist Jenna Corderoy’s evidence, as her investigation and appeals are the big reason that the evidence we have exists, and that the inquiry happened in the first place. 

A summary of the evidence given to the committee can be found in this twitter thread. 

Reaction to report

Many of the committee’s recommendations (such as better procedures and more regular data about the Clearing House) are non-controversial, and indeed would be very good to see implemented. However, a big area of concern is their recommendation around how the ICO should be funded. 

There is an excellent line taken through the report that FOI is a good thing, that makes good things happen. It does not see the Cabinet Office’s coordination role as illegitimate, but does see it as pointing in the wrong direction. The report has a vision of the government embracing the positive benefits of FOI, and the Cabinet Office playing a leading role in setting the standard, and maintaining the direction across all government departments. Rather than a secretive role,  the Clearing House should be public and transparent about how it goes about its business. The Cabinet Office should not just ensure a compliant response across government, but advocate for the principles and benefits of Freedom of Information. 

The report’s recommendations constrain themselves to actions for the government, and do not address the potential for legal change to address the problems. While we generally argue that legal change is needed to bring UK legislation up to the Scottish standard and set firm deadlines for internal review and public interest tests (to prevent the abuse described in PACAC’s report), it would obviously be a good outcome for the government to proactively meet those standards.

But to be realistic, this would be a big shift in culture that seems unlikely to happen spontaneously. The report recommended that the Cabinet Office allow the ICO to conduct an external and independent audit of their FOI procedures. The day before this report was released, after months of delay, the Cabinet Office instead announced the details of their planned internal audit. This is not a good start, and legislative change is still likely to be required to meet the objectives the committee has set out. 

The area where the report falls down is in addressing the urgent problem of ICO funding. While the report has a clear sense of the scale of the problem, their proposed fix is unlikely to improve the situation, and might make it worse. 

At the moment, while government FOI policy is set by the Cabinet Office, the “sponsor department” which provides the ICO with its FOI funding is the Department of Culture, Media and Sport (DCMS). This existing arrangement is bad because there is a clear conflict of interest in the government having the power to underfund its own regulator. As is already the case with the Scottish Information Commissioner, the most suitable sponsor is Parliament itself. 

However, PACAC has decided the problem is not government funding, but that policy and funding are split between two departments. They recommend that either the Cabinet Office takes over funding, or DCMS takes over policy. Given the general approach of the rest of the report (and the role of the Cabinet Office working across government), the effective recommendation is that the Cabinet Office should be the sponsor department of the ICO and responsible for managing its FOI funding. This seems like a bad idea without substantial changes at the Cabinet Office happening first. 

In the happy future where the Cabinet Office is an FOI champion, it is possible to imagine this arrangement working, but this is not where we are now.  The Cabinet Office is currently one of the least suitable departments to be trusted to treat the ICO appropriately. Evidence from the previous Information Commissioner stressed that one reason they are not able to use their full legal powers to address issues is a lack of funding. The Cabinet Office has fairly directly benefited from an underfunded ICO, in that the system of regulation meant to prevent the problems described in PACAC’s report just doesn’t have enough resources to work correctly. It would be a bizarre outcome of this process to give them even more direct power to make this happen. 

The positive understanding of FOI from this committee is very much welcome, as is the pushback against new bodies being excluded from Freedom of Information. But wider changes are needed to bring this vision of how FOI improves good governance into practice. As we have argued before (and will set out in more detail in the next few weeks), the most appropriate long term arrangement for funding the ICO is through a direct grant from Parliament. Legislative change to make key principles unambiguous, and to learn from best practice across the UK and further afield, are needed to take us further along the road into the positive future imagined. 

Summary of PACAC Recommendations 

For Cabinet Office

Clearing House

• Publish more data about the performance of the Clearing House.
  • Return to the previous standard of the number of referrals to the Clearing House split by department and month, quarterly.
    • Additionally, data on how casework volume is split by referral category and timeliness.
• Accept ICO’s offer for an audit to “to reassure the public that the Government’s approach to Freedom of Information requests is compliant with the Freedom of Information Act and that they are handled with the utmost professionalism”. 
  • Publish an action plan in response to this. 

FOI Procedure

• Adopt procedures that guarantee the legal standard of applicant-blind processes for requests.
• Guidance on limited circumstances Minister and Special Advisors can get involved in responding to FOI.
• Establish timetable for completion of internal review. 

Lead and model good FOI practice

• Cabinet Office should “drive a cultural shift from mere baseline compliance with the [FOI Act] to to a greater advocacy for the core principles and tenets of the Act”
• Cabinet Office should model best practice, and intervene elsewhere in government that doesn’t meet this best practice. 
• Issue guidance on the need to maintain public record given private messaging systems.

Wider government

Set the right tone and meet a higher standard

• PACAC wants to see stronger tone on the benefits to good government from Freedom of Information, and greater demonstrable action on steps taken to improve outcomes for Freedom of Information applicants. 
• Review decision to exclude ARIA from Freedom of Information Act, but only to ensure this is not a precedent. The report criticises the way FOI was discussed at the time, but does not directly say the decision should be reversed. 
• Government as standard (with Cabinet Office driving change) should respond to internal review within the 20 days suggested by the ICO. 

FOI Governance

• Reconcile the split between FOI Policy responsibility (Cabinet Office) and sponsor department (Department for Culture Media and Sport) by either shifting policy from Cabinet Office to DCMS, or the funding responsibility from DCMS to Cabinet Office.

 

The right to access information requires high quality oversight. Studies of effectiveness of Access to Information (ATI) legislation tells a clear story: the benefits of greater transparency and access to information can only be realised when this system is actively enforced. To be effective, the whole system of ATI review and appeal has to be designed as a system of cultural change. The system has to use limited resources in a strategic way to reform cultures of unnecessary secrecy in government that protect corruption and inefficiency in public life. 

Building on a comprehensive picture of appeal systems and processes across Europe, our new report argues for the value of specialised oversight bodies (Information Commissioners), who have independence from government and the power to compel compliance from authorities. In countries that use a system of internal review, better monitoring and interventions are necessary to ensure this system enhances rather than detracts from access to information.

The report can be read online, or downloaded as a pdf. 

Summary of recommendations:

• Better investment in the resources, capacities and independence of Information Commissioners improves the quality of the ATI regime, attacks corruption, and strengthens good governance.
• Specialist Information commissioners are preferable to general ombudsman, bringing more specific knowledge, and are a more suitable structure to shepherd the access to information regime. 
• The power to enforce decisions is a required tool for driving culture change in public authorities. 
• Systems of internal review should be replaced by commissioner-led systems of appeal, where information commissioners have understanding of appeals across the entire system, and can use internal review as a strategic choice, rather than a hurdle before an appeal can be considered. 
• In general, oversight bodies and civil society rarely have high quality information about full workings of the ATI system. We argue that better quality statistics are a valuable tool in demonstrating the value of the system, and in allowing targeted focus on problems. 

Fundamentally, good ATI regimes are important because of the effects they have in society, strengthening anti-corruption and good policy-making approaches. Better oversight is a cost-effective way of unlocking these wider benefits. This report explores how technical details of how the oversight system works are important in achieving these overall objectives.

The report can be read online, or downloaded as a pdf. 

mySociety’s climate programme is focused around reducing the carbon emissions that are within the power and influence of local authorities in the UK. 

A big area of ignorance for us was the public understanding of local government and its role in combating climate change. We anticipated that in general there might be low public understanding, but we wanted to know more about the shape of that understanding, and how that might affect how we approach our work.  

We commissioned Tom Sasse to write a literature review to help us understand more about this area. The report can be read online or downloaded as a PDF. 

The report has seven key messages:

Net zero is a national mission on a huge scale. It will mean changes to our landscapes and lifestyles, our physical infrastructure and energy systems, our homes, our diets, the way we travel. As well as being necessary, it should be a positive transition (think cleaner, healthier, more productive communities). But it will require some big investments, which need to be paid for fairly, and changes in people’s lives. Broad support will be critical.

To be successful, net zero will require strong local direction. This is the message from a growing chorus of voices (and not just diehard devolution advocates). Central government will have to take some decisions and set direction. It overwhelmingly controls funding. But the path to net zero will be different across the UK: rural areas face very different issues to dense urban ones. Councils are well placed to understand local concerns, needs and capabilities – and build support for local transition pathways. They can act as conveners, working with local citizens, businesses and community groups.

Local climate action is gearing up, but councils face constraints. Councils hold key levers in housing, transport and planning; in all, they exercise powers or influence over around a third of all UK emissions. Almost all have a net zero target and four fifths have published a climate action plan. But these vary in scope and detail, and many councils are still getting to grips with how to cut (or just measure) emissions. Some have started engaging citizens. But in general councils lack funding and capabilities, which is holding back action.

The public supports stronger climate action. In the last three years public concern about climate change has climbed from mid-ranking to be the third most important issue facing the country – and this change appears pretty robust. Cost of living pressures present challenges, but recent polling shows the public still overwhelmingly supports the UK’s net zero target. Most people think stronger action is needed to meet it.

The public supports more local involvement in net zero. Trust and satisfaction with government has been in decline, but attitudes towards local councils have held up much better (44% trust their local councillor vs 19% for government ministers). In particular, people much prefer councillors when asked who should make decisions about their local areas. They think local areas have a high degree of responsibility for tackling climate change, and believe central government should provide more funding to enable local action.

Public understanding of local government is relatively low. The public makes little distinction between different tiers of government. While people know councils are responsible for some highly visible services like waste collection, their understanding of the range of actions local authorities could take to tackle climate change is limited.

Climate action will need to be framed around local concerns. The public are particularly keen on certain net zero policies including frequent flier levies, carbon taxes, improved public transport, and support for replacing gas boilers and installing energy efficiency. But there has been less polling on preferences around local action. People have some concerns about how costs and impacts will fall, and when asked about priorities for their local area, they tend to raise wider issues like vibrant high streets or youth employment.

The report concludes by offering recommendations on four areas where mySociety could make a difference: supporting efforts to address information gaps and raise awareness, which could help provide a stronger foundation for local climate action; supporting wider use of public engagement, and helping councils to achieve wider reach via digital tools; helping to address knowledge gaps around public opinion and preferences for local net zero action; and supporting more effective ways of monitoring and tracking progress in local climate action.

Reflections on the report:

While we anticipated (and the report confirms) low public awareness around some aspects of local authorities, the process of writing the report helped expose some questions we’ve been dancing around the edges of, about how local authorities relate to their communities. For instance, our work is focused on the potential of local authorities to reduce emissions, but there are different opinions on exactly the proportion of emissions local authorities could have influence over. Depending who you ask, it can be anywhere between almost no emissions, and almost all emissions. This report helps navigate that area, in particular distinguishing between where local authorities are genuinely best placed to take action (and how this capacity could be improved), versus where central government may want to shift responsibility without properly enabling action. 

This report has been helpful in encouraging clearer thinking about what we mean when we talk about ‘local action’. Some sources slide between talk of local action (from communities) and local action (from local authorities). Accepting a framing that local authorities are not only better placed than central government to work with local communities (certainly true) but are the same thing as their local communities (definitely not true) hides a large problem area where local authorities’ ability to work in and with their community can be substantially improved. Being clear-eyed about the current situation, as well as the potential for change, is essential for the success of our programme of work. 

The report can be read online or downloaded as a PDF. 

__

Photo credit: Photo by Luke Porter on Unsplash