Links

• Online version of this document
• Application form
• Potential collaborators form
• Join the research commissioning mailing list/Research pool
• Download PDF

In one sentence

mySociety is looking for an individual, organisation or joint team to explore requests made through WhatDoTheyKnow.com to understand how requests for environmental information are being used in practice, highlighting success stories in releasing information at the local level, problems requesters encountered, and suggestions on how mySociety could better make use of the environmental information requests. 

About mySociety

mySociety is the charity behind UK civic services like TheyWorkForYou, WhatDoTheyKnow, FixMyStreet, and WriteToThem. We build open, digital solutions to help repower democracy, in the UK and around the world.

mySociety’s climate programme is funded by Quadrature Climate Foundation and the National Lottery Community Fund.

About mySociety’s existing work in this area 

mySociety’s Climate Programme is exploring how digital tools and approaches can work to reduce the third of UK emissions that local authorities have influence over. A key focus of this work is improving the quality and quantity of information that exists around local climate action. We make local authority plans more accessible and visible, campaign for better official data, and support pooling and crowdsourcing of data to improve knowledge and accountability for climate action. You can read more about mySociety’s Climate programme here.

WhatDoTheyKnow.com is a website built by mySociety, and administered by mySociety in partnership with volunteers, that helps people make Freedom of Information (FOI) requests and publishes the results in public. Over 900,000 FOI requests have been made through this platform. We have published research looking at the scale of FOI in local government, and in improving FOI through the UK, and Europe. 

About this project

Our ultimate goal in our climate work is to decrease UK carbon emissions that are either directly controlled or influenced by UK local government. Our main approach is by improving the information environment so that a variety of local institutions and actors are better able to understand the current situation, share knowledge on their approaches, and take more informed action. We have done this by making local authorities own plans and documents more discoverable and searchable, supporting crowdsourcing of information about plans and actions, pooling information held by different groups of campaigners, and arguing for improved publishing of official information. 

Another method open to us is to use (or facilitate the use of) Freedom of Information laws to get more official data and information into the public domain. Freedom of Information laws give a wide range of public access to information held by authorities. This is useful in releasing information about authorities’ own plans and actions, but is also useful in releasing underlying information which local authorities, so that other organisations (including companies, charities, and other public sector bodies) can use to inform their own approach and actions. 

The Environmental Information Regulations (EIR) substantially overlap with Freedom of Information. EIR requests make it easier to ask for and receive information specifically related to the environment, and covers a wider range of organisations than the general Freedom of Information law. We believe there is the potential to use improved support and guidance for Environmental Information Requests to facilitate more information relevant to local climate action being released, and being made discoverable by local authority actors, civil society and communities. This might take the form of modifications to WhatDoTheyKnow, or light-weight specialist services that sit on top of WhatDoTheyKnow’s FOI archive and infrastructure (similar to FragDenStaat’s Climate Helpdesk). 

But before we can do that, we want to understand more about Environmental Information Requests. WhatDoTheyKnow’s archive gives us access to a huge number of requests made for environmental information, but we understand very little about the contents of these requests. 

Based on a quick text search, we have a maximum of 70,000 requests that in some way mention environmental information regulations. As part of our wider pool of 1 million FOI requests, there may be other requests with environmental interests that have not explicitly mentioned EIR. We are looking for a contractor to explore the large WhatDoTheyKnow dataset, and highlight themes, patterns, and individual examples that show how EIRs are being used, obstacles users are encountering, and the implications of this for mySociety’s future work in this area. 

The available budget for this work is up to £8000-10,000 (inclusive of VAT), and the project would need to be completed by the end of July. There is a small amount of flexibility on this end date – but we are looking to make decisions in August about best ways to follow up the project in September. 

What we want to know more about

The broad goal of this project is to refine our understanding of how EIR works in practice. We’ve broken this into six areas we’d like to understand more about. 

1. EIRs have fewer exemptions, and a higher threshold for withholding material on public interest grounds, than FOI and. In principle, relevant information should be easier to get through EIR, but it is unclear how much this is significant in practice. We want to know more about the kinds of information that are successfully being released through EIR.
2. EIRs apply to more types of organisation – and organisations that have an important local footprint may be accessible to FOI. We want to know about the kinds of institutions and authorities who are or could be releasing information under EIR that is relevant to our local emissions goals.
3. Authorities should consider which regime a request falls under – and default to the more permissive EIR regime when a request is for environmental information. We want to know if there are cases where information is being withheld under FOI but should have been considered under EIR.
4. EIR might be more poorly understood by both requesters and authorities – information may be being incorrectly withheld that should be released. We want to understand any patterns of refusal that would inform advice tools. 
5. At the moment we don’t have an automatic way of identifying EIR responses (when authorities have judged a request under these rules), but it should be possible to create classification rules that separate them from FOI requests. We want to refine this approach so we can more consistently identify and analyse EIR responses, so they can be in current or future local climate services.
6. The big way in which EIRs are less accessible is that authorities can charge fees without a minimum amount of time taken, which is the case for FOI requests. Generally we think this doesn’t happen much (the ICO’s guidance is not to charge for a reasonable request), but we don’t know. We want a clearer sense of how often authorities ask that requesters pay a fee for environmental information.

What we are looking for in and from a partner

Expertise / skill set

We think there are a few different approaches to this project and are open to a range of approaches. In general we anticipate two main kinds of applicants: one leaning more on technical ability, and the other on specialist knowledge of FOI and EIR, or environmental data more broadly. For either kind of applicant, we can provide basic support in the other skillset. The ideal candidate would cover all of these bases (where a subject matter expert makes the technical search process sharper) – and we can help facilitate partnership applications between technical and specialist partners who might otherwise submit separately. If you are interested in this route, please fill out this collaboration form and we can help put you in contact. 

The key thing an applicant needs to have is an approach to searching the large quantity of information on WhatDoTheyKnow. At a minimum this requires some technical skills and ability to navigate the site – but beyond that might be accomplished by large scale text analysis, or a sampling and search method. WhatDoTheyKnow’s search allows searching for specific search terms – but the search count is not always accurate for larger queries. 

For technical strategies, we can provide data exports to enable searches off-platform. But for subject matter experts, we can also provide an Excel sheet of links to URLs that have triggered particular keywords, and authorities. 

Something that is important to us is being able to point to individual examples – and so approaches based on aggregate analysis (which might be the better approach to identify the scale of fees being charged, for instance, or automatically analyse themes of requests) need to also be able to drill down to individual requests. 

With this in mind, the following is a list of skills we would expect the candidate to have a couple but not necessarily all of the following:

• Technical skills
  • Experience with search and processing large amounts of text delivered in a mostly unstructured way.
  • Experience with Natural Language Processing (NLP) or automated topic extraction. 
• Subject matter skills
  • Knowledge and experience of Freedom of Information and/or Environmental Information Requests
    • Especially practical understanding of refusal and appeal grounds in both FOI and EIR.
  • Less important, but understanding of the wider European context of Environmental Information Regulations might provide additional understanding of potential approaches mySociety could take in the UK. 

Alignment with values and aims

Our Repowering Democracy strategy puts a special emphasis on embedding equity and inclusion in our work practices and services, and our work aims in general to fulfil values of equity/justice, openness and collaboration. 

Applicants should consider if this presents any obstacles to a working relationship, and think about how these values should be reflected in the project plan, either in terms of subject matter to investigate, or research approach.

Working practices

mySociety works flexibly and remotely, and there is no requirement to work from or visit an office. Applicants can distribute their work as appropriate over the time available, but we would expect regular check-ins on progress to be arranged over that period. A shared Slack channel and a specific contact person will be used to help coordinate and quickly share questions and information between mySociety and the researcher. 

Successful applicants would be expected to abide by the mySociety Code of Conduct in mySociety communications channels and events. 

Outputs and deliverables

The purpose of this project is to inform mySociety’s future projects, especially forming the base of knowledge around the use of EIRs for a prototyping week.

We are open to the form of outputs – this may take the form of one large report, briefings around our individual question areas, proposed amendments to guidance, etc. For technical submissions, analysis code under an appropriate licence would form part of the output.

The outputs should also work as a general contribution to knowledge of the current use of EIRs, and we might either publish or edit down and publish these briefings/reports.

Q&A and contact details

The application timeline includes a Q&A event, to which you can sign up at the link at the top of this document. The Q&A session will include an element to help individual researchers coordinate to form a joint submission (applications are also welcome from individual researchers). Answers will be made available in a video on this page for applicants who cannot take part. Questions can be emailed to the contact address below. 

Please send any queries or questions to research-commissioning+eir2023@mysociety.org and mention which project it is in regard to. Questions in advance are preferred and will be prioritised in the session. 

Your application

Applications can be submitted by individuals, organisations, or joint teams of individuals/organisations. These should be sent to research-commissioning+eir2023@mysociety.org by the closing date. 

You should submit a short application, of up to 4 pages of A4. A template for the response can be downloaded at the link at the top of this page, and covers: 

• Who you are (whether an individual, organisation, or joint team).
• A description of your previous experience/previous work and why you want to take on this project.
  • To the extent that this is possible, this should be anonymous and not include names of the org or members of the team (to help with anonymous stages of the recruitment process)
• How you would approach and deliver this project – a short project plan with approximate timings. 
  • This could include discussion of appropriate outputs for the project, and balancing technical and subject matter requirements. 
• The total value (£) of your proposal (including VAT), and high-level breakdown of costs (perhaps an indication of days per person, any other expenses). This does not need to include production costs of the report. 
  • Given the cost of the project, we will not be giving a great deal of weight to budget plans so please keep this short and high-level – we can dig into further details during interviews, if necessary.
• A short description of the individuals or team who will do the work, including biographies

There is a separate equalities monitoring form to fill out, which is processed separately from the main application (there is a link to the form in the application form). This is for understanding the reach of our method of distributing the call for proposals. 

If you are interested in joining a ‘researcher pool’ mailing list that we will contact with details of future projects, please see the link at the top of this document. 

Application timeline

If there are changes during this timeline, the table on the website version of this form will be updated. 

Stage	Date	Description
Call for proposals published	7 March 2023
Q&A Webinar	21 March 2023	An open, online public event for interested bidders to learn more about the project and ask questions. This will be recorded and available afterwards. You can submit questions in advance to research-commissioning@mysociety.org. Questions in advance are preferred.
Questions answered	23 March 2023	Video of the webinar to be made available to all potential bidders, in addition to answers to any other questions submitted via email
Deadline for applications	31 March 2023 (end of day)
Initial decisions	7th April 2023	Applicants to be informed whether they have made it through to a short panel interview (and may be asked for a sample of existing work). Applicants not progressing past this stage to be offered written feedback
Interviews	w/c 10 April 2023	Format to be decided, but this will likely be a one-hour panel interview with several people involved in the climate programme, towards the end of the week (14th, 15th April)
Final decision	w/c 18 April 2023	Remaining applicants to be informed of the final decision. Applicants not progressing to be offered feedback
Project briefing/kick-off meeting	End of w/c 18 April 2023	To include a brief introduction to mySociety, discussion of any onboarding required and approach to project management, communication and catch-ups
Project deadline	End of July 2023	End of project (Time range of project is a little flexible – we want it to inform decisions in August about any follow up work in September)

What happens after the project 

We intend to publish the report or briefs you produce, credited to you, on the mySociety website, licensed under a Creative Commons licence (see recent publications on research.mysociety.org for details). We may make some light edits (beyond proofreading) before we publish. You will be free to make publicly available your own version should you wish to, and any other material based on the research you conducted. 

We will convene a short ‘lessons learned’ session for the contractor and mySociety to discuss how the project went – what went well and anything that could have been improved. We will also discuss any future work based on the delivered project (e.g. if you are an academic and might want to co-author an article) and our ongoing relationship. We would also like to arrange a presentation on the project to mySociety staff, and there may also be an opportunity to promote the work in a public event held by mySociety (budgeting for this would be separate to the project above). 

Terms and conditions

Interested parties must be UK-based individuals or organisations.

 

Questions and Answers

 

For data analysis, what format is the data available in?

We create a regular research export of the database in CSV format. This doesn’t include the last few months of data and removes requester names where possible (this is generally available on the website, but we are trying to future proof against information being too available if redactions are needed in future).

There is a a data table of requests made by authority, and then a dataset of the individual messages (with full text) for each request. 

We currently don’t make the downloaded files themselves available in bulk (but are accessible through the site) – but the bulk export does include any cached conversion of a word document or pdf attached.

We can produce reduced data sets limited to specific authorities or keywords. For instance, to those flagged as potentially EIR projects at the start.

WhatDoTheyKnow data sharing policy for bulk data

What is the time scale for the research?

Broadly we’re looking for work to be completed by the End of July to help us inform how we spend our time in September. Depending on the nature of outputs, there may be some flexibility around this.

The retained EU law will ‘sunset’ EIR at the end of the year, unless explicitly retained. How does this work relate to that?

Our current default assumption is “everything will be fine” and this project can proceed as if EIR will continue past 2023. In the event it is looking like everything is not fine, this research helps us understand more about the impact of EIRs for campaigning purposes.

How have you handled projects like this before and how have they worked?

We’ve previously commissioned two pieces of research: one was about how we should commission research, the other was the role of local government in climate change. In both case we worked with a sole researcher, with regular check-in meetings and a shared slack channel. That said, we’re open to group applications (and this project may be appropriate for that). Neither of those projects was particularly data heavy, but we have worked to provide external researchers with data before. 

What support will you give the project?

The main support and contact for the project will be Alex Parsons (Senior researcher) from aa research and data perspective. There is limited available of the WhatDoTheyKnow on a day to day basis team – but I can either answer questions or get answers to questions we need to know.

—

If you have any other questions about this project or the application, please email : research-commissioning+eir2023@mysociety.org

As a joint project between mySociety and the Centre for Public Data, we have written a set of simple principles for how to get the most impact out of publishing public data.  You can read the report online, or download it as a PDF. 

Fragmented public data is a problem that happens when many organisations are required to publish the same data, but not to a common standard or in a common location. Data is published, but without work to join up the results, it rarely has the intended impacts. 

The results of this are frustrating for everyone. Data users cannot easily use the data, policy makers do not see the impact they want, and publishers in public authorities are required to produce data without seeing clear results from their work. 

Better and more consistent publication of data by local authorities helps enable understanding and action at scale across a range of areas. At the same time, we recognise that the technical advice given has assumed higher levels of technical capacity that in practice is possible for many data publishing tasks. Our goal has been to make sure our advice makes data more accessible, while having a realistic idea of technical capacities and support needed for data publishing. 

This report recommends three minimum features for a data publishing requirement to be successful: 

1. A collaborative (but compulsory) data standard to agree the data and format that is expected.
2. A central repository of the location of the published data, which is kept up to date with new releases of data.
3. Support from the data convener to make publication simple and effective – e.g. through validation and publication tools, coordinating returns, and technical support.

We recommend that:

• Whenever government imposes duties on multiple public authorities to publish datasets in future, it should also provide the staff and budget to enable these features.
• The Central Data and Digital Office should publish official guidance covering the above.

You can read the report online, or download it as a PDF. 

Better data publishing helps climate action

This project is informed by recurring frustrations we have run into in our work. Projects such as KeepItIntheCommunity, which mapped registered Assets of Community Value, were much more complicated than they needed to be because while transparency was required of authorities, coordination was not – meaning the task of keeping the site comprehensive and updated was enormously difficult. In principle, we could build a tool that empowered communities in line with the intentions of the original policy makers. In practice, a lack of support for basic data publishing made the project much harder than it needed to be.

This problem also affects our work around local government and reducing emissions. Local government has influence over one third of emissions, but much of that is indirect rather than from the corporate emissions of the authority directly.  As such, many activities (and datasets) of local government have climate implications, even if the work or data is not understood as climate data. For instance, the difficulty in accessing the asset data of local authorities makes it harder for civil society to cross-reference this information with the energy rating of properties, and produce tools to help councils understand the wider picture. 

In future we will be publishing in more detail the kind of data we think is needed to support local authorities in emission reduction – but emissions reduction cannot be isolated from the general work of local authorities. Improving the consistency of the data that is published helps everyone better understand the work that is happening, and makes local government more efficient. 

Sign up to hear more about our climate work, or to the monthly mySociety newsletter. 

Photo credit: Photo by Olav Ahrens Røtne on Unsplash

The ICO has launched a consultation on how it plans to prioritise Freedom of Information (FOI) complaints. 

We have prepared a draft response that explains the context of the consultation, and our current position. 

We are releasing this draft before the deadline for feedback so we have time to get input from WhatDoTheyKnow users or other interested organisations/individuals.

If you have any comments on our response, please fill out this survey.

If you’d like to make your own response, you can also respond to the ICO directly. You do not have to have an opinion on all questions to make a response. The deadline for submissions is 5pm on 19th December 2022.

A summary of our planned response:

• The ICO thinking strategically about how to prioritise case work is a really useful exercise; our main concern is whether there are better, higher impact uses of time than those proposed. 
• The purpose of the complaints system is not to release information, but to steer the whole FOI regime. Rather than focus on high impact requests, it would be better to look for strategic interventions to unblock common problems experienced by high impact requesters. 
  • Recommendation: Add a prioritisation criteria and process around administrative silence/stonewalling.
  • Recommendation: Use discretion to pay special attention to complaints from authorities where there are repeated problems. 
  • Recommendation: Build statistical knowledge of the whole FOI system (especially internal review) to support earlier and more effective intervention.
• We disagree in principle with prioritising classes of requesters (e.g.. journalists and civil society organisations), and on practical grounds do not think this is the most effective way of achieving the ICO’s goals.
  • Recommendation: reframe these as criteria for the kinds of requests from these groups that would benefit from faster treatment (e.g. time sensitive requests, stonewalling).
  • Recommendation: Alternatively, redefine criteria as kinds of activity rather than requester (‘journalism’ rather than ‘journalists’). Make these more porous categories (where many requests may qualify) and only fast-track a certain percentage of them to manage the overall volume. 
  • Recommendation: As one of these porous categories, if the destination of the request is to a public repository (like WhatDoTheyKnow), this should be seen as a request having a higher public impact. 
• The consultation reflects a view that the impact of FOI is primarily through intermediaries, making high impact requests in the public interest. We believe that the public interest needs to be viewed as wider than the result of high impact requests. There is enormous public value collectively even for privately motivated requests.
  • Recommendation:  Change the definition of frivolous from ‘low public interest in information requested’ to ‘low public interest in pursuing the complaint’.

Our reasoning can be read in full, and there are bullet pointed summaries for each consultation question towards the end of the document if you are looking for inspiration for your own response. 

Header image: Photo by Mihai Lazăr on Unsplash

---

---

Freedom of Information is a right that gives people power over public authorities. The case for FOI is often made on the high profile investigations and the public disclosure of scandals. But the value can be also seen in the quiet success of ordinary citizens being able to access information from the public authorities that hold power over their lives.

We ran a poll with Opinium to find out how widespread the use of Freedom of Information is. We found that 10% of UK adults have used FOI to try and get information they thought would be useful to themselves, their community or wider society. When including people who made a request as part of their work, this figure goes up to 14%. 

Giving evidence to Parliament, the former Information Commissioner said that “one in 1,000 citizens in the UK will file a Freedom of Information Act request, but journalists are standing in their shoes. It is through journalists that the public can understand or get to know why decisions are being made on their behalf. Journalists, public interest researchers and advocacy groups are important requesters”.  This is an important point about the value of Freedom of Information even if it is only used by a few, but based not just on our polling but the ICO’s own polling, this figure is 100 times too small. Freedom of Information is not a niche right, or mainly used by journalists,  but has been used by millions of people. 

Alongside a team of volunteers, mySociety runs WhatDoTheyKnow, a website that helps people make freedom of information requests (so far over 850,000 requests), and displays the results of those requests in public, so more people have access to the results. 

Through running this website, we’re very aware of the many different ways Freedom of Information is used by people, in ways that are not captured by official FOI statistics. We want more and more people to be able to use their information rights, and we want more and more of them to be using WhatDoTheyKnow so that what they find out is available to all.

We want to be doing more to support and advocate for people using their right to know, and see it as our role to be a voice for our users, and this large group of ordinary FOI users, in arguments about the future of Freedom of Information.

We’re proud of what we’re able to do with a small budget, but we want to do more. If you want to help us do that, there’s a number of things you can do:

• If you’ve benefited from Freedom of Information, or support our mission to make information more accessible, please consider making a regular or one-off donation to support our work. 
• If you’re not able to donate now, please join our newsletter so we can keep you up to date with our work and campaigns. 
• If you’re a journalist, researcher or campaigner, have a look at our pro service, which for £10 a month provides a wealth of features to help make and manage requests. 
• If you’re interested in volunteering time, WhatDoTheyKnow as a day-to-day service is run by a dedicated group of volunteers. If you’re interested in getting involved, you can learn more on the volunteering page. 

To read more about the numbers behind the polling, and how we validated it, please see this companion blog post. 

---

Header image: Photo by Samuel Regan-Asante on Unsplash

---

This blog post is a companion post to a shorter blog post explaining the significance of this polling to mySociety’s FOI work.

We know very little about the real picture of Freedom of Information use because there are not comprehensive statistics. Information on users of Freedom of Information is very hard to come by.  We have some information through a survey we run on WhatDoTheyKnow, but we know this only covers the minority of requesters who use our service. 

Knowing about this picture is important to us for several reasons.

The first reason is one of the big benefits to society of WhatDoTheyKnow is that we make public information easier to discover without explicitly asking for it. If we can know more about how many FOIs are being made in total, we can have a better sense of what proportion of this information we’re publishing (based on some of the maths below, It’s probably somewhere from 5-10%). 

The second reason is that conversations about the pros and cons of freedom of information can be dominated by the problems journalists experience in requesting information from the central government. This is a big and important problem, but it shifts the general understanding of the impact freedom of information has had on our society. Through WhatDoTheyKnow, we get a glimpse of a bigger world when citizens are making requests that affect them and their communities – but we don’t see everything, and getting more information about this is vital in informing how we approach our policy and campaigning work. 

As part of a “Giving Tuesday”, Opinium gave five survey questions (for a national representative survey panel)  to a number of charities, including mySociety. We used one of these questions to find out how many people had made a freedom of information request. The rest of this blog post explains the results of that survey. 

The question we asked

Our data comes from an Opinium survey of a representative selection of UK adults that ran between 30th November – 3rd December 2021. Respondents were asked:

The Freedom of Information Act gives you the right to request a large range of information from public authorities (government departments, local authorities, NHS trusts, schools, etc). These are called Freedom of information requests. Have you ever made a Freedom of Information request?

Respondents had the option of responding:

• No I haven’t made a request, and I am not aware of Freedom of Information
• No, I am aware of freedom of information but haven’t used it
• Yes, as part of my job
• Yes, to find out something that might be useful for me personally
• Yes, to find out something that might be useful to my community/society in general
• I’m not sure / NA

Results

mySociety/Opinium polling in 2021 found that 10% of UK adults have used FOI to try and get information they thought would be useful to themselves, their community or society. When including people who made a request as part of their work, this figure goes up to 14%. When looking just at personal use, the figure is 6%. Overall, a majority of people (62%)  had either used FOI or were otherwise aware of it. 

There was a small gender difference in both awareness and use of FOI, with men having higher awareness than women (68% to 57%), and greater use (16% to 11%). Our polling found that the 18-34 age group were the least aware of Freedom of Information (55%), but were also the age group most likely to have made an FOI request (25%). This is possibly partially explained by a much higher rate of using it as part of employment in younger demographics (12% compared to 6% overall), but the number using it for other reasons is still notably higher (some more discussion of this further down). Looking at respondents by nation/region, there was a less than expected proportion of people who made a request in Wales (6%) and Northern Ireland (2%), but a greater number who made a request in London (28%).

Validating these figures

When I first saw some of these figures, I was a little surprised and wanted to explore some different ways to validate the number.

Digging into it, I found that other polls asking different versions of the question show a similar figure, and back of the envelope calculations based on known statistics suggest the basic ballpark is right – there are millions, rather than hundreds of thousands, of people who have used the Freedom of Information Act. 

Part of the reason this figure might be surprising is that our statistical picture of Freedom of Information is so poor, we have  very little idea of the scale of it – and what we do know is misleading as to that scale.  For instance, a recent Financial Times article, when highlighting the (bad) trend of how central governments are withholding more and more information requested, falls into the trap of assuming that this picture represents all freedom of information requests. But departments and ministries are not the only public bodies that receive Freedom of Information requests. In fact there’s good reason to believe they receive only a small percentage of the overall total. 

Most FOI requests in the UK are not covered by official statistics. In 2017, we did a meta-FOI to ask local authorities about the number of FOIs they received. We calculated around 467k were made that year, compared to 46k made in that same year to the central government. From running WhatDoTheyKnow, we know that only 10% of requests made through the site go to the central government departments that are covered in the statistics. 

As the number of Freedom of Information requests is much higher than the official statistics show, this helps explain why the number of requesters can be far higher than expected. Not only are there many more public bodies outside central government, but these bodies are closer to people’s day to day lives, and so a broader range of people might want information, and find it through the Freedom of Information Act. 

Polling by the UK’s Information Commissioners

The clearest reassurance of the 10%ish figure is that a similar poll found a very similar number. Polling by the Information Commissioner’s Office in 2021 found 49% were aware of “the right to request information held by public organisations”. This is lower than our equivalent question, where 62% had either made use of FOI, or were aware of it and not used it. However in the same 2021 survey, 10% of respondents to the ICO’s survey said they had already made use of “the right to request information held by public organisations”. In the previous year this figure was 12%. This figure is very close to our figure of 10-14% making use of FOI, and it is reassuring to see something in this ball park come from a different survey company.

Both these surveys might be wrong of course, but polling by the Scottish Information Commission in 2022 found an even higher number. This poll found 36% of a weighted sample of Scottish respondents had at some time “asked for information from a Scottish public body by letter, email or online form”. 18% said they did this annually or more frequently. This is a much higher number than the other survey. There are several possible reasons why.

1. There is a genuine difference in awareness and use of rights between Scotland and the rest of the UK. 
2. The way this question is phrased should also include requests for personal information (subject access requests) as well as freedom of information requests.
3. This version of the question does not ask about a right, just if someone did something that might have engaged the Freedom of Information Act. This might catch people who get through the process unaware they may have benefited or made use of information rights.

How should we interpret this? There is no strong reason to believe the use of rights is significantly different in Scotland. The Scottish figure was 63% awareness of freedom of Information, which is higher than the ICO UK-wide, but in the same general area as our UK-wide polling, which did not show a significant difference for Scotland. Similarly, our survey found a statistically significant difference in use of FOI by respondents in London and did not find this for Scotland. 

As for subject access requests, we actually know from the (really good) statistics recorded in Scotland that around one-quarter of information requests are subject access requests. So even applying this correction, this question is still suggesting around double the figure from our survey. This is likely to be part of the explanation, but is not all of that. 

This leaves the possibility that by not prompting about rights or freedom of information, this is capturing a set of people who are coming into contact with information rights issues without noticing it. It is possible to exercise your freedom of information rights without being aware you are doing so. My first freedom of information request was made this way as a student, asking OFCOM if they held some information from the Broadcasting Complaints Commissions’s archive. If you email your local council wanting to know something, it should be processed under the Freedom of Information Act, even if you were unaware of it. This is likely to include some interactions with authorities that will have existed before FOI (and information may have been made available) but which is now covered formally through the Freedom of Information process. For example, a request for library opening times could be processed as an FOI request, but may well have been answered before the FOI act existed.

If this is an explanation for a higher number in response to the OSIC survey, it might also explain the higher proportion of 18-34 respondents in our poll who had used Freedom of Information for personal reasons. Contact with public bodies for information is more likely now to be by email, and trigger the formal FOI process. There is more to explore here around possible shifting patterns of first contact with FOI. 

Back of the envelope calculation based on WhatDoTheyKnow statistics

The relevant polling we have is supportive of our poll not being outrageously high. The other approach is to try a very back of the envelope approach based on known statistics to see if this is a reasonable amount of FOIs to have been made. 

Based on previous research by mySociety and the Constitution Unit, we have estimates for the number of FOIs made to local authorities in 2005-2010 and 2017. Filling in the extra years between those dates, extending forward, and doubling the number (roughly 48% of requests made through WhatDoTheyKnow are to local authorities – but we don’t know if this applies more generally or not), this gives roughly 11 million FOI requests all time. On WhatDoTheyKnow there are an average of six requests per user (again, don’t know if this applies more generally) – so applying that ratio gives roughly 2 million requesters all time. A figure of 10% of UK adults would expect roughly 4.3 million requesters all time. 

To get the two numbers more  into sync some combination of the following could be true:

• More local government requests have been made all time than this assumes.
  • Not impossible given this is based on three data points (all of which are incomplete surveys and require some amount of extrapolation).
• A greater proportion of requests being made to non-local government bodies than happens in WhatDoTheyKnow
  • No way of knowing this without a complete statistical picture.
  • OSIC statistics show a higher 60% statistic in Scotland being made to local government.
  • Given there are many  more non-local government public authorities in the rest of the UK, it is reasonable to guess it’s closer to the WhatDoTheyKnow statistic of 50%, but could it be lower than that?
• The ratio between requesters and requests is different outside of WhatDoTheyKnow.
  • Arguments both ways, WhatDoTheyKnow is missing all the ‘not intentionally using FOI’  one-offs, but also some of the bulk requesters who don’t want the results to be public on WhatDoTheyKnow. 

Given so many of these numbers are made-up or trying to generalise from WhatDoTheyKnow to all uses of FOI, there is no real reason not to prefer the figure two separate polls agree on. That said, it is reassuring it is in the right order of magnitude (still talking millions rather than hundreds of thousands of FOI users). This question would be helped by a complete statistical picture of FOI in the UK, and to be honest, that would be so useful, it’d be fine if it proved our current numbers wrong. 

---

Header image: Photo by Mitchell Luo on Unsplash

Demographic difference graphs

Download the polling tables.

The following graphs show the demographic split on aspects of the FOI polling. Where the percentage for a category is higher than would be expected statistically if there was no difference between groups, it is highlighted in blue. If it is smaller than would be expected, it is highlighted in red. For non-highlighted categories there is insufficient data to say the category differs from the general average.

Summary:

• We have updated our UK Index of Multiple Deprivation dataset with deciles and quintiles for local authorities and Westminster constituencies.
• We have updated our UK Rural Urban Classification dataset to sort local authorities and Westminster constituencies into four categories. 
• We want to help you use our data for climate work – see our previous blog post for more details. 

–

Back in 2020 we released a UK-wide version of the index of multiple deprivation (see original blog post). This is a dataset that uses multiple metrics of deprivation to rank all small neighbourhood sized chunks of the UK from most to least deprived.

This data is produced for each nation, but our dataset allows areas to be roughly compared across the whole UK (with a separate file for comparing just Great Britain, without Northern Ireland).

This is useful if you have postcode data you want to add information about deprivation to, but sometimes you want to be comparing the bigger areas like local councils and Westminster constituencies.

In the course of some of mySociety’s recent work,  we’ve added new sheets to the deprivation dataset that show the relative deprivation of UK councils and Westminster constituencies.

This works by using a population weighted average – where each neighbourhood’s raw score is multiplied by its population, added together for the authority/constituency and then divided by the total population. This new score can then be ranked and put into deciles.

Because local authorities and (to a lesser extent) constituencies, have different sizes at a national level, the deciles are based on the percentage of the population rather than number of councils or constituencies. So the 1st decile contains the councils with the highest deprivation, that collectively account for 10% of the population. 

If working with data that is entirely from Scotland, Wales or Northern Ireland, it is better to use one of the official datasets that are derived from the national index.

If you want to use our data for climate related purposes, I run drop-in hours on Thursdays and Friday to talk about our data, or just email me! For more information on our climate data, see our previous blog post. 

You can sign up to our data newsletter to keep up to date with future updates.

–

We’ve also updated the UK-wide Rural-Urban-Classification (RUC) dataset to include local authorities and Westminster constituencies. This is a dataset that combines the different measurements of whether a neighbourhood is urban or rural into a single UK wide dataset. 

Here my approach was slightly different. Our RUC dataset recognises three classifications (“Urban”, “Rural”, “Highly Rural”) as this was the best way of combining the different approaches from different nations. 

For each authority/constituency, we calculated the percentage of the population who lives in areas that fit these three criteria. Then, using a clustering approach, authorities/constituencies were split into four loose categories. 

• Urban – All, or the overwhelming majority of the population live in an urban area.
• Urban with rural areas – places with significant rural areas by volume, but generally where the population is concentrated in an urban area. 
• Rural – Less of the population is concentrated in urban areas.
• Sparse and rural – large rural areas with very dispersed populations.

All of these make sense on a spectrum, so at the margins some authorities will be more similar to ones in other classifications, than to the mean of that classification – but in broad terms these categories reflect different kinds of areas. The original population breakdowns are included if further processing work is useful.

If you want to use our data for climate change related purposes, I run drop-in hours on Thursdays and Friday to talk about our data, or just email me! For more information on our climate data, see our previous blog post. 

You can sign up to our data newsletter to keep up to date with future updates.

Header image: Photo by Héctor J. Rivas on Unsplash

Data users in the UK often encounter fragmented public data, where  public authorities are each spending money to publish data independently, but their outputs are difficult to find and join together. This means that a lot of effort is going into creating data which cannot be used to its full potential.

As a joint project between mySociety and the Centre for Public Data, we are writing what we hope is a simple approach to help address this problem. 

Our current thinking is that some level of central coordination is required to accompany a central mandate to publish. Public bodies need support and coordination to publish data to a lightweight common standard and in a common location.

We want to kick the wheels of this conclusion, identify existing attempts to fix the problem, and talk to people who produce and use the data to see what the obstacles are. If you’re interested, get in touch at research-publicdata@mysociety.org, or drop your views in this survey. 

What is fragmented public data?

Fragmented public data is when many public authorities are required to publish the same data, but not to a common standard (structure) or in a single location – so data becomes fragmented across multiple locations and multiple formats. 

For example, every English local authority is currently supposed to publish all spending over £500 each month. From Adur to Wyre Forest, council officers are working hard to publish monthly spending. 

In theory, this data should be being used by companies, researchers and journalists to provide insight into spending, spot fraud, and find opportunities to sell to councils.

But in practice, to use the data, you’ll need to search all 333 council websites each month, then import each spreadsheet into a central database – and you’ll spend a lot of time pulling your hair out, because the spreadsheets don’t use a consistent format. 

As a result, not much has actually been done with all this data. And the grand promises that spending data would unleash an ‘army of armchair auditors’ have largely failed to materialise. 

Why does this matter?

This is a problem because most of the effort is already being spent to do the job ineffectually. Councils do a lot of work to produce this data, and companies and analysts waste time fixing import scripts or crowdsourcing data, rather than creating new products or insights — and for many organisations, the skills and resources required to create national level datasets are beyond them.

This is not an isolated problem – there are many other examples of fragmented data. 

From assets of community value to election information to council land and property assets, data is often published in a fragmented and hard to reassemble way. 

For many datasets, while individual disclosures are useful, the combined data is much more than the sum of its parts because it allows real understanding of the picture across the whole country, and makes it easier to draw comparisons between different areas.

Across all these datasets the potential loss is huge – and just a bit of extra work could unlock huge amounts of the overall value of the data. We want to fix this for data that is already being published, and make sure that datasets in future are published in the best possible way. 

So what can we do?

The big problem is one of coordination. We think the UK’s central data teams just need to help public authorities do two things:

1. Use a lightweight common standard
2. Report the location of the data in a central register.

By taking things that individual authorities are doing anyway, but getting an agreed format and location, all the individual datasets become far more useful. But the details of the standard are less important than the fact that the government and legislators should be as interested in this side of the problem as they are in requiring the data to be published in the first place. 

We’re keen to learn lessons from previous attempts to do this, and reviewing old publication guides, our initial conclusion is that these over-complicated the idea of what open data is (with a fixation on file formats and linked standards), rather than simple interventions that help both publishers and users (where generally, the best approach is probably an Excel template with common headers). Common standards need to be a compromise between technical requirements and the people who work in local authorities who produce this data.

We’re encouraged by an approach taken by the Scottish Parliament, where publication of compliance with climate change duties was mandated in a particular format – and a consultation on this process found most organisations involved thought standard reporting was an improvement. We’re interested in any other examples of this kind of approach. 

What are we doing now?

We’re trying to explore this problem a bit more, to understand the scale of the problem, and the viability of our approach.

We are interested in talking to:

• People who have run into this issue, what they were trying to do, if they were able to overcome it, or if they had to give up.
• People who publish information in public bodies to understand what their restrictions are
• People or organisations with experience in trying to coordinate a single standard or location for public data releases.

Get in touch at research-publicdata@mysociety.org, or drop your views in this survey. 

—

Header image: Photo by Olav Ahrens Røtne on Unsplash

This is a more technical blog post in companion to our recent blog about local climate data. Read on if you’re interested in the tools and approaches we’re using in the Climate team to analyse and publish data. 

How we’re handling common data analysis and data publishing tasks.

Generally we do all our data analysis in Python and Jupyter notebooks. While we have some analysis using R, we have more Python developers and projects, so this makes it easier for analysis code to be shared and understood between analysis and production projects. 

Following the same basic ideas as (and stealing some folder structure from) the cookiecutter data science approach that each small project should live in a separate repository, we have a standard repository template for working with data processing and analysis. 

The template defines a folder structure, and standard config files for development in Docker and VS Code. A shared data_common library builds a base Docker image (for faster access to new repos), and common tools and utilities that are shared between projects for dataset management. This includes helpers for managing dataset releases, and for working with our charting theme. The use of Docker means that the development environment and the GitHub Actions environment can be kept in sync – and so processes can easily be shifted to a scheduled task as a GitHub Action. 

The advantage of this common library approach is that it is easy to update the set of common tools from each new project, but because each project is pegged to a commit of the common library, new projects get the benefit of advances, while old projects do not need to be updated all the time to keep working. 

This process can run end-to-end in GitHub – where the repository is created in GitHub, Codespaces can be used for development, automated testing and building happens with GitHub Actions and the data is published through GitHub Pages. The use of GitHub Actions especially means testing and validation of the data can live on Github’s infrastructure, rather than requiring additional work for each small project on our servers.

Dataset management

One of the goals of this data management process is to make it easy to take a dataset we’ve built for our purposes, and make it easily accessible for re-use by others. 

The data_common library contains a dataset command line tool – which automates the creation of various config files, publishing, and validation of our data. 

Rather than reinventing the wheel, we use the frictionless data standard as a way of describing the data. A repo will hold one or more data packages, which are a collection of data resources (generally a CSV table). The dataset tool detects changes to the data resources, and updates the config files. Changes between config files can then be used for automated version changes. 

Data integrity

Leaning on the frictionless standard for basic validation that the structure is right, we use pytest to run additional tests on the data itself. This means we define a set of rules that the dataset should pass (eg ‘all cells in this column contain a value’), and if it doesn’t, the dataset will not validate and will fail to build. 

This is especially important because we have datasets that are fed by automated processes, read external Google Sheets, or accept input from other organisations. The local authority codes dataset has a number of tests to check authorities haven’t been unexpectedly deleted, that the start date and end dates make sense, and that only certain kinds of authorities can be designated as the county council or combined authority overlapping with a different authority. This means that when someone submits a change to the source dataset, we can have a certain amount of faith that the dataset is being improved because the automated testing is checking that nothing is obviously broken. 

The automated versioning approach means the defined structure of a resource is also a form of automated testing. Generally following the semver rules for frictionless data (exception that adding a new column after the last column is not a major change), the dataset tool will try and determine if a change from the previous version is a MAJOR (backward compatibility breaking), MINOR (new resource, row or column), or PATCH (correcting errors) change. Generally, we want to avoid major changes, and the automated action will throw an error if this happens. If a major change is required, this can be done manually. The fact that external users of the file can peg their usage to a particular major version means that changes can be made knowing nothing is immediately going to break (even if data may become more stale in the long run).

Data publishing and accessibility

The frictionless standard allows an optional description for each data column. We make this required, so that each column needs to have been given a human readable description for the dataset to validate successfully. Internally, this is useful as enforcing documentation (and making sure you really understand what units a column is in), and means that it is much easier for external users to understand what is going on. 

Previously, we were uploading the CSVs to GitHub repositories and leaving it as that – but GitHub isn’t friendly to non-developers, and clicking a CSV file opens it up in the browser rather than downloading it. 

To help make data more accessible, we now publish a small GitHub Pages site for each repo, which allows small static sites to be built from the contents of a repository (the EveryPolitician project also used this approach). This means we can have fuller documentation of the data, better analytics on access, sign-posting to surveys, and better sign-posted links to downloading multiple versions of the data. 

The automated deployment means we can also very easily create Excel files that packages together all resources in a package into the same file, and include the meta-data information about the dataset, as well as information about how they can tell us about how they’re using it. 

Publishing in an Excel format acknowledges a practical reality that lots of people work in Excel. CSVs don’t always load nicely in Excel, and since Excel files can contain multiple sheets, we can add a cover page that makes it easier to use and understand our data by packaging all the explanations inside the file. We still produce both CSVs and XLSX files – and can now do so with very little work.

For developers who are interested in making automated use of the data, we also provide a small package that can be used in Python or as a CLI tool to fetch the data, and instructions on the download page on how to use it. 

At mySociety Towers, we’re fans of Datasette, a tool for exploring datasets. Simon Willison recently released Datasette Lite, a version that runs entirely in the browser. That means that just by publishing our data as a SQLite file, we can add a link so that people can explore a dataset without leaving the browser. You can even create shareable links for queries: for example, all current local authorities in Scotland, or local authorities in the most deprived quintile. This lets us do some very rapid prototyping of what a data service might look like, just by packaging up some of the data using our new approach.

Data analysis

Something in use in a few of our repos is the ability to automatically deploy analysis of the dataset when it is updated. 

Analysis of the dataset can be designed in a Jupyter notebook (including tables and charts) – and this can be re-run and published on the same GitHub Pages deploy as the data itself. For instance, the UK Composite Rural Urban Classification produces this analysis. For the moment, this is just replacing previous automatic README creation – but in principle makes it easy for us to create simple, self-updating public charts and analysis of whatever we like. 

Bringing it all back together and keeping people to up to date with changes

The one downside of all these datasets living in different repositories is making them easy to discover. To help out with this, we add all data packages to our data.mysociety.org catalogue (itself a Jekyll site that updates via GitHub Actions) and have started a lightweight data announcement email list. If you have got this far, and want to see more of our data in future – sign up!

—

Image: Sigmund

One of the things we want to do as part of our Climate programme is help build an ecosystem of data around local authorities and climate data. 

We have a goal of reducing the carbon emissions that are within the control of local authorities, and we want to help people build tools and services that further that ambition. 

We want to do more to actively encourage people to use our data, and to understand if there are any data gaps we can help fill to make everyone’s work easier. 

So, have we already built something you think might be useful? We can help you use it. 

Also, if there’s a dataset that would help you, but you don’t have the data skills required to take it further, we might be able to help build it! Does MapIt almost meet your needs but not quite? Let’s talk about it!

You can email us, or we are experimenting with running some drop-in hours where you can talk through a data problem with one of the team. 

You can also sign up to our Climate newsletter to find up more about any future work we do to help grow this ecosystem. 

Making our existing data more accessible

Through our previous expertise in local authority data, and in building the Climate Action Plan Explorer, we have gathered a lot of data that can overcome common challenges in new projects.

These include:

• A swiss-army knife/skeleton key/useful spreadsheet that lists all current local authorities, and helps transform data between different lookups. 
• Mapit An API that can take postcodes and tell you which local authority they’re in (and much more!) Free for low traffic charitable projects.
• Datasets of which authorities have published climate action plans.
• Datasets of which authorities have published net zero dates, and their scopes. 
• A massive 1GB zip of all the climate plans we know about.
• Measure of local deprivation across the whole UK.
• A simplified version of the BEIS local authority emissions data. 
• Measures of similarity between all local authorities (emissions, deprivation, distance, rural/urban and then all of those things together). 

All of this data (plus more) can be found on our data portal. 

We’ve also been working to make our data more accessible and explorable (example):

• Datasets now have good descriptions of what is in each column.
• Datasets can be downloaded as Excel files
• Datasets can be previewed online using Datasette lite.
• Providing basic instructions on how to automatically download updated versions of the data.

If you think you can build something new out of this data, we can help you out! 

Building more data

There’s a lot of datasets we think we can make more of — for example, as part of our prototyping research we did some basic analysis of how we might use Energy Performance Certificate data (for home energy in general, and specific renting analysis). 

But before we just started making data, we want to make sure we’re making data that is useful to people and that can help people tell stories, and build websites and tools. If there’s a dataset you need, where you think the raw elements already exist, get in touch. We might be able to help you out. 

If you are using our data, please tell us you’re using our data

We really believe in the benefit of making our work open so that others can find and build on it. The big drawback is that the easier we make our data to access, the less we know about who is using it.

This is a problem, because ultimately our climate work is funded by organisations who would like to know what is happening because of our work. The more we know about what is useful about the data, and what you’re using it for, the better we can make the case to continue producing it. 

Each download page has a survey that you can fill out to tell us about how you use the data. We’re also always happy to receive emails!

Stay updated about everything

Our work growing the ecosystem also includes events and campaigning activity. If you want to stay up to date with everything we do around climate, you can sign up to our newsletter.

—

Image: Emma Gossett

The goal of mySociety’s Climate programme is to reduce the carbon emissions that are either directly controlled or influenced by local government in the UK. 

From 5-8 July, we will run a prototyping week to understand what mySociety could bring to the problem of improving energy efficiency in the private rented sector.

If you’re interested in being involved with discussions, brainstorming, testing out whatever we build — or all of the above, please fill in this short application form. 

We’ve already run one prototyping week exploring conditional commitment and home energy, but the private rental sector has different challenges, and a different role for local authorities. 

Far more than for owner-occupiers, there is a strong opportunity for local authorities to enforce energy efficiency standards in the private rental sector. The strength of the required standard and the effort needed to enforce it will go up in the next few years. 

Currently there are major problems in enforcement, that unaddressed will mean failing the 2030 deadlines for substantial improvements to the private rented sector stock of houses. 

We want to think about how we could build a service that helps local authorities enforce standards, and/or helps tenants understand and use their existing rights. 

Our initial scoping research can be read online. It summarises existing research into the problems in enforcement, and identifies some potential areas where there might be a mySociety-style approach to the problem. 

If you think you have something to contribute to our discussions and would like to join us as we co-create and test a prototype, for as much or as little time as you can spare, you are very welcome. 

If you are renting from a private landlord right now, we’d really welcome your input. 

Of course, we’re also keen to hear from anyone who can bring lived experience or sector-based knowledge from every angle around the topic. Please tell us all about your areas of knowledge in our application form.

—

Image:  Tracey Whitefoot: pilot net zero retrofit, Melius Homes, Nottingham (CC/by/2.0)