A story in this week’s Financial Times [paywalled] has brought the EPC ratings of council-owned properties into the public conversation. This story was based on data obtained through FOI requests as part of the Council Climate Action Scorecards project, which we’ve been working on in partnership with Climate Emergency UK (CE UK).

What you can read in the FT is one story pulled from a wealth of data, but there’s more to come. Our WhatDoTheyKnow Projects tool allowed CE UK’s team of volunteers to conduct a nationwide survey of every council through well-placed FOI requests covering the use of renewable energy, plans for retrofitting, green skills training, road expansion and more. 

The data they gathered has allowed for the understanding of councils’ action on a nationwide scale. This level of oversight has not previously been possible: as with so much about the Scorecards project, it is allowing councils to take more informed action on climate, and individuals to clearly understand what is being done.

Why local action matters

In the UK, it is estimated that around one third of carbon emissions are in some way under the influence of local authorities. 80% of UK councils have declared a ‘climate emergency’ to indicate they recognise the scale of the problem of climate change, and are in a position to take practical steps to be part of the solution. To help local authorities achieve the goals they set themselves (and to push them to go further), we need to engage with the plans that local authorities are making, and the actions they are starting to take. 

In 2021, CE UK and mySociety worked together to produce the first Council Climate Plan Scorecards. CE UK’s upcoming launch is the second iteration of the Scorecards. It is much bigger and more ambitious in scope than the last: it scores not the plans, but the climate actions of every local authority in the UK. 

FOI requests were just one part of the process. As well as giving CE UK access to WhatDoTheyKnow Projects, we developed a crowdsourcing tool for volunteers to use while marking across the 90+ datapoints collected for each council. 

How do you score action?

CE UK moved from scoring plans to scoring actions. That required new approaches to gathering the information. 

The questions CEUK used in the new Scorecards are the result of a long and thorough process of research and refinement. Building on their own research and expertise, they conducted one-on-one consultations with approximately 80 organisations and sector-specific experts. An advisory group of environmental and local government experts provided further discussion and refinement, to help build a list of questions that would practically be possible to answer, and that would reveal important information about the climate actions of councils. 

The aim was to identify areas where information was publicly accessible; but also where gaps existed, especially in operational matters that aren’t often made public. Additionally, CE UK wanted to investigate whether councils are truly implementing the actions outlined in their climate action plans, including aspects like lobbying for additional powers.

Making use of Freedom of Information

Freedom of Information laws means that a huge range of information held by public authorities (including local councils) can be requested by any person who asks. This provides a legal tool to create greater transparency where information is not being published proactively.

For CE UK, the potential of FOI for the Scorecards project was clear – but there were concerns. In consultations with council staff, there was pushback regarding the use of FOI requests due to the potential time and financial burden on council officers who work on climate – with some requests for a more informal survey approach to be used. But the drawback of that would be making good data dependent on goodwill everywhere. FOI requests provided a way to make sure the scorecards were not just effective for councils who engaged with the process and provide an approach that was fair across the country. 

To balance a process where they want to encourage positive engagement from councils, with one that works without that, CE UK’s approach was to plan out the most efficient and least burdensome use of FOI requests. 

Based on feedback from the advisory group, and trial runs to a small number of councils, they eliminated questions that were less important and useful, made more ‘yes/no’ or ‘single number’ responses, and learned where certain questions weren’t relevant to certain areas or groups of councils. 

The subsequent FOI requests became more streamlined, and this resulted in quicker response times for the final requests than they had in the trial – as the information sought was more direct and concise.

In the end, CE UK submitted a total of over 4,000 FOI requests to councils across the UK. The questions were divided into 11 categories, with some being specific to certain types of councils, such as district councils or combined authorities. The next stage was taking these 4,000 requests and getting them into a form that can be used for the scorecards. 

Crowdsourcing and review process

CE UK used WhatDoTheyKnow to manage their FOI request process. mySociety’s WhatDoTheyKnow acts as a public archive for requests – requests made through the site have the responses shown in public to bring more information into the open  – making it more discoverable by other people interested in the information, and reducing the need for duplicate requests being made. As of 2023, a million requests for information have been made through the site, with hundreds of thousands of pieces of information being released. 

A feature we are trialling with a range of organisations is WhatDoTheyKnow Projects, which integrates crowdsourcing tools into WhatDoTheyKnow, and allows the task of extracting information into a new dataset to be spread out. The goal is that this helps organisations be more ambitious in finding out information and helps people work together to create genuinely new and exciting datasets, that no single organisation has ever seen. 

As CE UK’s approach already made heavy use of volunteers and crowdsourcing, this was a natural fit.  Alongside a wider group of 200 volunteers working on getting answers to the other questions, 15 volunteers specifically worked on the FOI requests. These volunteers were a mixture of people with prior experience or professional interest in FOI requests, campaigners well-versed in FOI processes, and individuals new to the concept but eager to engage in activism.

After the crowdsourcing of FOI data was complete, it joined the rest of the data in the new tool mySociety had developed for helping volunteers crowdsource information for the Scorecards.  

From here, councils were given access to the data collected about them and given a right of reply to correct any inaccuracies or point towards information not previously discovered or disclosed. The results of this process will then be reviewed to produce the final Scorecards data, which will be launched this month.

But the Scorecards data will not be the only useful thing that will come out of this process. Because of how WhatDoTheyKnow was used, to see evidence supporting the final Scorecards, people will be able to click through and see the original responses, for instance, to see what councils have lobbied on support for their climate work. 

Some of the FOIs are being used to construct datasets that have a broader impact, and here we come back to that FT story on the Energy Performance Certificate (EPC) ratings of council-owned houses. Building these new public datasets will be useful for councils to understand their own situation, and as we see with the news story, more broadly to understand the challenges ahead for local governments to meet net zero emissions goals. 

Onwards

The original Scorecards project has already been influential on how local governments understand their own plans, and how organisations like the UK’s Climate Change Committee understand the role and progress of local government in the challenges ahead. When the next generation of Scorecards is released, we hope that they continue to be useful in shaping and improving local government action around climate change.

mySociety believes that digital technology can be used to help people participate more fully in democracy, make governments and societies more transparent, and bring communities together to address societal challenges.

The Scorecards project showcases how the combination of digital tools, people power, and the right to information produces powerful results. We hope that the impact of this project can inspire and make possible similar approaches for other problems, or in other countries.

Following the PSNI and other recent data breaches, the ICO has issued guidance to public authorities. This guidance suggests a temporary stop on publishing Excel-style spreadsheets in response to FOI requests made via online platforms like WhatDoTheyKnow. The full advisory note is available online. 

The advisory note emphasises that this is not a reason not to disclose requested information. Instead, the ICO says to release the information from original source spreadsheets as a CSV file – a simpler format than Excel Workbooks, with less potential for including hidden sheets or metadata that can lead to an accidental breach.

–

A focus on file formats is a blunt measure, and one that will need to be superseded by better procedures and technical processes.

We support authorities releasing data in the most appropriate format for the information being requested. This may sometimes mean an extract from a table, and sometimes a complete document. Excel spreadsheets are legitimate public documents, and information released in this format can be hugely valuable. It’s important to develop processes where they can be released safely. 

Significant data breaches involving Excel files clearly show the risks when data management and release processes fail. These include not just breaches we see through WhatDoTheyKnow, but through disclosure logs and releases made directly to requesters. This is an opportunity for public authorities, the ICO and us at WhatDoTheyKnow to reflect on how we can best deliver the huge benefits of public transparency while safeguarding personal data. 

–

Modern authorities need to be good at handling data. Data breaches happen at the intersection of technical and human processes. The FOI team can be the last link in the chain of a data breach when they release the information, but the root cause often goes back to wider organisational issues with the handling of sensitive data.

In the short run, the ICO has recommended training for staff involved with disclosing data. Many teams already have excellent processes and do excellent work, but all authorities should take this opportunity to consider their responsibility on the data they hold, and have appropriate processes in place.

Long term progress means developing good universal processes that keep data safe, regardless of the format of the data or how the data is released. All FOI releases should in principle be treated as if they are being released to the public, because the authority’s ability to stop a data breach ends when the information is released. Making FOI responses public produces huge efficiencies for the public sector, increasing transparency in practice, and multiplying the benefit to society of the information released. 

Technology can also be part of the solution – we need to understand more about why existing technical ways of removing hidden information from Excel spreadsheets are not being used (as described in the ICO’s established guidance on disclosing information safely), and how new tools or guidance can make it easier to release data safely. 

–

A core part of our work at WhatDoTheyKnow is dealing with the practical reality of promoting public transparency while protecting personal information. We take data breaches seriously and have processes in place for dealing with them as promptly as possible. We continue to plan and work to help reduce the occurrences and impact of personal data breaches through both our procedures and technical approach. 

By monitoring how authorities respond to requests on WhatDoTheyKnow, we will seek to understand how this guidance is working in practice, and engage with the ICO and other organisations to promote effective long term approaches to this problem. 

---

Notes on the content of the advisory

Below is our understanding of the advisory note by subject matter:

Freedom of Information requests

• Continue to comply with FOI responsibilities. This guidance is about releasing information in a way that reduces risk of accidental disclosure. 
• Temporarily, do not release original source spreadsheets to online platforms like WhatDoTheyKnow. Instead – convert and release to CSV files.
• If that is not possible, then:
  • Ask if the Excel sheet can be sent to a separate (non-public) address. Proceed with the original address if they ask for this. 
  • In all releases, go through processes to ensure there is no data breach in the material. 

General data management

• Excel files are unsuitable working environments when they become very large (hundreds of thousands of rows). Authorities need to switch to appropriate data management systems that are more appropriate for managing larger amounts of data.  
• Staff who use data software and are involved in disclosing information need continuous training.  
• Understanding of pivot tables and their risks should be incorporated into data management.

The ICO plans to update their guidance on Disclosing Information Safely. 

The checklist released accompanying the advisory has several useful steps on checking for hidden data in Excel sheets. However, on the ‘considered alternative ways to disclose’ step, refer back to the steps in the advisory note. Information converted to CSV can be released to WhatDoTheyKnow in compliance with the advisory note. The advisory note says that the source dataset should continue to be released to WhatDoTheyKnow if it cannot be converted, the requester does not want to use an alternative route, and the authority is confident it does not contain a data breach.

Supported by the Wellcome Data Prize in Mental Health, mySociety are providing technical support to the organisation Black Thrive, whose work addresses the inequalities that negatively impact the mental health and wellbeing of Black people. Their question: does disproportional use of police stop and search impact the mental health of young Black people in England and Wales? This project aims to find out.

Late last year mySociety provided some low-key support during the Discovery phase of the project as Black Thrive were developing their statistics package to extract and enhance data from data.police.uk to make the data more accessible and research-ready (for example, by making it easier to combine the data with other datasets such as Understanding Society, a longitudinal household panel study).

We’re pleased to say that the project made an impact, and we’ll now be more actively sharing our expertise in creating data-heavy, citizen-focused services by collaborating with Black Thrive, King’s College London, and UNJUST CIC in the Prototyping phase of the Data Prize.

Now that Black Thrive have built an automated way of gathering the data and creating the analysis, the next phase focuses on presenting these in an open dashboard that’s straightforward for anyone to use and understand. We’ll also be looking at mechanisms for automatically keeping the data up to date, and adding new datapoints when they become available.

Having built several dashboards into our own services including WhatDoTheyKnow and FixMyStreet, we’ve got some great experience to build off; plus bags of experience in sourcing and munging lots of disparate data from our EveryPolitician project. More recently we’ve been making local council Climate Action Plans explorable and accessible, and also creating an easy-to-use data hub for data about local MPs, constituencies, public opinion and the climate and nature movement so this project sits comfortably within our wheelhouse.

In fact, as it happens, we’re currently using very similar tech on another project, our Local Intelligence Hub prototype, and we’re going to use it as the basis for this new dashboard. Nothing like a bit of crafty repurposing where it helps save time and effort! 

Here’s a sneak peek of how it’s looking at this early stage:

Watch this space and we’ll be sure to keep the updates coming as we progress.

—

Image: Chris White (CC by-nc-nd/2.0)

It finally feels like Spring is in the air, and you know it’s been a busy start to the year when we’re rolling the first two months into one set of monthnotes – in the middle of March!

So, what have we been up to?

Well – we’ve been adding datasets to and testing our alpha version of the Local Intelligence Hub tool that we built with The Climate Coalition. Feedback has been really good and this feels like something that’s really going to level up the ability of UK climate organisations to share data and coordinate their actions, at both a local and national level. We hope to share more about this project in the coming months, once it’s been made available to TCC members.

We submitted talks to a couple of conferences/events – and lo and behold, we’ll be in Sheffield (and online) for the Festival of Debate on May 24, with a panel of exceptional guests. Our topic? “What if you could reshape democracy for the better — and you had 20 years to do so?” Climate is sure to be part of the answer. Fancy joining us? Book here.

Between all this we’ve been working hard with our friends at Climate Emergency UK on the next round of the Council Climate Scorecards. Their draft methodology was released in November 2022, and the first round of marking started in January 2023. Part of our support has included building a Django application to store the marking data – and this has already dramatically improved the experience for Climate Emergency UK’s volunteers. 

Climate Emergency UK are also working with mySociety’s Transparency team, using WhatDoTheyKnow Projects (a WhatDoTheyKnow Pro beta feature that helps researchers crowdsource data out of batch FOI requests) to gather some of the data for the scoring. All their FOI requests will be published on WhatDoTheyKnow later this year.

Our IICT grants are coming to an end soon – we’ve put out a blog post about Lynsted Community Kitchen Garden and the data they’re collecting with the weather station we funded. They have a public event on March 25 if anyone lives near Lynsted and wants to visit to check it out! Updates from Possible and Better Futures should be coming soon.

On the research side, we launched our report on unlocking the value of fragmented public data, which is part of our work into the data ecosystem around climate data. Our plan over the next few months is to support a few research commissions which link in to this report and help to show use of climate data. 

We’ve confirmed a partnership with Dark Matter Labs – we’ll be moving forward with them and our Neighbourhood Warmth prototype, exploring how we could encourage neighbours to come together to take their first retrofit action, such as getting a house survey. We’ll be building a working prototype over the next few weeks, then testing it out with communities in three pilot areas around the UK, to ensure that what we’re building makes sense to the people we’re aiming to serve.

And finally, we met up in person! We had a team meeting in early February which was a wonderful chance for us all to take stock of the last year, and discuss the future. We’ve been making some plans for year 3 of the Climate programme and after widening our scope through prototyping, now we’re going to be focusing back in again on building and proving the impact of the services we’re running.

That’s a very whistlestop tour of our first months of 2023!

 

—

Image: Daniel James

As a joint project between mySociety and the Centre for Public Data, we have written a set of simple principles for how to get the most impact out of publishing public data.  You can read the report online, or download it as a PDF. 

Fragmented public data is a problem that happens when many organisations are required to publish the same data, but not to a common standard or in a common location. Data is published, but without work to join up the results, it rarely has the intended impacts. 

The results of this are frustrating for everyone. Data users cannot easily use the data, policy makers do not see the impact they want, and publishers in public authorities are required to produce data without seeing clear results from their work. 

Better and more consistent publication of data by local authorities helps enable understanding and action at scale across a range of areas. At the same time, we recognise that the technical advice given has assumed higher levels of technical capacity that in practice is possible for many data publishing tasks. Our goal has been to make sure our advice makes data more accessible, while having a realistic idea of technical capacities and support needed for data publishing. 

This report recommends three minimum features for a data publishing requirement to be successful: 

1. A collaborative (but compulsory) data standard to agree the data and format that is expected.
2. A central repository of the location of the published data, which is kept up to date with new releases of data.
3. Support from the data convener to make publication simple and effective – e.g. through validation and publication tools, coordinating returns, and technical support.

We recommend that:

• Whenever government imposes duties on multiple public authorities to publish datasets in future, it should also provide the staff and budget to enable these features.
• The Central Data and Digital Office should publish official guidance covering the above.

You can read the report online, or download it as a PDF. 

Better data publishing helps climate action

This project is informed by recurring frustrations we have run into in our work. Projects such as KeepItIntheCommunity, which mapped registered Assets of Community Value, were much more complicated than they needed to be because while transparency was required of authorities, coordination was not – meaning the task of keeping the site comprehensive and updated was enormously difficult. In principle, we could build a tool that empowered communities in line with the intentions of the original policy makers. In practice, a lack of support for basic data publishing made the project much harder than it needed to be.

This problem also affects our work around local government and reducing emissions. Local government has influence over one third of emissions, but much of that is indirect rather than from the corporate emissions of the authority directly.  As such, many activities (and datasets) of local government have climate implications, even if the work or data is not understood as climate data. For instance, the difficulty in accessing the asset data of local authorities makes it harder for civil society to cross-reference this information with the energy rating of properties, and produce tools to help councils understand the wider picture. 

In future we will be publishing in more detail the kind of data we think is needed to support local authorities in emission reduction – but emissions reduction cannot be isolated from the general work of local authorities. Improving the consistency of the data that is published helps everyone better understand the work that is happening, and makes local government more efficient. 

Sign up to hear more about our climate work, or to the monthly mySociety newsletter. 

Photo credit: Photo by Olav Ahrens Røtne on Unsplash

Data is at the core of everything we do at mySociety, and the better quality it is, the easier our work becomes — so the latest output from TICTeC Labs is particularly welcome. We would love everyone to know exactly what constitutes good quality data!

And, thanks to the members of the Action Lab #3 working group, now they can. They awarded a contract to the Canadian civic tech group Open North, to devise a course on Data Quality. This course is free to everyone, and we know it’ll be of huge benefit to the international civic tech community.

Available online in English and French (and hopefully with more languages to follow), the course provides users with a practical introduction to the topic, discussing key concepts and setting practical exercises.

Quality information for civic tech success

This output was the end result of our third TICTeC Labs Civic Surgery, which took place back in March 2022. That saw participants discussing the theme: ‘Accessing quality information for civic tech success: how can we overcome barriers to accessing good data and documentation?’ — it was within this session that the concept of a training course first arose.

This course uses Open North’s existing learning platform to provide training which covers:

• Understanding the importance of data quality
• Understanding the key terms when engaging with data
• Knowing how and where to find good quality data
• Recognising the barriers to accessing data and documentation
• Knowing how to evaluate the quality of a dataset

Collaborating with the Action Lab members throughout the process of planning and building the course, Open North have created an online educational resource that is suitable for a wide range of audiences. It provides a starting point for those already working with data, or those at the beginning of their journey. 

Take the course

You can find out more, and take the course by signing up to Open North’s Training Center and then looking for Data Quality (D103), with the French version at La qualité des données (D103F). In fact, once your account is activated you can take any of their free courses, so take a look around and you might find some more resources to try, as well.

November was another busy month for our Climate programme, with progress on a number of fronts – from the return of an old friend, in the shape of the Council Climate Scorecards; to the development of two new ones, as a result of our prototyping process earlier this year. We’ve also been working hard to share our data and tools with new audiences. Here’s a quick round up:

Constituency data for climate campaigners

As Alexander mentioned in October, we’ve been working on a Beta version of platform that brings together data about MPs, constituencies, and local climate action, as part of a project with The Climate Coalition. The aim is to help campaigners at both national and local levels to understand where to focus their efforts on enabling real local action on climate goals.

This month—thanks to the involvement of not only Struan and Alexander but also Graeme, on loan from our Transparency programme—we’ve made lots of progress, adding the features and importing the datasets we’ll need for testing out the minimum viable product with target users in the New Year. I look forward to sharing more with you in the coming months!

Exposing high-emissions local authority contracts

Another service that’s come out of one of our earlier prototyping weeks is ‘Contract Countdown’, which aims to give citizens advance notice of large, high-emissions local authority contracts that might be expiring in six, 12, or more months.

This November, Alexander finished developing the final pieces of a working Alpha version – including the use of real contracts from UK Contracts Finder and the Find A Tender service, and pulling in the details of local authority climate officers and councillors with climate/environment responsibilities (so we could test the idea of helping users contact these representatives).

And Siôn and I have been testing the alpha with target users – including local and national journalists, local authority climate officers and procurement officers, and local climate activists. We aim to continue getting feedback on the Alpha throughout December, and maybe January, after which point we can make a decision on whether to develop and launch a full service later in 2023.

Climate Action Scorecards 2023

Speaking of next year, preparations are already underway for next year’s follow-up to the Council Climate Scorecards project—this month saw Lucas and I work with Climate Emergency UK to design and publish their draft methodology for the assessment that will begin next year.

With CEUK’s assessors now looking at councils’ climate actions, in addition to their plans, we wanted to make it as easy as possible to understand precisely which questions your local authority will be scored on. I think we came up with a nice solution, where you can filter the list of draft questions by your local authority name or postcode, as well as by local authority type.

Sharing our data and tools

In other news, Alex updated our deprivation and urban/rural classification datasets to show relative figures for local authorities and Westminster parliamentary constituencies. We also published a local authorities lookup dataset that makes it easy to convert between the many names and codes used to identify local authorities.

If you want to use these new datasets—or any of our data in fact—Alex runs drop-in office hours on Thursdays and Fridays to talk about just that. We’re also happy to help collect or analyse climate-related data for free, as part of our work on supporting the UK’s climate data ecosystem – you can read more about that here.

Speaking of data ecosystems, you’ll now find a number of mySociety’s open climate datasets listed in Subak’s Data Catalogue, and Icebreaker One’s OpenNetZero catalogue.

Finally, Myf and Siôn in particular have continued to share and talk about our tools, and how people are using them to support local climate action, this month. Highlights include attending the Natural History Consortium’s Communicate conference; giving a hands-on workshop about all of mySociety’s tools for London’s small charities and community groups at Superhighways’ “Where’s The Power In Data” conference; and publishing a really exciting case study about how an officer at Surrey County Council used CAPE to share experiences and best practices with other similar councils elsewhere the UK.

—

Image: Designecologist

We spoke to Rebecca Sawyer of Brighton Peace and Environment Centre, to discover how their ReForest Brighton project interfaces with our own CAPE and Council Climate Scorecards sites.

As it turns out, our projects have a lot in common. Both aim to make it easier for everyone to understand and assess the progress a council is making towards cutting carbon emissions, a field where the picture can be complicated and difficult for the average person to follow. That starts with data.

Rebecca explained, “Identifying the path to carbon neutrality is not straightforward, and the data that would enable organisations to know where they currently are on this path is very weak.”

Visualising progress

To address this, ReForest Brighton is developing an interactive website to show in real time the progress that each local authority has made in relation to its individual carbon neutral targets. 

Naturally, the project began by looking at the organisation’s hometown of Brighton, which has a target of Net Zero carbon emissions by 2030. That’s just the start, though; the model is replicable for any other local authority in the UK, allowing their own carbon neutral targets and the actions determined by their climate action plans to be slotted in.  

“The aim,” says Rebecca, “is to provide real time quality data that will enable decision making around policy and practice.

“So for example, if it’s clear that maintaining the current level of action won’t bring a city to carbon neutrality by their set date, the council can refocus their efforts to reduce emissions and sequestrate more carbon.” 

The ultimate target? “To make local authority councils more ambitious.”

Carbon neutral dates

So where do CAPE and the Scorecards site come in? As Rebecca explained, CAPE was useful mainly for a single datapoint amongst the many that it provides.

“The main way we’ve been using it is to retrieve the carbon neutral dates of all the individual local authorities in the UK. 

“Without this data being easily accessible it would’ve taken us a long time and lot of resources to go through more than 300 local authorities and dig out their target dates.”

And as for the Scorecards site, this has been more of a sanity-check tool: “We used it once we’d completed our calculations, to check our ratings of each local authority against the Scorecards rating. 

“For example, if our calculations rated a local authority with high climate action but the Scorecards had it as low, then we’d analyse and reassess our ratings.” 

As well as the interactive map, their project will produce predictive data to show how much progress the council will have made by their target zero emissions date.

Forecast formula

For those who like the technical details, Rebecca is keen to oblige: “Our categorisation is based on a calculation of emission trends from 2016-2020. The trends allow us to predict where each local authority will be by the carbon neutral target date we downloaded from CAPE, using the ‘forecast’ formula (=FORECAST (x, known_ys, known_xs)). 

“There is actually 15 years’ worth of emissions data available, but we chose this five-year period because climate action has only started becoming a consideration for local authorities in the last few years. 

“Basically, we look at the predicted emissions on the authority’s carbon neutral date and categorise them accordingly — and if a local authority had no carbon neutral target or plans, it is automatically rated zero.”

A knock-on effect

ReForest Brighton wants to make it easier for the public to understand how their local authority is doing in achieving its carbon reduction targets — and they have another aim, too: 

“We would like the public to push local governments to take faster, more effective action, and we’re planning to help them do this by giving them the means to write to their elected representatives, and to share the website with their friends and contacts.  

“But even while hoping that councils will be making as much progress as possible, we’re also pushing for transparency. We’d even encourage an authority to push their carbon neutral target date further back if it gave a more honest picture of where they are at.”

Brighton Peace and Environment Centre are a registered charity and they welcome volunteers: get in touch if you would like to know more.

You can also make a donation to them, using this link.

—

Image: Aaron Burden

Last Wednesday a varied audience convened online for our Innovations In Climate Tech event.

The aim: to showcase some of the remarkable and effective projects being implemented in the UK and further afield, and to spark inspiration so that these, or similar projects, might be replicated in other UK regions.

mySociety has three £5,000 grants to give to innovators and local councils who work together and trial something they’ve seen, or been inspired by, during the event.

Missed the live version? Don’t worry: we have videos and notes, and you don’t have to have attended to be able to bid for a grant.

Rewatch or read up

Here’s where to find the various assets from the day:

• Watch a video of all the morning presentations, followed by the Q&A. This video features:
  • Annie Pickering from Climate Emergency UK, on how they scored councils’ Climate Action Plans;
  • Ariane Crampton from Wiltshire County Council on how they tackled outreach to diverse communities with their climate consultation;
  • Claus Wilhelmsen from Copenhagen City Council on the practical ways in which they are tackling carbon cutting within construction industries;
  • Ornaldo Gjergji from the European Data Journalism Network on how visualising temperature data from individual cities and towns helps people better understand the impacts of climate change;
  • Kasper Spaan from Waternet on creating green roofs across the city to aid urban cooling and biodiversity.
• The afternoon breakout sessions weren’t recorded, but you can read notes of the presentations and subsequent discussions for:
  • The Adaptation session (Padlet here) in which Josh Shimmin from Atamate talked about a data-driven approach to retrofitting housing;
  • The Engagement session (Padlet here) in which Susan Rodaway from Pennard Community Council presented on a community consultation tool that helped them decide what to spend budget on; and Arnau Quinquilla from Climate 2025 talked about mapping climate movements across the world;
  • The Spatial Planning session (Padlet here) in which Lora Botev from CitizenLab explained how their software enables councils to run consultations and grow an active group of residents who have a voice in decisions around climate;
  • The Equity, Diversion and Inclusion session (Padlet here) in which Emma Geen from the Bristol Disability Equality Forum explained how vital it is to include disabled people in a green transition, and the ways in which the group has taken action to make this happen.

What’s next?

On 19 October, we’ll be running an informal session online, explaining what we’re looking for in a grant pitch, and giving you the chance to explore your ideas with potential partners. Then, if you want to go forward and bid for one of three £5,000 grants, we’ll give you everything you need to make your pitch.

You do not have to have attended the first event to join us at this stage. Please explore the resources above, add your thoughts to the Padlets, and sign up for this event via Eventbrite.

What sort of projects will we be funding?

To be eligible to bid for one of the grants, you must either be:

• a council that wants to trial an idea; or
• an organisation that wants to work with a council to trial your project.

Partnerships can be between two or more organisations, but every partnership must include at least one local council (and might only consist of councils). But don’t worry if you haven’t got a partner in mind yet – you may find one at this event.

• You might have seen an idea in the presentations that is directly applicable to your council area, and want to simply replicate it.
• Or, in a less straightforward but equally valid scenario, you might simply have seen an organisation you’d like to work with, or had a completely new idea sparked by something you saw.
• You might have no ideas at all, but a commitment to try something new… bring an open mind and see if anything at the event grabs you!

Sign up for the upcoming event here.

 

This is a more technical blog post in companion to our recent blog about local climate data. Read on if you’re interested in the tools and approaches we’re using in the Climate team to analyse and publish data. 

How we’re handling common data analysis and data publishing tasks.

Generally we do all our data analysis in Python and Jupyter notebooks. While we have some analysis using R, we have more Python developers and projects, so this makes it easier for analysis code to be shared and understood between analysis and production projects. 

Following the same basic ideas as (and stealing some folder structure from) the cookiecutter data science approach that each small project should live in a separate repository, we have a standard repository template for working with data processing and analysis. 

The template defines a folder structure, and standard config files for development in Docker and VS Code. A shared data_common library builds a base Docker image (for faster access to new repos), and common tools and utilities that are shared between projects for dataset management. This includes helpers for managing dataset releases, and for working with our charting theme. The use of Docker means that the development environment and the GitHub Actions environment can be kept in sync – and so processes can easily be shifted to a scheduled task as a GitHub Action. 

The advantage of this common library approach is that it is easy to update the set of common tools from each new project, but because each project is pegged to a commit of the common library, new projects get the benefit of advances, while old projects do not need to be updated all the time to keep working. 

This process can run end-to-end in GitHub – where the repository is created in GitHub, Codespaces can be used for development, automated testing and building happens with GitHub Actions and the data is published through GitHub Pages. The use of GitHub Actions especially means testing and validation of the data can live on Github’s infrastructure, rather than requiring additional work for each small project on our servers.

Dataset management

One of the goals of this data management process is to make it easy to take a dataset we’ve built for our purposes, and make it easily accessible for re-use by others. 

The data_common library contains a dataset command line tool – which automates the creation of various config files, publishing, and validation of our data. 

Rather than reinventing the wheel, we use the frictionless data standard as a way of describing the data. A repo will hold one or more data packages, which are a collection of data resources (generally a CSV table). The dataset tool detects changes to the data resources, and updates the config files. Changes between config files can then be used for automated version changes. 

Data integrity

Leaning on the frictionless standard for basic validation that the structure is right, we use pytest to run additional tests on the data itself. This means we define a set of rules that the dataset should pass (eg ‘all cells in this column contain a value’), and if it doesn’t, the dataset will not validate and will fail to build. 

This is especially important because we have datasets that are fed by automated processes, read external Google Sheets, or accept input from other organisations. The local authority codes dataset has a number of tests to check authorities haven’t been unexpectedly deleted, that the start date and end dates make sense, and that only certain kinds of authorities can be designated as the county council or combined authority overlapping with a different authority. This means that when someone submits a change to the source dataset, we can have a certain amount of faith that the dataset is being improved because the automated testing is checking that nothing is obviously broken. 

The automated versioning approach means the defined structure of a resource is also a form of automated testing. Generally following the semver rules for frictionless data (exception that adding a new column after the last column is not a major change), the dataset tool will try and determine if a change from the previous version is a MAJOR (backward compatibility breaking), MINOR (new resource, row or column), or PATCH (correcting errors) change. Generally, we want to avoid major changes, and the automated action will throw an error if this happens. If a major change is required, this can be done manually. The fact that external users of the file can peg their usage to a particular major version means that changes can be made knowing nothing is immediately going to break (even if data may become more stale in the long run).

Data publishing and accessibility

The frictionless standard allows an optional description for each data column. We make this required, so that each column needs to have been given a human readable description for the dataset to validate successfully. Internally, this is useful as enforcing documentation (and making sure you really understand what units a column is in), and means that it is much easier for external users to understand what is going on. 

Previously, we were uploading the CSVs to GitHub repositories and leaving it as that – but GitHub isn’t friendly to non-developers, and clicking a CSV file opens it up in the browser rather than downloading it. 

To help make data more accessible, we now publish a small GitHub Pages site for each repo, which allows small static sites to be built from the contents of a repository (the EveryPolitician project also used this approach). This means we can have fuller documentation of the data, better analytics on access, sign-posting to surveys, and better sign-posted links to downloading multiple versions of the data. 

The automated deployment means we can also very easily create Excel files that packages together all resources in a package into the same file, and include the meta-data information about the dataset, as well as information about how they can tell us about how they’re using it. 

Publishing in an Excel format acknowledges a practical reality that lots of people work in Excel. CSVs don’t always load nicely in Excel, and since Excel files can contain multiple sheets, we can add a cover page that makes it easier to use and understand our data by packaging all the explanations inside the file. We still produce both CSVs and XLSX files – and can now do so with very little work.

For developers who are interested in making automated use of the data, we also provide a small package that can be used in Python or as a CLI tool to fetch the data, and instructions on the download page on how to use it. 

At mySociety Towers, we’re fans of Datasette, a tool for exploring datasets. Simon Willison recently released Datasette Lite, a version that runs entirely in the browser. That means that just by publishing our data as a SQLite file, we can add a link so that people can explore a dataset without leaving the browser. You can even create shareable links for queries: for example, all current local authorities in Scotland, or local authorities in the most deprived quintile. This lets us do some very rapid prototyping of what a data service might look like, just by packaging up some of the data using our new approach.

Data analysis

Something in use in a few of our repos is the ability to automatically deploy analysis of the dataset when it is updated. 

Analysis of the dataset can be designed in a Jupyter notebook (including tables and charts) – and this can be re-run and published on the same GitHub Pages deploy as the data itself. For instance, the UK Composite Rural Urban Classification produces this analysis. For the moment, this is just replacing previous automatic README creation – but in principle makes it easy for us to create simple, self-updating public charts and analysis of whatever we like. 

Bringing it all back together and keeping people to up to date with changes

The one downside of all these datasets living in different repositories is making them easy to discover. To help out with this, we add all data packages to our data.mysociety.org catalogue (itself a Jekyll site that updates via GitHub Actions) and have started a lightweight data announcement email list. If you have got this far, and want to see more of our data in future – sign up!

—

Image: Sigmund