In September we heard from inspiring speakers at our kick-off Innovations in Climate Tech event; in October, we took that inspiration and let you run with it when we hosted a series of online conversations. 

And now, we’re happy to present the teams who will be taking their ideas a little further with the help of our small grants.

We were looking for projects that could test a proof of concept or start something small but meaningful around climate in a local community. Proposals had to have at least one council on board.

Our successful applicants are all working in very different areas, but all of them have great potential to make a difference, and we’re excited to see what emerges from their work. So, let’s take a look at the grantees:

Lynsted Community Kitchen Garden

This collaborative food growing project in Kent will use tech to showcase sustainable approaches to gardening, with an emphasis on adapting to a changing climate. They’ll be working with Swale Borough Council.

Horticulture may be a new area for mySociety, but Lynsted Community Kitchen Garden made a compelling case for how they would collect data through a digital weather station, and use this information to develop adaptation methods which they could then share with other gardeners.

Data is data and we’re excited about its potential whether it’s around our familiar areas of democracy and transparency, or in this case precipitation, hours of sunshine and temperature! When correlated with plant growth and the amounts of watering required, this project should be providing some really tangibly useful outputs.

“We are hugely excited about using climate tech to improve the resilience of our community veg and fruit growing project to weather stress”, said LCKG. “A massive thank you to mySociety for this opportunity, and to Swale’s Climate and Ecological Emergency Officer for their support.”

Possible

Climate charity Possible is behind many innovative initiatives, including the Climate Perks scheme which mySociety subscribes to. For this project Possible will be working with Camden Borough Council to run feasibility studies around installing ground source heat pumps (GSHPs) in more challenging residential areas.

GSHPs have been heralded as a more sustainable option than the gas boilers currently found in most homes. However they are installed outdoors, presenting difficulties for tower block residents, or households with limited space or funds. Air source pumps can be affixed to the exterior of buildings, but this approach can fall foul of planning laws, and they can also be noisy.

Possible will be experimenting in Primrose Hill with a ‘shared loop’ system, in which the collector loops are installed beneath public green space to assess the technical and commercial viability of this approach. 

“One fifth of UK households live in flats, while one quarter live in terraced houses, so the untapped potential of this approach is vast”, say Possible.

Better Futures

 Sandwell Council will team up with Better Futures to research and scope a new project, Climate Interchange. This online database will showcase work undertaken to adapt to climate change challenges, from councils across the UK. 

The project has the kind of user-focused approach that we heartily approve of at mySociety: it will begin with asking officers in councils across the country what they need, before creating a  searchable project database of solutions and case studies.

“By opening up data and sharing we want to democratise climate adaptation solutions, putting actionable insights into the hands of those on the front line in communities and local government”, says Better Futures’ Rob Hale.

There are clear parallels here with the Scorecards work our partners at Climate Emergency UK are engaged in, and we hope that the two projects will benefit one another while providing richer resources to councils and the public.

 —

We’ll check in with our grantees to see what they achieve and what they learn along the way, so do watch this space for updates.

With Twitter’s future uncertain, we encourage friends and followers to subscribe to our newsletters, or to use the RSS feed which you can find on the right hand side of this blog page.

—

Image: Lynsted Community Kitchen Garden

Summary:

• We have updated our UK Index of Multiple Deprivation dataset with deciles and quintiles for local authorities and Westminster constituencies.
• We have updated our UK Rural Urban Classification dataset to sort local authorities and Westminster constituencies into four categories. 
• We want to help you use our data for climate work – see our previous blog post for more details. 

–

Back in 2020 we released a UK-wide version of the index of multiple deprivation (see original blog post). This is a dataset that uses multiple metrics of deprivation to rank all small neighbourhood sized chunks of the UK from most to least deprived.

This data is produced for each nation, but our dataset allows areas to be roughly compared across the whole UK (with a separate file for comparing just Great Britain, without Northern Ireland).

This is useful if you have postcode data you want to add information about deprivation to, but sometimes you want to be comparing the bigger areas like local councils and Westminster constituencies.

In the course of some of mySociety’s recent work,  we’ve added new sheets to the deprivation dataset that show the relative deprivation of UK councils and Westminster constituencies.

This works by using a population weighted average – where each neighbourhood’s raw score is multiplied by its population, added together for the authority/constituency and then divided by the total population. This new score can then be ranked and put into deciles.

Because local authorities and (to a lesser extent) constituencies, have different sizes at a national level, the deciles are based on the percentage of the population rather than number of councils or constituencies. So the 1st decile contains the councils with the highest deprivation, that collectively account for 10% of the population. 

If working with data that is entirely from Scotland, Wales or Northern Ireland, it is better to use one of the official datasets that are derived from the national index.

If you want to use our data for climate related purposes, I run drop-in hours on Thursdays and Friday to talk about our data, or just email me! For more information on our climate data, see our previous blog post. 

You can sign up to our data newsletter to keep up to date with future updates.

–

We’ve also updated the UK-wide Rural-Urban-Classification (RUC) dataset to include local authorities and Westminster constituencies. This is a dataset that combines the different measurements of whether a neighbourhood is urban or rural into a single UK wide dataset. 

Here my approach was slightly different. Our RUC dataset recognises three classifications (“Urban”, “Rural”, “Highly Rural”) as this was the best way of combining the different approaches from different nations. 

For each authority/constituency, we calculated the percentage of the population who lives in areas that fit these three criteria. Then, using a clustering approach, authorities/constituencies were split into four loose categories. 

• Urban – All, or the overwhelming majority of the population live in an urban area.
• Urban with rural areas – places with significant rural areas by volume, but generally where the population is concentrated in an urban area. 
• Rural – Less of the population is concentrated in urban areas.
• Sparse and rural – large rural areas with very dispersed populations.

All of these make sense on a spectrum, so at the margins some authorities will be more similar to ones in other classifications, than to the mean of that classification – but in broad terms these categories reflect different kinds of areas. The original population breakdowns are included if further processing work is useful.

If you want to use our data for climate change related purposes, I run drop-in hours on Thursdays and Friday to talk about our data, or just email me! For more information on our climate data, see our previous blog post. 

You can sign up to our data newsletter to keep up to date with future updates.

Header image: Photo by Héctor J. Rivas on Unsplash

Whilst the days get shorter, October is already over. Here’s what the Climate team has been up to this month!

Events

This month, we ran our second Innovations in Climate Tech event. It went well! Some project ideas from the first event began to build up momentum and excitement, and we closed our applications for grants at 23:59 on the last day of October. That means we’ll be announcing our decisions very soon! If you want to find out more about how the event went, check out Myf’s post here.

Communications

Myf graduated from the Weston Communicating Climate programme, and is feeding back to the team everything she’s learnt. It was an in-depth course, where she’s picked up lots of very valuable information to help us use communications around climate as effectively as possible.

Development

Off the back of our prototyping weeks over the year, we have been continuing developing.

Struan’s been working on development of the Local Intelligence Hub (which you can read about the prototype of here), and we look forward to borrowing Graeme from the Transparency team over the coming month to help bring this to life.

Meanwhile, Alexander has been working on Contract Countdown (with the prototype report for that here), getting it ready for focus groups, which Siôn has been hard at work to start getting together.

Our partners, Climate Emergency UK are working on their methodology for measuring actual climate action from councils. This is for the next iteration of their climate scorecards site, which up until now has only assessed councils’ plans rather than what progress they’ve made in implementing those plans. Full details of the criteria they’re working to will be released in November, before they dive into the rigorous process of scoring.

The Climate Action Plan Explorer is going to be undergoing some improvements – Myf has been looking at how to make the tool more accessible for non-specialist users and the team are now beginning to see how that can feed into development. Look out for changes between now and December.

Everything else

The Climate team has recently started to experiment with “fallow sprints”. Placing them at either end of a cycle, they’re allowing the team time to plan, and regroup, ready for the next sprint. This is helping us to feel more focussed in our work, and seems to be doing good for the team as a whole.

Even as the nights get longer and the days get colder, we’re not slowing down, so if you can’t get enough of what we’re up to, you can sign up to our newsletter to get updates in your inbox!

 

—

Image: Jonathan Cutrer (CC by-nc/2.0)

Yesterday was the second Innovations in Climate Tech event. People from councils and organisations came along and discussed all kinds of projects and ideas.

The key question? What they might do with a small injection of money designed to kickstart digitally based, local climate related projects.

If you’re ready to go ahead with your application, start here. Otherwise, read on.

Projects beginning to form

You can see all the ideas that were floated in our first meetup on our Padlet, but here are a few of the projects that emerged and appeared to be gaining the most momentum yesterday. 

• A national knowledge sharing tool This project would seek to create a comprehensive list of what has been done digitally around Climate Adaptation, showcasing lessons learned, successes and failures. The instigators could also develop playbooks, open source tools and a knowledge sharing forum for councils and citizens. Notes here.

• Community resilience to extreme weather events A plan to bring people together to embed community resilience, sharing information about flood risk, how to make your home more able to cope with the effect of climate change and extreme weather events. There was also a suggestion of broadening the existing community warden role to encompass community resilience issues. Notes here.

• Adaptation gardens Showing people how they could garden in a different way to adapt to a changing climate: eg with drought resistant plants, water conservation methods, pollinator friendly plants and other eco-friendly methods. Notes here.

• Digital toolkit for events Putting together a digital toolkit that people can use for climate-related community events, ensuring it’s accessible and reusable in lots of different situations. Notes here.

Seen a project that you’d like to try too?

Maybe you’re a council officer who thinks one of the ideas above would fit well within your constituency.

Or maybe you’re a community group that could help shape the project and replicate it in your area.

There may be an opportunity to join up with other folks working on the idea, and perhaps expanding their plans into more than one region. 

Feel free to fill in our form and indicate that you are open to working with others on one of the existing ideas. 

What you should know about the grants

• You do not have to have attended either of the prior sessions to bid, but please do give consideration to what we are looking for: small, locally-based trials of projects that work with a local council at the intersection of democracy (broadly defined) and climate. A local authority must be involved in the project.
• Need to find a partner council? Let us know and we’ll shout out on Twitter for you.
• This is seed funding, designed to allow for testing, planning and trying new approaches; things that aren’t possible with restricted grants. So don’t worry about having a detailed plan — your application can be short and simple.
• Applications close at 23:59 on Monday 31st October 2022. We aim to have made our decisions and awarded the grants by Monday 7th November 2022.
• Funding will cover the period until March 31 2023  — though your project may continue onwards for as long as you like. We’ll hold a wrap-up event in spring showcasing the work to date.

Apply now

Ready to bid? Apply here.

As we move into the season of the falling leaves, we look back on the activities that fell in September.

Most importantly we welcomed Alexander to the team, doubling both our developer count and the number of people on the team named Alex.

Events dear boy, events

We ran an event! About Climate Tech! It seemed to go quite well. There’s lots of detail in the blog post and links so you can rewatch people from Wiltshire to Copenhagen talking about how they used technology to help with everything from green roofs to community consultation.

The post also contains details of our follow up event about the small grants (£5,000) we have available for local councils and partners for trialling ideas for tackling climate change.

Internally we spent a bit of time thinking about how we might use some futures scenarios to test out our plans and explore any unspoken assumptions we might have about the way the world works. Failing that we could always use said scenarios to help run a creative writing workshop on dystopian fiction.

The work goes on

We have come to the end of our prototyping weeks and we’re now starting to look at  exploring some of them in more detail. The focus at the moment is on home energy, procurement and our most recent prototyping work with The Climate Coalition.

On the home energy front, Siôn has been continuing to speak to potential partners in the area while we work out the best way to turn this work into something concrete. If encouraging local communities to come together and improve the energy efficiency of their homes sounds interesting to you then get in touch.

Wasting no time, Alexander has been unknotting procurement and contracts data in order to turn our Contract Countdown prototype into something a little more functional. We’re still at an early stage with this, trying to work out if it’s practical to keep the data current. We’ll also be looking to show the more useful version to some potential users to see if it’s a service that has value.

Finally, we started work with The Climate Coalition on a beta of a tool to help them corral a range of data to more effectively help climate groups with campaigning. So far we’ve largely been talking about what data is both useful and available, and how to link it all up.

In non-prototyping work we’ve continued to chat to Climate Emergency UK about next year’s follow up to the Council Climate Plan Scorecards. This is very much in the planning stage at the moment.

Previously in blog posts

One of the side effects of our work on Climate is we’ve gathered a lot of data which we’d like more people to use. Alex wrote both about the data we have and also the process we use to gather and publish it. The first of these is of interest to anyone who would like some nice data, while the second is considerably more technical.

Speaking of people using our data, Myf published the latest in our series of case studies on how people are doing just that. This month it’s the turn of the Brighton Peace and Environment centre who’ve been using CAPE and the Council Climate Plan Scorecards to help with visualising council’s progress towards their Net Zero targets.

As ever, if you’ve used any of our data we’d love to hear from you. It helps us with both prioritising future work as well as when talking to current and potential funders.

While gathering all this data we’ve had some thoughts. Alex has started to work with the Centre for Public Data to turn these thoughts into some recommendations. There’s a survey!

If you’d like this sort of thing in your inbox then you can sign up to our monthly climate newsletter by clicking the subscribe link at the top right of that page.

—

Image: Mott Rodeheaver

We spoke to Rebecca Sawyer of Brighton Peace and Environment Centre, to discover how their ReForest Brighton project interfaces with our own CAPE and Council Climate Scorecards sites.

As it turns out, our projects have a lot in common. Both aim to make it easier for everyone to understand and assess the progress a council is making towards cutting carbon emissions, a field where the picture can be complicated and difficult for the average person to follow. That starts with data.

Rebecca explained, “Identifying the path to carbon neutrality is not straightforward, and the data that would enable organisations to know where they currently are on this path is very weak.”

Visualising progress

To address this, ReForest Brighton is developing an interactive website to show in real time the progress that each local authority has made in relation to its individual carbon neutral targets. 

Naturally, the project began by looking at the organisation’s hometown of Brighton, which has a target of Net Zero carbon emissions by 2030. That’s just the start, though; the model is replicable for any other local authority in the UK, allowing their own carbon neutral targets and the actions determined by their climate action plans to be slotted in.  

“The aim,” says Rebecca, “is to provide real time quality data that will enable decision making around policy and practice.

“So for example, if it’s clear that maintaining the current level of action won’t bring a city to carbon neutrality by their set date, the council can refocus their efforts to reduce emissions and sequestrate more carbon.” 

The ultimate target? “To make local authority councils more ambitious.”

Carbon neutral dates

So where do CAPE and the Scorecards site come in? As Rebecca explained, CAPE was useful mainly for a single datapoint amongst the many that it provides.

“The main way we’ve been using it is to retrieve the carbon neutral dates of all the individual local authorities in the UK. 

“Without this data being easily accessible it would’ve taken us a long time and lot of resources to go through more than 300 local authorities and dig out their target dates.”

And as for the Scorecards site, this has been more of a sanity-check tool: “We used it once we’d completed our calculations, to check our ratings of each local authority against the Scorecards rating. 

“For example, if our calculations rated a local authority with high climate action but the Scorecards had it as low, then we’d analyse and reassess our ratings.” 

As well as the interactive map, their project will produce predictive data to show how much progress the council will have made by their target zero emissions date.

Forecast formula

For those who like the technical details, Rebecca is keen to oblige: “Our categorisation is based on a calculation of emission trends from 2016-2020. The trends allow us to predict where each local authority will be by the carbon neutral target date we downloaded from CAPE, using the ‘forecast’ formula (=FORECAST (x, known_ys, known_xs)). 

“There is actually 15 years’ worth of emissions data available, but we chose this five-year period because climate action has only started becoming a consideration for local authorities in the last few years. 

“Basically, we look at the predicted emissions on the authority’s carbon neutral date and categorise them accordingly — and if a local authority had no carbon neutral target or plans, it is automatically rated zero.”

A knock-on effect

ReForest Brighton wants to make it easier for the public to understand how their local authority is doing in achieving its carbon reduction targets — and they have another aim, too: 

“We would like the public to push local governments to take faster, more effective action, and we’re planning to help them do this by giving them the means to write to their elected representatives, and to share the website with their friends and contacts.  

“But even while hoping that councils will be making as much progress as possible, we’re also pushing for transparency. We’d even encourage an authority to push their carbon neutral target date further back if it gave a more honest picture of where they are at.”

Brighton Peace and Environment Centre are a registered charity and they welcome volunteers: get in touch if you would like to know more.

You can also make a donation to them, using this link.

—

Image: Aaron Burden

Last Wednesday a varied audience convened online for our Innovations In Climate Tech event.

The aim: to showcase some of the remarkable and effective projects being implemented in the UK and further afield, and to spark inspiration so that these, or similar projects, might be replicated in other UK regions.

mySociety has three £5,000 grants to give to innovators and local councils who work together and trial something they’ve seen, or been inspired by, during the event.

Missed the live version? Don’t worry: we have videos and notes, and you don’t have to have attended to be able to bid for a grant.

Rewatch or read up

Here’s where to find the various assets from the day:

• Watch a video of all the morning presentations, followed by the Q&A. This video features:
  • Annie Pickering from Climate Emergency UK, on how they scored councils’ Climate Action Plans;
  • Ariane Crampton from Wiltshire County Council on how they tackled outreach to diverse communities with their climate consultation;
  • Claus Wilhelmsen from Copenhagen City Council on the practical ways in which they are tackling carbon cutting within construction industries;
  • Ornaldo Gjergji from the European Data Journalism Network on how visualising temperature data from individual cities and towns helps people better understand the impacts of climate change;
  • Kasper Spaan from Waternet on creating green roofs across the city to aid urban cooling and biodiversity.
• The afternoon breakout sessions weren’t recorded, but you can read notes of the presentations and subsequent discussions for:
  • The Adaptation session (Padlet here) in which Josh Shimmin from Atamate talked about a data-driven approach to retrofitting housing;
  • The Engagement session (Padlet here) in which Susan Rodaway from Pennard Community Council presented on a community consultation tool that helped them decide what to spend budget on; and Arnau Quinquilla from Climate 2025 talked about mapping climate movements across the world;
  • The Spatial Planning session (Padlet here) in which Lora Botev from CitizenLab explained how their software enables councils to run consultations and grow an active group of residents who have a voice in decisions around climate;
  • The Equity, Diversion and Inclusion session (Padlet here) in which Emma Geen from the Bristol Disability Equality Forum explained how vital it is to include disabled people in a green transition, and the ways in which the group has taken action to make this happen.

What’s next?

On 19 October, we’ll be running an informal session online, explaining what we’re looking for in a grant pitch, and giving you the chance to explore your ideas with potential partners. Then, if you want to go forward and bid for one of three £5,000 grants, we’ll give you everything you need to make your pitch.

You do not have to have attended the first event to join us at this stage. Please explore the resources above, add your thoughts to the Padlets, and sign up for this event via Eventbrite.

What sort of projects will we be funding?

To be eligible to bid for one of the grants, you must either be:

• a council that wants to trial an idea; or
• an organisation that wants to work with a council to trial your project.

Partnerships can be between two or more organisations, but every partnership must include at least one local council (and might only consist of councils). But don’t worry if you haven’t got a partner in mind yet – you may find one at this event.

• You might have seen an idea in the presentations that is directly applicable to your council area, and want to simply replicate it.
• Or, in a less straightforward but equally valid scenario, you might simply have seen an organisation you’d like to work with, or had a completely new idea sparked by something you saw.
• You might have no ideas at all, but a commitment to try something new… bring an open mind and see if anything at the event grabs you!

Sign up for the upcoming event here.

 

Data users in the UK often encounter fragmented public data, where  public authorities are each spending money to publish data independently, but their outputs are difficult to find and join together. This means that a lot of effort is going into creating data which cannot be used to its full potential.

As a joint project between mySociety and the Centre for Public Data, we are writing what we hope is a simple approach to help address this problem. 

Our current thinking is that some level of central coordination is required to accompany a central mandate to publish. Public bodies need support and coordination to publish data to a lightweight common standard and in a common location.

We want to kick the wheels of this conclusion, identify existing attempts to fix the problem, and talk to people who produce and use the data to see what the obstacles are. If you’re interested, get in touch at research-publicdata@mysociety.org, or drop your views in this survey. 

What is fragmented public data?

Fragmented public data is when many public authorities are required to publish the same data, but not to a common standard (structure) or in a single location – so data becomes fragmented across multiple locations and multiple formats. 

For example, every English local authority is currently supposed to publish all spending over £500 each month. From Adur to Wyre Forest, council officers are working hard to publish monthly spending. 

In theory, this data should be being used by companies, researchers and journalists to provide insight into spending, spot fraud, and find opportunities to sell to councils.

But in practice, to use the data, you’ll need to search all 333 council websites each month, then import each spreadsheet into a central database – and you’ll spend a lot of time pulling your hair out, because the spreadsheets don’t use a consistent format. 

As a result, not much has actually been done with all this data. And the grand promises that spending data would unleash an ‘army of armchair auditors’ have largely failed to materialise. 

Why does this matter?

This is a problem because most of the effort is already being spent to do the job ineffectually. Councils do a lot of work to produce this data, and companies and analysts waste time fixing import scripts or crowdsourcing data, rather than creating new products or insights — and for many organisations, the skills and resources required to create national level datasets are beyond them.

This is not an isolated problem – there are many other examples of fragmented data. 

From assets of community value to election information to council land and property assets, data is often published in a fragmented and hard to reassemble way. 

For many datasets, while individual disclosures are useful, the combined data is much more than the sum of its parts because it allows real understanding of the picture across the whole country, and makes it easier to draw comparisons between different areas.

Across all these datasets the potential loss is huge – and just a bit of extra work could unlock huge amounts of the overall value of the data. We want to fix this for data that is already being published, and make sure that datasets in future are published in the best possible way. 

So what can we do?

The big problem is one of coordination. We think the UK’s central data teams just need to help public authorities do two things:

1. Use a lightweight common standard
2. Report the location of the data in a central register.

By taking things that individual authorities are doing anyway, but getting an agreed format and location, all the individual datasets become far more useful. But the details of the standard are less important than the fact that the government and legislators should be as interested in this side of the problem as they are in requiring the data to be published in the first place. 

We’re keen to learn lessons from previous attempts to do this, and reviewing old publication guides, our initial conclusion is that these over-complicated the idea of what open data is (with a fixation on file formats and linked standards), rather than simple interventions that help both publishers and users (where generally, the best approach is probably an Excel template with common headers). Common standards need to be a compromise between technical requirements and the people who work in local authorities who produce this data.

We’re encouraged by an approach taken by the Scottish Parliament, where publication of compliance with climate change duties was mandated in a particular format – and a consultation on this process found most organisations involved thought standard reporting was an improvement. We’re interested in any other examples of this kind of approach. 

What are we doing now?

We’re trying to explore this problem a bit more, to understand the scale of the problem, and the viability of our approach.

We are interested in talking to:

• People who have run into this issue, what they were trying to do, if they were able to overcome it, or if they had to give up.
• People who publish information in public bodies to understand what their restrictions are
• People or organisations with experience in trying to coordinate a single standard or location for public data releases.

Get in touch at research-publicdata@mysociety.org, or drop your views in this survey. 

—

Header image: Photo by Olav Ahrens Røtne on Unsplash

This is a more technical blog post in companion to our recent blog about local climate data. Read on if you’re interested in the tools and approaches we’re using in the Climate team to analyse and publish data. 

How we’re handling common data analysis and data publishing tasks.

Generally we do all our data analysis in Python and Jupyter notebooks. While we have some analysis using R, we have more Python developers and projects, so this makes it easier for analysis code to be shared and understood between analysis and production projects. 

Following the same basic ideas as (and stealing some folder structure from) the cookiecutter data science approach that each small project should live in a separate repository, we have a standard repository template for working with data processing and analysis. 

The template defines a folder structure, and standard config files for development in Docker and VS Code. A shared data_common library builds a base Docker image (for faster access to new repos), and common tools and utilities that are shared between projects for dataset management. This includes helpers for managing dataset releases, and for working with our charting theme. The use of Docker means that the development environment and the GitHub Actions environment can be kept in sync – and so processes can easily be shifted to a scheduled task as a GitHub Action. 

The advantage of this common library approach is that it is easy to update the set of common tools from each new project, but because each project is pegged to a commit of the common library, new projects get the benefit of advances, while old projects do not need to be updated all the time to keep working. 

This process can run end-to-end in GitHub – where the repository is created in GitHub, Codespaces can be used for development, automated testing and building happens with GitHub Actions and the data is published through GitHub Pages. The use of GitHub Actions especially means testing and validation of the data can live on Github’s infrastructure, rather than requiring additional work for each small project on our servers.

Dataset management

One of the goals of this data management process is to make it easy to take a dataset we’ve built for our purposes, and make it easily accessible for re-use by others. 

The data_common library contains a dataset command line tool – which automates the creation of various config files, publishing, and validation of our data. 

Rather than reinventing the wheel, we use the frictionless data standard as a way of describing the data. A repo will hold one or more data packages, which are a collection of data resources (generally a CSV table). The dataset tool detects changes to the data resources, and updates the config files. Changes between config files can then be used for automated version changes. 

Data integrity

Leaning on the frictionless standard for basic validation that the structure is right, we use pytest to run additional tests on the data itself. This means we define a set of rules that the dataset should pass (eg ‘all cells in this column contain a value’), and if it doesn’t, the dataset will not validate and will fail to build. 

This is especially important because we have datasets that are fed by automated processes, read external Google Sheets, or accept input from other organisations. The local authority codes dataset has a number of tests to check authorities haven’t been unexpectedly deleted, that the start date and end dates make sense, and that only certain kinds of authorities can be designated as the county council or combined authority overlapping with a different authority. This means that when someone submits a change to the source dataset, we can have a certain amount of faith that the dataset is being improved because the automated testing is checking that nothing is obviously broken. 

The automated versioning approach means the defined structure of a resource is also a form of automated testing. Generally following the semver rules for frictionless data (exception that adding a new column after the last column is not a major change), the dataset tool will try and determine if a change from the previous version is a MAJOR (backward compatibility breaking), MINOR (new resource, row or column), or PATCH (correcting errors) change. Generally, we want to avoid major changes, and the automated action will throw an error if this happens. If a major change is required, this can be done manually. The fact that external users of the file can peg their usage to a particular major version means that changes can be made knowing nothing is immediately going to break (even if data may become more stale in the long run).

Data publishing and accessibility

The frictionless standard allows an optional description for each data column. We make this required, so that each column needs to have been given a human readable description for the dataset to validate successfully. Internally, this is useful as enforcing documentation (and making sure you really understand what units a column is in), and means that it is much easier for external users to understand what is going on. 

Previously, we were uploading the CSVs to GitHub repositories and leaving it as that – but GitHub isn’t friendly to non-developers, and clicking a CSV file opens it up in the browser rather than downloading it. 

To help make data more accessible, we now publish a small GitHub Pages site for each repo, which allows small static sites to be built from the contents of a repository (the EveryPolitician project also used this approach). This means we can have fuller documentation of the data, better analytics on access, sign-posting to surveys, and better sign-posted links to downloading multiple versions of the data. 

The automated deployment means we can also very easily create Excel files that packages together all resources in a package into the same file, and include the meta-data information about the dataset, as well as information about how they can tell us about how they’re using it. 

Publishing in an Excel format acknowledges a practical reality that lots of people work in Excel. CSVs don’t always load nicely in Excel, and since Excel files can contain multiple sheets, we can add a cover page that makes it easier to use and understand our data by packaging all the explanations inside the file. We still produce both CSVs and XLSX files – and can now do so with very little work.

For developers who are interested in making automated use of the data, we also provide a small package that can be used in Python or as a CLI tool to fetch the data, and instructions on the download page on how to use it. 

At mySociety Towers, we’re fans of Datasette, a tool for exploring datasets. Simon Willison recently released Datasette Lite, a version that runs entirely in the browser. That means that just by publishing our data as a SQLite file, we can add a link so that people can explore a dataset without leaving the browser. You can even create shareable links for queries: for example, all current local authorities in Scotland, or local authorities in the most deprived quintile. This lets us do some very rapid prototyping of what a data service might look like, just by packaging up some of the data using our new approach.

Data analysis

Something in use in a few of our repos is the ability to automatically deploy analysis of the dataset when it is updated. 

Analysis of the dataset can be designed in a Jupyter notebook (including tables and charts) – and this can be re-run and published on the same GitHub Pages deploy as the data itself. For instance, the UK Composite Rural Urban Classification produces this analysis. For the moment, this is just replacing previous automatic README creation – but in principle makes it easy for us to create simple, self-updating public charts and analysis of whatever we like. 

Bringing it all back together and keeping people to up to date with changes

The one downside of all these datasets living in different repositories is making them easy to discover. To help out with this, we add all data packages to our data.mysociety.org catalogue (itself a Jekyll site that updates via GitHub Actions) and have started a lightweight data announcement email list. If you have got this far, and want to see more of our data in future – sign up!

—

Image: Sigmund

One of the things we want to do as part of our Climate programme is help build an ecosystem of data around local authorities and climate data. 

We have a goal of reducing the carbon emissions that are within the control of local authorities, and we want to help people build tools and services that further that ambition. 

We want to do more to actively encourage people to use our data, and to understand if there are any data gaps we can help fill to make everyone’s work easier. 

So, have we already built something you think might be useful? We can help you use it. 

Also, if there’s a dataset that would help you, but you don’t have the data skills required to take it further, we might be able to help build it! Does MapIt almost meet your needs but not quite? Let’s talk about it!

You can email us, or we are experimenting with running some drop-in hours where you can talk through a data problem with one of the team. 

You can also sign up to our Climate newsletter to find up more about any future work we do to help grow this ecosystem. 

Making our existing data more accessible

Through our previous expertise in local authority data, and in building the Climate Action Plan Explorer, we have gathered a lot of data that can overcome common challenges in new projects.

These include:

• A swiss-army knife/skeleton key/useful spreadsheet that lists all current local authorities, and helps transform data between different lookups. 
• Mapit An API that can take postcodes and tell you which local authority they’re in (and much more!) Free for low traffic charitable projects.
• Datasets of which authorities have published climate action plans.
• Datasets of which authorities have published net zero dates, and their scopes. 
• A massive 1GB zip of all the climate plans we know about.
• Measure of local deprivation across the whole UK.
• A simplified version of the BEIS local authority emissions data. 
• Measures of similarity between all local authorities (emissions, deprivation, distance, rural/urban and then all of those things together). 

All of this data (plus more) can be found on our data portal. 

We’ve also been working to make our data more accessible and explorable (example):

• Datasets now have good descriptions of what is in each column.
• Datasets can be downloaded as Excel files
• Datasets can be previewed online using Datasette lite.
• Providing basic instructions on how to automatically download updated versions of the data.

If you think you can build something new out of this data, we can help you out! 

Building more data

There’s a lot of datasets we think we can make more of — for example, as part of our prototyping research we did some basic analysis of how we might use Energy Performance Certificate data (for home energy in general, and specific renting analysis). 

But before we just started making data, we want to make sure we’re making data that is useful to people and that can help people tell stories, and build websites and tools. If there’s a dataset you need, where you think the raw elements already exist, get in touch. We might be able to help you out. 

If you are using our data, please tell us you’re using our data

We really believe in the benefit of making our work open so that others can find and build on it. The big drawback is that the easier we make our data to access, the less we know about who is using it.

This is a problem, because ultimately our climate work is funded by organisations who would like to know what is happening because of our work. The more we know about what is useful about the data, and what you’re using it for, the better we can make the case to continue producing it. 

Each download page has a survey that you can fill out to tell us about how you use the data. We’re also always happy to receive emails!

Stay updated about everything

Our work growing the ecosystem also includes events and campaigning activity. If you want to stay up to date with everything we do around climate, you can sign up to our newsletter.

—

Image: Emma Gossett