mySociety’s Transparency team has developed a new tool, the Excel Analyser, which helps reduce the potential harms associated with accidental releases of large amounts of personal information.

The Excel Analyser scans spreadsheets before they are published on WhatDoTheyKnow, identifying metadata types that are often the cause of large data breaches, such as pivot cache data, hidden sheets, columns, rows, named ranges, and cached data from external links or data models.

If problematic metadata types and combinations are detected, the file is automatically prevented from being published on WhatDoTheyKnow.

This helps to reduce the risk that sensitive information is accidentally published online, and limits the harm that such releases can cause. The WhatDoTheyKnow team is alerted when a file has been blocked, which allows them to quickly delete any problematic material and inform the relevant authority that there has been a breach.

In cases where it’s unclear if a data breach has occurred, the authority is alerted that hidden data has been detected in their response, and given the opportunity to send a replacement file if necessary.

As well as Excel Analyser, the potentially problematic files are run through additional scripts that use Microsoft’s Presidio Analyzer tool to detect the presence of personally identifiable information within the hidden data itself. This enables the team to assess and address potential data breaches without needing to download or directly access the files themselves.

By communicating with authorities in this way, the ultimate hope is to reduce the number of data breaches involving Excel. In almost all cases, the relevant data could have been detected by authorities, and removed prior to release, using Excel’s built-in Document Inspector tool.

—
Image: Simon Lee

To react appropriately to the emergence of AI, we need to understand it. We’re making our internal AI Framework public as a way of being transparent about the kind of questions we’re asking ourselves about using AI in mySociety’s tools and services. 

At our recent TICTeC conference, there were several great examples of how generative AI approaches can be applied to civic tech problems.  But regardless of whether civic tech projects use AI approaches directly, it’s increasingly part of the tools we use,  and the context our services exist in is being changed by it. 

A key way mySociety works is by applying relatively mature technology (like sending emails) in interesting ways to societal problems (reporting problems to the right level of government; transforming Parliamentary publishing; building a massive archive of Freedom of Information requests, etc). This informs how we adapt and advance our technical approach – we want to have clear eyes on the problems we want to solve rather than the tools we want to use.

In this respect, generative AI is something new, but also something familiar. It’s a tool: it’s good at some things, not good at other things — and, as with other transformative tech we’ve lived through, we need to understand it and develop new skills to understand how to correctly apply it to the problems we’re trying to solve. 

We currently have some funding from the Patrick J. McGovern Foundation where we’re exploring how new and old approaches can be applied to specific problems in our long running services. Across our different streams of work, we’ve been doing experiments and making practical use of generative AI tools, working with others to understand the potential, and thinking about the implications of integrating a new kind of technology into our work. 

Our basic answer to “when should we use AI?” is straightforward. We should use AI solutions when they are the best way of solving problems, are compatible with our wider ethical principles and reputation, and can be sustainably integrated into our work.

Breaking this down further led us to questions in six different domains:

• Practical – does it solve a real problem for us or our users?
• Societal – does it plausibly result in the kind of social change we want, and have we mitigated change we don’t want?
• Legal/ethical – does our use of the tools match up to our wider standards and obligations?
• Reputational – does using this harm how others view us or our services?
• Infrastructural – have we properly considered the costs and benefits over time?
• Environmental – have we specifically accounted for environmental costs?

You can read the full document to see how we break this down further; but this is consciously a discussion starter rather than a checklist. 

Publishing this framework is similarly meant to be a start to a discussion — and an anchor around open discussion of what we’ve been learning from our internal experiments. 

We want to write a bit more in the open about the experiments we’ve been doing, where we see potential, where we see concerns. But this is all just part of the question at the root of our work: how can we use technology as a lever to help people to take part in and change society.

—

Image: Eric Krull

CAPE is our data-packed website that makes it easier for everyone to track and improve local action on climate change, built in collaboration with Climate Emergency UK.

When CAPE launched, it was basically a list of councils’ climate action plans — but we’ve continued to make regular improvements and additions, and it’s better, and more useful, than ever.

If you’re a council officer with climate duties; a researcher looking for climate data; an activist needing facts to back up your climate campaign; or simply an interested citizen, you’ll find some truly invaluable facts and figures on CAPE.

And if you haven’t visited for a while, you’re in for a surprise. Here’s what we’ve added this year:

Improved search

Now you don’t need to think of all the different ways that councils might have described the concept you’re looking for. We’ve integrated machine learning into the search function, to help surface your search term even in documents that use different words for the same concept. Interested to know more? We wrote it up in detail here. 

Updated for new councils

In April, a set of UK local authorities was abolished and merged into new unitary authorities.  When this happens, we do work to update the data that CAPE depends on. You can make use of it too!

We’ve updated:

• Our dataset of local authorities names and codes
• Our datasets of ‘nearest neighbour’ councils. 
• Our composite UK-wide multiple deprivation dataset for councils. 
• Our composite Rural Urban Classification dataset for councils. 

And this has all fed back into CAPE to make changes of local government structure transparent in the site. 

Local polling

We wanted to make it easier to see where there is support for climate action, so we took a few sets of MRP polling on people’s attitudes to net zero and energy sources and converted it from per-constituency to per-local council.

For each local authority (except in Northern Ireland) – see Croydon, for example – you can now see estimates for support for net zero (even when it is described as expensive), and for different kinds of renewable energy project.

Not sure what MRP polling is? We wrote something up about that. 

Climate Assemblies

In 2019, we were part of the UK-wide Climate Assembly, and we’ve kept an interest in citizens assemblies about climate change. There’s been one held for the whole of Scotland, and more widely we’ve found 17 (so far) held by local authorities across the UK.

We’ve uploaded the final reports of those assemblies into CAPE. It is now easy to see which authorities have held climate assemblies, and to search the results of those assemblies. Find all this at cape.mysociety.org/assemblies.

Hundreds of new documents have a fresh coat of paint

Thanks to people telling us about new plans, and us conducting an extensive search for new documents across council domains, we’ve updated and added hundreds of new documents to councils’ pages. 

With all that content, we’ve gone back and tidied up the design of our council page layout, giving us more space to add explainers and summaries to each section. Why not take a look for your area?

Council Climate Action Scorecards

Let’s not forget that while all this activity has taken place on CAPE, we were also helping Climate Emergency UK with the Council Climate Action Scorecards. At the time of their launch in October, we wrote about the part we played in building the site, wrangling the data, advising on the methodology and building a scoring interface.

The result of all this is, of course, that whatever your climate interests – be you council staff, researcher, journalist, campaigner or interested member of the public – you can dive in to any council’s page to see how they’re doing against several markers of climate action.

Since launch there has also been the significant addition of Question Pages, allowing a never before seen view of how each council scores on every individual action point. Julia wrote about that in detail, in this post.

Once again it’s time for our monthly roundup of what the Climate team has been doing in the last, er, two months. Plenty to write about at least.

First on the list is another milestone in the journey towards Climate Emergency UK’s Council Climate Action Scorecards – the start of the Right of Reply process. All the marking of councils’ climate actions has been completed by CEUK’s small army of volunteers, and now it’s over to councils to have a look at the results and provide any feedback. We’ve also pulled in the data from Freedom of Information requests which was gathered using our WhatDoTheyKnow Pro platform so they can check that over too.

A second launch is the Local Intelligence Hub project we’ve been working on with The Climate Coalition, to help climate campaigners across the UK wrangle climate related data. There was a bunch of work in the run up to this to improve how we were displaying information on the map to make it more accessible, plus adding yet more data. Now that TCC members have access to this we’ll be gathering feedback to decide on future work, as well as adding more data, before a full public launch.

Meanwhile, our Neighbourhood Warmth project with partners Dark Matter Labs has been moving gently but steadily forward. We’ve been meeting with organisations in our three chosen pilot areas, and fleshing out some basic content and design before we put together a very minimal working alpha, to test out with real neighbours on real streets. We’ve been thinking critically about some of our initial ideas on how to connect people interested in making energy saving improvements to their home, and have broadened out our definition of “neighbourhood” from people on the same street to people nearby – to capitalise on the connections people might have across a slightly wider local area. Alongside this we’ve been working out how we’re going to get this in front of users to gather feedback once we have something to show. You can read more about this in our first set of Neighbourhood Warmth monthnotes.

We’ve also had an update on what our second Innovations in Climate Tech grantee has been up to.

In the background we’ve been moving forward with plans for our Festival of Debate session (book here) and doing some thinking about what our Climate Programme will look like in 2024 and beyond.

Finally, with the spring new councils have bloomed which means updating CAPE to include these new councils, and to guide people looking at the old councils to their replacements.

—

Image: Olli Kilpi

It finally feels like Spring is in the air, and you know it’s been a busy start to the year when we’re rolling the first two months into one set of monthnotes – in the middle of March!

So, what have we been up to?

Well – we’ve been adding datasets to and testing our alpha version of the Local Intelligence Hub tool that we built with The Climate Coalition. Feedback has been really good and this feels like something that’s really going to level up the ability of UK climate organisations to share data and coordinate their actions, at both a local and national level. We hope to share more about this project in the coming months, once it’s been made available to TCC members.

We submitted talks to a couple of conferences/events – and lo and behold, we’ll be in Sheffield (and online) for the Festival of Debate on May 24, with a panel of exceptional guests. Our topic? “What if you could reshape democracy for the better — and you had 20 years to do so?” Climate is sure to be part of the answer. Fancy joining us? Book here.

Between all this we’ve been working hard with our friends at Climate Emergency UK on the next round of the Council Climate Scorecards. Their draft methodology was released in November 2022, and the first round of marking started in January 2023. Part of our support has included building a Django application to store the marking data – and this has already dramatically improved the experience for Climate Emergency UK’s volunteers. 

Climate Emergency UK are also working with mySociety’s Transparency team, using WhatDoTheyKnow Projects (a WhatDoTheyKnow Pro beta feature that helps researchers crowdsource data out of batch FOI requests) to gather some of the data for the scoring. All their FOI requests will be published on WhatDoTheyKnow later this year.

Our IICT grants are coming to an end soon – we’ve put out a blog post about Lynsted Community Kitchen Garden and the data they’re collecting with the weather station we funded. They have a public event on March 25 if anyone lives near Lynsted and wants to visit to check it out! Updates from Possible and Better Futures should be coming soon.

On the research side, we launched our report on unlocking the value of fragmented public data, which is part of our work into the data ecosystem around climate data. Our plan over the next few months is to support a few research commissions which link in to this report and help to show use of climate data. 

We’ve confirmed a partnership with Dark Matter Labs – we’ll be moving forward with them and our Neighbourhood Warmth prototype, exploring how we could encourage neighbours to come together to take their first retrofit action, such as getting a house survey. We’ll be building a working prototype over the next few weeks, then testing it out with communities in three pilot areas around the UK, to ensure that what we’re building makes sense to the people we’re aiming to serve.

And finally, we met up in person! We had a team meeting in early February which was a wonderful chance for us all to take stock of the last year, and discuss the future. We’ve been making some plans for year 3 of the Climate programme and after widening our scope through prototyping, now we’re going to be focusing back in again on building and proving the impact of the services we’re running.

That’s a very whistlestop tour of our first months of 2023!

—

Image: Daniel James

At our Freedom of Information service WhatDoTheyKnow, when faced with requests to remove material, we operate on the principle of removing the minimum amount possible.

Alaveteli, the codebase which underlies WhatDoTheyKnow and a number of other FOI sites around the world, gives moderators a range of options for removing content – with the ability to surgically remove text ranging from individual words and phrases, to individual messages, or even entire request threads. This is useful when we spot misuse of our service, for example.

What we’ve been lacking, up until now, was a way to apply these types of removals to attachments.

Back in the early days of WhatDoTheyKnow, attachments were less common, but now we see many more: there can often be several attachments to one individual message.

Over the last few years, there have been occasions where we’ve had to remove an entire message, which may contain several useful attachments, just because of a small issue with one of them.

We’d then go through an annoying manual process to download the publishable ones, upload them to our file server, and then annotate the request with the links – here’s an example.

Back in 2013, when the original suggestion for enabling finer grained control was raised, the site contained around 400,000 attachments. There are now more than 3,500,000! We don’t remove content often, but at this scale it’s inevitable that we need to intervene now and then.

After a little code cleanup we were able to make individual attachment removal a reality. This allows us much more control over how we balance preserving a historic archive of information released under Access to Information laws, and running the site responsibly and meeting our legal obligations under GDPR.

As an example, let’s imagine that the FOI officer replying to our request inadvertently makes a data breach when releasing some organisation charts in `organisation chart b.pdf`.

Previously we’d have had to have hidden the whole response. Now, we can go into the admin interface and inspect each individual attachment.

We can then set our usual “prominence” value – offering a few options from fully visible to completely hidden – and include a reason for why the content has been hidden. We always seek to run the site transparently and explain any actions taken.

On saving the form, you can see that only the problematic attachment has been removed, with the remainder of the response intact. This saves us considerable time when reviewing and handling material with potential data issues, and keeps as much information published as possible while we do so.

As an extra bonus, since the main body text of emails is also treated as an “attachment” in Alaveteli, we’re now able to hide potentially problematic material there without affecting the attachments we present.

We’ve already used this feature several times to republish material where we’d previously had to hide the entire message due to the technical limitations at the time.

—

Image: Kenny Eliason

Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

…or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

—

Image: Keila Hötzel

As we move into the season of the falling leaves, we look back on the activities that fell in September.

Most importantly we welcomed Alexander to the team, doubling both our developer count and the number of people on the team named Alex.

Events dear boy, events

We ran an event! About Climate Tech! It seemed to go quite well. There’s lots of detail in the blog post and links so you can rewatch people from Wiltshire to Copenhagen talking about how they used technology to help with everything from green roofs to community consultation.

The post also contains details of our follow up event about the small grants (£5,000) we have available for local councils and partners for trialling ideas for tackling climate change.

Internally we spent a bit of time thinking about how we might use some futures scenarios to test out our plans and explore any unspoken assumptions we might have about the way the world works. Failing that we could always use said scenarios to help run a creative writing workshop on dystopian fiction.

The work goes on

We have come to the end of our prototyping weeks and we’re now starting to look at  exploring some of them in more detail. The focus at the moment is on home energy, procurement and our most recent prototyping work with The Climate Coalition.

On the home energy front, Siôn has been continuing to speak to potential partners in the area while we work out the best way to turn this work into something concrete. If encouraging local communities to come together and improve the energy efficiency of their homes sounds interesting to you then get in touch.

Wasting no time, Alexander has been unknotting procurement and contracts data in order to turn our Contract Countdown prototype into something a little more functional. We’re still at an early stage with this, trying to work out if it’s practical to keep the data current. We’ll also be looking to show the more useful version to some potential users to see if it’s a service that has value.

Finally, we started work with The Climate Coalition on a beta of a tool to help them corral a range of data to more effectively help climate groups with campaigning. So far we’ve largely been talking about what data is both useful and available, and how to link it all up.

In non-prototyping work we’ve continued to chat to Climate Emergency UK about next year’s follow up to the Council Climate Plan Scorecards. This is very much in the planning stage at the moment.

Previously in blog posts

One of the side effects of our work on Climate is we’ve gathered a lot of data which we’d like more people to use. Alex wrote both about the data we have and also the process we use to gather and publish it. The first of these is of interest to anyone who would like some nice data, while the second is considerably more technical.

Speaking of people using our data, Myf published the latest in our series of case studies on how people are doing just that. This month it’s the turn of the Brighton Peace and Environment centre who’ve been using CAPE and the Council Climate Plan Scorecards to help with visualising council’s progress towards their Net Zero targets.

As ever, if you’ve used any of our data we’d love to hear from you. It helps us with both prioritising future work as well as when talking to current and potential funders.

While gathering all this data we’ve had some thoughts. Alex has started to work with the Centre for Public Data to turn these thoughts into some recommendations. There’s a survey!

If you’d like this sort of thing in your inbox then you can sign up to our monthly climate newsletter by clicking the subscribe link at the top right of that page.

—

Image: Mott Rodeheaver

We’re now six days into September. We’re feeling a strong ‘back to school’ vibe after the long summer in which many of the team were away on annual leave at one point or another. 

Prototyping is behind us and we’re shifting our focus to what we want to achieve in the last six months of the (project) year. Let’s take a quick look back at August and see what we achieved.

Innovations in Climate Tech event

We’ve been working flat out on finalising our agenda and speakers for our showcase event on 21st September. The lineup of speakers is shaping up to be really exciting, and they’ll be introducing key themes such as equity, diversity & inclusion, spatial planning, adaptation and engagement – all of which should spark some interesting new inspiration for councils.

Don’t forget, if you’re a local council working on climate change, and you spot a project in the event that you might like to trial, there’s the chance to bid for one of three £5,000 grants to help make it happen.

We’ve also been inviting people to the event and spreading the word. We’re running it as an affiliate event in Code for All’s amazing week-long Summit, packed full of other compelling climate presentations. So don’t just stop at our event, take a look at the full schedule and sign up for any that interest you – they’re all free, and it’s a great opportunity to learn more about what the global civic tech community is up to, in topics from climate and democracy to countering fake news and mass surveillance!

Product development

August was a light week for development, as Struan, our developer, enjoyed a well-earned break in Italy. (Travelling there by train, of course, and earning time back as part of our Climate Perks policy!)

He’s back now, and just in time to welcome our newest developer, Alexander, to the team. Together they’ve picked up work on the next stages of our prototype around the transparency of council procurement, Contract Countdown. Once the more developed prototype is live with real data in it, we’ll put a call out for journalists to help us test our assumptions, as part of a first focus group.

And finally we ran a stakeholder meeting around the Neighbourhood Warmth prototype – this was interesting and helped us start to shape our thoughts on where we could go next. We’ll be solidifying those over the next few months and hopefully forming a couple of strong partnerships which will help drive our thinking forward. Development on the next stage of this will start in January 2023.

Final prototyping week

We ran our final prototyping week in partnership with The Climate Coalition in early August, looking at how better data about local climate action, citizens, and MPs could help organisations campaign for better climate outcomes. We had a massive amount of input from a wide range of national organisations and local community groups – thank you to everyone who took part! The outcomes of the week were really exciting and we’ll be publishing a short write-up in September.

Communications

Now that our services have really bedded in and people are discovering how useful they can be, we’re able to produce a steady stream of case studies. The hope is that these inspire other folk to use CAPE and the Scorecards site for their own organisations. 

This month we’ve spoken to a professor using both services in her research on healthcare and climate; and a charity dedicated to making school dinners more sustainable. There are still more varied and interesting case studies coming soon.

A whole episode of Delib’s Practical Democracy podcast was dedicated to Myf and Siôn as together they explained our work – do have a listen if you like a chatty approach when taking things in! 

And our climate newsletter is now a regular monthly fixture – you can sign up from that link (top left of your browser) if you’d like to receive these in your email inbox.

—

Image: Pascal Debrunner

Having launched two services – the Climate Action Plan Explorer and Council Climate Scorecards – mySociety’s Climate Programme is running a series of six rapid prototyping weeks to explore what we could do next.

The next prototyping week, during the week starting 9 May 2022, will be all about access to nature. How can we use mySociety’s expertise in data and digital tools to help accelerate initiatives that integrate nature with the urban environment, open up rural spaces to a broader demographic, or encourage better stewardship, understanding and nurturing of our flora and fauna?

If you’d like to get involved, please fill this short form to express your interest and we’ll get back to you as soon as possible. We’ll also soon be announcing the topic of prototyping week #4.

Prototyping? What’s that?

If you’re wondering what happens during a prototyping week, look no further than my colleague Zarino’s report on week #1, which focused on enabling local authority emissions reductions through procurement.

Right now we’re midway through prototyping week #2, exploring the potential to catalyse local climate action on energy through conditional commitment. As with #1, we’ve had a busy couple of days with a great bunch of people from organisations outside of mySociety contributing thoughts on problems and potential solutions in this space. Several inspiring ideas emerged and we’ve whittled it down to one solution to prototype. Now our thoughts are turning towards building and testing that prototype with a few people before the end of this week.

So it’s a real rollercoaster, trying to quickly grasp what mySociety could contribute and taking steps towards understanding if it’s useful before going any further. We hope a couple of ideas will be strong enough for us to develop further, preferably in partnership. And by working openly we hope that this series of prototyping weeks provides possibilities for people outside of mySociety to pick up and pursue ideas that we aren’t able to commit to ourselves.

All that said, using this approach in this way – designing for the needs of society in the face of an ongoing emergency – is something of an experiment. So we’re reflecting and adapting as we go. This post is part of those broader efforts to continuously improve. We hope that by working in the open we’ll enable a richer range of feedback on, and involvement in, what we’re doing.

So, please do join us if access to nature is an area in which you have expertise or strong ideas, or pass this on to anyone suitable.

In line with our equity, diversity and inclusion strategy we’d be particularly grateful if you could also share this post in places that will help us in our policy of centring minoritised groups. We’ve been particularly inspired by orgs such as Black2Nature, Black Girls Hike and Nature is a Human Right but we know there must be more out there with relevant experience and expertise on access to nature — please do help us find them.