1. Access to Information network: data visualisation Show and Tell

    Organisations all around the world run Freedom of Information sites based on our Alaveteli platform. There are also sites running on other codebases. Either way, we’re all working to the same goals, and we believe that each site can learn from the others: for that reason, we’ve set up the Access To Information (ATI) network, a place for people who run FOI sites to meet, chat and share experiences.

    Within this framework, there’s the chance for practitioners to pair up and exchange specific skills under a mentorship arrangement — and last week, we heard from GONG in Croatia who have been learning from Georgia’s ForSet.

    ForSet began life as a data visualisation organisation, and these beginnings have allowed them to communicate data from their FOI site as compelling, visual stories: skills that GONG were keen to get to grips with. Sara, project co-ordinator at GONG, told us about working with Jubo from ForSet on data visualisations — and how her learnings will change the organisation’s work going forward.

    Sara explained that they agreed two main goals for this project: firstly, to improve GONG’s data visualisation skills; and secondly, to use data visualisation to promote their FOI site Imamo pravo znati (IPZ) to journalists and general users. They were successful on both counts, not only did Sara learn how to use new methods and tools; but their outputs also brought approximately 50 new users to IPZ, and two additional Pro users (Pro usage is free on the site, but numbers had been stagnant of late, so this was notable).

    So, how did they go about it? The mentorship comprised four stages: data collection, analysis, storytelling and visualisation, with the last being very interconnected.

    1. Data collection

    This stage began with both sides brainstorming potential topics for FOI requests that would be good candidates for data visualisation. An initial set of 12 topics was whittled down to five: local referendums in Croatia; special advisors (Spads) in the Croatian government; local participatory budgeting projects; local support for youth civic education; and local financing of civil society organisations. 

    GONG then sent 575 requests to local and county authorities, from which they received 525 responses — a pretty good response rate, and higher that expected. They didn’t hit many problems, although several authorities asked for the requester’s ID details, and there was one ministry that cited GDPR as a reason for refusing information on Spads. This case has now been referred to Croatia’s Information Commissioner. 

    2. Data analysis

    Jubo and Sara organised the responses they received into spreadsheets: they were looking for an angle or a story among the data, and tidying it up in this way was helpful for making comparisons. Could they find a story in there that aligned with GONG’s mission or advocacy?

    By organising the data in this way, the pair could easily see which data wasn’t really strong enough to take any further, and which lent itself to storytelling and visualisation. At this stage they rejected some of the angles they’d begun with, narrowing their projects down to local referendums, Spads, and lowering the voting age to 16 for the EU elections (this last project is pending; they’ll be launching a campaign in late Spring).

    3. Storytelling and visualisation

    Two pieces of software were used for the visualisations: Canva and Flourish. Sara was already familiar with Canva, as she uses it to create social media graphics; but Flourish was new to her, and she says she is very happy to have these new skills under her belt.

    Flourish allows you to create many different types of visualisations: you upload your data and it is fairly intuitive to create maps, charts, etc. Once these were in hand, they added a call to action for each story, encouraging people to use their FOI site and especially Pro.

    The visualisations

    Local referendums

    For the story on local referendums, GONG had requested from each local authority the number that had taken place; the topics under discussion; their outcomes; and the number of referendums that were suspended due to not being valid for whatever reason.

    They received more responses than expected, and this was also the first time nationwide data had been collected on the subject.

    Map showing where Croatian referendums were successful or otherwise in reaching quorate

    The first angle that GONG wanted to support with their data and visualisations was ‘Croatia needs a referendum law that recognises the importance of local democracy’. 

    The data showed that out of 47 local referendums that had been held, just 25 met the minimum turnout for the decision to be valid. Jubo and Sara mapped these, and paired their visualisations with the argument that a lower bar for turnout would encourage better participation in local democracy – demonstrated with human figures.

    Turnout quorum for a local referendum
    A local press outlet picked the story up, using the data to make their own story: they were the area that had had the highest number of referendums, so that was their focus. 

    Special Advisors

    The FOI requests returned the names of Special Advisors, the areas they were in charge of, and the fees they were receiving. As Sara explained, in Croatia Spads are not straightforwardly employees of the government, but they have a lot of influence, and in some cases receive huge fees.

    It became clear that there are two different types of advisors, under two laws; while each type has different functions, both are called Spads. First, there are external advisors who may or may not receive compensation; and secondly there is another class of Spads who are employed internally. Neither is subject to Croatia’s legislation on conflict of interest.

    Number of SPADS in each Croatian ministry

    A pie chart was put to service to clearly show how much compensation Spads had received. This varied widely from Spad to Spad, but the criteria dictating who received how much is still unclear: it appears to be at the discretion of the individual minister.

    Pie chart showing SPAD payment in Croatia

    In collecting this data, GONG unexpectedly unearthed a scandal, as they revealed one Spad who was abusing public funds. He was fired, along with the minister concerned; this resulted in nationwide coverage for the data; albeit again with the media’s own preferred focus.

    Lowering the voting age

    Sara says that it was a lot of work to find data to support the argument for lowering the voting age to 16 in Croatia. They wanted to show that, while young people see voting as the most efficient political action, it is denied to a large portion of them.

    Proving the absence of something is always tricky, and in this case they were uncovering that there isn’t any research to show that 16 year olds lack the cognitive abilities to vote responsibly. So they focused on other angles: in some EU countries, 16 year olds can vote, and they demonstrated that those countries are doing well in democratic processes: they score highly in the democracy index and have good voter turnout.

    Data visualisations around the voting age in Croatia

    Like many countries, Croatia’s population is ageing, so the young are in danger of being totally ignored. GONG plan to share their findings on social media in a simplified form with graphics cards, and a call to action to show support for the campaign.

    Questions and answers

    Once Sara had finished her presentation, members of the audience were invited to ask questions.

    Q: How did GONG and ForSet work together?

    A: At the beginning, they had lots of online video calls, and later on when the data had come in, they communicated a lot via comments on the spreadsheets.

    Q: It feels like each of these the topics would be applicable almost everywhere: perhaps it will spark other people’s interest to do something similar for their own country. Any advice if so?

    A: The questions asked in the first two sets of FOI requests were straightforward, which led to straightforward answers. The third topic was less so; Sara and Jubo had to go through lots of reports, and often the data from one source contradicted another. Also, an uncontentious topic is likely to result in more responses: something like referendums is politically neutral, unlike spads where the authorities may have reasons not disclose information.

    Q: When you put the requests together, were you already thinking about the format it would be best to receive the data in?

    A: In that respect, the best question is one with a yes/no answer. The reason for excluding many of the initial topics at a later stage was that the answers varied so widely that it was hard to pull out statistics or a simple throughline: you’d be comparing apples with pears. So, for example, when asking how much of a local authority’s budget is reserved for supporting civic education, and how civic education is delivered, the responses could range from “We are in favour of civic education, but leave it to schools”, to “We provide money for civic education and produce our own textbooks”. Meanwhile, some authorities wrote two pages of waffle in the free text box. 

    Q: Did you narrow down the topics before or after you had submitted the FOI requests?

    A: Both. There were 12 topics at the start; they decided which of them were best suited to FOI, then sent requests for five of them. One the answers had been received, they narrowed it down to three.

    Q: Could one make data visualisation about the other two? It’s hard to find ways to show that there’s no information. Saying that 80% of authorities don’t reply is not a very exciting way of showing things.

    A: While it might not fit in with the initial aim of the project, this sort of thing can be a great way to show how well or badly FOI is working in your country. Journalists often can’t get the information they need, so build stories around the fact that there’s no data about such an important issue.

    Q: We’ve seen how much GONG has benefitted from this mentorship. What, if anything, did ForSet get from this?

    A: Sara was so quick and flexible, she was great to work with. ForSet also learned from the project: for example, that it is better when requesting a large amount of data, that is sorted by the public institution, so it’s easier to work with. You can request it sorted in the way that you need for your story, which might be different from how it is in public.

    Also, Canva is such a great tool for visualisations. They’ve now merged with Flourish, so the have advanced data visualisation features. You just have to make sure you choose the right format: the type of charts or graphs that will show your findings the most clearly. 

    Finally, ForSet didn’t know about the topics that Sara suggested, so there was plenty to learn there, plus it was great to see the ways GONG employ to publish their stories on both social media and mainstream media. 

  2. Creating datasets from FOI data

    Responses obtained from a widespread FOI project can be difficult to analyse, until they are sorted into neat datasets. This allows you to make valid comparisons, pull out accurate statistics and ultimately ensure your findings are meaningful.

    In our third seminar within the Using Freedom of Information for Campaigning and Advocacy series, we heard from two speakers. Maya Esslemont from After Exploitation explained how to prepare for an FOI project to ensure you get the best results possible (and what to do if you don’t); and Kay Achenbach from the Open Data Institute explained the problems with ‘messy’ data, and how to fix them.

    You can watch the video here, or read the detailed report below.

    Preparing for an FOI project

    After Exploitation is a non-profit organisation using varied data sources, including FOI requests, to track the hidden outcomes of modern slavery in the UK.

    Maya explained that they often stitch together data from different sources to uncover new insights on modern slavery. She began with a case study showing some recent work they had done, using WhatDoTheyKnow to help them understand the longer term outcomes after survivors report instances of trafficking. This stood as an excellent example of how much work needs to be done before sending your requests, if you are to be sure to get the results you need.

    In this case, After Exploitation were keen to understand whether there is any truth in widely-held assumptions around why human trafficking cases are dropped before they are resolved: it’s often thought that there are factors such as the survivors themselves not engaging with the police, perhaps because of a nervousness around authorities.

    But what are these assumptions based upon? Actual information was not publicly available, so we wouldn’t know if cases were being dropped because of low police resource, a lack of awareness or more nuanced factors. Until the data could be gathered and analysed, the perceptions would continue, perhaps erroneously.

    Before starting, After Exploitation thought carefully about the audience for their findings and their ultimate aims: in this case the audience would be mostly the media, with the aim of correcting the record if the results flew in face of what was expected; but they knew that the data would also be of use to practitioners. For example, charities could use it to see which areas to target regionally for training and other types of intervention.

    They placed FOI requests with police forces across the country, making sure to ask for data using the crime codes employed by the forces: were cases dropped because of ‘lack of evidence’; did they have a status of ‘reported’ but not gone on to exist as an official crime record?

    The project had a good outcome: while some requests had to go to internal review, ultimately over 80% of the forces responded with quality data. The findings were worthwhile, too: general perceptions did indeed prove to be wrong and there was no indication that ‘no suspect identified’ was a result of the victim’s lack of involvement. The resulting story was able to challenge the general narrative.

    So, how can After Exploitation’s learnings be applied to the work of other organisations or campaigns?

    Maya says:

    • Planning, rather than analysis, is the majority of the work;
    • Identify the need and purpose before you even start to pick which authorities to send requests to;
    • Be clear who the audience for your findings is;
    • Consult with other stakeholders to make sure your parameters are really clear.

    Planning

    Before you even begin, make sure your project isn’t asking for data that has already been collected and is in the public domain — this might seem obvious but it’s easy to overlook. Check other people’s FOI requests (you can do this by searching on WhatDoTheyKnow); look for reports, research, inspectorate/watchdog outputs, and data released as part of parliamentary enquiries.

    That said, even if you do find previous data, there is sometimes value in requesting more up to date or more detailed information with a new set of FOI requests. If you see a national report collating data from every council for example, you could do an FOI project asking every council for a more detailed breakdown of what is happening in their region.

    But before sending a batch of requests to multiple authorities, ask yourself if there is a centralised source for your data. If so, then just one FOI request might be enough: for example, homelessness data is already collected by the Depts for Housing, Levelling Up and Communities, in which case one request to them would save time for both you, and more than 300 public authorities.

    Another question to ask before starting off on your project is “what is the social need?”. Does this need justify the resource you will expend? Mass FOI projects can be a bit of a time commitment, but the utility might not just be for your organisation: perhaps you can also identify a social benefit if the data would be of use to other groups, academics or journalists.

    Define your intended audience: will the data you gather be of interest to them? Do you have a sense of what they want? For example, MPs often like to see localised data that applies to their constituencies. Journalists like big numbers and case studies. If you think your findings are important but might have limited appeal, you could consider including an extra question to provide details that you don’t need for your own purposes, but which could provide a hook.

    Next, will the data that you gather actually be suitable for the analysis you want to perform? To avoid time-consuming mistakes, make sure the data you’ll receive is broken down in the way that you need. As an example, suppose you wanted to ask local authorities for details of programmes offered to children in different age bands: you might receive data from one council who has offerings for children ‘under 18 months’ and another ‘under two years old’ — and where units differ, they are difficult to compare and contrast. Be really precise in your wording so there’s no mismatch, especially if your request is going to a lot of authorities.

    Consider, too, whether you can you get enough responses to make your data meaningful: 2,000 people is the figure believed to be representative of the population as a whole. Decide how many responses you ideally need for your purposes — and, in a scenario where not all authorities respond, the minimum you can work with.

    You might want to contact other groups or organisations who could be interested in the same data, and ask if there are details that would be useful to their work.

    As suggested in Maya’s case study, try to use existing measurements where you can: if you shape your requests to the methodology the authorities themselves use to collect the information, such as KPIs or their own metrics of success, these will be much easier for them to supply.

    If you’re not sure what these metrics are, you can sometimes access internal guidance by googling the name of the authority plus ‘guidance’. Alternatively, submit scoping requests to a handful of authorities to ask how they measure success, etc.

    At this stage it’s also useful to decide what quality of data you will include or exclude. For example, if you ask about training materials and one authority says they offer training, but don’t include the actual materials, do you include it in your figures? The more authorities you ask, the more ambiguities like this you’ll normally encounter.

    Think about where and how you will log the data as it comes in. Maya recommended WhatDoTheyKnow Projects as a good tool for extracting data. Whatever you use, you should consider accessibility: can your platform be accessed by everyone you’re working with, across different communities? Especially if you are working with volunteers, it’s important to remember that not everyone has a laptop.

    Also consider the security of the platform: how much this matters will depend on how sensitive the data is, but recognise that Google sheets and many other platforms store the data in the cloud where it could be more vulnerable to abuse.

    After Exploitation take great pains to ensure that their data is accurate. They recommend that each response is assessed by two different people, making sure that everyone knows the criteria so they’re applied consistently; and doing regular spot checks on a handful of cases to make sure they are all logged in the same way and there’s no duplicate logging.

    This is time-intensive and arduous, but if you have other stakeholders they might be able to help with the data checking: for example, knowing that they would eventually place the story with the BBC, After Exploitation were happy to hand this task over to their inhouse data checkers.

    What if things go wrong?

    If you’ve done all the planning suggested above, it’s less likely that your project will go awry, but even if it does, Maya says that there’s always something you can do.

    No or few responses: ask yourself whether you have the capacity to chase no/late replies, and if you still don’t get a response, to refer them to the ICO. If not, consider prioritising the bodies that are most relevant to your work, eg the biggest authorities or those in areas with the densest populations; but be prepared to defend accusations that not every authority had a fair hearing unless you do them all.

    If you know your requests were well worded, but you’re not getting a lot of responses — perhaps because you’re dealing with a contentious issue, or simply because the authorities cash-strapped — you could shift to measuring the types of responses you get. If authorities aren’t able to answer the question, this can often be just as revealing.

    Responses that don’t tell you what you set out to understand: Consider whether there are any alternative angles in the data you do have: are there any additional themes, particularly in any free text fields? Or try a new round of requests asking for more detailed information.

    Responses don’t cover the whole country: If you can’t get data from everywhere, could you narrow down to just one area and still have useful findings? Even the most basic data can set the scene for other researchers or organisations to build on: you can put it out and outline the limitations.

    Results

    The impact of gathering data through FOI can be massively powerful, as After Exploitation’s work shows. They have revealed the wrongful detention of thousands of potential victims of human trafficking when the government were denying it could happen; opened the debate about locking up vulnerable people; and uncovered the flawed decision making in the Home Office on modern slavery cases. It was only through FOI requests that all this information came into the public domain and was picked up by mainstream media.

    Combining different sources of data to create datasets

    Kay Achenbach is a data trainer on the Open Data Institute’s learning team; the ODI works with government and companies to create a world where data is working for everyone.

    Kay shared a case study from the medical field, in which an algorithm was being designed to quickly assess high numbers of chest x-rays. The aim was to automate the process so that people identified as needing intervention would be sent to specialists right away.

    The developers wanted to make sure that different demographic groups weren’t being biased against, a common issue with algorithms built on existing data which can contain previously undetected biases.

    The test material was a set of x-rays from a diverse population, that had already been examined by specialists. They ran them past the algorithm to see if the diagnoses produced were the same as those made by human doctors.

    The doctors’ assessments came from three different datasets which, combined, comprised data from more than 700,000 real patients. As soon as you combine datasets from different sources, you are likely to come across discrepancies which can make analysis difficult.

    In this case, one dataset had diagnoses of 14 different diseases, and another had 15 — and from these, only eight overlapped. The only aspect that could for sure be compared was the “no finding” label, applied when the patient is healthy. That limitation set what the algorithm was asked to do.

    Other fields were problematic in various ways: only one of the three sources contained data on ethnicity; one source only contained data on the sickest patients; another was from a hospital that only takes patients with diseases that they are studying, meaning there were zero “no finding” labels. Two of the sources contained no socio-economic data. Sex was self-reported in two of the sources, but assigned by clinicians in the other, which could also affect outcomes.

    The advice from all this is that you should look carefully at each dataset before you combine them, to see what the result of combining them would be. In short: does it reflect real life?

    Ultimately the researchers found that the algorithm was reflecting existing biases: it was much more likely to under-diagnose patients from a minority group; more likely to make mistake with female patients, the under 20s, Black people, and those from low socio-economic groups. The bias was compounded for those in more than one of those groups.

    Cleaning up datasets

    Once you’ve obtained your datasets from different FOI requests, you’re highly likely to find mismatches in the data that can make comparisons difficult or even impossible — but cleaning up the data can help.

    For example, in spreadsheets you might discover empty fields, text in a numbers column, rows shifted, dates written in a variety of formats, different wording for the same thing, columns without titles, typos and so on.

    Kay introduced a tool from Google called Refine that will solve many of the issues of messy data, and  pointed out that the ODI has a free tutorial on how to use it, which you can find here.

  3. The Local Intelligence Hub: our latest launch helps you put climate on the agenda

    A general election is not far away, so get ready for heated conversations: on your doorstep, on social media and in the news.

    If you care about climate, we want you to be able to take part in those conversations with the facts at your fingertips. That’s why this week, we’re launching the Local Intelligence Hub, a powerful tool that provides a wealth of relevant national and local data in one place — and encourages you to combine it in a multitude of ways, to uncover useful new insights.

    mySociety worked in collaboration with The Climate Coalition, supported by Green Alliance to develop this site. The aim is to help you — whether you’re a citizen, climate campaigner or part of an organisation — to understand and share the places where there is a strong mandate for environmental action, ensuring commitments to climate and nature are put firmly onto party manifestos. We’ve demoed it in front of organisations who’ve told us it’s a total gamechanger!

    But enough words — let’s get straight to the action. Watch these short videos and you’ll immediately grasp the power of the Local Intelligence Hub.

    For individuals

    “I’m just one person: what difference can I make?” Well, with the Local Intelligence Hub’s data, you can make a lot of difference.

    As a first step, put your postcode into the Local Intelligence Hub and find out all about your local area.

    You might find some interesting data combinations: for example, what does public support for climate action look like in comparison to data on air pollution in your constituency? How about the measures of poverty against support for cheaper renewable energy?

    We hope you’ll use this kind of intel to inform conversations with canvassers or your MP. If you discover something notable, why not write to the newspaper — local or national — or share your findings with your community newsletter, Facebook group etc?

    In the run-up to an election public opinion has a lot of power, and all the more so when you can quote the data to prove it.

    Screenshot of the map page from Local Intelligence Hub

    For campaigning

    If you are part of a climate campaign that works nationally, the Local Intelligence Hub shows you at a glance where in the country to concentrate your activity for the most impact.

    Play about with the map page, selecting different datasets, and you’ll soon understand the insights they unlock. Every constituency with high support for renewable energy for example; or the constituencies where the MPs have the lowest majorities; or where the population is youngest… the possibilities are practically endless.

    If you’re more locally-based, dive into the constituency pages where a massive range of local data allows you to have a full picture of the area:

    • Public opinion: How much support is there for climate initiatives such as net zero or renewable energy?
    • Place: What factors affect people in the area, such as air pollution, flooding and levels of deprivation?
    • Movement: Which climate and environmental groups are active in the area, and what other relevant organisations have a presence?

    For each constituency, these three data collections are supplemented by information on the MP’s memberships, voting and activities. Note that you have the choice to see constituencies as they are now, or as they will be after the election when new boundaries come into play.

    Once you’ve dipped into the data, you should be able to shape your campaigns to more effectively speak to the right people about the issues that matter to them.

    We hope you find the Local Intelligence Hub useful. When you’ve had a chance to try it out, please do let us know how you’re using it!

  4. CE UK and mySociety are using people power and Freedom of Information to bring transparency to local climate action

    A story in this week’s Financial Times [paywalled] has brought the EPC ratings of council-owned properties into the public conversation. This story was based on data obtained through FOI requests as part of the Council Climate Action Scorecards project, which we’ve been working on in partnership with Climate Emergency UK (CE UK).

    What you can read in the FT is one story pulled from a wealth of data, but there’s more to come. Our WhatDoTheyKnow Projects tool allowed CE UK’s team of volunteers to conduct a nationwide survey of every council through well-placed FOI requests covering the use of renewable energy, plans for retrofitting, green skills training, road expansion and more. 

    The data they gathered has allowed for the understanding of councils’ action on a nationwide scale. This level of oversight has not previously been possible: as with so much about the Scorecards project, it is allowing councils to take more informed action on climate, and individuals to clearly understand what is being done.

    Why local action matters

    In the UK, it is estimated that around one third of carbon emissions are in some way under the influence of local authorities. 80% of UK councils have declared a ‘climate emergency’ to indicate they recognise the scale of the problem of climate change, and are in a position to take practical steps to be part of the solution. To help local authorities achieve the goals they set themselves (and to push them to go further), we need to engage with the plans that local authorities are making, and the actions they are starting to take. 

    In 2021, CE UK and mySociety worked together to produce the first Council Climate Plan Scorecards. CE UK’s upcoming launch is the second iteration of the Scorecards. It is much bigger and more ambitious in scope than the last: it scores not the plans, but the climate actions of every local authority in the UK. 

    FOI requests were just one part of the process. As well as giving CE UK access to WhatDoTheyKnow Projects, we developed a crowdsourcing tool for volunteers to use while marking across the 90+ datapoints collected for each council. 

    How do you score action?

    CE UK moved from scoring plans to scoring actions. That required new approaches to gathering the information. 

    The questions CEUK used in the new Scorecards are the result of a long and thorough process of research and refinement. Building on their own research and expertise, they conducted one-on-one consultations with approximately 80 organisations and sector-specific experts. An advisory group of environmental and local government experts provided further discussion and refinement, to help build a list of questions that would practically be possible to answer, and that would reveal important information about the climate actions of councils. 

    The aim was to identify areas where information was publicly accessible; but also where gaps existed, especially in operational matters that aren’t often made public. Additionally, CE UK wanted to investigate whether councils are truly implementing the actions outlined in their climate action plans, including aspects like lobbying for additional powers.

    Making use of Freedom of Information

    Freedom of Information laws means that a huge range of information held by public authorities (including local councils) can be requested by any person who asks. This provides a legal tool to create greater transparency where information is not being published proactively.

    For CE UK, the potential of FOI for the Scorecards project was clear – but there were concerns. In consultations with council staff, there was pushback regarding the use of FOI requests due to the potential time and financial burden on council officers who work on climate – with some requests for a more informal survey approach to be used. But the drawback of that would be making good data dependent on goodwill everywhere. FOI requests provided a way to make sure the scorecards were not just effective for councils who engaged with the process and provide an approach that was fair across the country. 

    To balance a process where they want to encourage positive engagement from councils, with one that works without that, CE UK’s approach was to plan out the most efficient and least burdensome use of FOI requests. 

    Based on feedback from the advisory group, and trial runs to a small number of councils, they eliminated questions that were less important and useful, made more ‘yes/no’ or ‘single number’ responses, and learned where certain questions weren’t relevant to certain areas or groups of councils. 

    The subsequent FOI requests became more streamlined, and this resulted in quicker response times for the final requests than they had in the trial – as the information sought was more direct and concise.

    In the end, CE UK submitted a total of over 4,000 FOI requests to councils across the UK. The questions were divided into 11 categories, with some being specific to certain types of councils, such as district councils or combined authorities. The next stage was taking these 4,000 requests and getting them into a form that can be used for the scorecards. 

    Crowdsourcing and review process

    CE UK used WhatDoTheyKnow to manage their FOI request process. mySociety’s WhatDoTheyKnow acts as a public archive for requests – requests made through the site have the responses shown in public to bring more information into the open  – making it more discoverable by other people interested in the information, and reducing the need for duplicate requests being made. As of 2023, a million requests for information have been made through the site, with hundreds of thousands of pieces of information being released. 

    A feature we are trialling with a range of organisations is WhatDoTheyKnow Projects, which integrates crowdsourcing tools into WhatDoTheyKnow, and allows the task of extracting information into a new dataset to be spread out. The goal is that this helps organisations be more ambitious in finding out information and helps people work together to create genuinely new and exciting datasets, that no single organisation has ever seen. 

    As CE UK’s approach already made heavy use of volunteers and crowdsourcing, this was a natural fit.  Alongside a wider group of 200 volunteers working on getting answers to the other questions, 15 volunteers specifically worked on the FOI requests. These volunteers were a mixture of people with prior experience or professional interest in FOI requests, campaigners well-versed in FOI processes, and individuals new to the concept but eager to engage in activism.

    After the crowdsourcing of FOI data was complete, it joined the rest of the data in the new tool mySociety had developed for helping volunteers crowdsource information for the Scorecards.  

    From here, councils were given access to the data collected about them and given a right of reply to correct any inaccuracies or point towards information not previously discovered or disclosed. The results of this process will then be reviewed to produce the final Scorecards data, which will be launched this month.

    But the Scorecards data will not be the only useful thing that will come out of this process. Because of how WhatDoTheyKnow was used, to see evidence supporting the final Scorecards, people will be able to click through and see the original responses, for instance, to see what councils have lobbied on support for their climate work. 

    Some of the FOIs are being used to construct datasets that have a broader impact, and here we come back to that FT story on the Energy Performance Certificate (EPC) ratings of council-owned houses. Building these new public datasets will be useful for councils to understand their own situation, and as we see with the news story, more broadly to understand the challenges ahead for local governments to meet net zero emissions goals. 

    Onwards

    The original Scorecards project has already been influential on how local governments understand their own plans, and how organisations like the UK’s Climate Change Committee understand the role and progress of local government in the challenges ahead. When the next generation of Scorecards is released, we hope that they continue to be useful in shaping and improving local government action around climate change.

    mySociety believes that digital technology can be used to help people participate more fully in democracy, make governments and societies more transparent, and bring communities together to address societal challenges.

    The Scorecards project showcases how the combination of digital tools, people power, and the right to information produces powerful results. We hope that the impact of this project can inspire and make possible similar approaches for other problems, or in other countries.

  5. ICO advisory note on publishing spreadsheets

    Following the PSNI and other recent data breaches, the ICO has issued guidance to public authorities. This guidance suggests a temporary stop on publishing Excel-style spreadsheets in response to FOI requests made via online platforms like WhatDoTheyKnow. The full advisory note is available online

    The advisory note emphasises that this is not a reason not to disclose requested information. Instead, the ICO says to release the information from original source spreadsheets as a CSV file – a simpler format than Excel Workbooks, with less potential for including hidden sheets or metadata that can lead to an accidental breach.

    A focus on file formats is a blunt measure, and one that will need to be superseded by better procedures and technical processes.

    We support authorities releasing data in the most appropriate format for the information being requested. This may sometimes mean an extract from a table, and sometimes a complete document. Excel spreadsheets are legitimate public documents, and information released in this format can be hugely valuable. It’s important to develop processes where they can be released safely. 

    Significant data breaches involving Excel files clearly show the risks when data management and release processes fail. These include not just breaches we see through WhatDoTheyKnow, but through disclosure logs and releases made directly to requesters. This is an opportunity for public authorities, the ICO and us at WhatDoTheyKnow to reflect on how we can best deliver the huge benefits of public transparency while safeguarding personal data. 

    Modern authorities need to be good at handling data. Data breaches happen at the intersection of technical and human processes. The FOI team can be the last link in the chain of a data breach when they release the information, but the root cause often goes back to wider organisational issues with the handling of sensitive data.

    In the short run, the ICO has recommended training for staff involved with disclosing data. Many teams already have excellent processes and do excellent work, but all authorities should take this opportunity to consider their responsibility on the data they hold, and have appropriate processes in place.

    Long term progress means developing good universal processes that keep data safe, regardless of the format of the data or how the data is released. All FOI releases should in principle be treated as if they are being released to the public, because the authority’s ability to stop a data breach ends when the information is released. Making FOI responses public produces huge efficiencies for the public sector, increasing transparency in practice, and multiplying the benefit to society of the information released. 

    Technology can also be part of the solution – we need to understand more about why existing technical ways of removing hidden information from Excel spreadsheets are not being used (as described in the ICO’s established guidance on disclosing information safely), and how new tools or guidance can make it easier to release data safely. 

    A core part of our work at WhatDoTheyKnow is dealing with the practical reality of promoting public transparency while protecting personal information. We take data breaches seriously and have processes in place for dealing with them as promptly as possible. We continue to plan and work to help reduce the occurrences and impact of personal data breaches through both our procedures and technical approach. 

    By monitoring how authorities respond to requests on WhatDoTheyKnow, we will seek to understand how this guidance is working in practice, and engage with the ICO and other organisations to promote effective long term approaches to this problem. 


    Notes on the content of the advisory

    Below is our understanding of the advisory note by subject matter:

    Freedom of Information requests

    • Continue to comply with FOI responsibilities. This guidance is about releasing information in a way that reduces risk of accidental disclosure. 
    • Temporarily, do not release original source spreadsheets to online platforms like WhatDoTheyKnow. Instead – convert and release to CSV files.
    • If that is not possible, then:
      • Ask if the Excel sheet can be sent to a separate (non-public) address. Proceed with the original address if they ask for this. 
      • In all releases, go through processes to ensure there is no data breach in the material. 

    General data management

    • Excel files are unsuitable working environments when they become very large (hundreds of thousands of rows). Authorities need to switch to appropriate data management systems that are more appropriate for managing larger amounts of data.  
    • Staff who use data software and are involved in disclosing information need continuous training.  
    • Understanding of pivot tables and their risks should be incorporated into data management.

    The ICO plans to update their guidance on Disclosing Information Safely

    The checklist released accompanying the advisory has several useful steps on checking for hidden data in Excel sheets. However, on the ‘considered alternative ways to disclose’ step, refer back to the steps in the advisory note. Information converted to CSV can be released to WhatDoTheyKnow in compliance with the advisory note. The advisory note says that the source dataset should continue to be released to WhatDoTheyKnow if it cannot be converted, the requester does not want to use an alternative route, and the authority is confident it does not contain a data breach.

  6. mySociety and Black Thrive: Stop & Search Dashboard

    Supported by the Wellcome Data Prize in Mental Health, mySociety are providing technical support to the organisation Black Thrive, whose work addresses the inequalities that negatively impact the mental health and wellbeing of Black people. Their question: does disproportional use of police stop and search impact the mental health of young Black people in England and Wales? This project aims to find out.

    Late last year mySociety provided some low-key support during the Discovery phase of the project as Black Thrive were developing their statistics package to extract and enhance data from data.police.uk to make the data more accessible and research-ready (for example, by making it easier to combine the data with other datasets such as Understanding Society, a longitudinal household panel study).

    We’re pleased to say that the project made an impact, and we’ll now be more actively sharing our expertise in creating data-heavy, citizen-focused services by collaborating with Black Thrive, King’s College London, and UNJUST CIC in the Prototyping phase of the Data Prize.

    Now that Black Thrive have built an automated way of gathering the data and creating the analysis, the next phase focuses on presenting these in an open dashboard that’s straightforward for anyone to use and understand. We’ll also be looking at mechanisms for automatically keeping the data up to date, and adding new datapoints when they become available.

    Having built several dashboards into our own services including WhatDoTheyKnow and FixMyStreet, we’ve got some great experience to build off; plus bags of experience in sourcing and munging lots of disparate data from our EveryPolitician project. More recently we’ve been making local council Climate Action Plans explorable and accessible, and also creating an easy-to-use data hub for data about local MPs, constituencies, public opinion and the climate and nature movement so this project sits comfortably within our wheelhouse.

    In fact, as it happens, we’re currently using very similar tech on another project, our Local Intelligence Hub prototype, and we’re going to use it as the basis for this new dashboard. Nothing like a bit of crafty repurposing where it helps save time and effort! 

    Here’s a sneak peek of how it’s looking at this early stage:

    Black Thrive Stop & Search Dashboard proof of concept using the Local Intelligence Hub prototype

    Watch this space and we’ll be sure to keep the updates coming as we progress.

    Image: Chris White (CC by-nc-nd/2.0)


    Like what we do? Help us bring more projects like this into the world.
    Donate now
  7. Climate monthnotes: Jan/Feb 2023

    It finally feels like Spring is in the air, and you know it’s been a busy start to the year when we’re rolling the first two months into one set of monthnotes – in the middle of March!

    So, what have we been up to?

    Well – we’ve been adding datasets to and testing our alpha version of the Local Intelligence Hub tool that we built with The Climate Coalition. Feedback has been really good and this feels like something that’s really going to level up the ability of UK climate organisations to share data and coordinate their actions, at both a local and national level. We hope to share more about this project in the coming months, once it’s been made available to TCC members.

    We submitted talks to a couple of conferences/events – and lo and behold, we’ll be in Sheffield (and online) for the Festival of Debate on May 24, with a panel of exceptional guests. Our topic? “What if you could reshape democracy for the better — and you had 20 years to do so?” Climate is sure to be part of the answer. Fancy joining us? Book here.

    Between all this we’ve been working hard with our friends at Climate Emergency UK on the next round of the Council Climate Scorecards. Their draft methodology was released in November 2022, and the first round of marking started in January 2023. Part of our support has included building a Django application to store the marking data – and this has already dramatically improved the experience for Climate Emergency UK’s volunteers. 

    Climate Emergency UK are also working with mySociety’s Transparency team, using WhatDoTheyKnow Projects (a WhatDoTheyKnow Pro beta feature that helps researchers crowdsource data out of batch FOI requests) to gather some of the data for the scoring. All their FOI requests will be published on WhatDoTheyKnow later this year.

    Our IICT grants are coming to an end soon – we’ve put out a blog post about Lynsted Community Kitchen Garden and the data they’re collecting with the weather station we funded. They have a public event on March 25 if anyone lives near Lynsted and wants to visit to check it out! Updates from Possible and Better Futures should be coming soon.

    On the research side, we launched our report on unlocking the value of fragmented public data, which is part of our work into the data ecosystem around climate data. Our plan over the next few months is to support a few research commissions which link in to this report and help to show use of climate data. 

    We’ve confirmed a partnership with Dark Matter Labs – we’ll be moving forward with them and our Neighbourhood Warmth prototype, exploring how we could encourage neighbours to come together to take their first retrofit action, such as getting a house survey. We’ll be building a working prototype over the next few weeks, then testing it out with communities in three pilot areas around the UK, to ensure that what we’re building makes sense to the people we’re aiming to serve.

    And finally, we met up in person! We had a team meeting in early February which was a wonderful chance for us all to take stock of the last year, and discuss the future. We’ve been making some plans for year 3 of the Climate programme and after widening our scope through prototyping, now we’re going to be focusing back in again on building and proving the impact of the services we’re running.

    That’s a very whistlestop tour of our first months of 2023!

     

    Image: Daniel James

  8. Unlocking the value of fragmented public data

    As a joint project between mySociety and the Centre for Public Data, we have written a set of simple principles for how to get the most impact out of publishing public data.  You can read the report online, or download it as a PDF

    Fragmented public data is a problem that happens when many organisations are required to publish the same data, but not to a common standard or in a common location. Data is published, but without work to join up the results, it rarely has the intended impacts. 

    The results of this are frustrating for everyone. Data users cannot easily use the data, policy makers do not see the impact they want, and publishers in public authorities are required to produce data without seeing clear results from their work. 

    Better and more consistent publication of data by local authorities helps enable understanding and action at scale across a range of areas. At the same time, we recognise that the technical advice given has assumed higher levels of technical capacity that in practice is possible for many data publishing tasks. Our goal has been to make sure our advice makes data more accessible, while having a realistic idea of technical capacities and support needed for data publishing. 

    This report recommends three minimum features for a data publishing requirement to be successful: 

    1. A collaborative (but compulsory) data standard to agree the data and format that is expected.
    2. A central repository of the location of the published data, which is kept up to date with new releases of data.
    3. Support from the data convener to make publication simple and effective – e.g. through validation and publication tools, coordinating returns, and technical support.

    We recommend that:

    • Whenever government imposes duties on multiple public authorities to publish datasets in future, it should also provide the staff and budget to enable these features.
    • The Central Data and Digital Office should publish official guidance covering the above.

    You can read the report online, or download it as a PDF

    Better data publishing helps climate action

    This project is informed by recurring frustrations we have run into in our work. Projects such as KeepItIntheCommunity, which mapped registered Assets of Community Value, were much more complicated than they needed to be because while transparency was required of authorities, coordination was not – meaning the task of keeping the site comprehensive and updated was enormously difficult. In principle, we could build a tool that empowered communities in line with the intentions of the original policy makers. In practice, a lack of support for basic data publishing made the project much harder than it needed to be.

    This problem also affects our work around local government and reducing emissions. Local government has influence over one third of emissions, but much of that is indirect rather than from the corporate emissions of the authority directly.  As such, many activities (and datasets) of local government have climate implications, even if the work or data is not understood as climate data. For instance, the difficulty in accessing the asset data of local authorities makes it harder for civil society to cross-reference this information with the energy rating of properties, and produce tools to help councils understand the wider picture. 

    In future we will be publishing in more detail the kind of data we think is needed to support local authorities in emission reduction – but emissions reduction cannot be isolated from the general work of local authorities. Improving the consistency of the data that is published helps everyone better understand the work that is happening, and makes local government more efficient. 

    Sign up to hear more about our climate work, or to the monthly mySociety newsletter

    Photo credit: Photo by Olav Ahrens Røtne on Unsplash

  9. Learn how to find good quality data

    Data is at the core of everything we do at mySociety, and the better quality it is, the easier our work becomes — so the latest output from TICTeC Labs is particularly welcome. We would love everyone to know exactly what constitutes good quality data!

    And, thanks to the members of the Action Lab #3 working group, now they can. They awarded a contract to the Canadian civic tech group Open North, to devise a course on Data Quality. This course is free to everyone, and we know it’ll be of huge benefit to the international civic tech community.

    Available online in English and French (and hopefully with more languages to follow), the course provides users with a practical introduction to the topic, discussing key concepts and setting practical exercises.

    Quality information for civic tech success

    This output was the end result of our third TICTeC Labs Civic Surgery, which took place back in March 2022. That saw participants discussing the theme: ‘Accessing quality information for civic tech success: how can we overcome barriers to accessing good data and documentation?’ — it was within this session that the concept of a training course first arose.

    This course uses Open North’s existing learning platform to provide training which covers:

    • Understanding the importance of data quality
    • Understanding the key terms when engaging with data
    • Knowing how and where to find good quality data
    • Recognising the barriers to accessing data and documentation
    • Knowing how to evaluate the quality of a dataset

    Collaborating with the Action Lab members throughout the process of planning and building the course, Open North have created an online educational resource that is suitable for a wide range of audiences. It provides a starting point for those already working with data, or those at the beginning of their journey. 

    Take the course

    You can find out more, and take the course by signing up to Open North’s Training Center and then looking for Data Quality (D103), with the French version at La qualité des données (D103F). In fact, once your account is activated you can take any of their free courses, so take a look around and you might find some more resources to try, as well.

  10. Climate monthnotes: November 2022

    November was another busy month for our Climate programme, with progress on a number of fronts – from the return of an old friend, in the shape of the Council Climate Scorecards; to the development of two new ones, as a result of our prototyping process earlier this year. We’ve also been working hard to share our data and tools with new audiences. Here’s a quick round up:

    Constituency data for climate campaigners

    As Alexander mentioned in October, we’ve been working on a Beta version of platform that brings together data about MPs, constituencies, and local climate action, as part of a project with The Climate Coalition. The aim is to help campaigners at both national and local levels to understand where to focus their efforts on enabling real local action on climate goals.

    This month—thanks to the involvement of not only Struan and Alexander but also Graeme, on loan from our Transparency programme—we’ve made lots of progress, adding the features and importing the datasets we’ll need for testing out the minimum viable product with target users in the New Year. I look forward to sharing more with you in the coming months!

    Exposing high-emissions local authority contracts

    Another service that’s come out of one of our earlier prototyping weeks is ‘Contract Countdown’, which aims to give citizens advance notice of large, high-emissions local authority contracts that might be expiring in six, 12, or more months.

    This November, Alexander finished developing the final pieces of a working Alpha version – including the use of real contracts from UK Contracts Finder and the Find A Tender service, and pulling in the details of local authority climate officers and councillors with climate/environment responsibilities (so we could test the idea of helping users contact these representatives).

    And Siôn and I have been testing the alpha with target users – including local and national journalists, local authority climate officers and procurement officers, and local climate activists. We aim to continue getting feedback on the Alpha throughout December, and maybe January, after which point we can make a decision on whether to develop and launch a full service later in 2023.

    Climate Action Scorecards 2023

    Speaking of next year, preparations are already underway for next year’s follow-up to the Council Climate Scorecards project—this month saw Lucas and I work with Climate Emergency UK to design and publish their draft methodology for the assessment that will begin next year.

    With CEUK’s assessors now looking at councils’ climate actions, in addition to their plans, we wanted to make it as easy as possible to understand precisely which questions your local authority will be scored on. I think we came up with a nice solution, where you can filter the list of draft questions by your local authority name or postcode, as well as by local authority type.

    Sharing our data and tools

    In other news, Alex updated our deprivation and urban/rural classification datasets to show relative figures for local authorities and Westminster parliamentary constituencies. We also published a local authorities lookup dataset that makes it easy to convert between the many names and codes used to identify local authorities.

    If you want to use these new datasets—or any of our data in fact—Alex runs drop-in office hours on Thursdays and Fridays to talk about just that. We’re also happy to help collect or analyse climate-related data for free, as part of our work on supporting the UK’s climate data ecosystem – you can read more about that here.

    Speaking of data ecosystems, you’ll now find a number of mySociety’s open climate datasets listed in Subak’s Data Catalogue, and Icebreaker One’s OpenNetZero catalogue.

    Finally, Myf and Siôn in particular have continued to share and talk about our tools, and how people are using them to support local climate action, this month. Highlights include attending the Natural History Consortium’s Communicate conference; giving a hands-on workshop about all of mySociety’s tools for London’s small charities and community groups at Superhighways’ “Where’s The Power In Data” conference; and publishing a really exciting case study about how an officer at Surrey County Council used CAPE to share experiences and best practices with other similar councils elsewhere the UK.

    Image: Designecologist