-
/RSS FeedIt’s March 4 2025, and we’re releasing a bunch of new data on TheyWorkForYou, around each MPs’ financial interests: that’s whether they have second jobs, what donations helped them campaign ahead of the general election, and whether they’ve received gifts such as Taylor Swift tickets.
In the course of assembling this data — with the help of our brilliant team of volunteers — we’ve come to understand exactly what the problems with the current system of reporting are.
If you’re seeing this on the morning of release, we’ll also be launching a report at 1pm today, and you’re welcome to join us. (Don’t worry if you’re too late; we’ll be sharing the video afterwards. Just make sure you’re signed up for our newsletter to be alerted when it’s available).
Don’t forget to check out your own MP, to see who funds them, on TheyWorkForYou.com. And if you have any questions about this project, the data, or MPs’ financial interests in general, send them to us at whofundsthem@mysociety.org.
If you appreciate this type of work, please help us do more of it by making a one-off (or even better, a regular) donation. Thank you!
Transcript
[0:00] Julia: If you’ve ever wondered if your MP has a second job, what donations they received, or if they were one of the ones that got a free Taylor Swift ticket, we’ve got the answers for you.
[0:08] Hi, I’m Julia, and at my work at mySociety, over the last six months I have personally looked through every single MP’s Register of Members Financial Interests.
[0:18] That means I’ve looked at every donation, second job, shareholding, trip abroad. Everything that has been declared to the Register by every MP, my eyes have seen it. And we’ve learnt a lot in this process. And I’ve been supported by 50 amazing volunteers and the rest of the mySociety team.
[0:33] 50 volunteers helped us go through all of this information about second jobs, gifts and donations for all 650 MPs.
[0:40] We’re making that available on TheyWorkForYou on Tuesday the fourth of March. There’s new summaries and extra information about whether your MP received money from oil and gas companies, gambling companies, or if they took visits to countries which are scored ‘not free’ by Freedom House.
[0:56] In the process of doing all of this work and adding all this information to the Register, we’ve got some opinions on how it can work better, both in terms of collecting the data so that it’s accurate and so that we know who is donating, but also changing some of the wider rules to make sure that there is less of an influence of private money in politics.
[1:13] The first recommendation that’s coming out of the WhoFundsThem project is about the data itself. And that might sound boring, but it’s really fundamental to us being able to have trust in the system.
[1:23] At the moment, the forms that are being used by MPs to declare their donations and their second jobs just aren’t capturing the right information. There’s all sorts of things going on here.
[1:32] Sometimes the questions that are being asked aren’t very good. Sometimes the questions are right, but they’re not required, so MPs can skip them. Or sometimes the right questions aren’t even being asked.
[1:43] All of the rules seem to be interpreted in slightly different ways by new MPs or by older MPs, and fundamentally, it just made the project really hard, because the same questions were being answered in such different ways that when we’re trying to compare all MPs, it felt like comparing apples and oranges.
[1:58] We’ve got more recommendations later on in the report, which have bigger implications on how we fund politics. But we have to start from the beginning. We have to ask the right questions to get the right data, to even understand what’s going on.
[2:10] The second recommendation from our report is about scrutiny, which is a boring word for an important thing, which is, how do we check that the information that we’re being given is accurate and it’s correct?
[2:19] Our first recommendation was about changing the forms and doing better data collection to begin with. But that’s not enough. We also need to have mechanisms in place to check that the data is accurate, and fundamentally, we think that that has to be Parliament’s job.
[2:32] This is tricky, because the capacity in Parliament is stretched, and we totally appreciate that, but there’s some key things that can be done.
[2:38] We’re recommending a quarterly audit of just a handful of MPs to check the donations are being reported. At the moment, we’re surprised that there are some quite high profile MPs who haven’t declared a single donation over the last year.
[2:49] This may be the case, but I think an audit process would improve compliance and improve trust as a whole.
[2:54] There’s also a problem that we’re not sure that the data is up to date. And so, for example, there are a lot of new MPs that declared that they were councillors who aren’t councillors any more now, we found that out in our research, but they hadn’t updated the register to say that.
[3:06] At the moment, it seems that MPs can just choose whether or not to update the Register, whereas we’d like that to be changed so that every quarter at least, an MP has to do a declaration to the Register, even if it’s just to say that there’s been no changes.
[3:20] One more little thing in this category is that when MPs ask a question in Parliament, especially if it’s a written question, on the form where they submit the question, they have to say whether they have a relevant interest.
[3:30] And just for some strange reason, at the moment, Parliament will tell you whether or not they ticked yes, they have an interest or not, but they don’t tell you what that interest is. And we’d like that to change.
[3:39] If an MP has declared to Parliament that they have an interest that relates to the topic they’re asking a question about, that interest should be made public.
[3:47] So there’s a few of our ideas in this kind of second category of recommendations, which is all about improving the checks on the donations and the other information that gets declared to the Register.
[3:57] The third set of recommendations, as part of our Who Funds Them project are all about the rules themselves. So our first set of recommendations were about stronger data collection to get better data in the first place. Then the second set of recommendations were about checks for that data: are we sure it’s accurate? Are we sure that everything is being declared?
[4:15] And this third set is actually about the rules themselves, what qualifies as meeting the threshold to be declared. And in lots of the cases, we think that the thresholds need to be lowered so that more information gets declared.
[4:27] At the moment, MPs have to declare donations over £1,500 . Overall, we think that this threshold should be lowered to £1,000, and that will just capture more – and you know, £1,000 is a lot of money.
[4:40] We think the rules on gifts need to be stricter: both that more gifts need to be declared. At the moment, the threshold is £300, whereas in the civil service, it’s often £20. And so we think that needs to be lowered dramatically.
[4:51] And we also think there needs to be more rules on what should be accepted.
[4:54] Our fourth set of recommendations we’re calling Systematic Reform, and that’s about trying to decrease the influence of private money in politics.
[5:02] The biggest part of this fourth set of recommendations is this idea that we think there should be a citizens assembly on money in politics.
[5:10] A citizens assembly is when a group of people get brought together that are broadly representative of the public as a whole. They get given a really complicated or controversial policy question, and then over a few sessions, they get invited to hear from experts and witnesses, and they debate and discuss and come to conclusions about what should happen going forward.
[5:30] These have been used on issues that are really controversial, such as climate change, abortion and assisted dying.
[5:36] And we get that our recommendations as part of this report, where we’re saying that private money should play a lesser role, and therefore public money needs to play more of a role, like taxpayers’ money will be going more to political parties, is controversial.
[5:50] On the whole, people don’t generally support more money going into politics from taxpayers’ money, but we think it’s necessary, or at least we think that there are some trade offs to be had here. There’s a conversation that needs to happen.
[6:01] Are we happy that there is the amount of private money going into politics that there currently is? A citizens assembly seems to us a good way to draw out some of those debates.
[6:10] I have all of this information in my head now, and a lot of it has gone into the report, but the report is really long, and so if you have any questions on how money and politics works in the country, any questions about your individual MPs or what we’re going to do with the data, then let me know. But don’t forget that you can read the report in full on our website.
-
/RSS Feed
We’ve got updates from Julia on this Parliament’s first Register of Financial interests, showing what second jobs and gifts, etc, MPs have declared; and on the startlingly diminished list of All Party Parliamentary Groups (APPGs).
Meanwhile, Gareth tells us how to get a discount on WhatDoTheyKnow Pro, and we hear from AccessInfo about a new award – the winner will be invited to Madrid to present their work.
Alongside all of that, Myf explains how a WhatDoTheyKnow user harnessed the power of Reddit to verify the responses they were receiving to their FOI requests.
Enjoy!
Links
- Blog post on the Register of Financial Interests spreadsheet; and more details on what it contains
- Blog post about Reddit, WhatDoTheyKnow, and Physician Associates
- Blog post on the diminishing number of APPGs
- AccessInfo Impact Awards
- Full details on how to get a discount on WhatDoTheyKnow Pro by linking to your outcomes
- Our TikTok account
- Our Bluesky account
Music: Chafftop by Blue Dot Sessions.
Transcript
[0:04] Myf: Hello. Thank you very much for tuning in.
[0:07] This is our second monthly collection of news and updates from mySociety, and my name is Myf Nixon. I’m mySociety’s Communications Manager.
[0:15] This month, I’m going to share with you five pieces of news — two from our democracy work, and three from our transparency side. (more…)
-
/RSS FeedIt’s our first ever podcast at mySociety! Heeey how about that?
Myf, our Communications Manager, runs you through all the stuff we’ve been doing at mySociety over the last month. It’s amazing what we manage to fit into just 30 days: you’ll hear about a meeting of Freedom of Information practitioners from around Europe; our new (and evolving) policy on the use of AI; a chat with someone who used the Climate Scorecards tool to springboard into further climate action… oh, and there’s just the small matter of the General Election here in the UK, which involved some crafty tweaking behind the scenes of our sites TheyWorkForYou and WriteToThem.
Links
- TICTeC videos on YouTube
- TICTeC photos on Flickr
- Browse the TICTeC 2024 schedule, find slides etc
- Matthew’s post on updating TheyWorkForYou on election night
- Sign up to get an email whenever your MP speaks or votes
- Democracy resources and our future plans in Alex’s post
- Local Intelligence Hub lets you access and play with data around your constituency
- Matt Stempeck’s summary of the Access to Information meetup
- Our summary of Matt’s summary of the meetup
- Updates from all those ATI projects around Europe
- New in Alaveteli: importing & presenting blog posts; request categories and exploring csvs in Datasette
- Fiona Dyer on how volunteering for Scorecards upped her climate action
- Where to sign up if you fancy volunteering as well
- mySociety’s approach to AI
- Contact us on hello [at] mysociety.org if you have any questions or feedback.
Music: Chafftop by Blue Dot Sessions.
Transcript
0:00
Well, hello and welcome to mySociety’s monthly round-up.
My name is Myf Nixon, Communications Manager at mySociety.
0:11
This is part of an experiment that we’re currently running where we’re trying to talk about our work in new formats, to see if that makes it easier for you to keep up with our news. (more…)
-
CSV is a great format for releasing sets of structured data in response to Freedom of Information requests. Indeed, on WhatDoTheyKnow we’ve seen several thousands of CSVs released.
We’ve recently added the ability to explore CSV files via a Datasette instance. Here’s an example.
Opening the CSV in Datasette makes it easy to explore and analyse it in an interactive website.
If you’re not familiar, Datasette converts the CSV to an sqlite database, which means you can then query the data using SQL.
Alaveteli uses the publicly available lite.datasette.io instance by default, but you can host your own instance and configure it at theme level like we’ve done for WhatDoTheyKnow.
You can see the implementation details at mysociety/alaveteli/#7961.
—
Banner image: Joshua Fuller
-
My colleague Alex has already written about looking forward from this election, so here I am going to look back at the technical work that was involved for the election, and in getting all the new MPs into TheyWorkForYou.
Boundary changes
This election was the first UK Parliament election with boundary changes since 2010. Due to the long-running nature of TheyWorkForYou, which has been around now for over 20 years, this can throw up some interesting challenges. In this particular case, it turned out we were using two different JSON data lists of constituencies – both containing the same data, but one also included the other Parliaments and Assemblies, whilst the other included alternative names for some constituencies. I took the opportunity presented to merge these together and update the bits of code to use the one consolidated dataset, and then added in the 650 new constituencies to the JSON data.
Loading the new constituency data into TheyWorkForYou then threw up another historical problem – the constituency table was still using the very old Latin-1 character set encoding, rather than a more modern encoding such as UTF-8, that almost everything we have uses. This had been fine until now, with even Ynys Môn covered by that encoding, but the new constituency of Montgomeryshire and Glyndŵr contained a letter that Latin-1 could not cope with, leading to a quick emergency upgrade of the table to UTF-8 (thankfully this is a backwards compatible encoding, so worked without issue).
We had already generated data of the new constituencies and loaded these into our lookup service MapIt before Christmas. Ordnance Survey more recently published the official dataset of the boundaries, which we could then import via our usual processes, though even this raised a small issue to be resolved. It turned out in the last data release OS had given the parts of two county council electoral divisions with detached parts (Lightwater, West End and Bisley and Thorpe St Andrew) different identifiers, which they had reverted in their new release, causing our import script to get a bit confused – resolved with a small manual script.
Displaying on TheyWorkForYou
In the period before the election, we knew people would be using our site as a postcode lookup, perhaps to look up their previous MP but perhaps also expecting something useful for the upcoming election, which we wanted to provide, and so we used Democracy Club’s API to show election candidates and link to their WhoCanIVoteFor and WhereDoIVote services. We also displayed your boundary changes using the new constituency data mentioned above.
TheyWorkForYou isn’t just the UK Parliament, though, it also covers the Scottish and Welsh Parliaments, and the Northern Ireland Assembly, so we also had to maintain the provision of that information to people – email alerts for those bodies continued throughout as usual, and the postcode lookup kept showing people their representatives in the devolved nations.
Once the election closed, we automatically updated our messaging, and the next day switched back to our normal behaviour of taking you directly to your MP page in England, and showing you your MP and other representatives elsewhere.
We had a fun issue where some people were getting their new MP, whereas some were getting the old MP – during the period of dissolution, when there are no MPs, we have a configuration flag to enable the site to know it should return the latest result even if it’s not current (you don’t want this all the time, when e.g. an MP has resigned or died), but once new data was being loaded in, one database query was returning results in a random order; fixed by adding some sorting by descending end date.
Election result data
At the last election in 2019, we took a live feed of election results from Democracy Club, who have collected all the candidate information for their Who Can I Vote For service – which all began as the result of a mySociety project back in 2010.
Democracy Club were performing the same service this time, and gratifyingly it was quite a small change to have our 2019 code work with any 2024 changes to the source information (incidentally, there aren’t a lot of narrative doctests in our codebase, but I quite like the one in use there!).
This script would do half the job, of taking in some source data (who has been elected, and including their TheyWorkForYou identifier if they already had one due to being a previous representative of some sort) and amending our source JSON data to add the newly elected representative.
The other half is loading that source data into the TheyWorkForYou database for display on the site. Our normal loading script works fine, but looks through all the source data to see if there have been any changes to take account of. For the election, we don’t need it to do all that, so I tweaked the script to only do the minimal necessary to load in newly created information.
These two scripts were then added to a cron on our server, running every few minutes through the night. I did stay up long enough to check that the first few worked okay, before leaving it to itself from then on. I also set it up to pipe its output to our Slack channel, so people could see it operating:
This also meant as the final few trickle through, it’s popping up reminding us it’s still doing its job:
All the results (bar the one we’re still waiting for) are now committed to the repository, joining all our other open data.
Support TheyWorkForYou and our work
TheyWorkForYou and WriteToThem are run by mySociety, a small UK charity. We’re a very efficient operation and do a lot with a small team; if we had bit more money, we could achieve a lot more.
We want to see a transparent, resilient democracy, with equal access to information, representation and voice for citizens. If you believe in this vision please donate today to enable greater transparency and accountability of the next government.
—
Image: Moritz Kindler
-
The ICO have today announced that they intend to fine the Police Service of Northern Ireland (PSNI) for their accidental release of staff’s personal information in August 2023. This data was released in response to a Freedom of Information request made using WhatDoTheyKnow.
mySociety is a charity; we run WhatDoTheyKnow as a vital tool to help anyone exercise their right to information held by public authorities. We understand the repercussions of a breach like this, which serves to demonstrate that public authorities must be good at dealing with personal information. We welcome the ICO’s emphasis on the importance of robust release processes to ensuring that information that is important to the public interest can be released safely.
We take the responsibilities that come with operating a large platform extremely seriously, especially around the personal data breaches that can occur when authorities’ release processes fail. Following this breach, we’ve undertaken a significant programme of technical and process work to play our part in reducing the risks of this kind of incident.
We’ve developed a new piece of code which analyses spreadsheets as they come in as responses to FOI requests on WhatDoTheyKnow, and holds them for review if they are detected to contain hidden data. The deployment of this code has proven successful and we will be continuing to improve it. In its first three months, this spreadsheet analyser has screened 3,064 files and prevented the release of 21 spreadsheets that have been confirmed to contain data breaches, and 53 which were likely to contain data breaches (around 2% of the files screened in total).
In an ideal world, such measures would not be necessary; we continue to work with authorities making such releases to help them understand the reasons for data breaches, the potential severity of their impact, and how to avoid them.
This blog post was updated at 10:04 on 23 May to correct the figures around the number of spreadsheets screened.
—
Image: Pietro Jeng
-
Tl;dr: We’ve added lots of local council data to the Local Intelligence Hub.
In February, we launched the Local Intelligence Hub, and today we’ve released a huge new update.
We designed the Local Intelligence Hub — in collaboration with The Climate Coalition and supported by Green Alliance — to provide all the data you need, either about one constituency or across the whole country, on issues around climate. It helps you gain a deep understanding of public opinion, demographics, political considerations, and much, much more. In short, it’s an extremely powerful tool, free to use, and invaluable for anyone pushing for better climate action.
At launch, we divided the data by UK Parliamentary constituency — but with this huge new update, you can now also explore data at the local council level.
As ever, there are several different ways to view this data:
- by individual authority, so you can deep dive into your local area
- as a table, so you can compare councils by metrics that matter to you
- plotted onto a map, so you can see where to find hot- and cold-spots of action
And it can all be downloaded as a spreadsheet for use on your own desktop.
What kind of data are we talking about?
We’re pulling together data from multiple different sources. What does it all have in common? We reckon that it provides new insights for climate campaigners, researchers, journalists and organisations — especially when it’s combined in new ways, as Local Intelligence Hub allows you to do quickly and simply.
Sources include national polling data, information from our services CAPE and Scorecards, and other Climate Coalition member organisations, like the National Trust and the RSPB.
And we’re always looking for more data, so do get in touch if you know of a useful source we haven’t yet included!
What can I do with it?
You will know best how this rich data could inform your work, but here are a few ideas to get you started.
1. Build a profile of your local council
Dip into the local council page and see what data awaits you! Here’s an example of the top-level stats you can find for Leeds City Council:
- The area has a strong mandate for climate action. MRP polling suggests we’d see 88% of Leeds City residents support onshore wind compared to 83.5% national average, and just 10% oppose net zero compared to 12% national average.
- Leeds City Council is doing better than most councils, but could be doing more. It scored 53% on the Climate Action Scorecards, gaining its highest scores in Planning and Land Use, but with the biggest room for improvement on Transport.
- Emissions are huge, but so is the population. Leeds City Council serves 798,786 residents compared to the average of 307,712. According to BEIS data, Leeds City Council has influence over 2,822 kilotons of CO2 emissions, which is more than twice the national average of 1,168.3.
- There’s an active climate movement. In Leeds city there were more Great Big Green Week events than average in both 2022 and 2023.
2. Design a national campaign strategy
If you’re a campaigning organisation looking to work out where and how to allocate resources, the table-builder and CSV download could form an essential part of your planning process. Here we’ve generated the single-tier councils with Net Zero target dates that fall within the coming decade, and sorted by their Action Scorecards overall score, alongside useful data about public opinion and emissions.
Council Name Action Scorecards overall score Net Zero target date Population Oppose Net Zero % Total emissions (ktCO2) IMD Trussell Trust foodbanks Support onshore wind Wolverhampton City Council 21 2028 264407 12 854 1 0 82.0 Middlesbrough Council 21 2029 141285 12 558 1 7 78.0 Bromley Council 26 2027 332752 12 938 5 4 88.1 Dumfries and Galloway Council 28 2025 148290 15 864 3 3 80.0 Oldham Borough Council 32 2025 237628 12 690 1 2 80.1 Cheshire East Council 33 2025 386667 13 1860 4 2 87.9 Highland Council 35 2025 235430 13 1268 4 7 82.6 Nottingham City Council 42 2028 337098 9 1038 1 10 78.0 Haringey Borough Council 52 2027 266357 7 617 2 1 79.3 Tower Hamlets Borough Council 53 2025 331969 6 1019 2 0 79.8 Bristol City Council 55 2025 465866 8 1295 2 13 86.5 3. Visualise your goals
Local Intelligence Hub helps you zero in on the areas of the country that meet specific criteria. For example, where are the district councils who have declared a climate emergency but haven’t published a climate action plan? Here’s a map that shows you — just one of hundreds of maps that you can generate with a few clicks, and no expertise required:
What to do with all this lovely local data?
Thanks to this update, it’s now easier than ever to push for local climate action. With these rich new insights, you now have a number of talking points with which to engage your local councillors or council climate officers — and a wealth of facts and figures to back them up.
What next?
We need you to use the Hub and tell us what works, and what doesn’t! Give us your feedback — and if you’d like to know whenever we add something new, sign up to updates and we’ll let you know when there’s new data to play with.
Photo by Daniil Korbut on Unsplash
-
Organisations all around the world run Freedom of Information sites based on our Alaveteli platform. There are also sites running on other codebases. Either way, we’re all working to the same goals, and we believe that each site can learn from the others: for that reason, we’ve set up the Access To Information (ATI) network, a place for people who run FOI sites to meet, chat and share experiences.
Within this framework, there’s the chance for practitioners to pair up and exchange specific skills under a mentorship arrangement — and last week, we heard from GONG in Croatia who have been learning from Georgia’s ForSet.
ForSet began life as a data visualisation organisation, and these beginnings have allowed them to communicate data from their FOI site as compelling, visual stories: skills that GONG were keen to get to grips with. Sara, project co-ordinator at GONG, told us about working with Jubo from ForSet on data visualisations — and how her learnings will change the organisation’s work going forward.
Sara explained that they agreed two main goals for this project: firstly, to improve GONG’s data visualisation skills; and secondly, to use data visualisation to promote their FOI site Imamo pravo znati (IPZ) to journalists and general users. They were successful on both counts, not only did Sara learn how to use new methods and tools; but their outputs also brought approximately 50 new users to IPZ, and two additional Pro users (Pro usage is free on the site, but numbers had been stagnant of late, so this was notable).
So, how did they go about it? The mentorship comprised four stages: data collection, analysis, storytelling and visualisation, with the last being very interconnected.
1. Data collection
This stage began with both sides brainstorming potential topics for FOI requests that would be good candidates for data visualisation. An initial set of 12 topics was whittled down to five: local referendums in Croatia; special advisors (Spads) in the Croatian government; local participatory budgeting projects; local support for youth civic education; and local financing of civil society organisations.
GONG then sent 575 requests to local and county authorities, from which they received 525 responses — a pretty good response rate, and higher that expected. They didn’t hit many problems, although several authorities asked for the requester’s ID details, and there was one ministry that cited GDPR as a reason for refusing information on Spads. This case has now been referred to Croatia’s Information Commissioner.
2. Data analysis
Jubo and Sara organised the responses they received into spreadsheets: they were looking for an angle or a story among the data, and tidying it up in this way was helpful for making comparisons. Could they find a story in there that aligned with GONG’s mission or advocacy?
By organising the data in this way, the pair could easily see which data wasn’t really strong enough to take any further, and which lent itself to storytelling and visualisation. At this stage they rejected some of the angles they’d begun with, narrowing their projects down to local referendums, Spads, and lowering the voting age to 16 for the EU elections (this last project is pending; they’ll be launching a campaign in late Spring).
3. Storytelling and visualisation
Two pieces of software were used for the visualisations: Canva and Flourish. Sara was already familiar with Canva, as she uses it to create social media graphics; but Flourish was new to her, and she says she is very happy to have these new skills under her belt.
Flourish allows you to create many different types of visualisations: you upload your data and it is fairly intuitive to create maps, charts, etc. Once these were in hand, they added a call to action for each story, encouraging people to use their FOI site and especially Pro.
The visualisations
Local referendums
For the story on local referendums, GONG had requested from each local authority the number that had taken place; the topics under discussion; their outcomes; and the number of referendums that were suspended due to not being valid for whatever reason.
They received more responses than expected, and this was also the first time nationwide data had been collected on the subject.
The first angle that GONG wanted to support with their data and visualisations was ‘Croatia needs a referendum law that recognises the importance of local democracy’.
The data showed that out of 47 local referendums that had been held, just 25 met the minimum turnout for the decision to be valid. Jubo and Sara mapped these, and paired their visualisations with the argument that a lower bar for turnout would encourage better participation in local democracy – demonstrated with human figures.
A local press outlet picked the story up, using the data to make their own story: they were the area that had had the highest number of referendums, so that was their focus.Special Advisors
The FOI requests returned the names of Special Advisors, the areas they were in charge of, and the fees they were receiving. As Sara explained, in Croatia Spads are not straightforwardly employees of the government, but they have a lot of influence, and in some cases receive huge fees.
It became clear that there are two different types of advisors, under two laws; while each type has different functions, both are called Spads. First, there are external advisors who may or may not receive compensation; and secondly there is another class of Spads who are employed internally. Neither is subject to Croatia’s legislation on conflict of interest.
A pie chart was put to service to clearly show how much compensation Spads had received. This varied widely from Spad to Spad, but the criteria dictating who received how much is still unclear: it appears to be at the discretion of the individual minister.
In collecting this data, GONG unexpectedly unearthed a scandal, as they revealed one Spad who was abusing public funds. He was fired, along with the minister concerned; this resulted in nationwide coverage for the data; albeit again with the media’s own preferred focus.
Lowering the voting age
Sara says that it was a lot of work to find data to support the argument for lowering the voting age to 16 in Croatia. They wanted to show that, while young people see voting as the most efficient political action, it is denied to a large portion of them.
Proving the absence of something is always tricky, and in this case they were uncovering that there isn’t any research to show that 16 year olds lack the cognitive abilities to vote responsibly. So they focused on other angles: in some EU countries, 16 year olds can vote, and they demonstrated that those countries are doing well in democratic processes: they score highly in the democracy index and have good voter turnout.
Like many countries, Croatia’s population is ageing, so the young are in danger of being totally ignored. GONG plan to share their findings on social media in a simplified form with graphics cards, and a call to action to show support for the campaign.
Questions and answers
Once Sara had finished her presentation, members of the audience were invited to ask questions.
Q: How did GONG and ForSet work together?
A: At the beginning, they had lots of online video calls, and later on when the data had come in, they communicated a lot via comments on the spreadsheets.
Q: It feels like each of these the topics would be applicable almost everywhere: perhaps it will spark other people’s interest to do something similar for their own country. Any advice if so?
A: The questions asked in the first two sets of FOI requests were straightforward, which led to straightforward answers. The third topic was less so; Sara and Jubo had to go through lots of reports, and often the data from one source contradicted another. Also, an uncontentious topic is likely to result in more responses: something like referendums is politically neutral, unlike spads where the authorities may have reasons not disclose information.
Q: When you put the requests together, were you already thinking about the format it would be best to receive the data in?
A: In that respect, the best question is one with a yes/no answer. The reason for excluding many of the initial topics at a later stage was that the answers varied so widely that it was hard to pull out statistics or a simple throughline: you’d be comparing apples with pears. So, for example, when asking how much of a local authority’s budget is reserved for supporting civic education, and how civic education is delivered, the responses could range from “We are in favour of civic education, but leave it to schools”, to “We provide money for civic education and produce our own textbooks”. Meanwhile, some authorities wrote two pages of waffle in the free text box.
Q: Did you narrow down the topics before or after you had submitted the FOI requests?
A: Both. There were 12 topics at the start; they decided which of them were best suited to FOI, then sent requests for five of them. One the answers had been received, they narrowed it down to three.
Q: Could one make data visualisation about the other two? It’s hard to find ways to show that there’s no information. Saying that 80% of authorities don’t reply is not a very exciting way of showing things.
A: While it might not fit in with the initial aim of the project, this sort of thing can be a great way to show how well or badly FOI is working in your country. Journalists often can’t get the information they need, so build stories around the fact that there’s no data about such an important issue.
Q: We’ve seen how much GONG has benefitted from this mentorship. What, if anything, did ForSet get from this?
A: Sara was so quick and flexible, she was great to work with. ForSet also learned from the project: for example, that it is better when requesting a large amount of data, that is sorted by the public institution, so it’s easier to work with. You can request it sorted in the way that you need for your story, which might be different from how it is in public.
Also, Canva is such a great tool for visualisations. They’ve now merged with Flourish, so the have advanced data visualisation features. You just have to make sure you choose the right format: the type of charts or graphs that will show your findings the most clearly.
Finally, ForSet didn’t know about the topics that Sara suggested, so there was plenty to learn there, plus it was great to see the ways GONG employ to publish their stories on both social media and mainstream media.
-
Responses obtained from a widespread FOI project can be difficult to analyse, until they are sorted into neat datasets. This allows you to make valid comparisons, pull out accurate statistics and ultimately ensure your findings are meaningful.
In our third seminar within the Using Freedom of Information for Campaigning and Advocacy series, we heard from two speakers. Maya Esslemont from After Exploitation explained how to prepare for an FOI project to ensure you get the best results possible (and what to do if you don’t); and Kay Achenbach from the Open Data Institute explained the problems with ‘messy’ data, and how to fix them.
You can watch the video here, or read the detailed report below.
Preparing for an FOI project
After Exploitation is a non-profit organisation using varied data sources, including FOI requests, to track the hidden outcomes of modern slavery in the UK.
Maya explained that they often stitch together data from different sources to uncover new insights on modern slavery. She began with a case study showing some recent work they had done, using WhatDoTheyKnow to help them understand the longer term outcomes after survivors report instances of trafficking. This stood as an excellent example of how much work needs to be done before sending your requests, if you are to be sure to get the results you need.
In this case, After Exploitation were keen to understand whether there is any truth in widely-held assumptions around why human trafficking cases are dropped before they are resolved: it’s often thought that there are factors such as the survivors themselves not engaging with the police, perhaps because of a nervousness around authorities.
But what are these assumptions based upon? Actual information was not publicly available, so we wouldn’t know if cases were being dropped because of low police resource, a lack of awareness or more nuanced factors. Until the data could be gathered and analysed, the perceptions would continue, perhaps erroneously.
Before starting, After Exploitation thought carefully about the audience for their findings and their ultimate aims: in this case the audience would be mostly the media, with the aim of correcting the record if the results flew in face of what was expected; but they knew that the data would also be of use to practitioners. For example, charities could use it to see which areas to target regionally for training and other types of intervention.
They placed FOI requests with police forces across the country, making sure to ask for data using the crime codes employed by the forces: were cases dropped because of ‘lack of evidence’; did they have a status of ‘reported’ but not gone on to exist as an official crime record?
The project had a good outcome: while some requests had to go to internal review, ultimately over 80% of the forces responded with quality data. The findings were worthwhile, too: general perceptions did indeed prove to be wrong and there was no indication that ‘no suspect identified’ was a result of the victim’s lack of involvement. The resulting story was able to challenge the general narrative.
So, how can After Exploitation’s learnings be applied to the work of other organisations or campaigns?
Maya says:
- Planning, rather than analysis, is the majority of the work;
- Identify the need and purpose before you even start to pick which authorities to send requests to;
- Be clear who the audience for your findings is;
- Consult with other stakeholders to make sure your parameters are really clear.
Planning
Before you even begin, make sure your project isn’t asking for data that has already been collected and is in the public domain — this might seem obvious but it’s easy to overlook. Check other people’s FOI requests (you can do this by searching on WhatDoTheyKnow); look for reports, research, inspectorate/watchdog outputs, and data released as part of parliamentary enquiries.
That said, even if you do find previous data, there is sometimes value in requesting more up to date or more detailed information with a new set of FOI requests. If you see a national report collating data from every council for example, you could do an FOI project asking every council for a more detailed breakdown of what is happening in their region.
But before sending a batch of requests to multiple authorities, ask yourself if there is a centralised source for your data. If so, then just one FOI request might be enough: for example, homelessness data is already collected by the Depts for Housing, Levelling Up and Communities, in which case one request to them would save time for both you, and more than 300 public authorities.
Another question to ask before starting off on your project is “what is the social need?”. Does this need justify the resource you will expend? Mass FOI projects can be a bit of a time commitment, but the utility might not just be for your organisation: perhaps you can also identify a social benefit if the data would be of use to other groups, academics or journalists.
Define your intended audience: will the data you gather be of interest to them? Do you have a sense of what they want? For example, MPs often like to see localised data that applies to their constituencies. Journalists like big numbers and case studies. If you think your findings are important but might have limited appeal, you could consider including an extra question to provide details that you don’t need for your own purposes, but which could provide a hook.
Next, will the data that you gather actually be suitable for the analysis you want to perform? To avoid time-consuming mistakes, make sure the data you’ll receive is broken down in the way that you need. As an example, suppose you wanted to ask local authorities for details of programmes offered to children in different age bands: you might receive data from one council who has offerings for children ‘under 18 months’ and another ‘under two years old’ — and where units differ, they are difficult to compare and contrast. Be really precise in your wording so there’s no mismatch, especially if your request is going to a lot of authorities.
Consider, too, whether you can you get enough responses to make your data meaningful: 2,000 people is the figure believed to be representative of the population as a whole. Decide how many responses you ideally need for your purposes — and, in a scenario where not all authorities respond, the minimum you can work with.
You might want to contact other groups or organisations who could be interested in the same data, and ask if there are details that would be useful to their work.
As suggested in Maya’s case study, try to use existing measurements where you can: if you shape your requests to the methodology the authorities themselves use to collect the information, such as KPIs or their own metrics of success, these will be much easier for them to supply.
If you’re not sure what these metrics are, you can sometimes access internal guidance by googling the name of the authority plus ‘guidance’. Alternatively, submit scoping requests to a handful of authorities to ask how they measure success, etc.
At this stage it’s also useful to decide what quality of data you will include or exclude. For example, if you ask about training materials and one authority says they offer training, but don’t include the actual materials, do you include it in your figures? The more authorities you ask, the more ambiguities like this you’ll normally encounter.
Think about where and how you will log the data as it comes in. Maya recommended WhatDoTheyKnow Projects as a good tool for extracting data. Whatever you use, you should consider accessibility: can your platform be accessed by everyone you’re working with, across different communities? Especially if you are working with volunteers, it’s important to remember that not everyone has a laptop.
Also consider the security of the platform: how much this matters will depend on how sensitive the data is, but recognise that Google sheets and many other platforms store the data in the cloud where it could be more vulnerable to abuse.
After Exploitation take great pains to ensure that their data is accurate. They recommend that each response is assessed by two different people, making sure that everyone knows the criteria so they’re applied consistently; and doing regular spot checks on a handful of cases to make sure they are all logged in the same way and there’s no duplicate logging.
This is time-intensive and arduous, but if you have other stakeholders they might be able to help with the data checking: for example, knowing that they would eventually place the story with the BBC, After Exploitation were happy to hand this task over to their inhouse data checkers.
What if things go wrong?
If you’ve done all the planning suggested above, it’s less likely that your project will go awry, but even if it does, Maya says that there’s always something you can do.
No or few responses: ask yourself whether you have the capacity to chase no/late replies, and if you still don’t get a response, to refer them to the ICO. If not, consider prioritising the bodies that are most relevant to your work, eg the biggest authorities or those in areas with the densest populations; but be prepared to defend accusations that not every authority had a fair hearing unless you do them all.
If you know your requests were well worded, but you’re not getting a lot of responses — perhaps because you’re dealing with a contentious issue, or simply because the authorities cash-strapped — you could shift to measuring the types of responses you get. If authorities aren’t able to answer the question, this can often be just as revealing.
Responses that don’t tell you what you set out to understand: Consider whether there are any alternative angles in the data you do have: are there any additional themes, particularly in any free text fields? Or try a new round of requests asking for more detailed information.
Responses don’t cover the whole country: If you can’t get data from everywhere, could you narrow down to just one area and still have useful findings? Even the most basic data can set the scene for other researchers or organisations to build on: you can put it out and outline the limitations.
Results
The impact of gathering data through FOI can be massively powerful, as After Exploitation’s work shows. They have revealed the wrongful detention of thousands of potential victims of human trafficking when the government were denying it could happen; opened the debate about locking up vulnerable people; and uncovered the flawed decision making in the Home Office on modern slavery cases. It was only through FOI requests that all this information came into the public domain and was picked up by mainstream media.
—
Combining different sources of data to create datasets
Kay Achenbach is a data trainer on the Open Data Institute’s learning team; the ODI works with government and companies to create a world where data is working for everyone.
Kay shared a case study from the medical field, in which an algorithm was being designed to quickly assess high numbers of chest x-rays. The aim was to automate the process so that people identified as needing intervention would be sent to specialists right away.
The developers wanted to make sure that different demographic groups weren’t being biased against, a common issue with algorithms built on existing data which can contain previously undetected biases.
The test material was a set of x-rays from a diverse population, that had already been examined by specialists. They ran them past the algorithm to see if the diagnoses produced were the same as those made by human doctors.
The doctors’ assessments came from three different datasets which, combined, comprised data from more than 700,000 real patients. As soon as you combine datasets from different sources, you are likely to come across discrepancies which can make analysis difficult.
In this case, one dataset had diagnoses of 14 different diseases, and another had 15 — and from these, only eight overlapped. The only aspect that could for sure be compared was the “no finding” label, applied when the patient is healthy. That limitation set what the algorithm was asked to do.
Other fields were problematic in various ways: only one of the three sources contained data on ethnicity; one source only contained data on the sickest patients; another was from a hospital that only takes patients with diseases that they are studying, meaning there were zero “no finding” labels. Two of the sources contained no socio-economic data. Sex was self-reported in two of the sources, but assigned by clinicians in the other, which could also affect outcomes.
The advice from all this is that you should look carefully at each dataset before you combine them, to see what the result of combining them would be. In short: does it reflect real life?
Ultimately the researchers found that the algorithm was reflecting existing biases: it was much more likely to under-diagnose patients from a minority group; more likely to make mistake with female patients, the under 20s, Black people, and those from low socio-economic groups. The bias was compounded for those in more than one of those groups.
Cleaning up datasets
Once you’ve obtained your datasets from different FOI requests, you’re highly likely to find mismatches in the data that can make comparisons difficult or even impossible — but cleaning up the data can help.
For example, in spreadsheets you might discover empty fields, text in a numbers column, rows shifted, dates written in a variety of formats, different wording for the same thing, columns without titles, typos and so on.
Kay introduced a tool from Google called Refine that will solve many of the issues of messy data, and pointed out that the ODI has a free tutorial on how to use it, which you can find here.
-
A general election is not far away, so get ready for heated conversations: on your doorstep, on social media and in the news.
If you care about climate, we want you to be able to take part in those conversations with the facts at your fingertips. That’s why this week, we’re launching the Local Intelligence Hub, a powerful tool that provides a wealth of relevant national and local data in one place — and encourages you to combine it in a multitude of ways, to uncover useful new insights.
mySociety worked in collaboration with The Climate Coalition, supported by Green Alliance to develop this site. The aim is to help you — whether you’re a citizen, climate campaigner or part of an organisation — to understand and share the places where there is a strong mandate for environmental action, ensuring commitments to climate and nature are put firmly onto party manifestos. We’ve demoed it in front of organisations who’ve told us it’s a total gamechanger!
But enough words — let’s get straight to the action. Watch these short videos and you’ll immediately grasp the power of the Local Intelligence Hub.
For individuals
“I’m just one person: what difference can I make?” Well, with the Local Intelligence Hub’s data, you can make a lot of difference.
As a first step, put your postcode into the Local Intelligence Hub and find out all about your local area.
You might find some interesting data combinations: for example, what does public support for climate action look like in comparison to data on air pollution in your constituency? How about the measures of poverty against support for cheaper renewable energy?
We hope you’ll use this kind of intel to inform conversations with canvassers or your MP. If you discover something notable, why not write to the newspaper — local or national — or share your findings with your community newsletter, Facebook group etc?
In the run-up to an election public opinion has a lot of power, and all the more so when you can quote the data to prove it.
For campaigning
If you are part of a climate campaign that works nationally, the Local Intelligence Hub shows you at a glance where in the country to concentrate your activity for the most impact.
Play about with the map page, selecting different datasets, and you’ll soon understand the insights they unlock. Every constituency with high support for renewable energy for example; or the constituencies where the MPs have the lowest majorities; or where the population is youngest… the possibilities are practically endless.
If you’re more locally-based, dive into the constituency pages where a massive range of local data allows you to have a full picture of the area:
- Public opinion: How much support is there for climate initiatives such as net zero or renewable energy?
- Place: What factors affect people in the area, such as air pollution, flooding and levels of deprivation?
- Movement: Which climate and environmental groups are active in the area, and what other relevant organisations have a presence?
For each constituency, these three data collections are supplemented by information on the MP’s memberships, voting and activities. Note that you have the choice to see constituencies as they are now, or as they will be after the election when new boundaries come into play.
Once you’ve dipped into the data, you should be able to shape your campaigns to more effectively speak to the right people about the issues that matter to them.
We hope you find the Local Intelligence Hub useful. When you’ve had a chance to try it out, please do let us know how you’re using it!
