Over the last few years we have stopped publishing several statistics on some of our services, but haven’t really talked publicly about why. This blog post is about the problems we’ve been trying to address and why, for the moment, we think less is better.
TheyWorkForYou launched with explicit rankings of MPs but these were quickly replaced with more “fuzzy” rankings, acknowledging the limitations of the data sources available in providing a concrete evaluation of an MP. Explicit rankings on the ‘numerology’ section of an MPs profile were removed in 2006. In July 2020, we removed the section altogether.
This section covered the number of speeches in Parliament this year, answers to written questions, attendance at votes, alongside more abstract metrics like the reading age of the MP’s speeches, and ‘three-word alliterative phrases’ which counted the number of times an MP said phrases like ‘she sells seashells’.
This last metric intended to make a point about the limits of the data, along with a disclaimer that countable numbers reflect only part of an MP’s job.
Our new approach is based on the idea that, while disclaimers may make us feel we have adequately reflected nuance, we don’t think they are really read by users. Instead, if we do not believe data can help make meaningful evaluations or requires large qualifications, we should not highlight it.
Covid-19 and limits on remote participation also mean that the significance of some participation metrics is less clear for some periods. We’re open to the idea that some data may return in the future, if a clear need arises that we think we can fill with good information. In the meantime the raw information on voting attendance is still available on Public Whip.
WriteToThem responsiveness statistics
When someone sends a message to a representative through WriteToThem, we send a survey two weeks later to ask if this was their first time writing, and whether they got a response.
This answer was used annually to generate a table ranking MPs by responsiveness. In 2017 we stopped publishing the WriteToThem stats page. The concerns that led to this were:
- There are systemic factors that can make MPs more or less likely to respond to correspondence (eg holding ministerial office).
- As the statistics only cover the last year, this can lead to MPs moving around the rankings significantly, calling into question the value of a placement in any particular year. Does it represent improvement/decline, or is the change random?
- MPs receive different types of communication and may prioritise some over others (for example, requests for intervention rather than policy lobbying). Different MPs may receive different types of messages, making comparisons difficult.
- The bottom rankings may be reflecting factors outside MPs’control (eg a technical problem with the email address, or health problems), which can invalidate the wider value of the rankings.
The original plan was to turn this off temporarily while we explored how the approach could be improved, but digging into the complexity has led to the issue dragging on and at this point it is best to say the rankings are unlikely to return in a similar form any time soon.
The reasons for this come from our research on WriteToThem and the different ways we have tried to explore what these responsiveness scores mean.
There are structural factors that make direct comparisons between MPs more complicated. For instance, we found that when people write to Members of the Scottish Parliament there are different response rates for list and constituency members. What we don’t know is whether this reflects different behaviour in response to the same messages, or whether list and constituency MSPs were getting different kinds of messages, some of which are easier to respond to. Either way, this would suggest an approach where we judge these separately or need to apply a correction for this effect (and we would need to have different processes for different legislatures).
There are also collective factors that individual representatives do contribute to. For instance, if MPs from one party are more responsive to communication, controlling for this factor to make them easier to compare to other MPs individually is unfair as it minimises the collective effort. Individuals are part of parties, but also parliamentary parties are a collection of individuals. Clear divides are difficult in terms of allocating agency.
One of the other findings of our paper on the Scottish Parliament was that there was an effect in the Holyrood and Westminster Parliaments where female MPs had a systematically lower responsiveness score than male MPs (roughly 7% lower in both cases, and this remains when looking at parties in isolation). Is this a genuine difference in behaviour, or does it reflect a deeper problem with the data? While responsiveness scores are not quite evaluations it seems reasonable to be cautious in user-generated data that is systematically leading to lower rankings for women, especially when the relevant literature suggests that women MPs had spent more time on constituency service when the question was studied in the 1990s.
One concern was if abusive messages sent through the platform were leading to more emails not worth responding to. This was of special concern given online abuse against women MPs through other platforms. While WriteToThem only accounts for 1-2% of emails to MPs, it is a concern if we cannot rule out if a gendered difference in abusive messages is a contributor to a difference in a metric we would then use to make judgements about MPs.
Our research in this area has found some interaction between the gender of the writer and recipient of a message. We found a (small) preference for users to write to representatives who shared their gender, but without more knowledge of the content of messages we cannot really understand if the responsiveness difference results from factors that are fair or unfair to judge individual representatives on. Our policy that we should maintain the privacy of communications between people and their MP as much as possible means direct examination is not possible for research projects, and returning to publishing rankings without more work to rule this out would be problematic. We are exploring other approaches to understand more about the content of messages.
Content and needs
We could in principle adjust for differences that can be identified, but we also suspect there are other differences that we cannot detect and remove. For instance, constituents in different places have different types of problems, and so have different needs from their MP. If these different kinds of problems have different levels of responsiveness, what we are actually judging an MP on is their constituents, rather than their own behaviour.
A finding from our analysis of how the index of multiple deprivation (which ranks the country on a variety of different possible measures of deprivation) relates to data in WriteToThem is that messages to MPs from more deprived areas are less likely to get a response than those from less deprived areas. The least deprived decile has a response rate about 7% higher than the average and the most deprived decile is 6% lower. However, when looking at rates per decile per individual MP there is no pattern. This suggests this is a feature of different MPs covering different areas (with different distributions of deprivation), rather than individual MPs responding differently to their own constituents.
At the end of last year, we experimented with an approach that standardised the scores via a hypothetical average constituency. This was used by change.org as one metric among many in a People-Power index. While this approach addresses a few issues with the raw rankings, we’re not happy with it. In particular, there was an issue with an MP who was downgraded because more of their responses were in a more deprived decile, and this was averaged down by lower responses in higher deciles.
If we were to continue with that approach, a system that punishes better responsiveness to more deprived areas is a choice that needed a strong justification. This approach is also becoming more abstract as a measure, and less easy to explain what the ranking represents. Are we aiming to provide useful comparisons by which to judge an MP, or a guide to WriteToThem users as to whether they should expect a reply? These are two different problems.
We are continuing to collect the data, because it is an interesting dataset and we’re still thinking about what it can best be used for, but do not expect to publish rankings in their previous form again.
FixMyStreet and WhatDoTheyKnow
Other services are concerned with public authorities rather than individual representatives. In these cases, there is a clearer (and sometimes statutory) sense of what good performance looks like.
Early versions of FixMyStreet displayed a “league table”, showing the number of reports sent to each UK council, along with the number that had been fixed recently. A few years ago we changed this page so that it only lists the top five most responsive councils.
There were several reasons for this: FixMyStreet covers many different kinds of issues that take different amounts of time to address, and different councils have more of some of these issues than others. Additionally, even once a council resolves an issue, not all users come back to mark their reports as fixed.
As a result the information we have on how quickly problems are fixed may vary for reasons out of a council’s control. And so while we show a selection of the top five “most responsive” councils on our dashboard page, as a small way of recognising the most active councils on the site, we don’t share responsiveness stats for all councils in the UK. More detail on the difference in the reported fix rate between different kinds of reports can be seen on our data explorer minisite.
WhatDoTheyKnow similarly has some statistics summary pages for the FOI performance of public authorities. We are reviewing how we want to generate and use these stats to better reflect our goals of understanding and improving compliance with FOI legislation in the UK, and as a model for our partner FOI platforms throughout the world.
In general, we want to be confident that any metric is measuring what we want to measure, and we are providing information to citizens that is meaningful. For the moment that means publishing slightly less. In the long run, we hope this will lead us to new and valuable ways of exploring this data.
MapIt is a mySociety service that can take UK postcodes and return which administrative boundaries those postcodes are inside. This can be used to find out what council or constituency an area is in — you can test it at https://mapit.mysociety.org/.
Over the last few months we’ve been updating some existing boundaries in MapIt, and adding new kinds of geographies.
What has changed
Local authorities and Clinical Commissioning Groups have been updated to their new April 2020 boundaries. Small census areas have been updated to their latest version across all UK nations.
There is also a new kind of statistical geography: Travel To Work areas. These are areas that include the home and work location of 75% of people inside them. They are a way of visualising the commuting boundaries of an area, which may be significantly different from the administrative ones (See maps of all Travel To Work areas in England).
Small census areas are small statistical areas that cover a neighbourhood sized areas (although what locals consider the neighbourhood to be may vary). Many sets of official statistics are mapped against these small areas, making them an important intermediary between postcode or coordinate data and measures such as the indices of multiple deprivation.
As many statistics are produced separately for different UK nations, there are different kinds of small areas in different nations.
- Lower Super Output Areas (LSOA) – England and Wales
- Datazones (DZ) – Scotland
- Super Output Areas (SOA) – Northern Ireland
- Middle Layer Super Output Area (MSOA) – England and Wales
- Intermediate Zones (IZ) – Scotland
In MapIt, all of these boundaries are present, available under the English geography names (LSOA/ MSOA) to avoid needing more complicated lookups when working with postcode data from across different nations.
What can I use this for?
Mapping from user postcode data to LSOA helps build a picture of the environment of users. As we’ve done with FixMyStreet, this can be used to understand patterns of use. It can also help researchers with existing postcode datasets to find the equivalent statistical areas to expand the dataset.
You can see some of these areas in practice powering the postcode lookup on this minisite looking at the new 2019 maps of multiple deprivation in England.
Last week, I wrote about our project collecting council climate action plans, and making them easier for anyone to explore in a public, open database.
Today, I’m happy to say that the first version of this service is now at https://data.climateemergency.uk.
You can enter your postcode to quickly see if your council has an action plan, browse and compare different councils’ plans, and search over the text of all the plans. We’ve also added some headline emissions data for each council’s area, to start to put the plans in context.
There’s a lot more that could be done here, and we’ve got some ideas we’re really excited about, but given the urgency of the need to act, we want to make this service available right now, so that people can start using it, and so that we can learn from feedback, and have more informed conversations about where to take it next.
We’ll also be demoing the service at 2pm today at the Climate and Ecological Emergency: Taking Action Together online conference – at 2pm in Room 3 (Action Plans). You can sign up for the session here.
So please do join us there or go and take a look at the service yourself and let us know what you think by emailing us at firstname.lastname@example.org.
Additionally, do join us at our upcoming TICTeC seminar where we’ll be discussing what civic tech’s role in mitigating the climate crisis should be.
When we asked this question on Twitter, the first person to reply was Tim Morton, who told us how he’d used our services to get a useful addition to his local neighbourhood. The story began on FixMyStreet, but really came to fruition thanks to WriteToThem.
Tim says that he’s been using FixMyStreet since 2008: “If you look at my reports, the vast majority relate to the street I live in, and my local park” — and indeed, that’s the scene for the success he tweeted to tell us about: the story of the Grit Bin.
It began with a report, back in 2010:
“I pressed send,” says Tim, “and waited for something to happen”.
But unfortunately, nothing did — had Tim’s message been lost in the internal workings of his council?
It was radio silence until four weeks later when FixMyStreet’s automated mail arrived, asking whether the report had been seen to. If you click ‘no’, you’re taken to a screen suggesting a few ideas for escalating your issue, one of which is to contact your local councillors through WriteToThem.
Tim decided that this was a good idea, and posted an update on his FixMyStreet report to say so:
“Again, though, there was a period of silence… and I’d almost forgotten about it,” says Tim.
But sometimes these things take a bit of time. Because, seven weeks later, and just in time for Christmas:
Tim’s simple request had brought about a useful and tangible change for his community.
OK, so, ideally it would have happened quickly and with full communication from the council, after that first FixMyStreet report. But on the other hand, this is a great example of how sometimes you have to persevere, and try another route, before you get success.
“The grit bin is still there: occasionally I ask for a refill, and when the snow falls I trudge along the road and shovel grit across the junction.”
So the benefit has lasted — and is allowing Tim to do his bit for his community even now, a decade later.
Tim rates FixMyStreet so much that he’s demonstrated it to community groups and on training courses. He explains, “I think the great thing about FixMyStreet is its ease of use, and the very visible audit trail.
“One thing I always point out is the timestamps on my initial reports. I often make reports in the evening, or at weekends: they’re done in the moment and not by trying to get through to the council on Monday morning or when the office is open. I find if I had to wait, I’d forget about the issue.
“Leicester Council has been good at responding to my requests, and I always post their replies in the comments on my reports.” (Leicester is not currently a FixMyStreet Pro client, so their responses are not automatically published on the website, but sent to the report-maker by email.)
Being an expert user, of course Tim knows all about FixMyStreet’s more advanced features.
“I’ve recommended that community groups use the local alerts function. This means they can see what other people are reporting in their area, which they may be unaware of.
“If they’re a group that focuses on neighbourhood improvement, it will identify potential issues for them to work on, and in fact, may introduce them to potential new activists in their area. I’ve pointed Ward Councillors to this, as well, as it can be really helpful in their work”.
Thanks so much to Tim for telling us all about the grit bin and his efforts to help spread the word about FixMyStreet. A grit bin may seem like a small win, but when you consider how many thousands of reports are made up and down the country every week on FixMyStreet, and how many messages are sent to councillors on WriteToThem to ask for a neighbourhood improvement, you can see that the net effect could be massive.
And on that note, if you have brought change by writing to your MP or councillor, by making a FixMyStreet report or perhaps by using one of our other services, please let us know — we’re all ears.
We’re using these stories as part of a training module that helps young people understand how democracy functions in the UK, and how to work within it to make positive change. Your stories will help us to show this in action, rather than just theoretically, so you’ll be helping us to help those who need it. Thanks!
Next Friday (13 November), two years after the first climate emergency declaration by a UK council, we’ll be demoing a new online service to help people find and understand councils’ climate action plans at the Climate Emergency: Taking Action Together online conference.
The conference will explore how councils, other public organisations, businesses, charities and communities can all work together to develop radical action plans to deliver on their climate commitments.
Back in March, we kicked off a small crowdsourcing project gathering councils’ climate action plans in an open spreadsheet. A lot has changed since then, but the urgency of responding to climate change becomes ever more acute. With the pandemic providing proof that we can change our behaviour in extraordinary ways, and now that many of us have, of necessity, narrowed our focus to the world on our doorstep, this work seems more important, more challenging, and yet more possible than ever.
Three guiding principles
In September, Climate Assembly UK, the citizens’ assembly commissioned by the UK parliament to answer the question of how should the UK meet its target of net zero greenhouse gas emissions by 2050, produced its final recommendations. We were proud to be part of the team working on the assembly, and particularly happy to be able to make the comprehensive report available in readable, navigable, accessible and mobile-friendly HTML online.
The randomly selected people from all walks of life and all across the UK who made up the assembly chose and agreed a set of principles to guide their work. The top three were:
- Informing and educating everyone (the public, industry, individuals and government)
- Fairness within the UK, including for the most vulnerable (affordability, jobs, UK regions, incentives and rewards) in actions, not just words
- Leadership from government that is clear, proactive, accountable and consistent
We’re committed to a climate response that follows these principles, and believe that local government and local communities – individuals, institutions, and businesses – have a key and difficult role to play together.
As the recent Institute for Government report on getting to Net Zero noted,
“The local level has become a key outlet for public enthusiasm to address climate change. This is one reason why it is important to address the co-ordination and capability problems that are holding back local efforts – or else this enthusiasm will turn to disillusionment as aspirations cannot be achieved.”
This is a huge challenge, and getting the right information is part of it. We’re hoping to use our data and service design skills to play a part in helping councils learn from each other’s ideas and successes, and in helping citizens find and engage with their councils’ climate plans.
An open dataset of action plans
With your help, and working with ClimateEmergency.uk, we’ve created a first basic dataset of all the council climate action plans that are publicly available. The headline is that 269 out of 414 councils we researched (around 65%) have a current public plan outlining their response to the climate emergency.
In the last few months of this year, we’re doing research to better understand the challenges of producing and improving these plans, and of understanding, discussing and scrutinising them.
Helpful for councils — and citizens
We know that people working inside councils to produce plans are looking for inspiration – “What’s worked in other places like ours? How do you do it on a budget? How do I persuade my colleagues that it can be done? How do I talk to residents about the options?”
Citizens who want to have a say in their council’s plan may struggle to find it in the first place, or to understand what the council can and can’t do, how to influence them, or how their plan compares to others.
We’ve also been working on a minimal viable digital service that will meet some of the basic needs that people have around these challenges – one that supports quickly finding plans and starts to put them in context.
How to find out more
So if you can, join us at the Climate Emergency UK: Taking Action Together online conference next week on Friday 13th November. We’ll be giving the first public demo of that service, which will allow anyone to quickly and easily find out if their council has a plan, and to filter and search within all these action plans.
We think that will be useful in itself and we’re really excited to be putting it out into the world – but we’re also going to be developing our ideas on how to sustain and expand the service. This is still an early stage project for us, but we think it’s one where we believe our skills can play a part in catalysing action and enabling people to come together to make these plans reality.
Image: Master Wen
Many thanks, too, to our panelists, who spoke so knowledgeably and engagingly about the experiences of parliaments around the world that have been forced to make a quick switch to digital technologies during the COVID months.
Julia Keutgen of the Westminster Foundation for Democracy, Avinash Bikha of the Centre for Innovation in Parliament and Lord Purvis of Tweed from the UK’s House of Lords were led in conversation by mySociety’s Head of Research Dr Rebecca Rumbul.
We heard about parliaments in Morocco, Brazil, Chile, the Maldives, and of course the UK, with a rounded view of the benefits of quick digitisation against the challenges and inconveniences. Naturally, parliaments and their members come in all shapes and sizes around the world, and their readiness or suitability for transferring to online methods vary accordingly.
On the negative side, some representatives have struggled to adapt, especially if older; and all may be missing the nuances of face to face conversations with their colleagues.
But there are positives too, with MPs able to spend more time in their constituencies helping constituents, and (close to mySociety’s heart, this one) a quicker turnaround of digital data on voting results.
Watch the video to hear plenty more detail on this engrossing topic.
The third and final TICTeC Seminar in our autumn 2020 series will focus on civic tech’s role in the climate crisis and will take place next month — date TBC. Sign up for TICTeC updates and we’ll send you an alert once timings are confirmed.
Finally, if you work on, use, fund or research civic technology, we would be really grateful if you could spare some time to help us shape the future of TICTeC by filling in this survey.
Image: Joakim Honkasalo.
A forthcoming tribunal will examine the blocking of FOI requests that have been placed by people living outside the UK.
To those at mySociety and WhatDoTheyKnow, the matter is quite simple: the UK’s Freedom of Information Act was written explicitly to allow “any person” to request information from a public body. There’s no restriction to say that the requester must be a UK citizen.
A phrase often used is that the FOI process must be ‘applicant blind’. An authority doesn’t have the right to refuse information because of what it knows about the requester. That applies to nationality as much as to any other characteristic.
We vehemently defend this principle, not least because we have seen first hand that important investigations can result from cross-border collaborations — right now, we’re working to support journalists across Europe working on several stories that cannot be confined to one territory.
Associates across the international FOI network are proof positive that this kind of collaboration is invaluable in getting to the truth. Last year, Arne Semsrott of German FOI site FragDenStaat told us of a project they are running in tandem with Spain-based AccessInfo, to find out more about the treatment of migrants in many countries.
“You can file FOI requests for Frontex documents anywhere in Europe”, he said, “so we’re asking in different countries for ‘serious incident’ reports: these will tell you of human rights violations”.
If each country insisted that its information was only accessible to its own citizens, there would be significantly less opportunity to uncover such cross-border instances of mistreatment, not to mention stories of corruption, malpractice, misspending and cronyism. And as we know, such phenomena are unlikely to respect jurisdictional boundaries.
For a view from closer to home, we can consult a member of the experienced WhatDoTheyKnow volunteer team. Richard Taylor comments, “If UK FOI requests were restricted to British citizens or to those living in the UK, that could, depending on how it was implemented, seriously impact our ability to provide WhatDoTheyKnow’s service.
“Providing proof of nationality or residence would be a significant additional hurdle for people making requests, and for us in managing them.”
We question why there is a need for a tribunal to examine a point of the Act that is already quite clear — and, since there is to be one, call upon them to make a judgement that adheres to the letter, and spirit, of this country’s information law.
Image: Max Böhme
We’re longstanding supporters of LocalGovCamp, the conference where innovators in Local Government come together to share knowledge on how to improve services.
This year we’re both sponsoring it and running a couple of hands-on, interactive sessions. All online, of course, given the way things are these days.
On Tuesday 6 October, join a mySociety-led discussion with Mark and Zarino, on how consistent data standards across councils could open the doors to much better innovation.
We’ll be looking at our own Keep It In The Community project, nodding to our Council Climate Action Plans database, and inviting attendees to join a wider discussion on how we can encourage better joined-up data across councils.
And on Weds 7 October, our designer Martin will be running a mock ‘consequence scanning’ exercise. He’ll take participants through a new and useful way of assessing and mitigating risks in new government services, as conceived by Dot Everyone, recently taken up by Future Cities Catapult, and now used successfully in service design workshops by SocietyWorks.
We hope you’ll come along and enjoy some good discussion and deep dives into local government service improvement: find out more and book your place here.
This work will support users in taking the next steps, if appropriate, when their requests for information are denied.
A bit of background
In the last few years, there has been a significant and sustained decline in FOI requests being granted by the UK government.
According to the Institute for Government, the proportion of refused FOI requests reached a record level in the third quarter of 2019, with departments refusing to comply in full with more than half of all FOI requests that they received. This compares to around 40% in 2010 and around 30% in 2005.
And yet, our research found that, when challenged, a large proportion of refusals were overturned, suggesting that the fault did not lie with the type of request being made. 22% of internal reviews resulted in the full or partial release of information, and a further 22% of appeals to the ICO led to all or some of the information being released.
For local authorities, up to half of internal reviews – and just over half of all ICO appeals – led to the release of all or some of the information requested. In Scotland, with its own FOI regime, 64% of appeals to the Information Commissioner resulted in the full or partial release of information.
And so, while acknowledging that some refusals are certainly legitimate, there is a clear case for challenging such responses. But to do so is daunting, especially for novice requesters who can understandably be discouraged by an official response citing exemptions in legalese.
This new funding will allow us to approach the issue from four different, but interlinked directions, each intended to inform and support users in challenging government refusals of FOI requests.
- When a WhatDoTheyKnow user confirms that they’ve received a refusal, we’ll be integrating context-sensitive advice. This will inform the user of their right to appeal, give clear guidance on how to assess whether the authority has complied with the law, and also advise on other channels, beside FOI, by which information may be obtained.
- We’ll automatically identify which exemption has been cited in the refusal, giving us the ability to help users better understand why their request has been turned down.
- Based on this finding, we’ll offer context-specific advice for the exemption identified. For example, if the request has been turned down because of cost, we’ll show how to reframe it to fall below the ‘appropriate limit’.
- Finally, once the user has been fully informed, we’ll offer the support they need to escalate the request to an appeal.
Ultimately we hope that this work will help reset the balance on the public’s right to access information, better enabling citizens, journalists and civil society to effectively scrutinise and hold authorities to account.
As always, we’ll also be thinking hard about how to make all of this apply more universally, across the various legislatures that apply in jurisdictions where people are running sites on the Alaveteli platform.
If this interests you, watch this space. We’ll be sure to update when we’ve made some progress on the project.
Image: Tim Mossholder
Open Data: an essential, not just ‘nice to have’
Would societies around the world be better able to respond to the pandemic, if more or better open data were freely available?
That was the question put to our expert panel on Tuesday, in the first in our series of online TICTeC Seminars.
Karabo Rajuili of Open Ownership, Olivier Thereaux from Open Data Institute and Fabrizio Scrollini of the Open Data Latin American Initiative (ILDA) were led in a discussion by our own Head of Research Dr Rebecca Rumbul.
We heard of the need for — and simultaneously the impossibility of — a rapidly-constructed open data standard; the benefits and dangers of releasing data about COVID to a potentially uninformed public; and the need for good ownership data to be freely available in a fast-moving procurement environment in which there may not be the tools to investigate where money is being spent.
After the speakers had laid out their positions, the floor was opened for questions, each of which ignited still more informed debate. Finally, attendees were invited to a quick (and optional!) networking session in which they could speak to other attendees more directly.
There are still two more TICTeC Seminars in this series to go, so do join us to take part in the conversation.
On 20 October, panelists will discuss why it’s taken a pandemic for more parliaments to digitise; while in November (date TBC) the topic will be the climate crisis. Find full details for both sessions here, and don’t forget you can sign up for TICTeC updates.
Also: if you work on, use, fund or research civic technology, we would be really grateful if you could spare some time to help us shape the future of TICTeC by filling in this survey.