Tl;dr: We’re now releasing our register of interests data as a spreadsheet.

High quality data about the external interests of our MPs and ministers is vital to identifying conflicts of interests, and discouraging politicians from having conflicts of interest in the first place.

Lack of clarity on the interests and income streams of MPs is a corruption risk. The problem with second jobs and outside interests is less that MPs might be distracted from their main job – but that when they stand in Parliament, they may be representing groups beyond their constituents, asking questions (or not asking questions) depending on their outside work.

When outside interests exist, it’s vital they are clear and transparent. The Register of Members Interests contains a list of disclosures MPs are required to make of financial interests or benefits which “others might reasonably consider to influence his or her actions or words as a Member of Parliament”. Following the Owen Patterson scandal, there was renewed interest in this data, as it was clear that there were a number of potential stories and scandals hidden in plain sight – just requiring someone to join up the data.

Building a data ecosystem

A key problem is that the data is not easy to work with. The data is released (roughly fortnightly) on the parliament.uk website as a HTML document for each MP. This process technically releases the information, but makes it hard to compare releases of the same MP over time, or to make comparisons between different MPs.

TheyWorkForYou improves on this by creating structured data from the HTML release. Using this we can highlight the changes in each release from the previous release. This is useful for journalists and campaigners in quickly understanding what has changed in each release. For instance, the change in Rishi Sunak’s register over time can be seen here.

We want to avoid people doing the same work of cleaning the data over and over. We make our version of the data available publicly, so other people can use our work to do things that we haven’t done ourselves. For instance, Open Innovations have built on top of the data we publish to link the data to other datasets and create a Register of Members’ Financial Interests Explorer.

While projects like the Tortoise/Sky News Westminster Accounts create new value in joining up datasets and cleaning the data for their own work – ultimately the new datasets they have created are only usable by those organisations. That’s their right as the people doing the work – but we think there is a bigger (and more sustainable) impact to be had in improving the data in public.

Making our data more accessible

Previously, we have published our interests data as a series of XML files, which is useful for programmers, but harder for other specialists to work with. We did some thinking with OpenDemocracy last year to explore if there were small changes we could make that would make the work we already do more useful.

As well as the XML files, we now publish an experimental spreadsheet version of all data since 2000, and the register for the current 2019 Parliament.

These sheets show the earliest and latest disclosure of an interest, and include some (very) basic NLP analysis to extract mentioned orgs from the free text and make it easier to quickly parse when scrolling.

This data can also be explored through Datasette, which can be used to query the datasets in the browser, and save the queries as links that can be shared.

For instance, the following links go to specific queries (we’re using an in-browser version for prototyping and this might take a minute to load):

• Paid visits to outside UK mentioning the UAE
• Gifts from England Lawn Tennis Club
• Declarations involving a helicopter
• Declarations new in latest release

We want to continue to improve our approach here – and welcome feedback from anyone this spreadsheet helps.

Parliament can do better data publication

A key problem run into by everyone working with the data is that it’s broken to start with. MPs fill things out in inconsistent ways that makes the overall data different to analyse without cleaning first (see both the Open Innovations and Tortoise/Sky News methodology notes). Fixing this up is a key first step towards aggregate analysis – and the easiest place to fix it is with validation when the data is collected at the start.

While work can be done to improve the data after the fact (and experiments with Generative AI have found it to be quite good at fixing inconsistent formatting), improving the initial data collection is the most effective way of improving the quality of the data. There are active moves in Parliament to fix some of these problems. Producing more information in machine readable formats, and adding methods to make sure the data is correct to start with, will make the transparency process simpler at every stage.

Similar issues apply to the register published for All-Party Parliamentary Groups (APPGs), which should publish as “machine readable” data the range of data that the groups are formally supposed to make publicly available. APPGs are semi-official groups that MPs can form around specific interests or issues. Many of these are useful ways of having discussions, but these can also be an avenue for corruption, with outside interests supporting the group and its activities. The register includes the officers of groups and financial assistance and gifts received by the group – but not the overall membership. APPGs are separately required to disclose their wider membership on their website (or if they don’t have a website, if someone asks) but this isn’t included in the register, and so can’t be consistently scraped to produce data. While MPs are supposed to disclose benefits from groups on their individual disclosure, clearer data on what is officially “public” memberships would help ensure that there is nothing missed between these two datasets.

Separately there is a register of ministerial interests that applies to MPs who also have government positions. This is in principle more strict, requiring disclosures of relevant interests of family members, and avoiding even perceived conflicts of interest. However, in practice the information does not contain the specific financial value of gifts or benefits, just that they exist. The disclosure cycle is also longer, being published every six months rather than monthly. In practice – this means that relevant interests may not be public for a significant time after a minister is appointed (and potentially never published, if the minister has again moved on by then).

There is a lot of work that can be done from the outside to build on official data. But the more Parliament does things that it is uniquely able to do, the more we can focus on analysis and data comparisons that are best done outside.

What mySociety can do

A very basic thing we can do is beat the drum (and work with those who have been doing this for ages) for better publication of data from Parliament.

But if this happens or not, we can do work to make the data better. If it looks like Parliament’s data is unlikely to be fixed at the source, then a project of improving the data in public in a way that multiple projects could then build on would be useful. But if the data gets better, then we can better spend our time doing more work on top of this data. This might include joining up the official data with other datasets (including those of the UK’s other Parliaments and Assemblies) to draw out connections and better analysis.

But our work here isn’t just about producing good data – it’s about displaying it in a way that’s useful and understandable by people. Chris Bryant MP (former Chair of the Standards Committee) has argued that Parliament’s own display of the history of registers should match what’s provided by TheyWorkForYou. If Parliament improved its own display to the public of registers of members’ interests this would be fantastic news – and we in turn would need to think about if there are new approaches that would be useful on top of that.

One approach we are thinking about would be to find out what people wanted to know the answers to about their MPs interests, and then using volunteers to answer a set of common questions. This is the kind of editorialising that Parliament itself would find much harder to do – while providing something different from aggregate analysis of the data all together. This is something we could do with the data as it exists, but is something where better data would let us create new tools so volunteers could answer more complicated questions.

Making MPs’ interests clearer and easier to understand is key to spotting conflicts of interest and keeping politicians accountable. We hope our new spreadsheet version of the data helps make the work we’re already doing more useful and accessible – while we think about the road we want to take in future to improve TheyWorkForYou and the project of a transparent democracy.

Image: Wilhelm Gunkel on Unsplash.

Donations to MPs are in the news again, and TheyWorkForYou allows users to easily see what any individual MP has received. In fact, the site has carried a copy of the Register of Members’ Financial Interests (in which, as Parliament’s website explains, “MPs must register within 28 days any interest which someone might reasonably consider to influence their actions or words as an MP“) since at least 2005.

This hasn’t always been straightforward, and has recently become slightly trickier.

The official register is published as static HTML or PDF, with a simple list of all MPs. We scrape that HTML, convert it into light XML and import it onto the site – which means you can easily see not only the current entry on an individual MP’s page, but also see a complete history of their register without having to view many different copies of the official register.

The XML contains all the data from the official register, but it only parses out basic information like the category of interest. Providing more detail would be great, but is quite a hard problem to tackle.

Recently, Parliament has started using Cloudflare’s bot-protection technology. We assume this change was made with good reason, but as a side effect it has prevented effective scraping of the website, as Cloudflare don’t distinguish between good and bad bots or scrapers.

We know that Parliament was working on an API at least as far back as 2016, from their now-removed data blog, but if this is still in development, it is yet to see the light of day. What they said at the time still stands: their website is still the only means of accessing this data. We don’t think it’s necessary to protect purely static HTML pages such as the Register in quite such a heavy-handed manner.

We do have ways of continuing to get the Register, and TheyWorkForYou is still up to date, so anyone else who has been scraping the official site and has hit issues because of this is welcome to use our data, either via the XML or our API.

—

Image: Adeolu Eletu

One service we offer on TheyWorkForYou is an email alert: this lets you know when there is new data published on the site that either contains a word/phrase that you’ve subscribed to, or that indicates new activity from your selected Member/s of Parliament.

(Didn’t know this? Go and sign up now!)

We send around 400,000 of these emails a month. For many years, the look has remained exactly as it was when we first developed them: plain text, which has the benefit of being lightweight and unlikely to get scrambled by email clients. The downsides are that it doesn’t exactly make for a compelling email, visually speaking, and that some find it hard to identify which sections are of interest in a uniform block of unformatted text.

We’ve now finally transformed alert emails into a much more polished HTML format, and at the same time we’ve also improved the look and feel of four other vital elements of TWFY: profile images, the API, the sign-up page, and the Contact page.

As usual, before starting work, we did a bit of research into who uses this feature and why, so we could be sure we were answering their needs. You can see more about this in Alex’s post here.

Photos of MPs

Where there is a more recent and higher quality image available, we’ve updated the profile image we use for MPs. In some cases, this has replaced some pretty youthful faces — it’s clearly high time we caught up with this particular ticket! 

Higher resolution or larger images also mean that they’ll be more useful to developers using the images (which are all available under an open licence) on other sites and apps.

Clearer access to the API

The API page (where developers and researchers can access TheyWorkForYou data) has been given a slick new design. We’ve updated it with new examples of how the API might be used, and streamlined the language and content to make it easier to understand. 

We hope that all of these features will make it easier and more pleasant for you to use TheyWorkForYou, either when you’re checking up on what’s happened in Parliament for yourself, or using our data to make other parliamentary apps and sites.

—

Image: David Pisnoy

The climate emergency is, of course, a massive concern, and that’s why we often urge you to contact your MPs and councillors to demand faster, better, greener progress.

And that’s important — but also, we really should take the time to give positive feedback, thanking those councils and politicians who are doing the right thing.

This year, we’re taking part in the Climate Coalition’s Good News Day which, since 2015, has asked “organisations, institutions, household names and millions of people to use the power of green hearts to join together and ask politicians to put aside their differences and tackle the climate crisis.”

Here’s how you can get involved

1. On Friday February 12, use our Climate Action Plans database to search for your local council and see if they have a plan in place.
2. If they have, drop your councillors a line on our WriteToThem service to let them know you appreciate it.
   Local authorities and councillors who are taking action need to know they’re supported in their actions, some of which may be radical or taking them into new territories — so let’s thank them for everything they’ve done so far, and maybe give them the support to go further, too.
3. If they haven’t? Let them know you care about any climate-related action the council have taken, and urge them to get a wider plan approved.
4. Maximise the power of your action by shouting about it on social media. Use the hashtag #ShowTheLove, and use a picture of a green heart (we’ve added links to some royalty-free images below you can download or copy and paste) to join in with the national Good News Day movement. Or, if you want to go all out, make your own crafty green heart: there are some ideas on the Climate Coalition’s worksheet and on cafod.org.uk.
5. If you’d like to do more, see the Climate Coalition’s collection of downloadable resources.

If you’re on a roll…

There are other ways you can #showthelove, too.

We think the prompt to ‘ask politicians to put aside their differences and tackle the climate crisis‘ is a particularly important one, so:

• You could also use WriteToThem to email your MP with this message…
• …or go public and tweet them!

And finally, there is encouragement to share everything your own organisation is doing to help the climate. With that in mind:

• Here’s mySociety’s own environment policy — which any other organisation is welcome to copy and adapt for your own needs.
• And a reminder of our post showing the various ways in which our services can be used to help the climate.
• If you’re one of our many friend and partner organisations in the UK and around the world, please consider joining Good News Day and getting the word out to your own followers.

Green heart pictures

Pictures on Unsplash are free to use and you don’t even have to credit the photographer, although if we’re talking about showing the love, we should of course do the same for the creative people whose work we benefit from!

Top row L-R: Ronak Valobobhai, Siora Photography, Adithya Vinod.
Bottom row: Volodymyr Hryshchenko, Patrick Fore, Bekky Bekks.

On 18 January we received a letter from Robert Largan MP regarding our parliamentary site TheyWorkForYou. He requested that we ‘correct a misrepresentation’ in the way that the site displays how he and his fellow MPs have voted on measures to prevent climate change:

The letter was co-signed by around 50 members of his party, and identified three votes not currently included in our climate change vote calculations, with the request that they be taken into account on their voting records pages.

This was not an unusual message: we often receive emails from MPs to TheyWorkForYou, asking us to explain or reconsider the data we publish on them — and the most common subject is the voting records pages.

The only differences with this letter were that Mr Largan had gathered the support of so many other MPs; and that it was covered in the press and shared on Twitter quite a few days before we actually had receipt of it. 

So we’ve treated it in the same way that we would any other, but given the amount of exposure the issue has already had, we thought we’d also share our considered response here. We’re glad to have this opportunity to illustrate how we run the site, and the judgements that we have to make in order to run the fairest, most factual service we can.

—

Image: UK Parliament

Humanity & Inclusion is a charity working to combat the injustices faced by people with disabilities and vulnerable populations in situations of poverty and exclusion, conflict and disaster.

Their recent campaign, ‘Stop Bombing Civilians’, encourages supporters to protest the bombardment of innocent citizens in areas of conflict like Syria, Iraq, Yemen, and Afghanistan.

As their website explains, when explosive weapons are used in populated areas, 90% of casualties are civilians. Many are left injured or disabled with their lives and livelihoods completely ruined — surely all would agree that this is not a just or desired outcome of bombardment, no matter what your view on the conflicts generally.

And so we were gratified to learn that WriteToThem, our ‘contact your MP’ service, is playing a part in this campaign.

A toolkit for speaking out

Humanity & Inclusion have put together a set of actions that supporters can take, from signing a petition to taking a selfie and sharing it on social media — or writing to your MP.

“WriteToThem was the obvious choice.”

And that’s where we came in: this last action is managed through integrating WriteToThem on the charity’s campaign page (something that any campaign can do, for free).

We asked Tom Shelton from Humanity & Inclusion to explain more about how they used our service within their integrated campaign.

Up to date contacts

Tom explained that a central part of the campaign is the petition, and it is easy enough for them to run petitions by using the forms on their own website.

However, when they’re asking supporters to directly email MPs, it’s just too complex to maintain and implement the dataset of politicians’ contacts themselves.

“Yes, this data is publicly available, but like many small organisations, we have no capacity for maintaining its integrity”.

Flexible and free

So the charity looked around to see what tools were available.

“There are some impressive tools out there, but most of them are pretty expensive given our modest needs. In previous years, we have used a relatively low cost paid tool for this type of ‘email your MP’ campaign.

“We needed a tool that was simple and safe for our supporters to use.”

“However, given that this new campaign was quite targeted, we were expecting a relatively low volume of emails, so we needed something that was easy to implement on our website, and we didn’t want to make any investment in a paid tool that would involve setup costs.

“We also wanted to avoid an ongoing subscription cost as we knew that our campaign would probably be paused at various points and then re-activated later (say, during elections, parliamentary recess etc).

“In particular, we needed a tool that was simple and safe for our supporters to use, and would help them to approach MPs in a way that is appropriate and would get the best response.

“Based on this, WriteToThem was the obvious choice.”

For all levels of coding knowledge

How easy was it to add the tool to their website?

Tom says that, for anyone with basic web skills, the postcode box option is very simple to set up.

“The more complex integration is also quite straightforward, but due to time constraints, we opted to integrate the postcode finder widget.

“The documentation on the WriteToThem website is excellent.”

“This fitted nicely in with our website and immediately worked. The documentation on the WriteToThem website is excellent, as is the guidance for how best to use the tool for effective campaigns.”

Humanity & Inclusion are actually a great example of an organisation who have read the guidelines and included them into their campaigning plans: if you visit the ’email your MP’ page of their campaign, you’ll see that they encourage you to write messages in your own words, while providing inspiration for some of the points that might be included.

This is because WriteToThem blocks mass copies of identical messages, based on evidence that these tend to be regarded as a nuisance by politicians, rather than having the desired effect.

Thank you very much to Tom and Humanity & Inclusion for sharing their experience of using WriteToThem as one part of a simple but effective online campaign.

And now, if you have been convinced of their cause, we suggest that you take advantage of their campaign pages, and email your MP.

—
Image: ©Peter Biro/HI
Nada, 10, was injured in a bombing with her father in Mosul. As a result of her injuries her leg was amputated below the knee and she will need jaw surgery to help make eating less difficult.

We are living in a historic age.

There are plenty of ways to see the truth of that right now. And here’s one more indicator: WriteToThem user numbers have exploded. Over the site’s lifetime (more than 15 years), we’ve never seen so many people using it to contact their representatives as we have during the last week or so.

We operate according to a strict privacy policy, so of course we can’t say for certain what people are writing to their MPs and councillors about. But it is worth noting that this boom has occurred while lockdown, the R rate, schools reopening, police brutality, racial inequality, and the toppling of statues were all part of the public conversation.

See that big spike on the right? That represents almost 35,000 messages sent to MPs in the first twelve days of June, against a normal monthly average of around 4,000.

In total, so far this month you’ve already sent 55,000 messages to every type of representative, as the UK’s coronavirus death toll rose ever higher, and Black Lives Matter protests spread from the US to the UK.

A previous peak on 24/25 May coincided with the Dominic Cummings story. That week, 11,756 messages were sent to MPs.

Referrals have largely come through social media, as people share the easy way to contact representatives about the issues that have gripped them — but there have also been welcome links from mainstream media, including youth culture and style publications like i-D and Dazed. We hope this might indicate a welcome broadening of our userbase to include more young and diverse citizens — and if so, we hope they’ll come back in the future every time they need to make contact with those in power.

WriteToThem exists so that anyone can contact their elected representatives, and feed into the democratic process. We make it as easy as possible for you to tell your politicians what you expect of them, to share your beliefs and opinions, and to ask for their support. We are glad that so many citizens are doing just that during this increasingly momentous era.

If you’d like to know more about what WriteToThem is and how it works, see this post.

—

Image: James Eades

When you woke up this morning to check the election results, you may have visited TheyWorkForYou.

And you’d have found it bang up to date, thanks to the new MP data that was added through the night, as the election results came in. More than a fifth of you have a new MP, and whether you voted for them or not we know you’ll want to keep them accountable.

Donate to help us keep this service going.

We’ve just now added one final MP — for St Ives, since weather conditions prevented ballot boxes coming over from the Isles of Scilly earlier.

We’ll be helping you hold all MPs, new and returning, to account over the next few years, as we publish their debates and votes, expenses, interests and contact details.

We make it as simple as possible for everyone to understand what’s going on in Parliament, and how you can play a part in your own democracy.

Right now, you can get a headstart:

• Check who your MP is
• Sign up to receive alerts whenever they vote or speak

If you’re a developer, researcher or just a good old data junkie, you might additionally like to:

• Use the API in the knowledge that it’s delivering the current MPs
• Download a spreadsheet of the current MPs

Now we need you to help us

We’re determined to carry on providing these services, but we still need your help to do so.

There are seven days left to run on our crowdfunder. Thanks to the generosity of hundreds of donors, we’ve already raised almost £10,000, for which we are enormously grateful.

But we still need to raise another £15,000 so that we can continue providing these services, as well as adding new features that will improve the site and make Parliament easier for everyone to follow.

Please donate now.

—

Image: ©UK Parliament/Jessica Taylor/ Stephen Pike (CC by-nc/2.0)

If you’re a user of mySociety’s Democracy-focused sites, such as TheyWorkforYou and WriteToThem, you may notice a few changes during the election period.

Generally speaking, the sites just work. Sure, there are a bunch of tasks we’re managing on a daily basis behind the scenes, but none of those need bother you, the user. To employ a tired old metaphor, the sites glide swanlike, while under the water there’s some busy paddling to ensure that the latest debates, votes and representatives’ contact details are all present and correct.

During an election, though, that paddling becomes a bit more visible, and some services may be interrupted.

WriteToThem

You want to contact your MP? Here’s the thing: officially, you don’t have one at the moment.

Parliament has dissolved. The representatives formerly known as MPs are no longer allowed to refer to themselves as such, and their parliamentary email addresses have been withdrawn.

So when you visit WriteToThem, you’ll see this message where we normally provide the link for writing to your MP:

Note that you can still use WriteToThem to contact all your other representatives, from local councillors to MSPs, Assembly members, MEPs, etc — provided that your issue is relevant to them (you’ll see a short list of the types of issue each representative deals with, on the site).

If you’ve got something to say about the current political situation or a matter that you’d like your MP to vote on, though, you’ll just have to wait. Even if your former MP is standing for re-election, they’re most likely dedicating a lot of their time to canvassing, and of course they won’t be taking any issues into the debating chamber just now because Parliament is not in session.

Where it becomes a little more tricky is if you have a constituency issue you want an MP to help with. Perhaps consider if it’s something your local councillor/s may be able to help with instead — it’s always worth asking them, anyway. If not, and if it’s an urgent matter, it may be worth calling your former MP’s office, as some (especially those standing for re-election) will still be running a bare bones service.

If your issue is not urgent, then wait until a couple of weeks after the election. In particular, if you find yourself with a brand new MP they’ll be finding their feet, setting up staff and office equipment, etc.

TheyWorkForYou

You’ll see the word ‘former’ used a lot, if you visit TheyWorkForYou over the next few weeks. For example, the homepage generally has a prominent link to direct you towards your own MP’s page. These days, it looks like this:

And if you do click through to any MP’s page, you’ll see that they now have this below their name:

On the page where we list all MPs, you’ll see this factually accurate message at the top:

If you want a list of who the MPs were, it’s still there, you just have to click the link.

And then there’s one more thing: of course, as there are no debates taking place in Parliament, we’re not sending out Westminster email alerts (you’ll still get those from Scottish Parliament and the London Assembly, though).

When will everything be back to normal?

Our friends at Democracy Club collate the election results as they come in, producing data that we can then import. Thanks to them we’re generally able to update TheyWorkForYou pretty much in real time. So, when you wake up in the morning you’ll hopefully be able to:

• Check who your MP is;
• If it’s someone new, sign up for alerts so you get an email when they speak.

For a little while, of course, new MPs will have very little content on their pages: you’ll see a message to say that data will start to appear once they’ve done a bit more.

WriteToThem takes a little longer to get back up to speed: that’s because we need to import all the MPs’ email addresses, and these can take a while to come through. If we’re using an official parliamentary email address, experience shows that they may not even be set up by Parliament for a short while.

So please be patient — as we mentioned earlier, it’s probably best to wait a couple of weeks before contacting your brand new MP in any case.

Meanwhile…

While mySociety sites are fully operational in the periods between elections, there are other organisations who swing into action and do their best work during this time.

So here are a few things you can do, thanks to those other orgs, while you wait for mySociety’s democracy services to return to normal.

• Visit WhoCanIVoteFor and WhereDoIVote from Democracy Club to discover who your local candidates for the General Election are, what they stand for, and where to find your nearest polling station.
• Upload scans of the political mailouts coming through your door to ElectionLeaflets, and help build a permanent archive of promises that elected representatives can be held to account for further down the line.
• Get the Who Targets Me extension on your browser to see clearly who is behind the political ads you’re being served on Facebook.

And finally: if you have questions about the whole electoral process, read the beginner’s guide to the UK General Elections we put together in 2017. While the names and dates have changed since then, the facts are still the same.

—

Image: Reproduced with the permission of Parliament

With so very much going on in politics right now, and so many MPs in the spotlight at any given moment, there has been a lot of sharing of TheyWorkForYou’s voting records on social media.

Of course, we’re all for it, if it helps people understand MPs’ voting history and the stances they’ve taken during their careers: we even include little share buttons beneath each voting record section to help you do this.

But as from a couple of weeks ago, you’ll also see a new addition to these sections: we’ve added a link saying ‘please share these voting records responsibly’ — and if you click on it, you’ll see a page setting out lots more information about votes, including the data that feeds the voting information on the site, and what you can — and what you definitely shouldn’t — conclude from it.

What TheyWorkForYou has always tried to do is take the complex, sometimes messy, often arcane and opaque business of Parliament and make it easy for the everyday person to understand, even if they don’t have a degree in Politics or lifelong membership of a political party.

The trouble is, as our users and MPs themselves can be very quick to point out, when you try to simplify a complicated area, some nuance is always lost. There are things everyone should know before they charge onto Twitter or Facebook, hoping to win an argument or denigrate an MP by brandishing their record on foreign policy or social issues. And so we’ve set these points out on one page.

A key question that arises when writing a page like this is: if we can’t present everything (either because the data doesn’t exist, or because including it would complicate the overall picture so much that we would risk losing our aim of making things easy to understand) should we present anything at all?

We ask ourselves this question fairly often, and so far our answer has always been ‘yes’. Please read our page so that you fully understand the reasons behind the decisions we make.

—

Image: MP speaking at Theresa May’s last Prime Minister’s Questions, 24 July 2019, CC-BY-NC, Copyright UK Parliament / Jessica Taylor.