An analysis, with code and data, of which Commons votes would have had different results, if Scottish MPs’ votes hadn’t been counted since 1997.
By Richard Taylor and Anna Powell-Smith.
PublicWhip is a wonderful thing. Founded and still run by independent volunteers, it contains the results of every House of Commons vote since 1997, scraped from the official web pages and presented as simple structured data. Here at mySociety, we’ve used it to power TheyWorkForYou for many years.
Most recently, it helped our staffer Richard create the new voting analyses on TheyWorkForYou’s MP pages. Want a quick, simple summary of your MP’s voting history on same-sex marriage or climate change, or on any of 62 other major issues? You’ll now find the answer on your MP’s TheyWorkForYou page, all based on PublicWhip data.
But here’s the most exciting thing about PublicWhip. If you know how to get around its slightly forbidding exterior, it contains a treasure-trove of data on MPs’ voting patterns, all structured, openly-licensed and ready for anyone to analyse.
A data challenge
Recently, while discussing the upcoming Scottish referendum, Richard posed a question to Anna: could PublicWhip data tell us which House of Commons votes would have had different results, if Scottish MPs’ votes hadn’t been counted?
This is interesting because if the Scottish people vote “yes” to independence on September 18th, we may see (probably not as soon as 2015, but perhaps soon thereafter) a House of Commons without Scottish MPs. No-one really knows how such a Parliament would be different.
While it was widely reported that that Scottish MPs’ votes carried the decision to introduce student tuition fees and foundation hospitals in England, those were just two high-profile votes. To our knowledge, no-one has published a comprehensive analysis of all votes that were carried by the Scottish MPs.
Anna chose to accept Richard’s challenge, and to use PublicWhip data to carry out this analysis. You can see all their code, and the data they produced, on GitHub.
The headline finding is that only 21 votes (out of nearly 5000 since 1997) would have gone differently if Scottish MP’s votes hadn’t been counted. This surprised Anna, who expected more.
Secondly, if there’s any visible pattern, it’s that English MPs seem to have a stronger civil-libertarian bent than their Scottish counterparts. High-profile votes on 42-day detention, “glorifying terrorism”, allowing the Lord Chancellor to suspend inquests, and on control orders: according to Anna’s analysis, all would have gone differently if Scottish MPs had not been in the chamber.
Other than that – Anna comments – the key finding is perhaps the absence of any other strong trend.
Here is the full list of votes that would have gone differently – click on the date to see the full vote details on PublicWhip. If Scottish MPs hadn’t been in the chamber:
- 5 Sep 2014 The majority of MPs would have voted to send the Affordable Homes Bill to a Select Committee rather than a Public Bill Committee.
- 29 August 2013 The majority of MPs would have voted to agree that a strong humanitarian response to the use of chemical weapons in Syria was required from the international community, and that it may, if necessary, require military action. (You may remember that David Cameron called MPs back from their summer break to vote on this, and MPs rejected the motion.)
- 29 Jan 2013 The majority of MPs would have voted against postponing a review of the boundaries of parliamentary constituencies until 2018 and against delaying a review of the effect of reducing the number of MPs.
- 31 Oct 2012 The majority of MPs would have voted against calling on the UK Government to seek a real-terms cut in the European Union budget.
- 24 Apr 2012 The majority of MPs would have voted to require products containing halal and kosher meat to be labelled as such.
- 24 Feb 2010 The majority of MPs would have voted for restrictions on the amount of carbon dioxide electricity generation plants are permitted to emit.
- 9 Nov 2009 The majority of MPs would have voted against allowing the Lord Chancellor (a minister) to suspend an inquest and replace it with an inquiry and against allowing the use of intercepted communications evidence in inquests.
- 8 Dec 2008 The majority of MPs would have voted to immediately starting the proceedings of a committee of MPs to investigate the House of Commons procedures in light of the seizure by the police of material belonging to Damian Green MP.
- 12 Nov 2008 The majority of MPs would have voted to require membership of new regional select committees to be determined taking account of the proportion of members of each party representing constituencies in the relevant region and for at least one member from each of the three largest parties to be on each committee.
- 11 Jun 2008 The majority of MPs would have voted against extending the period of police detention without making any criminal charges of terrorist suspects from 28 days to 42 days.
- 2 Jun 2008 The majority of MPs would have voted to require the National Policy Statement to contain policies which contribute to the mitigation of, and adaptation to, climate change.
- 15 Mar 2006 The majority of MPs would have voted against a proposed timetable for the Parliamentary consideration of the Education and Inspections Bill.
- 2 Nov 2005 The majority of MPs would have voted against making glorifying the commission or preparation of acts of terrorism an offence.
- 2 Nov 2005 The majority of MPs would have voted to make the offence of Encouragement of Terrorism only apply to cases where an individual intended their actions to encourage terrorism.
- 28 Feb 2005 The majority of MPs would have voted to give a greater role to the courts in relation to the imposition of control orders.
- 22 Apr 2004 The majority of MPs would have voted against installing a security screen separating the public gallery from the House of Commons Chamber.
- 31 Mar 2004 The majority of MPs would have voted against the introduction of variable university tuition fees (top-up fees) of up to £3,000 per year in place of the previous fixed fee of £1,250 per year.
- 27 Jan 2004 The majority of MPs would have voted against allowing university tuition fees to increase from £1,125 per year to up to £3,000 per year, and against making other changes to higher education funding and regulation arrangements.
- 19 Nov 2003 The majority of MPs would have voted against introducing NHS foundation trusts, bodies with a degree of financial and managerial independence from the Department of Health.
- 4 Feb 2003 The majority of MPs would have voted for an 80% elected House of Lords.
- 29 Oct 2002 The majority of MPs would have voted against starting sittings of the House of Commons on Tuesdays at 11.30am rather than 2.30pm.
In the 1997-2001 Parliament, Anna’s code found no votes that would have had different results.
IMPORTANT DISCLAIMER! We can’t conclude that all of the above would necessarily have become law if Scottish MPs had not been in the chamber. Bills don’t become law until they have passed through the House of Lords – not to mention the many other forces of history that would have acted differently.
Get the code and the data
You can see the code used for this analysis, and the full datasets, on GitHub. You can adapt it yourself if you want to do your own analyses.
This analysis is the work of one volunteer: we welcome any corrections. Like PublicWhip itself, the whole point is that it is out in open for anyone to analyse and improve.
Image by Catherine Bebbington. Parliamentary copyright image reproduced with the permission of Parliament.
One of the most popular features on TheyWorkForYou is the plain English, non-judgemental vote analyses on MP pages that say things like “voted strongly against introducing a smoking ban“. We call these ‘policies’, and they are authored by skilled people using the volunteer run PublicWhip website.
Making each one of these policies is a painstaking task, requiring good knowledge of how the Parliamentary voting system works, good writing skills, patience, and the strength of character not to let your own views about the issues cloud the analysis. It is of utmost importance to both mySociety and our users that these policies are fair and trustworthy.
Earlier this year we started to update the process by which we made new policies to make it even more rigorous, which we wrote about here. Marcus Fergusson and Stephen Young came onboard and did sterling work, but they have now moved on to greater things, and so we’re looking to recruit two to three new people to do this job. Uber volunteer Richard Taylor has been helping out recently, but this is really a job for two or more people.
You might very well ask ‘why two people, given the work is part time?’. The answer is that we really want every new policy to be cross-checked by two different people every time it is added or amended. This is to help eliminate possible mistakes, and prevent any unconscious biases.
We pay for this work on a piece work basis – £160 a time for a combination of one new policy authored, and one other policy double-checked. This money comes mainly from people making small donations, which I think helps keep everyone focussed on how important it is to get these right. We hope to add about two new policies a month, once the new team is up to speed.
If you’d like to be considered, please email firstname.lastname@example.org with ‘mspolicies’ in the subject line. Applications close 22nd November 2010.
mySociety is lucky enough to have a number of small donors who give us monthly donations, normally ranging from £5-£20 (if you like our work and want to support us, please do join them!). Today we’re announcing a change to TheyWorkForYou which is supported by these donations.
One of the most popular features on TheyWorkForYou is the vote analysis – the bit that tells you that your MP “voted strongly against introducing a smoking ban” and so on. These voting analyses cut through a massive wall of parliamentary opacity whilst still allowing visitors to examine the details first hand. Despite each analysis resulting in just a single line on TheyWorkForYou, each one is rather time-consuming to construct, and TheyWorkForYou has not updated them as much as our users deserve.
Thanks to our small donors we’ve now been able to commission two part time researchers, Marcus Fergusson and Stephen Young, to help add new vote analyses more regularly. We’re pleased to say that we’ve just rolled out the first new policies, covering issues relating to schools, inquests and the House of Lords. We aim to add a couple of new vote analyses a month for the foreseeable future.
We take the business of authoring analyses that are scrupulously fair and neutrally worded extremely seriously. To this end we have replaced our previously ad-hoc approach with a newly instituted process designed to ensure the maximum rigour and balance, and to ensure we focus on issues which MPs thought were important even if they were not so well covered by the media.
The new process for analysing MP’s positions works like this:
- A list of votes in the current Parliament, ordered with the highest turnout at the top, is taken as a starting point. The turnout figure used is corrected to account for party abstentions.
- mySociety’s researchers work down that list writing explanations in easy to read terms describing what the vote was about; they also identify other related votes on the same issue and research those too.
- A “policy position on the issue” is then chosen against which MP’s votes are compared to determine to what extent they agree or disagree with it. Policy positions are written to be intelligible and interesting to a wide range of users and in such a way that votes to change the status quo are ultimately described on TheyWorkForYou MP pages as votes “for” that change.
We hope you find these analyses useful. Thanks to Richard Taylor for his divisions list and help with this post.
A few weeks ago mySociety and Politik Digital held a small unconference in Berlin. The idea was to get together some of the best practioners building and running democracy websites across Europe, regardless of their size or status.
I’ll try to write this up more fully soon, but for the moment I wanted to share some email interviews I did with some of the participants after the event. The first is with Guglielmo Celata from the Italian group D.E.P.P. We first came across them a couple of years ago when they borrowed some code from PublicWhip.org.uk (the independent volunteer vote analysis project run by Julian Todd and mySociety senior developer Francis Irving) for their website OpenPolis.
Anyway, enough for the context – D.E.P.P have some great, boundary-pushing work coming up and I thought people in the English speaking community would want to know.
What is the organisation you work for?
The name of the Association is D.E.P.P., that stands for Electronic Democracy and Public Partecipation.
It’s a relatively small group of people (four) who work on e-partecipation projects with local administrations (the municipality of Rome and the Regione Lazio, for examples). We also have an self-financed project, named Openpolis, to map politicians, their charges, their declarations, both at a national and at a local level. More on this later.
What is the main purpose of the site(s) that you run?
We have a project named eDem 1.0 which has been so far installed twice: municipiopartecipato.it focuses in enabling e-participation of local communities on the “participatory budget”; and edem-regione on the budget of the Regione Lazio (the link points to an alpha version).
I think the participatory budget for the local community is far more interesting. The site shows a list of issues categorized by theme and territory. Registered users can vote up issues and make them emerge as important. Issues are created by the users. Users can also create proposals related to issues and vote them. The integration with Google Maps, allows user to see how issues and proposals distribute in their territory; it makes the user interface immediate (and of course makes the site sooo stylish).
The proposals emerging as the most voted are approved and follow a workflow to be actually financed and implemented.
Online activities and offline physical assemblies (which exist), are linked together by a group of paid people, called enablers. They take care of moderating both offline and online activities, too.
The other project has almost the same features, but applied to the budget document of the Regione Lazio. Of course, the issues here are not created by the citizens, being the chapters (or sections) of the official budget document. The citizens can create and rate proposals, but such proposals are never going to be implemented.
This happens a lot, administrators are intersted in e-participation projects, but they want to reduce the possibility of issues emerging directly from citizens, and€ of course they try to change the nature of the project from a participative one, into a consultative one. A kind of Poll 2.0, if one wants to be cynical.
Can you tell us about your next site, the one you showed us in Berlin?
Openpolis is a project to gather informations on our political class and make them transparent. How they vote once elected, what laws they propose, their charges in institution, political parties and private organisations, public declarations, financial interests, judicial positions etc. The aim of the project is to revive the bond between the citizens and their representatives. We would like to give individuals or organized group of citizens, a set of tools to enable them to perform lobbying activities.
We want to work both at a national level and at a very local level, and to do this we plan to allow users to create part of the content on the site, and hope this way to create communities, wiki-style.
However, the site is not a wiki, since content has to be well-structured; we want to export statistics and make analysis on data added by users.
You are planning to combine information gathered from formal sources, and submitted by users. Can you tell us where you’re getting the formal information from, and how you are going to handle the information submitted by users?
We have different levels, and correspondingly different sources. At a national level, we are harvesting the official web sites of the Camera and Senato (the two houses of national representatives) and the web site of European Parliament. At local levels we rely on official biographical data from the Ministero degli Interni (Interior Ministry). We double check politician’s data for the 20 major cities in italy, but of course can’t possibly dream of doing that for the 109 provinces and 8100 municipalities.
For data on charges, declarations, financial interests and judicial positions, and for a complete double check on details and biographical data, we plan to leverage the community of users. The more users, the more data and verification.
Of course, data inserted by the users must be always connected to sources (i.e a web link, a reference to a book, an article in a newspaper, or a radio or television program). Data will be verified by moderators, and the community of moderators will grow on trust basis (using a karma-based system, so that when a subscribed user reach a certain treshold of trust, he is proposed as moderator to the board of administrators). We all know that this part is a real challenge and that handling a community online is a daunting task, but, hey, let’s try.
Users can be banned and content can be censored (after publishing), but any banning or censorship will be performed transparently, so that anyone, in any moment will be able to know the reason why a user was banned.
Do you ever face claims that the effects you have on politicians aren’t entirely positive? If so, how do you respond?
We actually have not yet started, but we do plan on receiving a lot of such claims. Of course we are trying to create something that the politicians should use, as well, so the most interested and active users should be the politician themselves.
Are there any other features of your site that you think are unusual or unique?
We plan to release an API, in order to make integration of our data and analysis possible directly from other web applications. Starting from RSS feed, to a proper API, it should be possible to integrate pieces of our applications directly into people’s blog or other similar applications.
What other projects around the world excite you the most, and why?
Well, of course the TheyWorkForYou project was a real kick off, we just thought: “wow, we have to do that here in Italy!” Then I really appreciate the work at GovTrack.us, especially from the technical standpoint, for the innovative way of using RDF and the Semantic Web approach.
Here in Italy, a project I forgot to mention in Berlin is: http://fainotizia.radioradicale.it.
FaiNotizia means Make Your Own News, it’s a project by Radio Radicale, an historical radio broadcast of the Italian Radical Party. It provides one of the first citizens journalism website in Italy and we plan to integrate with them in the future.
Do you use the law to help you get information? If so, how have you gone about it, and what have you obtained?
We haven’t so far, every information that we gathered was publicly available, we just wrote tons of parser code.
We plan to push the release of data on financial interests and judicial positions, though. Those data are public, but poorly accessible (no electronic format, no scanning, phtos or copies possible). This will require some legal actions or some fantasy to get them. We’ll see.
So there we are. If you’ve any further questions or clarifications, just post a comment here and I’ll update this post with Guglielmo’s help.
The first new councillor details have begun to automatically arrive in our database, thanks to GovEval. 34 councils were reactiviated on WriteToThem today, from Alnwick to Wokingham. It would have been 38 but the other 4 councils have had boundary changes that we don’t have the data for yet.
16 of the 40 Welsh Assembly constituencies did not change their boundaries at the election (this took some time to work out, as the Press Association said it was 18, and the official report from the Boundary Commission for Wales said it was 17 🙂 ). Those 16 Assembly Members are now also reactiviated on WriteToThem, along with their regional AMs.
Other than that, I’ve continued tweaking Neighbourhood Fix-It and started some work on TheyWorkForYou – the first step of which is to deal with the large backlog of mail that’s accumulated, leading to a number of bugfixes. Apologies to anyone who was trying to look at Brian Wilson MLA‘s page and found themselves stuck in an infinite loop of being told there were two Brian Wilsons. We also had a couple of emails asking us why Gordon Brown didn’t have a voting record on equal gay rights like other MPs. This was easy to answer – he’s never voted in any division that is included by that PublicWhip policy – and so an MP’s page now states if they’ve been absent from every vote on a particular policy.
I seem to keep ending up in the North of England on work this Autumn, which is good as that’s where I’m from.
The Liverpool event on Friday went really well. Over the day, we probably had about 10 people altogether. It ended up more like a rolling seminar series – we all introduced and talked about what we were up to. Of course, Matthew on the TheyWorkForYou API, Tom on what mySociety doing these days. Also Julian (who does lots of the Parliament screen scraping that TheyWorkForYou depends on) talked about using Public Whip for collaborative research on Parliamentary votes. We spent a while talking about Fredom of Information with Steve Wood, who lives in Liverpool.
Ben (noii) told us about Tad Hirsch at MIT, who has done lots of interesting projects, which are inspiring for mySociety style project ideas. I’m sure we can do a lot with automatic routing of voice calls to volunteers, and with text message lists. Another Ben showed us an early version of some great work he is doing extracting information about newspapers and journalists. Thanks very much to Aidan from Blue Fountain who hosted us in the beautiful India building. We’ll be having two more events to promote the TheyWorkForYou API, and talk about other mySociety matters, later in the year. One in London, one somewhere else, suggestions welcome.
Earlier in September, all of us went to work together in the Lake District (as mentioned in my last post). It’s important to meet up for a solid chunk of time in the year when we spend most of it spread out across the Internet. Matthew took lots of photos, one of which is above (Tom, Louise, Matthew, Chris, Francis and Anna from left to right). It might look from the photo like we were brainstorming solutions to thorny problems, clearing back logs of customer support, writing code, and creating todo lists as long as your arm. But actually we just climbed mountains, hung out by tarns, drank beer and had a good time.
And next week I’m back off up North, to Betws-y-Coed in Wales this time. To give a talk to Bloc about TheyWorkForYou.
mySociety is really keen to know what we can build, modify or improve to make our services more useful to people who run blogs. We’ve not really done any work on things like banners, buttons, postcode boxes or interesting snippets of free HTML to make our work more attractive to stick down the side of blogs, and we’d like to hear what we could do.
So, if you’re in London, join us in the Dover Castle tomorrow night (Tuesday 20th December 2005) from 7.30PM, or simply drop us an email with suggestions. We’ll even take suggestions for our non-mySociety sister sites TheyWorkForYou.com and Publicwhip.org.uk too.
Over the weekend, and this morning, I’ve been updating the Public Whip and TheyWorkForYou database of MPs. This was much easier this year thanks, amazingly to Macromedia Flash. The BBC have a fantastic animated constituency map which is made in flash. When you click on a constituency it gives you the results. Now, one little know thing about flash is that it uses XML to communicate with the server. This means that any data it downloads must be in XML.
Further investigating reveals that you can download results data from URLs like http://news.bbc.co.uk/1/shared/vote2005/flash_map/resultdata/200.xml. So I wrote a script to download these and convert them to XML. May I present a list of all the new MPs. Now, just need to wait for them to start talking and voting so we can build up a record of them. Meanwhile, on to some mySociety work; fixing up WriteToThem…
Praise be to Macromedia!
This morning I woke up quite late because I didn’t sleep well (and had been building Public Whip’s How To Vote). Today I’m not really working for mySociety, as I’m going to London this afternoon, but Tom had a couple of important bits for me to do. There was a bug in the admin page, and a missing link on the confirmation page. I fixed these up, improved the test suite a bit, and deployed to the main site.
PledgeBank is now ready for people to start using it in earnest. We’re still in testing, as we’re sure there’ll be lots of changes needed to it as it is used in the real world. But all the basic features, and fancier ones such as SMS and the auto-generated flyers, work. Tom’s just been on the radio, and he’s making lots of specific example pledges like this one about Shropshire. So, the hunt is now on.
My next jobs are to tidy up outstanding tickets which we have already fixed, and fix any bugs in there. The next feature we’re adding to the site is comments, so people can discuss the pledge.