If you need data on the people who make up your parliament, another country’s parliament, or indeed all parliaments, you may be in luck.
What’s more, it’s all provided as Open Data to anyone who would like to use it to power a civic tech project. We’re thinking parliamentary monitoring organisations, journalists, groups who run access-to-democracy sites like our own WriteToThem, and especially researchers who want to do analysis across multiple countries.
But isn’t that data already available?
Yes and no. There’s no doubt that you can find details of most parliaments online, either on official government websites, on Wikipedia, or on a variety of other places online.
But, as you might expect from data that’s coming from hundreds of different sources, it’s in a multitude of different formats. That makes it very hard to work with in any kind of consistent fashion.
Every Politician standardises all of its data into the Popolo standard and then provides it in two simple downloadable formats:
- csv, which contains basic data that’s easy to work with on spreadsheets
- JSON which contains richer data on each person, and is ideal for developers
This standardisation means that it should now be a lot easier to work on projects across multiple countries, or to compare one country’s data with another. It also means that data works well with other Poplus Components.
What can I do with it?
Need a specific example? Yesterday, we introduced Gender Balance, the game that gathers data about women in politics.
As you’ll know if you’ve already given it a try, Gender Balance works by displaying politicians that make up one of the world’s legislatures, one by one.
That data all comes from Every Politician, and it’s meant that the developers have been able to concentrate on making a smooth and functional interface, knowing that the data side of things has already been taken care of.
That’s just one way to use Every Politician data, though. If you’d like to use it in your own site or app, you can find out more here.
We still need more data
As you may have noticed, there are more than 100 parliaments in the world. In fact, despite having reached what feels like a fairly substantial milestone, we’re still barely half way to getting some data for every parliament.
So we could use your help in finding data for the parliaments we don’t yet cover, and historic information for the ones we do. Read more about how you can help out.
From Afghanistan to Zimbabwe, there are over 180 parliaments in the world — but what percentage of their members are female?
The crazy thing is, there’s no definitive figure*.
So we created Gender Balance, an easy game that crowd-sources gender data across every parliament in the world. Try it! We hope you’ll find it fun.
Gender Balance isn’t just an enjoyable way to fill half an hour, though: users will be helping to build up a dataset that will be useful for researchers, campaigners, politicians, and sociologists. As the results emerge, we’ll be making them available in an open format for anyone to use, to answer questions like:
- Which country has the highest proportion of women in parliament?
- Do women vote differently on issues like defence, the environment, or maternity benefits?
- Exactly when did women come into power in different countries, and did their presence change the way the country was run?
Gender Balance’s underlying data comes from another mySociety project—EveryPolitician, a database which aims to collect information on every politician in the world.
And while it’s nailing down those stats on gender balance across every country, Gender Balance also aims to be a showcase of what can be done with the open data from EveryPolitician. That data is free for anyone who wants to build tools like this, and it’s easy to use, too. Find out more about that here.
*While the Inter-Parliamentary Union does collect figures, they are self-reported, often out of date, and only cover its own members.
Private data, containing personal details of the general public, is accidentally released by public authorities at least once a fortnight, say mySociety.
The volunteer team behind WhatDoTheyKnow, mySociety’s freedom of information website, have dealt with 154 accidental data leaks made by bodies such as councils, government departments and other public authorities since 2009, and these are likely to represent only the tip of the iceberg.
On the basis of this evidence, we are again issuing an urgent call for public authorities everywhere to tighten up their procedures.
How WhatDoTheyKnow works
Under the Freedom of Information act, anyone in the UK may request information from a public body.
WhatDoTheyKnow makes the process of filing an FOI request very easy: users can do so online. The site publishes the requests and their responses, creating a public archive of information.
Public authorities operate under a code of conduct that requires personal information is removed or anonymised before data is released: for example, while a request for the number of people on a council housing waiting list may be calculated from a list including names, addresses and the reason for housing need, the information provided should not include those details.
Accidental data releases become particularly problematic when the data requested concerns the details of potentially vulnerable people.
Hidden data is not always hidden
When users request information through WhatDoTheyKnow, it’s often provided in the form of an Excel spreadsheet. But unfortunately, private data is sometimes included on those spreadsheets, usually because the staff member who provides it doesn’t understand how to anonymise it effectively.
For example, data which is in hidden tabs, or pivot tables, can be revealed by anyone who has basic spreadsheet knowledge, with just a couple of clicks.
By its very nature, data held by our public authorities can be extremely sensitive: imagine, for example, lists of people on a child protection register, lists of people who receive benefits, or as happened back in 2012, a list of all council housing applicants, including each person’s name and sexuality.
Our latest warning is triggered by an incident earlier this month, in which Northamptonshire County Council accidentally published data on over 1,400 children, including their names, addresses, religion and SEN status. Thanks to the exceptionally fast work of both the requester and the WhatDoTheyKnow volunteers, it was removed within just a few hours of publication, and the incident has been reported to the Information Commissioner’s Office. Concerned residents should contact the ICO or the council itself.
Advice for FOI officers
Back in June 2013, we set out the advice that we think every FOI officer should know. That advice still stands:
- Don’t release Excel pivot tables created from spreadsheets containing personal information, as the source data is likely to be still present in the Excel file.
- Ensure those within an organisation who are responsible for anonymising data for release have the technical competence to fulfil their roles.
- Check the file sizes. If a file is a lot bigger than it ought to be, it could be that there are thousands of rows of data still present in it that you don’t want to release.
- Consider preparing information in a plain text format, eg. CSV, so you can review the contents of the file before release.
Part of a larger picture
Not every FOI request is made through WhatDoTheyKnow—many people will send their requests directly to the public authority. Moreover, we can only react to the breaches that we are aware of: there are, in all probability, far more which remain undiscovered.
But because of WhatDoTheyKnow’s policy of making information accessible to all, by publishing it on the site, it’s now possible to see what an endemic problem this kind of treatment of personal data is.
When we come across incidents like these, we act very rapidly to remove the personal information. We then inform the public authority who provided the response. We encourage them to self-report to the Information Commissioner’s Office, and where the data loss is very serious, we may make an additional report ourselves.
Have you ever typed a phrase like ‘what’s the time in New York’ or ‘what is 28 km in miles’ into Google? If you’ve done so in the last couple of years, you’ll have seen answers given on the results page itself, inside dedicated answer boxes.
We’re delighted to share the news that these answers are being supplied as open standard data from Democracy Club’s brilliant, volunteer-powered YourNextMP project, as well as our own long running TheyWorkForYou. The aim is to make information easily accessible to anyone who seeks it: we are one of a few data sources to be supplying Google.
We think that adoption of open standards data by companies as big as Google points towards a promising world in which there is a lot more good quality, open standard data on political issues of all kinds.
How was this made possible?
We’re so delighted to see essential civic data being brought to the search engine’s vast numbers of users. It’s all down to the power of open standards and re-usable open source software.
YourNextMP provides feeds via an API, which are available for anyone—large organisations like Google, or individual people like you, perhaps—to use in their own projects.
YourNextMP’s use of the Popolo open standard for government means that the data is clean, machine-readable and easy to slot in anywhere—including Google.
We think a few shout-outs are due. This simple but far-reaching usage of YourNextMP data is only possible because of many good people bringing good things together:
- Democracy Club, and the huge amount of work that this volunteer-run organisation put in to gather and check candidate data
- The many volunteers who gave their time in adding and refining that data
- Popolo, driven by the work of James McKinney
- Poplus, because YourNextMP is based on the people-and-positions storing Component, PopIt
- Google.org, whose funding allowed the creation of Poplus in the first place.
What YourNextMP has achieved
Just as was hoped, YourNextMP data has been used to underpin a variety of projects by many individuals and organisations.
Google may be the latest and the biggest, but we’re no less pleased to see how it has made possible numerous tools to educate or inform the public before the election, as well as powering stories and infographics in several national newspapers. See our previous post for more details on this.
And there’s more. YourNextMP, because it’s built on Open Source code, won’t be going into hibernation until the next election in this country.
The UK voting may be over on May 8, but there are elections all over the world still to come. Our friends in Latin America will be taking the code and adapting it for use in Argentina whose election process starts in August. ¡Viva YourNextMP!
The Knight Foundation’s News Challenge offers funding to innovative projects. We wonder whether they’ve ever had a bid whose collaborators span six different countries before.
You can read more about the plans on the bid page—and please click the little pink heart to give us ‘applause’!
In short, we want to build on the success that the YourNextMP crowdsourcing platform has had here in the UK.
Right now, YourNextMP offers open data on every candidate for the UK general election. That data is being used by major media companies and internet giants, and underlies several innovative online tools. On top of that, it’s getting thousands of visits every day from people who simply want more information about who’s standing in their area.
With some modification, other countries could use the same tech in advance of their own elections, giving their citizens the same opportunities to become more informed about those standing, and to develop still more useful online tools.
This is a ‘Yay for Poplus’ moment
Because Poplus is an international federation of organisations with similar needs, we can come together to forge plans that will benefit all of us, and then work together to make them a reality.
Our plans wouldn’t just benefit those six countries, either. Like every bit of Poplus tech, it’d be available as open source software for anyone to use, anywhere in the world. And that’s what Poplus is all about: maximum impact from every bit of code.
Join the Poplus mailing list to find out more about Poplus activities
Give some applause to our Knight Foundation News Challenge bid
Users of Lothian Buses are more satisfied with the value for money of their bus journeys than anyone else in the country.
Passengers on the Oxford Park and Ride service find the seats the most comfortable. And the drivers of Trent Barton in Nottinghamshire give a friendly enough greeting, according to 95% of passengers.
It’s an extension of the work we did last year to help the transport watchdog display their train satisfaction data. We’ve introduced a new design which, we hope, makes it much easier to explore the results of Passenger Focus’ annual passenger satisfaction survey.
We’ve used a new visual approach that is appropriate for the bus data: it makes it really easy to browse through 32 different survey categories, from cleanliness to safe driving.
When you have that many categories, drop-downs aren’t really an option, and we’re pleased with what we came up with to make it easy to make the most important categories prominent, while still allowing easy and intuitive access to the others.
We’ve also used responsive design, which means it performs beautifully whether viewed on mobile or at the desktop. Check it out for yourself here – be sure to resize your browser to see the mobile version kick in!
We at mySociety build and popularise digital tools worldwide that help citizens exert power over institutions and decision-makers. Or do we?
Wanting to know whether our well-meaning civic tech is actually making a difference, mySociety recently created the post of Head of Research. My name is Dr Rebecca Rumbul, and I have now been installed in that role for about 6 weeks. I want to know if civic tech like ours is having an impact on citizens and governments, and how such sites operate and negotiate issues not just in the UK, but in the 50 or so countries that we know have digital democracy websites operating in them.
There is enormous scope for interesting and important research to be conducted using sites such as the ones that mySociety and our partners operate. The digital nature of our focus means that we can collect large volumes of data online at a low cost.
That said, there is nothing quite like making connections on the ground or meeting people face to face. mySociety is a small NGO, and does not have the capacity to conduct all of the research activities it would like on its own.
Therefore, we are actively seeking to work with academic partners on both qualitative and quantitative research focusing on the impact of civic tech.
We are planning to conduct research in the following countries. If you are an academic based in one of these countries and interested in our research agenda, please get in touch. We will be very happy to hear from you. Contact me via firstname.lastname@example.org
- South Africa
We conduct and disseminate research regularly. If you would like to hear more about our activities and events, sign up for our newsletter.
Image credit: Into the Unknown by Gary Gao (angrytoast), CC BY-NC 2.0
An analysis, with code and data, of which Commons votes would have had different results, if Scottish MPs’ votes hadn’t been counted since 1997.
By Richard Taylor and Anna Powell-Smith.
PublicWhip is a wonderful thing. Founded and still run by independent volunteers, it contains the results of every House of Commons vote since 1997, scraped from the official web pages and presented as simple structured data. Here at mySociety, we’ve used it to power TheyWorkForYou for many years.
Most recently, it helped our staffer Richard create the new voting analyses on TheyWorkForYou’s MP pages. Want a quick, simple summary of your MP’s voting history on same-sex marriage or climate change, or on any of 62 other major issues? You’ll now find the answer on your MP’s TheyWorkForYou page, all based on PublicWhip data.
But here’s the most exciting thing about PublicWhip. If you know how to get around its slightly forbidding exterior, it contains a treasure-trove of data on MPs’ voting patterns, all structured, openly-licensed and ready for anyone to analyse.
A data challenge
Recently, while discussing the upcoming Scottish referendum, Richard posed a question to Anna: could PublicWhip data tell us which House of Commons votes would have had different results, if Scottish MPs’ votes hadn’t been counted?
This is interesting because if the Scottish people vote “yes” to independence on September 18th, we may see (probably not as soon as 2015, but perhaps soon thereafter) a House of Commons without Scottish MPs. No-one really knows how such a Parliament would be different.
While it was widely reported that that Scottish MPs’ votes carried the decision to introduce student tuition fees and foundation hospitals in England, those were just two high-profile votes. To our knowledge, no-one has published a comprehensive analysis of all votes that were carried by the Scottish MPs.
Anna chose to accept Richard’s challenge, and to use PublicWhip data to carry out this analysis. You can see all their code, and the data they produced, on GitHub.
The headline finding is that only 21 votes (out of nearly 5000 since 1997) would have gone differently if Scottish MP’s votes hadn’t been counted. This surprised Anna, who expected more.
Secondly, if there’s any visible pattern, it’s that English MPs seem to have a stronger civil-libertarian bent than their Scottish counterparts. High-profile votes on 42-day detention, “glorifying terrorism”, allowing the Lord Chancellor to suspend inquests, and on control orders: according to Anna’s analysis, all would have gone differently if Scottish MPs had not been in the chamber.
Other than that – Anna comments – the key finding is perhaps the absence of any other strong trend.
Here is the full list of votes that would have gone differently – click on the date to see the full vote details on PublicWhip. If Scottish MPs hadn’t been in the chamber:
- 5 Sep 2014 The majority of MPs would have voted to send the Affordable Homes Bill to a Select Committee rather than a Public Bill Committee.
- 29 August 2013 The majority of MPs would have voted to agree that a strong humanitarian response to the use of chemical weapons in Syria was required from the international community, and that it may, if necessary, require military action. (You may remember that David Cameron called MPs back from their summer break to vote on this, and MPs rejected the motion.)
- 29 Jan 2013 The majority of MPs would have voted against postponing a review of the boundaries of parliamentary constituencies until 2018 and against delaying a review of the effect of reducing the number of MPs.
- 31 Oct 2012 The majority of MPs would have voted against calling on the UK Government to seek a real-terms cut in the European Union budget.
- 24 Apr 2012 The majority of MPs would have voted to require products containing halal and kosher meat to be labelled as such.
- 24 Feb 2010 The majority of MPs would have voted for restrictions on the amount of carbon dioxide electricity generation plants are permitted to emit.
- 9 Nov 2009 The majority of MPs would have voted against allowing the Lord Chancellor (a minister) to suspend an inquest and replace it with an inquiry and against allowing the use of intercepted communications evidence in inquests.
- 8 Dec 2008 The majority of MPs would have voted to immediately starting the proceedings of a committee of MPs to investigate the House of Commons procedures in light of the seizure by the police of material belonging to Damian Green MP.
- 12 Nov 2008 The majority of MPs would have voted to require membership of new regional select committees to be determined taking account of the proportion of members of each party representing constituencies in the relevant region and for at least one member from each of the three largest parties to be on each committee.
- 11 Jun 2008 The majority of MPs would have voted against extending the period of police detention without making any criminal charges of terrorist suspects from 28 days to 42 days.
- 2 Jun 2008 The majority of MPs would have voted to require the National Policy Statement to contain policies which contribute to the mitigation of, and adaptation to, climate change.
- 15 Mar 2006 The majority of MPs would have voted against a proposed timetable for the Parliamentary consideration of the Education and Inspections Bill.
- 2 Nov 2005 The majority of MPs would have voted against making glorifying the commission or preparation of acts of terrorism an offence.
- 2 Nov 2005 The majority of MPs would have voted to make the offence of Encouragement of Terrorism only apply to cases where an individual intended their actions to encourage terrorism.
- 28 Feb 2005 The majority of MPs would have voted to give a greater role to the courts in relation to the imposition of control orders.
- 22 Apr 2004 The majority of MPs would have voted against installing a security screen separating the public gallery from the House of Commons Chamber.
- 31 Mar 2004 The majority of MPs would have voted against the introduction of variable university tuition fees (top-up fees) of up to £3,000 per year in place of the previous fixed fee of £1,250 per year.
- 27 Jan 2004 The majority of MPs would have voted against allowing university tuition fees to increase from £1,125 per year to up to £3,000 per year, and against making other changes to higher education funding and regulation arrangements.
- 19 Nov 2003 The majority of MPs would have voted against introducing NHS foundation trusts, bodies with a degree of financial and managerial independence from the Department of Health.
- 4 Feb 2003 The majority of MPs would have voted for an 80% elected House of Lords.
- 29 Oct 2002 The majority of MPs would have voted against starting sittings of the House of Commons on Tuesdays at 11.30am rather than 2.30pm.
In the 1997-2001 Parliament, Anna’s code found no votes that would have had different results.
IMPORTANT DISCLAIMER! We can’t conclude that all of the above would necessarily have become law if Scottish MPs had not been in the chamber. Bills don’t become law until they have passed through the House of Lords – not to mention the many other forces of history that would have acted differently.
Get the code and the data
You can see the code used for this analysis, and the full datasets, on GitHub. You can adapt it yourself if you want to do your own analyses.
This analysis is the work of one volunteer: we welcome any corrections. Like PublicWhip itself, the whole point is that it is out in open for anyone to analyse and improve.
Image by Catherine Bebbington. Parliamentary copyright image reproduced with the permission of Parliament.
Our team member Richard has now analysed every single one of those votes, and his findings have been added to each MP’s information on TheyWorkForYou.
We hope that’s great news for users: it means that we can now present a really full picture of how your MP voted on key topics.
It’s also potentially useful for developers, eDemocracy hackers and campaign groups, who can pick up our data and use it as they please.
So what exactly is the data?
Often MPs vote on motions which are, at first glance, rather incomprehensible and cryptic. They might vote for example on a motion to accept:
Amendments (a) to (d) proposed in lieu of Lords amendments 1 to 4 and 6.
We’ve done the research to determine what MPs were actually voting on in each case, and turned their archaic language into plain English.
For every vote we’ve written a sentence to describe the effect of voting either “aye” or “no”. In relation to one MP’s vote on the evening of the 9th of July 2014 we write:
Mark Pawsey MP, Rugby voted for a residence test as an eligibility criteria for civil legal aid; subject to exceptions for refugees and those who have sought asylum.
In addition to describing every vote, we have decided whether it should be considered relevant to the topics we list on each MP’s page (see an example MP here, or check your own MP by inputting your postcode on the homepage, then clicking ‘voting record’ on your MP’s page).
If a vote was relevant to one of the statements we show on TheyWorkForYou, we then determined whether voting ‘aye’ or ‘no’ was a vote for or against the statement and if the vote was very important, or less important. By clicking on the green ‘details’ button beside each statement on an MP’s voting record you can see exactly which individual votes contributed to it as well as how we calculated which wording such as “moderately for” or “strongly against” to apply in each case.
Matters MPs have voted on since the 2010 general election have ranged from bankers’ bonuses to same sex marriage; from food banks to the “bedroom tax” (all of which have contributed to statements we show on TheyWorkForYou); from daylight saving to the regulation of hairdressers (neither included) – and plenty more. (We’ve written previously about how we select which topics to show on TheyWorkForYou.)
Of course, Parliament continues to hold votes, and we’ll be continuing to analyse the results as they come in – but it is good to know that we are bang up to date.
How can this data be used?
We have plenty of ideas ourselves, and we want to hear yours, too. With the forthcoming general election, one obvious use is for ‘who should I vote for?’ tools, which match users’ opinions with those of each party.
There’s also potential for comparisons between what constituents believe and what their elected representative has voted for.
No doubt there are many other ideas that haven’t even occurred to us yet – please do get in touch if you have ideas and you’d like to use this data.
The Written Answer is a noble parliamentary tradition, dating back almost 300 years. MPs and peers use them to hold the government to account, getting facts and figures on the record.
But wriggling out of answering them is also a recognised Parliamentary skill – and one that, while often applied with dexterity, can impede the process of democracy.
That’s the primary reason that, beside each Written Answer on TheyWorkForYou, we poll our users on a single point:
“Does this answer the above question?”
Last month marked the tenth birthday of TheyWorkForYou, and over that time, this unassuming poll has amassed more than 275,000 of these yes or no responses on a total of around 130,000 written answers.
That’s a substantial sample for us to analyse. Running that data through a few tickertape machines and putting the results in order means that we can now see just how many written answers actually address the question in hand – and which government departments are the best and worst at giving a straight answer.
Is the current administration more slippery?
It seems that ministers are getting worse at returning a straightforward answer.
In the previous government: 47% of written answers that were voted on got more ‘yes’ answers than ‘no’s from our users.
In the current administration: That figure has dropped to 45%. Even within the current term, the figure has been falling year on year, with a 49% ‘yes’ rate in 2010 comparing to a 42% rate in 2013.
Best and worst departments for a straight answer
Breaking down the data by department is also eye-opening – some departments are decidedly more likely to be judged as prevaricators by TheyWorkForYou’s users.
Accolade for ‘most improved’ goes to the Wales office, who managed an 86% ‘yes’ rate in the current government, against 48% in the last. Worst of the bunch – as perceived by TheyWorkForYou’s users – is the Department of Work and Pensions, with just 31% in this administration.
We’ve put the full rankings below, for those of you who would like to delve deeper into these figures.