Amazing—we did it!
When we decided to mark Global Legislative Openness Week with a drive to get the data for 200 countries up on EveryPolitician, in all honesty, we weren’t entirely sure it could be done.
And without the help of many people we wouldn’t have got there. But last night, we put live the data for North Korea and Sweden, making us one country over the target.
The result? There is now consistently-structured, reusable data representing the politicians in 201 countries, ready for anyone to pick up and work with. We hope you will.
That’s not to say that our job is over… far from it! There’s still plenty more to be done, as we’ll explain below.
Here’s how it happened
Getting the data for each country was a multi-step process, aided by many people. First, a suitable online source had to be located. Then, a scraper would be written: a piece of code that could visit that source and pull out the information we needed—names, districts, political parties, dates of office, etc—and put it all in the right format.
Because each country’s data had its own idiosyncrasies and formatting, we needed a different scraper for every country.
Once written, we added each scraper to EveryPolitician’s list. Crucially, scrapers aren’t just a one-off deal: ideally they’ll continue to work over time as legislatures and politicians change.
The map above shows our progress during GLOW week, from 134 countries, where we began, up to today’s count of 201.
mySociety’s Tony, Lead on the EveryPolitician project, worked non-stop this week to get as many countries as possible online. But this week we’ve seen EveryPolitician reach some kind of momentum, as it takes off as a community project. It’s an ambitious idea, and it can only succeed with the help of this kind of community effort. Thanks to everyone who helped, including (in no particular order):
Duncan Walker for writing the scraper for Uganda; Joshua Tauberer for helping with the USA data; Struan Donald for handling Ecuador, Japan, Hong Kong, Serbia and the Netherlands; Dave Whiteland, with ThaiNetizen helpfully finding the data source for Thailand; Team Popong for South Korean data; Jenna Howe for her work on El Salvador; Rubeena Mahato, Chris Maddock, Kätlin Traks, François Briatte, @confirmordeny, and @foimonkey for lots of help on finding data; Henare Degan and OpenAustralia who made the scraper for Ukraine; Matthew Somerville for covering the Falkland islands and Sweden; Liz Conlan for lots of help with Peru and American Samoa; Jaroslav Semančík who provided data for, and assistance with, Slovakia; Mathias Huter who supplied current data for Austria while Steven Hirschorn wrote a scraper for the historic data; Andy Lulham who wrote a scraper for Gibraltar; Abigail Rumsey who wrote a scraper for Sri Lanka; everyone who tweeted encouragement or retweeted our requests for help.
But there’s more
There are still 40 or so countries for which we have no data at all: you can see them here. This week has provided an enormous boost to our data, but the site’s real target is, just like the name says, to cover every politician in the world.
And once we’ve done that, there’s still the matter of both historic data, and more in-depth data for the politicians we do have. Thus far, we mostly have only the lower houses for most countries which have two — and for many countries we only have the current politicians. Going into the future we need to include much richer data on all politicians, including voting records, et cetera.
Meanwhile, our first target, to have a list of the current members of every national legislature in the world, is starting to look like it’s not so very far away. If you’d like to help us reach it, here’s how you still can.
International emergency aid charity Médecins Sans Frontiers are one of the biggest purchasers of medicine worldwide, and naturally it’s important that the drugs they buy are cost-effective. Where possible, they choose generics—white label medicines that contain the same ingredients even if they don’t carry the well-known brand names: think ‘ibuprofen’ or ‘aspirin’ rather than ‘Nurofen’ or ‘Anadin’.
But when a specific medicine is only available as a patented product from a big drugs company and with an equally big price tag attached, MSF, like everyone else, has little choice but to pay.
Curiously, this turned out to be a problem that can be solved, in part, through good web design. Here’s the story.
Obviously, drugs companies have an interest in keeping their medicines under patent. As MSF explained, patents, and in particular the practice of ‘evergreening’ them (extending their life indefinitely by making slight modifications to the medicine’s make-up), give pharmaceutical companies a monopoly on pricing, and can impede access to patients who would benefit from them.
MSF’s online project, the Patent Oppositions Database (PODB) is a resource for helping people challenge medicine patents. PODB helps groups around the world to find each other and work on cases together, and to share previous examples of art and arguments used in lawsuits which may help others in future oppositions.
The site was already up, running and functional, and the concept was sound. But it wasn’t attracting much take-up. On analysis, it became clear that this was because there was no focused experience on the site, encouraging users towards the core interactions which would power the whole concept of collaborating and sharing knowledge.
Where design came in
MSF asked us to suggest improvements that would enable groups to communicate about specific cases, and to improve the sense of community. Our solutions will add intuitive user paths that lead people to existing opposition cases and the information they need, then encourage them to join in by placing discussions and information about contributors on the page.
It’s crucial for MSF that the project reaches its full potential, and with the in-depth design changes we’ve suggested, and have now been asked to implement, we know it will.
You can read more about how we approached this project in our latest case study, over at the mySociety Services website.
We received a tweet this morning wondering how many emails there had been to MPs on the subject of the ‘Refugees Welcome’ campaign, and whether WriteToThem, our contact-a-politician website, might have some relevant data.
Even if we could, WriteToThem is a completely non-partisan service, and users may be writing on either side of an issue.
We do use Google Analytics, which collects entirely anonymous statistics on how many people visit the site, how long they stay on it, etc. There is one clear indication that the site is being used more than usual: user numbers on Thursday and Friday of last week were about 5 or 6 times higher than the norm. There was a dip at the weekend — there generally is — and numbers have continued to climb on Monday and today.
Google Analytics also allows us to see which websites have referred people to our site. Over this period, it seems it was mostly Facebook and the petitions site Avaaz.
With most websites, you can regard visitor numbers as a pretty good indication of your success — if they’re going up, then at least something’s right.
With WriteToThem though, user numbers regularly fluctuate so wildly that you could be fooled into thinking we’re on the brink of disaster, or the brink of world domination, from one week to the next.
In the normal way of things, there seems to be a baseline at which the UK populace will toddle along. A small percentage of us will write to our politicians whenever we have an opinion that we want to express, but most of us are content with a few acerbic Facebook updates or heated discussions down the pub.
Then, now and then, an issue comes along which grips the nation. This week, that would indeed appear to be the issue of refugees.
Of course, we’re always glad to see the site used, and we hope that people who are referred to it because of an issue they care about will also remember it’s there whenever they need to contact their representatives in the future.
Incidentally, if you are running any kind of campaign and you would like to harness WriteToThem’s functionality on your own site, don’t forget that we’ve written a guide to doing just that.
Public health teams, policy-makers, councillors and NGOs often need facts about a specific area. If they’re looking for data on things like the number of smokers, the demand for hospital beds, or the birth rate, they turn to their regional Health and Wellbeing Board.
These local authority committees are required to produce a document known as a JSNA (Joint Strategic Needs Assessment) every few years. It’s a snapshot of the demographics and healthcare needs of the local population, and is used by a variety of stakeholders including policy makers and strategy groups.
Like most local authority committees, the London Borough of Hackney and the City of London Health and Wellbeing Board have previously produced their JSNA as a simple read-only PDF document. But, in the digital age, they knew that there was more they could do to make this document accessible, useful, and engaging.
That’s when they called us in — not to build the final digital version of the JSNA, but to help them understand the possibilities and ensure that they were heading down the right path.
We’ve written up the whole process in a case study, so, if you’d like to know more, read on.
Just how quickly can we hit the 200 countries mark on EveryPolitician? That’s what we’ll be finding out this week, and one thing’s for sure — we’ll get a lot further with your help.
This week is GLOW, the Global Legislative Openness Week, and we’re marking it with a concerted drive for more data.
Tony, the project lead, has consistently added one new country every day since EveryPolitician launched four months ago, and now it’s time to put a rocket behind our efforts.
The site currently contains data for 134 countries. We’ll be going flat out to see how quickly we can reach 200, and as the excitement ramps up, we hope you will help spread the word and get involved, too. Tony will carry on working as hard as he can to fill in the gaps, but we need your help to get further, faster.
What is EveryPolitician?
How can I help?
- Help us find data for more countries! We don’t currently know where to find the politician data for many countries. Here’s a list of the ones we need and here’s a page about how to contribute. If you get stuck, give us a shout.
- Write a scraper If you have the know-how, you can help us enormously by helping scrape the data from the places we do know about. See this page for guidance on how to go about writing a scraper. You’ll find lots of examples here.
- You can also help by spreading the word – tell your friends, tweet, blog, get up on a platform and talk, and just generally share this post. Thank you!
Why do we need this data?
Politician data is readily available for most countries, but it comes in a massive variety of inconsistent formats. Most of those formats aren’t ‘machine readable’, that is to say, the data can’t readily be extracted and re-used by programmers, and pretty much every country differs on what information it provides about each politician.
That being the case, anyone who wants to build an online tool that deals with politicians from more than one country, or who would like their tool to be available to people in other places, or would like to adapt an existing tool to be used elsewhere, would first have to adapt their tool to cope with the data.
EveryPolitician saves them the trouble, and the structured format also means that the tools they build will be compatible with any other tools that use it.
What kind of tools?
EveryPolitician data will be useful for all kinds of projects.
It’ll be much easier to build a website that shows people how to contact a politician. Or one that holds a government to account and educates people about what politicians are doing. Or one that helps voters make choices by displaying facts about what their politicians believe.
It can go further than that, though — with these building blocks in place, developers can really use their imagination to put together all kinds of projects, many of which we haven’t even begun to imagine. And don’t forget that, if a tool has been built to use the standardised data, it’ll also be easy for others to redeploy elsewhere.
If you’d like to see a concrete way in which the data’s already being used, check out Gender Balance.
How can I keep up to date?
We’ll be putting out regular updates via Twitter as the number of countries covered increases — plus you can watch the map turn green on http://everypolitician.org/countries.html as we progress.
If you use FixMyStreet to make a report in the Royal Borough of Greenwich, you won’t notice anything different from the norm. But once you click submit, your report is doing something a little bit different—it’s using a standard called ‘Open311′ to place your request directly into the council’s systems.
You might be thinking, “Yawn-o! What do I care, so long as my pothole gets filled?” and—well, that’s a fair point. But there’s a wider issue here, which we think is one that’s worth getting excited about.
Greenwich have taken a forward-thinking and sensible step—because Open311 doesn’t just let FixMyStreet reports come into their systems smoothly. It also opens up their data in a way that allows other developers to create exciting applications that can work with it, talk to their systems or provide new interfaces for us to do so.
What might those be? Well, one of the great things about technology is that it’s very hard to predict how users will behave in even the near future. Just a few years ago, who would have guessed that we’d be chatting to companies, organisations and our MPs in snappy, public 140-character soundbites, for example?
With Open311 in place, Greenwich do in fact have the option of receiving reports via Twitter, Facebook, and, crucially, whatever the next big platforms happen to be. Meanwhile they benefit from FixMyStreet reports dropping directly into their workflow.
Reports sent by email (which FixMyStreet does by default) can be a bit of an inconvenience for councils using CRM systems, because staff have to copy and paste the details in. But Open311 sends your report, along with every detail the council needs to know, into their chosen systems.
You can read more about the nitty-gritty of that here, but in the meantime, all you need to know is that Greenwich have proactively taken the step to allow FixMyStreet to send reports in this way, installing our Open 311 endpoint, and taking advantage of our offer to connect for free.
This is quite separate from the option of installing FixMyStreet for Councils as their main reporting system, which incidentally Greenwich also does.
So it’s a big high five for Greenwich, who with this simple step have allowed a wealth of potential applications, services and developers to interact with them over the web. Now—any other councils want to follow their example?
As players were quick to notice, decisions made on our politician-sorting game Gender Balance were final. Thanks to volunteer coder Andy Lulham, that’s now been rectified with an ‘undo’ button.
Gender Balance is our answer to the fact that there’s no one source of gender information across the world’s legislatures—read more about its launch here. It serves up a series of politicians’ names and images, and asks you to identify the gender for each. Your responses, along with those of other players, helps compile a set of open data that will be available to all.
Many early players told us, however, that it’s all too easy to accidentally click the wrong button. (The reasons for this may be various, but we can’t help thinking that it’s often because there are so many males in a row that the next female comes as a bit of a surprise…)
In fact, this shouldn’t matter too much, because every legislature is served up to multiple players, and over time any anomalies will be ironed out of the data. That doesn’t stop the fact that it’s an upset to the user, though, and in the site’s first month of existence, an undo button has been the most-requested feature.
Thanks to the wonders of open source, anyone can take the code and make modifications or improvements, and that’s just what Andy did in this case. He submitted this pull request (if you look at that, you can see the discussion that followed with our own developers and our designer Zarino). We’ve merged his contribution back into the main code so all players will now have the luxury of being able to reverse a hasty decision. Thanks, Andy!
Are you passionate about using digital tools to empower citizens and improve governments? Do you have significant research experience in civic technology or a related field? If so, why not apply for mySociety’s new US Civic Technologies Research contract?
mySociety have received a generous gift from Microsoft to explore civic tech usage at the sub-national level within the USA. We are therefore looking for a partner (individual(s) or organisation) in the USA to lead on this research for us.
We are offering a contract for approx 200 days work and a clear research remit. You’d need to be based in, and able to travel within the USA.
Interested? For further details and to apply, click here.
Back in November 2013, we asked you what improvements you’d like to see on TheyWorkForYou.
One answer dominated: you wanted more information about how MPs vote.
Adding information on voting has been the single biggest project on the site since its launch, and has required several different phases of development. We announced each of these as it happened, but now that we’re at the end of this large piece of work, it seems like a good time for a complete overview.
So let’s take a look at exactly what it has involved—and, more importantly, what it means for you.
We’ll start with a rundown of features, then go into more detail about how they are created at the end of the post, for those who are interested.
What vote information means for you
1. You can easily see how your MP voted
Just how much do you know about how your MP voted on the stuff that matters? Most of us would have a hard time keeping up with every vote, simply because it isn’t information that’s widely publicised.
On TheyWorkForYou, you can see a run-down of how any MP has voted on key policies, by visiting their page on the site and clicking the ‘voting record’ tab (see image, above). We’ve created summaries of their stance on all kinds of matters, including the EU, same-sex marriage, NHS reform and a lot more.
Each of these summaries is compiled from every vote the MP has made on a motion that impacts on that policy.
You can click ‘show votes’ (see image above) to see the specific votes that go to make up any particular stance, and we’ve laid them all out in plain English so that it’s easy to grasp exactly what the issue is.
And from there you can click through to the website Public Whip, where you can explore votes in more detail, including lists of who voted for or against any given motion.
2. You can find out how strongly your MP feels
When we first presented voting information, we said that an MP had voted ‘strongly for’ or ‘moderately against’ certain policies, which led to quite a large postbag from people asking, “How can you vote strongly, surely you either vote for or against?”.
We wrote in the second half of this blog post about the wording changes we made to clarify the fact that these stances are calculated from a number of votes.
3. You can assess if your MP is a sheep or a lone wolf
We’ve pulled out all the votes which differ substantially from the way that the majority of each MP’s party voted. If your MP has voted against the flow, you’ll see something like this on their page:
Why do we highlight this type of vote? Because we think they’re a really good indication of where an MP feels strongly enough about something to risk sticking their neck out. It’s also a great way to check the truth when people say, “MPs? They’re all the same”.
4. You can understand the background to the votes
Generally speaking, there’s a debate before any vote takes place in Parliament, covering all the matters which may be topmost in MPs’ minds before they cast their lot.
Clicking on the ‘show full debate’ link from the topic pages (see image above) will give you the full context.
How we compile vote information
If that all seems nice and simple, well, great! That was our aim.
Putting it all together definitely wasn’t so simple, though. Voting information has never been previously presented all in one place in quite this way before—on TheyWorkForYou or anywhere else, to the best of our knowledge—so we had to figure out how to import the data and how best to display it.
As with much of our work, it’s a mixture of manual graft and automating whatever we can. Some things, like rewriting votes so that everyone can understand them, can’t be done by a computer. But many of our users are surprised to learn just how much of what we publish out is untouched by human hand.
Our Developer Struan, who did the most recent round of work on the voting records, said:
We get all our voting data from PublicWhip, a site set up by Francis Irving (once of mySociety) and Julian Todd. Public Whip takes the data we [TheyWorkForYou] produce from Hansard and extracts only the information on votes (or divisions in Parliamentary jargon) that take place in Parliament. It then allows you to look up how an MP or a Lord voted.
Let’s just think about that for a moment. We’re looking at a process where Parliament publishes Hansard, TheyWorkForYou scrapes the data and re-presents it, Public Whip extracts the voting information and presents that, and TheyWorkForYou takes that voting information back for its own voting pages. Simple…
One of the first things we did was to ‘translate’ the votes into plain English, so that it was very clear what was being voted for or against— and if you want to read more about that process, we talked about it in a blog post back in July 2014.
That allowed us to move to the next phase, as Struan explains:
Public Whip groups related votes together into policies, e.g renewing Trident, so you can see how an MP voted on the policy as a whole.
It does this by saying which way an MP would have to vote each of the divisions in the policy if they agreed with the policy. It then takes the MP’s votes on each division in the policy and assigns a score to it based on how they voted. These scores are then added up and compared to the score they would get if they always voted in agreement with the policy. The closer the MP’s score is to the score of an MP who always voted in agreement with the policy, the more they agree with the policy.
Thanks to Public Whip’s grouping, we were able to start compiling our MPs’ voting records along those same policy lines.
One of the most fiddly parts of the process was figuring out how to ensure that the information we present is a true, non-biased representation of the MP’s intentions. You might think that a vote is quite a simple matter – it’s either a yes or a no for a particular motion. But as soon as we started displaying votes within a policy, things got a bit trickier.
Some divisions in a policy can be marked as important and voting with the policy in those divisions is worth more points. This is to prevent voting in agreement on a set of minor votes, e.g “Parliament will commission a report on the future of Trident”, outweighing voting against something important, e.g. “Renew Trident”. It also reflects the way Parliament works, often with several smaller votes on parts of a bill and then a vote on the bill as a whole.
For clarity I should point out here that sometimes voting no in a division is a vote for the policy, e.g voting no in a “This house believes Trident should not be renewed” division would clearly be a vote for our example “Renew Trident” policy.
This approach also helps where one vote straddles several topics: for example, consider a vote against the Budget when the Budget contains many proposals including, say, the capping of VAT. It’s quite possible that an MP may be for the capping of VAT but broadly against several other motions covered by the Budget, and so decides to vote against it on balance. So long as we mark the Budget vote as a weak vote for the capping of VAT, its significance should be properly accounted for.
Where we don’t have enough information to show a stance, for example where an MP never voted on the topic, is too new to have had a chance to vote on the topic, or all their votes on the topic have been labelled as ‘weak’, we say so:
A final little subtlety is the difference between “Never voted” on a policy and votes where the MP was absent. If it says an MP has never voted on a policy that means they were elected after all the divisions in the policy took place so did not have a chance to vote on them. Absent means they could have voted in the divisions but did not.
Absent votes count towards your score but at half the rate of voting in agreement with the policy. This is so that an MP who votes in agreement with the policy in one division and then misses all the other divisions shows as agreeing with the policy rather than against as it would if no score was assigned to absent votes. That does currently mean that if they were always absent it shows, slightly unhelpfully, as “a mixture of for and against”.
It’s not an ideal system as it does produce some odd results occasionally but it mostly works.
To show where an MP has voted against the majority of their party, we have to figure out a similar score across the party as a whole.
This is exactly the same process, only we add up all the votes by all the MPs but the maths is pretty much the same.
All in a day’s work
As mentioned at the top of this post, vote information was our most-requested addition. And rightly so! Our MPs represent us, so naturally we want to see their track records, quickly and easily.
If you’re not an expert, you might not have known how to find this information before. And that’s essentially what TheyWorkForYou aims to do: make the workings of Parliament more accessible for everyone.
Parliamentary copyright images are reproduced with the permission of Parliament.
If you’ve got a problem with your planning applications, we’ve got a little something for you…
Back in September, we wrote about our project with Hampshire Hub to build a prototype, open source web application that would help members of the public find out more about planning applications in their area:
The planning process can be baffling if you’re new to it and this tool aims to help make it easier to understand. We’ll be helping people answer some of the most common questions they have about planning applications: What applications are happening near me? What decisions have been made in the past on applications like mine? How likely is it that my application will be dealt with on time?
The site helps people browse planning application data by location — whether a postcode or a street address — and by type — whether it’s an extension, a loft conversion, or a major development like a retail park or commercial warehouse.
That project can now be seen at http://openplanning.hampshirehub.net/: go and have a poke around!
How it works
OpenPlanning displays planning applications clearly on a map. Users can browse their local area and learn more about how to complete their own request by looking at the success or failure of what has gone before.
This benefits everyone, from residents who are less likely to have their applications turned down, to the council who will find themselves dealing with higher-quality submissions.
By aggregating planning application details from multiple councils, the site allows users to browse irrespective of administrative boundaries or the authority in charge. After all, neither of those considerations are very high on the list of the resident’s priorities.
There’s nothing new about putting planning applications online, of course: they can already be browsed and submitted in many places across the web. This project isn’t hoping to replace those tools, but to complement them, providing links to existing data sources where possible, all accessible via a much more user-friendly interface.
We know many councils and residents struggle with planning applications on a daily basis, and we hope that OpenPlanning will provide the first step towards making the whole process easier for all parties.
The future for OpenPlanning
OpenPlanning is the first iteration of a new product. At this early stage, we haven’t included a facility to submit a planning application – that’s something we could slot in cost-effectively at a later phase though, and of course we’d be happy to hear from any councils who would be interested in adopting that approach.
The code is based on Open Australia Foundation’s PlanningAlerts platform, which means it’s already been tried and tested by a wide community down under. It’s still under active development and, thanks to the joys of Open Source code, we’ll be able to contribute improvements back to the original codebase too.
We’ve really enjoyed working with Hampshire Hub: a forward-thinking partnership of councils and other public organisations, led by Hampshire County Council, which aims to provide useful open data for the county. Hampshire understands the benefits, both direct and indirect, of open source tools and open data.
Now we’re seeking local councils who are struggling with the quality of planning applications, perhaps processing large volumes of applications that are not granted. If that sounds like you then please get in touch to speak to mySociety Services about what OpenPlanning can do for you.