Last year, when we were helping to develop YourNextMP, the candidate-crowdsourcing platform for the General Election, we made what seemed like an obvious decision.
We decided to use PopIt as the site’s datastore — the place from which it could draw information about representatives: their names, positions, et cetera. We’d been developing PopIt as a solution for parliamentary monitoring sites, but we reckoned it would also be a good fit for YourNextMP.
That turned out to be the wrong choice.
YourNextMP was up and running in time for the election, but at the cost of many hours of intensive development as we tried to make PopIt do what was needed for the site.
Once you’ve got an established site in production, changing the database it uses isn’t something you do lightly. But on returning to the codebase to develop it for international reuse, we had to admit that, in the words of mySociety developer Mark Longair, PopIt was “actually causing more problems than it was solving”. It was time to unpick the code and take a different approach.
Mark explains just what it took to decide to change course in this way, over on his own blog.
The post contains quite a bit of technical detail, but it’s also an interesting read for anyone who’s interested in when, and why, it’s sometimes best to question the decisions you’ve made.
There are websites built on mySociety code in many countries across the world.
Whatever the site you’re planning, you’ll find it a lot easier with our support and development help.
Our quarterly call for applications closes on October 30, so make sure you have yours in soon. Want to know exactly what’s involved? Start here.
Just how quickly can we hit the 200 countries mark on EveryPolitician? That’s what we’ll be finding out this week, and one thing’s for sure — we’ll get a lot further with your help.
This week is GLOW, the Global Legislative Openness Week, and we’re marking it with a concerted drive for more data.
Tony, the project lead, has consistently added one new country every day since EveryPolitician launched four months ago, and now it’s time to put a rocket behind our efforts.
The site currently contains data for 134 countries. We’ll be going flat out to see how quickly we can reach 200, and as the excitement ramps up, we hope you will help spread the word and get involved, too. Tony will carry on working as hard as he can to fill in the gaps, but we need your help to get further, faster.
What is EveryPolitician?
How can I help?
- Help us find data for more countries! We don’t currently know where to find the politician data for many countries. Here’s a list of the ones we need and here’s a page about how to contribute. If you get stuck, give us a shout.
- Write a scraper If you have the know-how, you can help us enormously by helping scrape the data from the places we do know about. See this page for guidance on how to go about writing a scraper. You’ll find lots of examples here.
- You can also help by spreading the word – tell your friends, tweet, blog, get up on a platform and talk, and just generally share this post. Thank you!
Why do we need this data?
Politician data is readily available for most countries, but it comes in a massive variety of inconsistent formats. Most of those formats aren’t ‘machine readable’, that is to say, the data can’t readily be extracted and re-used by programmers, and pretty much every country differs on what information it provides about each politician.
That being the case, anyone who wants to build an online tool that deals with politicians from more than one country, or who would like their tool to be available to people in other places, or would like to adapt an existing tool to be used elsewhere, would first have to adapt their tool to cope with the data.
EveryPolitician saves them the trouble, and the structured format also means that the tools they build will be compatible with any other tools that use it.
What kind of tools?
EveryPolitician data will be useful for all kinds of projects.
It’ll be much easier to build a website that shows people how to contact a politician. Or one that holds a government to account and educates people about what politicians are doing. Or one that helps voters make choices by displaying facts about what their politicians believe.
It can go further than that, though — with these building blocks in place, developers can really use their imagination to put together all kinds of projects, many of which we haven’t even begun to imagine. And don’t forget that, if a tool has been built to use the standardised data, it’ll also be easy for others to redeploy elsewhere.
If you’d like to see a concrete way in which the data’s already being used, check out Gender Balance.
How can I keep up to date?
We’ll be putting out regular updates via Twitter as the number of countries covered increases — plus you can watch the map turn green on http://everypolitician.org/countries.html as we progress.
As players were quick to notice, decisions made on our politician-sorting game Gender Balance were final. Thanks to volunteer coder Andy Lulham, that’s now been rectified with an ‘undo’ button.
Gender Balance is our answer to the fact that there’s no one source of gender information across the world’s legislatures—read more about its launch here. It serves up a series of politicians’ names and images, and asks you to identify the gender for each. Your responses, along with those of other players, helps compile a set of open data that will be available to all.
Many early players told us, however, that it’s all too easy to accidentally click the wrong button. (The reasons for this may be various, but we can’t help thinking that it’s often because there are so many males in a row that the next female comes as a bit of a surprise…)
In fact, this shouldn’t matter too much, because every legislature is served up to multiple players, and over time any anomalies will be ironed out of the data. That doesn’t stop the fact that it’s an upset to the user, though, and in the site’s first month of existence, an undo button has been the most-requested feature.
Thanks to the wonders of open source, anyone can take the code and make modifications or improvements, and that’s just what Andy did in this case. He submitted this pull request (if you look at that, you can see the discussion that followed with our own developers and our designer Zarino). We’ve merged his contribution back into the main code so all players will now have the luxury of being able to reverse a hasty decision. Thanks, Andy!
Remember the UK General Election? Yes, we know it’s a distant memory now, and you’ve probably forgotten YourNextMP, too. But the project is far from dormant!
YourNextMP successfully crowd-sourced information on every election candidate, and made it available as open data for anyone who wanted to use it to build useful websites and online tools.
And while here in the UK we won’t have further use for it until 2020, the great news is that the underlying code can be repurposed to work for other elections around the world. Thanks to Yo Quiero Saber, the first of these is now live and collecting data for Argentina at http://investigacion.yoquierosaber.org/, and there are also plans for DataMade Chicago to use it in the USA.
In Argentina, the crowdsourcing component sits as part of a wider voter informing project. Martín Szyszlican from Yo Quiero Saber explains more:
We just launched Yo Quiero Saber and it’s had a great reception. You’re welcome to visit our main site, where we feature the game and full profiles for candidates for presidency and governors of four provinces.
You can also see our YourNextRepresentative instance (we renamed it, since MP is not a relevant term for us) where, in just two weeks, we’ve already had more than 100 registered users, and have also managed to add all the official candidates from DINE (the national elections office).
We’re still missing city-level and provincial-level candidates from the site, but that’s going to be improved before the October general elections.
So far, we’ve had 350,000 unique users and a million page views since launch. That means we are close to reaching 1% of the total number of voters in the country. Neatly, the number of people who have used the site is roughly equivalent to the number of voters a party needs to pass from this election to the next ones.
Media reception has been great with online portals big, small and regional mentioning our site and some of them embedding our game in their articles. We’ve also been kept busy with radio interviews and some tv programmes featuring the game. In Argentina, the media is deeply split down party lines, and we very much like the fact that we’ve surfed that divide, being featured in media from both sides of the political spectrum.
This is just the beginning: we’re working as an alliance of local NGOs, and our bid for a prototype grant from the Knight Foundation has been successful, meaning that we can forge ahead with our plans. We’ve also had support from HacksLabs, a data journalism accelerator. The full list of partners can be found on the footer of both sites.
We’re really glad to hear of this success—it’s great to see the code get another lease of life, which is, of course, what the Poplus project is all about.
Naturally, the YourNextRepresentative codebase also available to other countries who want to help inform their electorates, and what’s more, Martín says they’ll be glad to offer help to anyone who wants it. That goes for us here at mySociety too.
If you need data on the people who make up your parliament, another country’s parliament, or indeed all parliaments, you may be in luck.
What’s more, it’s all provided as Open Data to anyone who would like to use it to power a civic tech project. We’re thinking parliamentary monitoring organisations, journalists, groups who run access-to-democracy sites like our own WriteToThem, and especially researchers who want to do analysis across multiple countries.
But isn’t that data already available?
Yes and no. There’s no doubt that you can find details of most parliaments online, either on official government websites, on Wikipedia, or on a variety of other places online.
But, as you might expect from data that’s coming from hundreds of different sources, it’s in a multitude of different formats. That makes it very hard to work with in any kind of consistent fashion.
Every Politician standardises all of its data into the Popolo standard and then provides it in two simple downloadable formats:
- csv, which contains basic data that’s easy to work with on spreadsheets
- JSON which contains richer data on each person, and is ideal for developers
This standardisation means that it should now be a lot easier to work on projects across multiple countries, or to compare one country’s data with another. It also means that data works well with other Poplus Components.
What can I do with it?
Need a specific example? Yesterday, we introduced Gender Balance, the game that gathers data about women in politics.
As you’ll know if you’ve already given it a try, Gender Balance works by displaying politicians that make up one of the world’s legislatures, one by one.
That data all comes from Every Politician, and it’s meant that the developers have been able to concentrate on making a smooth and functional interface, knowing that the data side of things has already been taken care of.
That’s just one way to use Every Politician data, though. If you’d like to use it in your own site or app, you can find out more here.
We still need more data
As you may have noticed, there are more than 100 parliaments in the world. In fact, despite having reached what feels like a fairly substantial milestone, we’re still barely half way to getting some data for every parliament.
So we could use your help in finding data for the parliaments we don’t yet cover, and historic information for the ones we do. Read more about how you can help out.
La Constitución De Todos allows citizens to discuss, vote on and propose changes to each article of the constitution online, using code that originates from Morocco’s Legislation Lab from GovRight.
The launch comes in the context of the new Chilean president announcing that there will be a widespread public consultation on a constitution for the nation.
The two organisations might never have met, if it hadn’t been for the Poplus kick-off conference back in 2014, where the idea was first mooted, and GovRight stepped in to offer help.
The Poplus federation was founded on the idea that sharing civic code and knowledge can benefit organisations worldwide: this project is another superb example of exactly that.
The Poplus community is spread all over the world—but that doesn’t stop us getting together whenever we can.
Poplus is a worldwide federation of people and organisations with an interest in civic tech. This Friday, we will be holding a virtual Show and Tell, hearing from two very different projects:
- Andrew Mandelbaum from SimSim in Morocco will be speaking about Nouabook. This is an application which enables anyone to contact their politicians in public, through Facebook. It uses the Poplus Component WriteIt.
- Matthew Landauer from OpenAustralia Foundation will be speaking about Cuttlefish, one of the latest pieces of software to be certified as a Poplus Component.
As well as hearing all about these projects, there’ll be a chance to catch up and have a chat about all things Poplus/civic tech. Everyone is welcome.
We’ll be using an online platform called QiqoChat to host this call: sign up here.
You can create a free account using Facebook/Google/LinkedIn/Meetup/Twitter or a regular email address. Instructions for connecting by phone or computer microphone are available when you sign in and click “Participate”.
This Friday, 12 June. Times are as follows:
• 5 AM – US Pacific
• 7 AM – US Eastern
• 8 AM – Chile/Argentina
• 12 PM – UK
• 7 PM – Taiwan and Malaysia
• 9 PM – Sydney
• 11 PM- New Zealand
As you’ll know if you’re a regular reader of this blog, YourNextMP crowd-sourced details of every candidate who stood in the UK general election.
But, just because our own election is over, doesn’t mean we’ll be letting YourNextMP gather dust. On the contrary—we want to see it being re-used wherever there are elections being held, and citizens needing information! We’re already seeing the first re-use case, and we’d love to see more.
Opening up data
YourNextMP’s main purpose was to provide a free, open database of candidates, so that anyone who wanted to could build their own tools on top of it, and it was very successful with that aim.
The traditional source of candidate data for such projects has been through expensive private providers, not least because the official candidate lists are published just a few days before the election.
Thanks to YourNextMP’s wonderful crowd-sourcing and triple-checking volunteers, we reckon that we had the most complete, most accurate data, the earliest. And it was free.
Directly informing over a million citizens
YourNextMP also came into its own as a direct source of information for the UK’s electorate. This hadn’t been the priority when the project was launched, but it was helped greatly by the fact that constituency and candidate pages ranked very highly in search engines from early on, so anyone searching for their local candidates found the site easily.
Once they did so, they found a list of everyone standing in their constituency, together with contact details, links to their online profiles such as web pages, social media and party websites, and feeds from spin-off projects (themselves built on YourNextMP data) such as electionleaflets.org and electionmentions.com.
YourNextMP had more than a million unique users. In the weeks just prior to the UK general election, it was attracting approximately 20,000 visitors per day, and on the day before the election, May 6th, there was suddenly a massive surge: that day the site was visited by nearly 160,000 people.
So, in a nutshell: YourNextMP has not only enabled a bunch of projects which helped people become more informed before our election—it also directly informed over a million citizens.
A reusable codebase
And, in the spirit of Poplus, the codebase is open for anyone to re-use in any country.
It’s already being pressed into use for the upcoming elections in Argentina, and we hope that developers in many other countries will use it to inform citizens, and inspire great web tools for the electorate, when their own elections come around.
If that’s something that interests you, please come and talk, ask questions and find out what’s involved, over on the Democracy Club mailing list.
The Knight Foundation’s News Challenge offers funding to innovative projects. We wonder whether they’ve ever had a bid whose collaborators span six different countries before.
You can read more about the plans on the bid page—and please click the little pink heart to give us ‘applause’!
In short, we want to build on the success that the YourNextMP crowdsourcing platform has had here in the UK.
Right now, YourNextMP offers open data on every candidate for the UK general election. That data is being used by major media companies and internet giants, and underlies several innovative online tools. On top of that, it’s getting thousands of visits every day from people who simply want more information about who’s standing in their area.
With some modification, other countries could use the same tech in advance of their own elections, giving their citizens the same opportunities to become more informed about those standing, and to develop still more useful online tools.
This is a ‘Yay for Poplus’ moment
Because Poplus is an international federation of organisations with similar needs, we can come together to forge plans that will benefit all of us, and then work together to make them a reality.
Our plans wouldn’t just benefit those six countries, either. Like every bit of Poplus tech, it’d be available as open source software for anyone to use, anywhere in the world. And that’s what Poplus is all about: maximum impact from every bit of code.
Join the Poplus mailing list to find out more about Poplus activities
Give some applause to our Knight Foundation News Challenge bid