Placing EveryPolitician on hold

After over five years of active development we have decided to pause work on the EveryPolitician project for the foreseeable future.

In this post we’ll outline where we are leaving things, how you can make use of the data that does exist, and how you might be able to help migrate or transfer some of what we’ve collected over to services like Wikidata.

What’s in place today

The EveryPolitician project is, as its name suggests, based on the simple idea to gather accurate and up-to-date data on every politician in the world, collated and shared in a consistent format for free download and use by researchers, democracy projects, campaigners and individual citizens.

Over the course of the project we have gathered, structured and shared data on 78,382 politicians from 233 countries and territories presented on EveryPolitician.org via hundreds of scrapers run on morph.io and hosted on GitHub, producing the data on everypolitician-data.

Mostly the data covers the main chambers of recent parliaments around the world, but it also includes thousands of entries for previous parliaments, in some cases going back decades.

This has been a sizeable undertaking, involving a handful of very talented developers and colleagues within mySociety, as well as contributions from dozens of other organisations and individuals, many of whom make use of the data within their own projects.

The reality is that this work is hugely time consuming, complex and requires not just expert knowledge but a commitment to go deep into the intricacies of parliamentary data in order to make it comprehensible to a wider group of users. And looking to the next couple of years this task is only ever going to increase in complexity — too much for one underfunded organisation.

We therefore intend to freeze the current data as it currently stands, and it will continue to be available for download and reuse. We just can no longer commit to keeping this data up to date.

Always playing catch up

The challenge with data projects like EveryPolitician, beyond the complexity of understanding the structures and relationships within hundreds of individual parliaments (every parliament is an edge case in some way), is that the data is always steadily going out of date.

Across the world’s national parliaments there is an election somewhere roughly once a week, and that’s often when parliaments choose to update their websites, sometimes breaking our scrapers and changing the format of the data. Throughout the life of a parliament you might expect a few percent of MPs to change, sometimes more in different systems, so keeping on top of those individual changes is a sizeable task – especially where errors or duplications occur.

In addition to managing the hundreds of scrapers, we also included data from other sources — increasingly from Wikidata. Over the past 18 months we’ve been attempting to migrate more and more of what we’ve learned on EveryPolitician over to Wikidata via the WikiProject every politician.

Where the project goes next

EveryPolitician was built on the many years of work we had already delivered in this area, through PopIt, Poplus and working with Popolo. We knew what was needed, what worked and what didn’t.

We saw the potential to create an Open Corporates for political data, and hoped that EveryPolitician would be able to attract grant funding to grow, and potentially develop appropriate commercial services in support.

However, after five years of significant investment we just don’t have the funding to continue this work on our own.

In time we hope to be able to continue to contribute again to the wider availability of political data, and with hindsight it’s clear that Wikidata should be the natural global home for this type of data – benefitting from much greater reach, the contribution of motivated individuals in each country, and from the wider Wiki community.

As part of our contribution to Wikidata, we’ve created numerous tools to support the cross-referencing, verification, and supported update of data between EveryPolitician and the Wikiproject. This is still something of a work in progress, but we see it as a key way that others might contribute and take on aspects of the project in the future.

In the meantime we hope that many people continue to make use of the wealth of data that’s already been collected.

If you have a specific interest in a country, group of legislatures or some other combination, perhaps you can consider adding the kind of data that  EveryPolitician has collected to Wikidata. We have no further resources to devote to this work; however if you do have an interest in taking some of this on then we will try to advise what options might best suit.

Image: Jelle van Leest