1. GLOW is calling: help increase the world’s knowledge of politicians

    If you know your way around Wikidata, we’d love you to join in with the global string of events taking place for GLOW next week.

    We’re very keen to get as many people as possible helping to improve the quality of Wikidata’s information on politicians. Why? Well, let’s take a quick look at a recent story that hit the news.

    A new Bundestag

    With Germany’s new parliament gathering for the first time on October 24, der Spiegel took the opportunity to examine their male-to-female balance, in the context of legislatures across the world. At around 31% female, they noted, the Bundestag now sits at the better end of the scale: parliaments almost everywhere are male-dominated.

    How were they able to make such an assessment? As they note at the foot of their article, they used data on politicians’ gender from our EveryPolitician project.

    A further exploration looked at age — they discovered that on average their parliamentarians were very slightly younger than in previous years — and they note as an aside that here in the UK, we have in Dennis Skinner the oldest MP in Europe, while Mhairi Black is the second-youngest by a whisker.

    These are the kind of insights we seek to increase through our work with Wikidata as we help to boost the quality of their politician data: we consider such analysis not only interesting, but important. Whether or not countries wish to encourage fair representation across age groups and gender — not to mention many other categories — their decisions should at least be based on facts.

    As things stand, there are only a handful of countries where data is good enough to be able to make such comparisons: in our vision, journalists, researchers — and anyone else — will be able to turn to Wikidata to find what they need. The forthcoming Global Legislative Openness Week (GLOW) gives us all an opportunity to put a rocket under the quality and quantity of data that’s available to people making analyses like these, that stand to benefit us all.

    How to get involved

    GLOW runs from next Monday until the 30th November, and we’re encouraging people — wherever you live in the world — to get together and improve the data on national-level politicians for your country.

    We’re already expecting a good number of groups to run events. Get-togethers are confirmed in Slovenia, Bulgaria, Italy, Greece, Spain and more — once final details are firmed up, we think there’ll be action in other countries across the globe. Now how about you? As we said in our post last month, a concentrated effort from a small group of people can really make a difference.

    We’re especially keen to encourage folk who have some experience of contributing to Wikidata: we reckon that, for this particular drive, you need to already know your way around a bit. So if that’s you, do come forward!

    Start by having a look at this page, which outlines what we hope to achieve; we’ll be adding more detail this week too.  You can add your country to the list if you’d like to, or explore what’s missing in the data of those countries already listed.

    Or, if Wikidata’s all new to you, why not put out some feelers and see if there’s anyone who can show you the ropes while you work together? One good way is to see if there’s a Wikimedia User Group local to you.

    What exactly will you be doing?

    Here’s a bit more detail on what a workshop will look like.

    The idea is to improve information in Wikidata about members of your country’s legislature. The ‘Progress Indicators’ on this page will give you guidance: typically you’ll be working through tasks like adding any missing “position held” statements and biographical data. We’re asking folk to prioritise current politicians, with information for historic members an added bonus if time permits.

    Once sufficient data is available in Wikidata, the real fun begins! Your workshop attendees will be able to query the data to answer questions such as:

    1) Can the gender breakdown and average age of members of the current legislature be calculated?
    2) Can that be broken down per political party/group, or (where appropriate) by region?
    3) Can you compare those figures for the legislature vs. the cabinet?
    4) How far back can you generate those for?

    And if the ideas start to flow, building queries and visualisations to answer other questions will also be very useful.

    Let us know if you have any questions before the week begins — we’re going to be very busy during GLOW, but we’ll do our absolute best to help.


    Image: Alex Iby (Unsplash)

  2. Join the GLOW: run a Wikidata workshop in your country

    Global Legislative Openness Week (GLOW) celebrates open, participatory legislative processes across the globe.

    Back in 2015 we marked GLOW by setting up a challenge: could we get politician data for 200 countries up on EveryPolitician within the week, with the help of the global community? The answer was a resounding yes, and the challenge was a massive success. We ended up with data for 201 countries in the end, thanks to help from awesome people from all over the world.

    This year, we’re running another challenge: to get as many Wikidata workshops focusing on political data to happen during GLOW as possible.

    Fancy helping with this challenge? Read on…

    Why?

    This is all part of our Wikidata/EveryPolitician project.

    The project aims to improve political data in Wikidata, so that it can be used more easily for projects, research or investigations that hold politicians to account. Examples of where good political data is vital include in parliament-tracking websites (like in Zimbabwe, Nigeria and Ukraine) and in cross-border journalistic investigations (like the Panama Papers).

    Providing this data in consistent and structured formats across countries means the people running these accountability projects spend less of their time gathering the data and more on actually using and interpreting it, to keep tabs on those in power. This project is all part of our mission to contribute to the Democratic Commons.

    One of the best ways to improve and use political data on Wikidata is to get people together in person to work on their country’s data. So, that’s the aim with this latest GLOW challenge, and we’d love for as many groups around the world to host Wikidata workshops as possible!

    The aim of these Wikidata workshops is to:

    • Improve political information in Wikidata so that developers, researchers and journalists (or anyone!) can use the data in their investigations and accountability projects.
    • Use and query existing political data in Wikidata to see what interesting questions can be answered when data is available in consistent and structured formats.

    Workshop attendees will go away with:

    • Increased knowledge of how Wikidata works and how to contribute to it
    • A better understanding of why good political data is so vital and how it can be used
    • New connections to the global community of people who care about accountability issues
    • A warm, fuzzy feeling of satisfaction that they’ve helped with the global accountability movement 😉 (We hope so anyway!)

    Not sure what such an event might look like? Read up on our recent Wikifying Westminster workshop: it really showed us how much can be done when a few people get together in a room.

    Funding and support available

    Thanks to the Wikimedia Foundation, we’re able to offer some support to individuals/groups who are interested in running Wikidata workshops like these during GLOW. This will differ on a case by case basis but includes:

    • A small amount of funding to help cover event costs
    • A review of your country’s existing political information in Wikidata and some pointers about possible next steps
    • Ideas for how you and your attendees can:
      a) Use the data for interesting research and projects, and
      b) Improve the data for future research queries/projects
    • In-person support during your event – if you’d like one of our EveryPolicitian/Wikidata team to come to your event to present and participate, we can do this (if our budget allows!)
    • Access to a dedicated Slack channel which connects you with other groups around the world who are also running events during GLOW.

    Workshops can take place at any time within GLOW week, which is from 20th-30th November 2017 (yes, that is a long week!).

    Get involved!

    So, if you’d like to be part of this global challenge to improve and use political information in Wikidata, we’d be thrilled to hear from you. Please do get in touch: gemma@mysociety.org

  3. Democratic Commons: working with Facebook

    Earlier this month, Mark laid out the concept of a Democratic Commons for the Civic Tech community: shared code, data and resources where anyone can contribute, and anyone can benefit.

    He also talked about exploring new models for funding the kind of work that we do in our Democracy practice at mySociety.

    For many years, our Better Cities work has been proof of concept for one such model: we provide data and software as a service (FixMyStreet, MapIt, Mapumental) to paid clients, the revenue from which then funds our charitable projects. Could a similar system work to sustain our Democracy practice?

    That’s the hope, and with Facebook who we first worked with during the UK General Election in June, providing the data that helped people see and connect with their elected representatives, we’ve already seen it in action.

    This kind of project is positive on multiple levels: it brings us an income, it brings the benefits of democratic engagement to a wider audience than we could reach on our own, and it contributes data back into EveryPolitician and Wikidata, that everyone can use.

    Interesting challenges

    The UK election was only the first for which we did this work: we’ve gone on to provide the same service for the French elections and more recently for the rather more eventful Kenyan ones — currently on hold as we await the re-run of the Presidential election next month. And now we’re doing the same for the German elections, where candidate data is being shared this week.

    As we’re learning, this is definitely not one-size-fits-all work, and each country has brought its own interesting challenges. We’re learning as we go along — for example, one significant (and perhaps obvious) factor is how much easier it is to work with partners in-country who have a better understanding of the sometimes complex political system and candidates than we can ever hope to pick up. Much as we might enjoy the process, there’s little point in our spending days knee-deep in research, when those who live in-country can find lists of candidates far more quickly, and explain individual levels of government and electoral processes far better.

    Then, electoral boundaries are not always easy to find. We’ve used OpenStreetMap where possible, but that still leaves some gaps, especially at the more granular levels where the data is mainly owned and licensed by the government. It’s been an exercise in finding different sources and putting them all together to create boundary data to the level required.

    Indeed, that seems to be a general pattern, also replicated across candidate data: at the national level, it’s easy to find and in the public domain. The deeper you go, the less those two descriptors hold true. It was also at this point that we realised how much, here in the UK, we take for granted things like the fact that the spelling of representatives’ names is usually consistent across a variety of sources — not always a given elsewhere, and currently something that only a human can resolve!

    Giving back

    What makes all the challenges more worthwhile, though, is that we know it’s not just a one-off push that only benefits a single project. Nor is the data going straight into Facebook, never to be seen again.

    Much of what we’re collecting, from consistent name data to deep-level boundaries data, is  to be made available to all under open use licenses. For example, where possible we can submit the boundaries back to OpenStreetMap, helping to improve it at a local granular level across whole countries.

    The politician data, meanwhile, will go into Wikidata and EveryPolitician so that anyone can use it for their own apps, websites, or research.

    There are also important considerations about how this type of data will be used and where and when it is released in the electoral process; finding commercial models for our Democracy work is arguably a more delicate exercise than on some of our other projects. But hopefully it’s now clear exactly how a project like this can both sustain us as a charity, and have wider benefits for everyone — the holy grail for an organisation like us.

    At the moment it’s unclear how many such opportunities exist or if this is a one-off. We’re certainly looking for more avenues to extend the scope of this work and keen to hear more ideas on this approach.


    Your donations keep our sites running.
    Donate now


    Photo: Justin Tietsworth, Unsplash

  4. When Westminster Was Wikified

    Last Saturday (August 19th) at Newspeak House in London, mySociety and Wikimedia UK held the “Wikifying Westminster” workshop, a day-long event to encourage people to get involved with Wikidata, but also to give a taste of what people can build with the data that is already there.

    The vision: one day, complex investigations which currently take researchers a lot of time, such as “how many MPs are descended from people who were also MPs” or “how many people named X were MPs in year Y”, will be answerable with data from Wikidata using a single SPARQL query…

    …but we’re not quite there yet. Currently, some data is scattered all over separate databases (which sometimes get shut down or disappear); some is just plain missing; and most frustrating of all, some is in place but there’s no apparent way to get it out of the database.

    In order to make this vision a reality, we need to experiment with the data, find ways to check how complete it is, and explore what questions we can currently answer with it. Events like Wikifying Westminster are the perfect opportunity to do just that.

    After a brief introduction to Wikidata and the EveryPolitician project, we split into two groups: one focused on learning how to use Wikidata, while the other focused on working on mini-projects.

    Here’s a taste of what happened…

    Learning track

    The learning track began by introducing new users to the basic Wikidata editing principles (or “getting data into Wikidata”). Participants were able to put their new skills into action immediately, by adding missing data on British MPs, who were mostly lacking dates and places of birth.

    By the end of the first session, good progress had been made, particularly on obtaining dates of birth for current British parliamentarians. For some reason, though, it proved much harder to find these for women than for men: we can only speculate as to why that might be (do some still adhere to the idea that a woman shouldn’t reveal her age?!).

    Lucas query Wikidata

    Lucas Werkmeister from Wikimedia Deutschland shows how Wikidata query service works (photo credits: Lucy Chambers)

    We were also given an introduction to SPARQL, a language used to query information on databases (or “getting data out of Wikidata”). Lucas Werkmeister introduced the Wikidata query service and explained a few tricks to help with using it. Participants were later able to put this to the test by running progressively difficult test queries such as “All current UK MPs” or “Who is the youngest current MP?”

    Also, Navino Evans showed us the potential of reusing data, talking about Histropedia, which he co-created with Sean McBirnie. Histropedia is an awesome tool that lets you visualise thousands of topics on interactive timelines: you can browse through existing ones or create a new one from scratch.

    Hacking track

    This group both worked on improving data and looked at how well we could answer some simple “stepping stone” queries (i.e. small questions to which we already knew some of the answers) as a heuristic of how good the data in Wikidata already is. You can see and contribute questions to the list of test queries here.

    Some more details:
    Improving data. The focus here was on the Northern Ireland Assembly, for which Wikidata now has full membership history back to the foundation of the Assembly, and on adding academic degrees of cabinet ministers. Starting from an excellent spreadsheet of the undergraduate universities and subjects of UK politicians and ministers (going back to John Major’s cabinets), we tried to upload that data on the relevant items, adding the qualifier “academic major” (P812) to the property “educated at” (P69). In this case, the key problem we found was that we weren’t sure how to model when people did joint subjects, like “Maths and Politics”, convincing us to concentrate on the more obvious subjects first.

    Answering some unusual and/or intriguing questions. Inspired by a prior finding that there are more FTSE 100 CEOs named John than there are female ones, and that John is historically the most common name of UK parliamentarians, we thought we’d find out when exactly the John-to-female balance was toppled amongst the UK’s MPs (hint: not until 1992).

    Going back further in history, we queried the first time each given name was recorded in Parliament, this was inspired by a recent news article about an MP who claimed he was the first “Darren” in the Commons.

    Some ideas were also born that we weren’t able to see through, for various reasons. For example, could we discover which, if any, MPs are descended from people listed in the UCL’s ‘Legacy of British Slave-owners’ database? An interesting question, but at the moment, the answer is ‘no’, partly because child-parent relationships are currently inconsistently modelled in Wikidata, and partly because of the nature of Wikidata and ancestry: if there is someone who doesn’t exist in Wikidata (e.g. Grandad Bob, the painter) in the family chain, Wikidata can’t bridge the gap between a present day MP and the slave owner who might be their ancestor.

    This is just the beginning

    Work, of course, is still ongoing: all pre-1997 UK data is still to be inserted or improved on Wikidata, and so much more is missing – family connections, academic degrees, links to other databases, and all sorts of “unusual stuff” that can be used for interesting queries.

    This data is crucial if we want to be able to answer the really big questions which Wikidata should one day be capable of helping us explore, about what politicians do.

    We can do that together!

    Wikifying Westminster - Build cool things with Wikidata


    (photo credits: Lucy Chambers)

    We hope that events like this give people an easy way in to Wikidata and also show them what’s already possible to achieve with the data. Over the coming months, we are hoping to support more events of this type around the world. If you are interested in getting involved, here’s how:

    • Want to improve your country’s data? Events like this can be a great way to help kickstart activities and find other people who share your goals. We are happy to help out and support people in other countries to do so.
    • Are you already organising or planning to organise a similar workshop around Wikidata? Make sure it is listed on the Wikidata Event page!
    • Do you want to attend future workshops? Follow us on Twitter to stay updated about events that we are running, and ones that other people are too!

    Feature image credits: Mark Longair

  5. EveryPolitician: the road ahead

    This post is by Tony Bowden and Lucy Chambers from the EveryPolitician team. Today we officially launch our collaboration with Wikidata – here’s what to expect…  

    The story so far

    You might have been following the progress: since 2015, through our project EveryPolitician, we’ve been gathering data for every national legislature in the world, from thousands of sources, and sharing it.

    Now, two years on, we’ve started to see some great results. For example:

    • It’s much easier to build simple Parliament Tracking sites (as mySociety partner organisations have in Zimbabwe and Nigeria). Those running the sites can work on providing information and context to hold politicians accountable — and don’t have to worry about wrangling data and software.
    • Tools that allow citizens to write to their representatives (like Majlis Nameh in Iran,  based on the WriteInPublic software, or Oxfam’s UK and Australia Campaigning Tool) can now be deployed in days rather than weeks, allowing groups to focus on local customisation.
    • Vote-tracking sites like TheyVoteForYou in Australia can be more easily adapted to other countries — such as Ukraine, again without worrying about the major task of sourcing the politician data.
    • Projects that highlight politicians’ activity can be augmented to show extra information against those politicians — so for example, Politwoops can show party affiliation next to politicians’ tweets.
    • Investigative journalists can cross-match our lists of politicians to other sources (e.g. investigating shell companies for a recent Private Eye article or with the Panama Papers).
    • mySociety’s collaboration with Facebook makes it easy for people to connect with their newly elected representatives.

    All of this is possible because of two main tenets of EveryPolitician: having gathered the data, we structure it consistently, and we share it freely.

    Not just current data

    We’ve also discovered that there is a huge value for everyone in retaining historic information, too:

    • Old data tends to disappear from many official sites, for example, when a new government comes into power and / or when parliaments decide to remodel their websites.
    • Sometimes even the official sources no longer exist: for example, in Burkina Faso the Parliament building was burned down by protesters in 2014 and many crucial documents were lost.
    • Research projects and sites run by civil society organisations sometimes run out of funding and have to shut down — meaning that the data people were relying on can vanish overnight.

    The future: going one step further

    So that’s all great, but we think EveryPolitician can do still more to help the worldwide community of Civic Tech coders and activists.

    In particular, we want to go beyond data showing who the politicians are, and also provide information on what they do. That’s because we can see real value in the ability to answer questions like:

    • When we look at politicians who vote on issues such as gay marriage, smoking bans, tax on sugary drinks, etc, anywhere in the world — are there any broad correlations like age or gender?
    • As countries elect more women MPs, do those women also gain equivalent representation in committees? Does this affect attendance and participation rates?
    • Are there standard political career paths that can be observed anywhere in the world? So for example, do certain Cabinet positions limit future progression; and if so, would it be feasible to spot politicians who are on the way up, or on their way out?
    • Do politicians who move from the lower to upper house act, and vote, differently?
    • Do politicians change their voting activity after certain types of intervention, for example after receiving funding from oil companies?

    … and there are undoubtedly millions more questions, each one just as interesting and with answers that could enrich our understanding of the world. We are aiming for a future in which each should be answerable within minutes, rather than form the basis of a multi-year post-graduate research project.

    How we plan to get there

    You may remember our recent proposal to integrate more deeply with Wikidata. We’re delighted to say that our proposal was accepted, and that makes EveryPolitician’s path very clear.

    Wikipedia is fast becoming one of the best sources of political information in many countries: it’s often updated more quickly than major news outlets or official parliamentary sites are.

    Which is great, but there’s an issue when it comes to using that information for projects like the ones we’ve mentioned above: Wikipedia contains largely unstructured information (that is to say, information that comes in a wide variety of different formats) — as you’d expect from any project with multiple contributors and, often, a free-text input.

    There are also a lot of differences between Wikipedias in different countries. Some countries’ data (particularly countries where the Wikipedia community is larger) gets updated very quickly. Smaller language Wikipedias can’t rely on such a large pool of editors, and it tends to be longer before they are updated.

    Additionally, and as you might expect, Wikipedia content will appear fastest in the countries most directly affected by the change being documented, so for example when there are elections in Estonia, the Estonian Wikipedia may show the results almost immediately, but it can take a while for those changes to trickle into all languages.

    The solution to all this? We believe it’s Wikidata: the structured sibling of Wikipedia. If Wikidata is new to you, there’s an easy introduction here and you can also take a tour to learn more.

    But of course, nothing’s ever quite that simple. While Wikipedia has a wealth of unstructured political information, on Wikidata there’s still an awful lot of data missing. You may recall that we recently ran a drive to ensure that every country at least had its head of government entered, but that’s just the beginning: in order to answer the kinds of questions we mention above, we really need to ensure there’s consistently-structured information for all legislators, all elections… and much more.

    What you’ll see in the coming months

    While we plan for EveryPolitician to retain its own identity and keep its own front end, we’re excited to say that over the next few months, we’ll be teaming up with Wikidata communities across the world.

    Our first objective is see if we can bring Wikidata up to the same level as EveryPolitician for as many countries as possible.

    And once we hit that target, we plan to go much further. The beauty of Wikidata is that you can add pretty much any information you want to add to politicians (indeed, to anything!), so local communities can decide for themselves which information is most pertinent and make sure that it’s included.

    To help us make all of this happen, we’re expanding. If you’ve read this far and you’re still finding the project interesting, the chances are you’d be a great addition to the team! We’re looking for a new community co-ordinator, working from anywhere in the world (compatible timezones allowing): you can see all the details here.

    How you can get involved

    There are plenty of ways that you can help with this drive to improve the structured information on Wikidata. Here are the obvious ones:

    • Would you be interested in organising a push to Improve Wikidata for your country? Get in touch and we’ll do what we can to help.
    • If this all sounds interesting, but you’re not sure where to start, or you’re unfamiliar with Wikidata – drop us an email and tell us what you’re interested in. We’re more than happy to help you get started.

    Image: Aftab Uzzaman (CC by-nc/2.0)

  6. Help us answer key questions about politicians using Wikidata

    Last you heard from from the EveryPolitician team we were talking about integrating EveryPolitician more deeply with Wikidata.

    Here’s a quick update on what to expect in the coming months and a note on how you can get involved – we are going to need a lot of help!

    Before we can make headway on a deeper integration, there are some pretty foundational pieces of the puzzle we need to put in place.

    Some key data is missing, and you may be surprised to discover that it is pretty basic stuff. Take a look at this for example:

    Report 1313/1308

    This is from a report we have generated to highlight gaps in the links between items. Notice how not even all of the offices have been filled in on the country page, let alone who occupies those offices?

    This is the type of foundational data we are hoping to get in place over the next little while, and you can help

    How you can get involved

    Over the coming weeks we’re going to be conducting a few experiments in public and trying to get data into Wikidata. We’ll need as much help as we can get!

    The experiments will focus on using Wikidata to attempt to answer some questions we find interesting and see how we can expose gaps and inconsistencies in the data. In doing this, we’ll be pointing to specific reports we have generated and asking you to help us fill in the gaps.

    Our first question will be:

    “What is the gender breakdown of heads of government across the world?”

    We’ll be blogging about the investigation over on Medium. The first post is already there:

    Help us find the offices of heads of governments across the world!

    Keen to dive straight in? Help us fill in the blanks you see in this report! If you are familiar with Wikidata, you will probably be able to get started straight away, but there are also tips and pointers in the Medium post if you want a bit more guidance.

    You can get notified of the challenges to complete the data as soon as they are published by following the EveryPolitician bot on Twitter.

    Want even more details on our plans?

    We should also mention that we’ve updated our proposal to the Wikimedia Foundation to make a couple of things clearer about the problem we are trying to solve and our proposed solution (the proposal is still being reviewed). If you are interested in the nitty-gritty, that’s the best place to get the full overview of what we are planning.

    Image: Dylan (CC by/2.0)

  7. Using EveryPolitician data: contact details

    Our EveryPolitician project makes data on the world’s politicians available in a useful, consistent format for anyone to use. If you’ve been following our progress, you’ll know we’ve already collated a lot of data (over 72,000 politicians from 233 countries). The work on adding to the depth and breadth of that data is ongoing, but EveryPolitician data is already being used to do interesting things.

    Previously we looked at Politwoops as an example of EveryPolitician data being used to augment existing data.

    In that case, the useful data for Politwoops was the politicians’ party affiliation. But our team (a handful of humans and one very busy bot) collects richer data than just that. EveryPolitician data includes contact information for politicians.

    At mySociety, we know how powerful this particular kind of data can be. For example, our WriteToThem site makes it easy for UK citizens to contact their representatives (WriteToThem grew out of the earlier online service FaxYourMP, and uses the now more common technology of email).

    Of course, there’s nothing especially radical about collecting email addresses of politicians… or phone numbers, Twitter handles, or Facebook pages. Indeed, many individuals and groups do just that. But an important difference with EveryPolitcian is that we’re not just collecting data (which happens to include those things, as well as a host of others) but also making it available so it’s easy to use. We do that by putting it out in consistent, useful formats.

    For many projects, downloading a CSV of current politicians from EveryPolitician will be enough. That can be opened as a spreadsheet, and if one of those columns is called email, you’re good to go.

    Opening a spreadsheet is just one way of accessing the data. Our own use of EveryPolitician data to power the “Write in Public” MajlisNameh site for ASL19 (see this blog post for more about that) demonstrates a more programmatic approach.

    But the whole point of making data available like this isn’t so that we can use it. It’s for other people, other groups. Anyone can build more nuanced or complex services with this data too.

    represent-banner

    For example, the people at Represent.me have built a sophisticated platform for gathering opinions and votes that can be shared with politicians and constituency MPs. It’s a system of information-gathering that has a network of citizens at one end feeding into their political representatives at the other. They use EveryPolitician’s data to populate their system with information about those representatives, including contact details, for each country they operate in.

    And, because we make sure our data is consistently formatted, it’s a good general solution. As they cover more areas, they can expect the code they’ve written to ingest the EveryPolitician data in the countries they’re already operating in to also work as they expand into others.

    If you’re running a project that needs such data, you could invest time and effort finding and collecting it all yourself. But it’s almost inevitable that you’d be using the same public sources that we are anyway — after all, we try to identify and use all the sources we can, merging them together into one, collated whole — so really it makes sense to simply take the data from EveryPolitician. Remember, too, that once our bot has been told about a source, it checks it daily for changes and updates too. So instead replicating the effort we’re already doing to gather the same data you need, you’re free to focus on developing the way your project uses that data… while we hunker on down and get on with collecting it.

    Inevitably, as with all software projects, there’s always lots more to do, but already the value of providing useful data — and especially contact information — in a consistent format is clear.


    Image: Telegraph Chambers (Montreal) CC BY-NC-ND 2.0 by Andre Vandal

  8. Help us share EveryPolitician’s data where it will make even more impact

    Over the last two years, we’ve gathered data on the top-level politicians of almost every country in the world, and made it accessible to developers everywhere through our project EveryPolitician.

    Now we’d like to take a step that we believe will benefit more people, and further extend the usefulness of this extensive dataset. We’re proposing to integrate more deeply with Wikidata, to fill the gaps in their coverage and provide consistent, linked data to their global community.

    Wikidata is the central storage for the structured data each of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wikisource, and others. Wikidata also provides support to many other sites and services beyond just Wikimedia projects so the combination of EveryPolitician’s data with the reach of Wikidata’s community is pretty compelling.

    So in many places, the aims of the EveryPolitician and Wikidata projects are already aligned. We already synchronise EveryPolitician data with the good quality data available in Wikidata where we find it, and we feed back our own additions. As our datasets improve, it seems prudent to combine efforts, and resources, in one place.

    You can see our proposal to make this happen here.

    If you play an active part within the Wikidata community, or are someone who would benefit from this initiative, we’d very much appreciate your support. Please do add your endorsements or thoughts at the foot of that page if you’d like to see the project go ahead.

    Image: Opensource.com (CC by-sa/2.0)

  9. How’s that Gender Balance data?

    International Women’s Day seems like a good time to check in on our project Gender Balance, the crowdsourcing website that invites users to help gather gender data on the world’s politicians.

    As you may recall, our aim was not simply to present top-level numbers: data already existed that allows us to, say, understand which legislatures have the most even-handed representation, genderwise.

    No, Gender Balance seeks to go more in-depth: by attaching gender data to individual politicians, and making that data available via structured datasets, we hope to allow for more subtle comparisons to be made.

    For example, researchers may like to test theories such as, ‘do women vote differently from men?’, or ‘do women politicians make different laws around childcare?’ — or a whole host of other questions, all of which can only be answered when gender data relates to specific public figures, or when it is viewed in combination with other data.

    The data that is collected when you play Gender Balance goes, with data from other sources, into EveryPolitician, our project that seeks to provide structured, downloadable, open information across all the world’s legislatures.

    Not right away, mind you. To ensure that the data really is accurate, we make sure that each politician on Gender Balance is presented to at least five different players, all of whom give the same answer, before we consider it verified.

    Data on Kazakhstan's politicians, including gender, from EveryPolitician

    EveryPolitician currently contains data for about 73,000 politicians in total. In some cases, that data came to us along with a trusted gender field, so we don’t need to run that through the Gender Balance mill, but the majority of parliamentary sites don’t provide this data.

    We can sometimes obtain that information from other sources, but Gender Balance has been invaluable in filling in lots of the gaps. Thanks to our players, it has already provided us with gender information for over 30,000 politicians (and in some cases, pointed out discrepancies in the data we obtained from elsewhere).

    There’s still plenty to go, though, if you’d like to help; and, as elections happen around the world, Gender Balance will continue to refresh with any politicians for whom we can’t find trusted gender data. As we speak, approximately 22,000 politicians still need sorting.

    That might sound like quite a lot, but each politician need only take seconds — and every little helps. So, if you’d like to help contribute a little more gender data, just step this way.


    Image: India’s Prime Minister Narendra Modi at the valedictory session of the National Conference of Women Legislators in Parliament House CC BY-SA 2.0, via Wikimedia Commons

  10. Using EveryPolitician data: Politwoops

    Politwoops tracks politicians’ tweets, and reports the ones that are deleted.

    Often those tweets are deleted because of a typo: everyone makes simple mistakes with the buttons on their devices, and politicians are no less human than the rest of us.

    But Politwoops’s targets are public servants who use Twitter to communicate with that public. And sometimes the contents of the tweets they delete are not simply the result of bad typing. Those tweets can be especially interesting to people whom those politicians are representing: sometimes they may be evidence of a usually-suppressed prejudice, or an attempt to remove evidence of a previously held opinion that is no longer convenient.

    In effect, Politwoops is a public archive of direct quotes that would otherwise be lost.

    And also… EveryPolitician

    Our EveryPolitician project is an ever-growing collection of data on every politician in the world (we’re not there yet, but we’re over 230 countries and 72,700 politicians in, and counting).

    Like Politwoops, our data includes politicians’ Twitter handles. But also a lot more besides.

    We make that data useful by putting it out in consistent, simple formats — the simplest of which is a comma-separated value (CSV) file for each term of a legislature. In practice, that means if you want a spreadsheet of the current politicians in your country’s parliament, then EveryPolitician is probably the place for you.

    Put them together…

    Now, Politwoops predates EveryPolitician by several years, and they’ve being doing their thing without needing our data just fine. In fact, Politwoops has been happily politwooping since 2010 (Politwoops is a project of the Open State Foundation, based in the Netherlands).

    Behind the scenes, it works pretty much the way you’d expect: with a list of politicians’ Twitter handles for each country where it’s running.

    But… who doesn’t want to add something extra for free? Our data also includes Twitter handles (mostly but not entirely from the same public sources Politwoops were using). So that meant they could take our CSVs and match each line—all that extra data!—with the Twitter handle.

    Better, for free

    So last year, they augmented their data with ours for one very simple win: they get to know party affiliation for the politician associated with each of those twitter handles. Well, actually they get to know lots of other things besides party — gender, date of birth, or… well, all our other data, if they wanted it. But just party? That’s also fine.

    This all means that Politwoops now shows the party of each tweet’s deleter, just because they merged our CSV with theirs. Lovely!

    A tiny detail...  party affiliation (arrowed) added to @deletedbyMPs tweets [screenshot of https://politwoops.co.uk/]

    A tiny detail… party affiliation (arrowed) added to @deletedbyMPs tweets [screenshot of https://politwoops.co.uk/]

    Although party affiliation was the detail Politwoops went for, it turns out the other data from EveryPolitician was a little too tempting for them to ignore… So recently they’ve been doing some playful analysis on their statistics using the gender breakdown that EveryPolitician data makes possible too. You can see more on the Politwoops website.

    You can too

    To be clear: Politwoops did this, not us. We’re committed to doing the groundwork of finding, collecting and collating the data, and making it available (and, additionally, endlessly checking for updates… if you’re interested in how this all works, you can read our bot’s own blog). We do this so people who want to get on with using the data can do just that. As did, in this case, Breyten and his team at Politwoops.

    EveryPolitician’s data is available as plain CSVs for this kind of thing, but we also provide a richer JSON version too if that’s more useful to you. All the files are downloadable from the website. If you’re a coder who wants to dive in, there are libraries to make it even easier for you (the EveryPolitician team works in Ruby, so we wrote the everypolitician gem, but there are also ports to Python and PHP).

    For more information see the docs.everypolitian.org.

    The EveryPolitician bot wrote its own version of this blog post, which goes into a little more detail of the process.


    Image: Tom Maladova (CC by-nc-nd/2.0)