Our EveryPolitician project makes data on the world’s politicians available in a useful, consistent format for anyone to use. If you’ve been following our progress, you’ll know we’ve already collated a lot of data (over 72,000 politicians from 233 countries). The work on adding to the depth and breadth of that data is ongoing, but EveryPolitician data is already being used to do interesting things.
Previously we looked at Politwoops as an example of EveryPolitician data being used to augment existing data.
In that case, the useful data for Politwoops was the politicians’ party affiliation. But our team (a handful of humans and one very busy bot) collects richer data than just that. EveryPolitician data includes contact information for politicians.
At mySociety, we know how powerful this particular kind of data can be. For example, our WriteToThem site makes it easy for UK citizens to contact their representatives (WriteToThem grew out of the earlier online service FaxYourMP, and uses the now more common technology of email).
Of course, there’s nothing especially radical about collecting email addresses of politicians… or phone numbers, Twitter handles, or Facebook pages. Indeed, many individuals and groups do just that. But an important difference with EveryPolitcian is that we’re not just collecting data (which happens to include those things, as well as a host of others) but also making it available so it’s easy to use. We do that by putting it out in consistent, useful formats.
For many projects, downloading a CSV of current politicians from EveryPolitician will be enough. That can be opened as a spreadsheet, and if one of those columns is called
Opening a spreadsheet is just one way of accessing the data. Our own use of EveryPolitician data to power the “Write in Public” MajlisNameh site for ASL19 (see this blog post for more about that) demonstrates a more programmatic approach.
But the whole point of making data available like this isn’t so that we can use it. It’s for other people, other groups. Anyone can build more nuanced or complex services with this data too.
For example, the people at Represent.me have built a sophisticated platform for gathering opinions and votes that can be shared with politicians and constituency MPs. It’s a system of information-gathering that has a network of citizens at one end feeding into their political representatives at the other. They use EveryPolitician’s data to populate their system with information about those representatives, including contact details, for each country they operate in.
And, because we make sure our data is consistently formatted, it’s a good general solution. As they cover more areas, they can expect the code they’ve written to ingest the EveryPolitician data in the countries they’re already operating in to also work as they expand into others.
If you’re running a project that needs such data, you could invest time and effort finding and collecting it all yourself. But it’s almost inevitable that you’d be using the same public sources that we are anyway — after all, we try to identify and use all the sources we can, merging them together into one, collated whole — so really it makes sense to simply take the data from EveryPolitician. Remember, too, that once our bot has been told about a source, it checks it daily for changes and updates too. So instead replicating the effort we’re already doing to gather the same data you need, you’re free to focus on developing the way your project uses that data… while we hunker on down and get on with collecting it.
Inevitably, as with all software projects, there’s always lots more to do, but already the value of providing useful data — and especially contact information — in a consistent format is clear.
Image: Telegraph Chambers (Montreal) CC BY-NC-ND 2.0 by Andre Vandal
Over the last two years, we’ve gathered data on the top-level politicians of almost every country in the world, and made it accessible to developers everywhere through our project EveryPolitician.
Now we’d like to take a step that we believe will benefit more people, and further extend the usefulness of this extensive dataset. We’re proposing to integrate more deeply with Wikidata, to fill the gaps in their coverage and provide consistent, linked data to their global community.
Wikidata is the central storage for the structured data each of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wikisource, and others. Wikidata also provides support to many other sites and services beyond just Wikimedia projects so the combination of EveryPolitician’s data with the reach of Wikidata’s community is pretty compelling.
So in many places, the aims of the EveryPolitician and Wikidata projects are already aligned. We already synchronise EveryPolitician data with the good quality data available in Wikidata where we find it, and we feed back our own additions. As our datasets improve, it seems prudent to combine efforts, and resources, in one place.
If you play an active part within the Wikidata community, or are someone who would benefit from this initiative, we’d very much appreciate your support. Please do add your endorsements or thoughts at the foot of that page if you’d like to see the project go ahead.
International Women’s Day seems like a good time to check in on our project Gender Balance, the crowdsourcing website that invites users to help gather gender data on the world’s politicians.
As you may recall, our aim was not simply to present top-level numbers: data already existed that allows us to, say, understand which legislatures have the most even-handed representation, genderwise.
No, Gender Balance seeks to go more in-depth: by attaching gender data to individual politicians, and making that data available via structured datasets, we hope to allow for more subtle comparisons to be made.
For example, researchers may like to test theories such as, ‘do women vote differently from men?’, or ‘do women politicians make different laws around childcare?’ — or a whole host of other questions, all of which can only be answered when gender data relates to specific public figures, or when it is viewed in combination with other data.
The data that is collected when you play Gender Balance goes, with data from other sources, into EveryPolitician, our project that seeks to provide structured, downloadable, open information across all the world’s legislatures.
Not right away, mind you. To ensure that the data really is accurate, we make sure that each politician on Gender Balance is presented to at least five different players, all of whom give the same answer, before we consider it verified.
EveryPolitician currently contains data for about 73,000 politicians in total. In some cases, that data came to us along with a trusted gender field, so we don’t need to run that through the Gender Balance mill, but the majority of parliamentary sites don’t provide this data.
We can sometimes obtain that information from other sources, but Gender Balance has been invaluable in filling in lots of the gaps. Thanks to our players, it has already provided us with gender information for over 30,000 politicians (and in some cases, pointed out discrepancies in the data we obtained from elsewhere).
There’s still plenty to go, though, if you’d like to help; and, as elections happen around the world, Gender Balance will continue to refresh with any politicians for whom we can’t find trusted gender data. As we speak, approximately 22,000 politicians still need sorting.
That might sound like quite a lot, but each politician need only take seconds — and every little helps. So, if you’d like to help contribute a little more gender data, just step this way.
Image: India’s Prime Minister Narendra Modi at the valedictory session of the National Conference of Women Legislators in Parliament House CC BY-SA 2.0, via Wikimedia Commons
Politwoops tracks politicians’ tweets, and reports the ones that are deleted.
Often those tweets are deleted because of a typo: everyone makes simple mistakes with the buttons on their devices, and politicians are no less human than the rest of us.
But Politwoops’s targets are public servants who use Twitter to communicate with that public. And sometimes the contents of the tweets they delete are not simply the result of bad typing. Those tweets can be especially interesting to people whom those politicians are representing: sometimes they may be evidence of a usually-suppressed prejudice, or an attempt to remove evidence of a previously held opinion that is no longer convenient.
In effect, Politwoops is a public archive of direct quotes that would otherwise be lost.
And also… EveryPolitician
Our EveryPolitician project is an ever-growing collection of data on every politician in the world (we’re not there yet, but we’re over 230 countries and 72,700 politicians in, and counting).
Like Politwoops, our data includes politicians’ Twitter handles. But also a lot more besides.
We make that data useful by putting it out in consistent, simple formats — the simplest of which is a comma-separated value (CSV) file for each term of a legislature. In practice, that means if you want a spreadsheet of the current politicians in your country’s parliament, then EveryPolitician is probably the place for you.
Put them together…
Now, Politwoops predates EveryPolitician by several years, and they’ve being doing their thing without needing our data just fine. In fact, Politwoops has been happily politwooping since 2010 (Politwoops is a project of the Open State Foundation, based in the Netherlands).
Behind the scenes, it works pretty much the way you’d expect: with a list of politicians’ Twitter handles for each country where it’s running.
But… who doesn’t want to add something extra for free? Our data also includes Twitter handles (mostly but not entirely from the same public sources Politwoops were using). So that meant they could take our CSVs and match each line—all that extra data!—with the Twitter handle.
Better, for free
So last year, they augmented their data with ours for one very simple win: they get to know party affiliation for the politician associated with each of those twitter handles. Well, actually they get to know lots of other things besides party — gender, date of birth, or… well, all our other data, if they wanted it. But just party? That’s also fine.
This all means that Politwoops now shows the party of each tweet’s deleter, just because they merged our CSV with theirs. Lovely!
Although party affiliation was the detail Politwoops went for, it turns out the other data from EveryPolitician was a little too tempting for them to ignore… So recently they’ve been doing some playful analysis on their statistics using the gender breakdown that EveryPolitician data makes possible too. You can see more on the Politwoops website.
You can too
To be clear: Politwoops did this, not us. We’re committed to doing the groundwork of finding, collecting and collating the data, and making it available (and, additionally, endlessly checking for updates… if you’re interested in how this all works, you can read our bot’s own blog). We do this so people who want to get on with using the data can do just that. As did, in this case, Breyten and his team at Politwoops.
EveryPolitician’s data is available as plain CSVs for this kind of thing, but we also provide a richer JSON version too if that’s more useful to you. All the files are downloadable from the website. If you’re a coder who wants to dive in, there are libraries to make it even easier for you (the EveryPolitician team works in Ruby, so we wrote the everypolitician gem, but there are also ports to Python and PHP).
For more information see the docs.everypolitian.org.
The EveryPolitician bot wrote its own version of this blog post, which goes into a little more detail of the process.
Last time we updated you about Alaveteli professional, the Freedom of Information toolset for journalists that we’re building, we were just coming out of our discovery phase.
Since then, we’ve made strides through the alpha and early beta part of our development process. In alpha, the idea is to build dummy versions of the tool that work in the minimum way possible — no bells and whistles — to test concepts, and our assumptions. Having thought hard about the potential problems of Alaveteli Professional, now is the time for us to try the approaches that we believe will solve them, by making prototypes of how the tool might work and testing them with a very small group of users.
In the early stages of beta, our priority has been to get to the point where a Freedom of Information request can go through all its various processes, from composition to response, with the features that a journalist user would need. Once that’s in place, it allows us to add other features on top and see how they would integrate.
This pattern — discovery, alpha, beta, release — is a well-tested method by which to produce a final product that works as it should, while avoiding costly mistakes.
Alpha and beta testing, perhaps unexcitingly, are all about the reduction of risk: in the words of the startup mantra, it’s good to ‘fail fast’— or rather, it’s better to know early on if something doesn’t work, rather than spend time and money on something that doesn’t fit the bill.
So, for Alaveteli Professional, what are the risks that have been keeping us awake at night?
We think the biggest priority is to ensure that there’s actually added value for journalists in using a service like this. Clearly, the Freedom of Information process is already available to all, whether via our own site WhatDoTheyKnow, or directly.
We need to be able to demonstrate tangible benefits: that Alaveteli Professional can save journalists time; help them be more efficient in managing their requests; maybe help them get information that otherwise wouldn’t be released; and give them access to rich data they wouldn’t otherwise be able to access.
For all we said about failing fast, the alpha phase also meant committing to some fairly big technical decisions that, ideally, we wouldn’t like to reverse.
Decisions like, do we build the service into the existing Alaveteli codebase, or go for a new standalone one (we went for the former)? From the user’s point of view, should Alaveteli Professional look like a totally different site, or like a registration-only part of WhatDoTheyKnow (we chose the latter)?
And onto beta
As we move from alpha to beta, we’re finding out what happens when real users make real requests through the service, and making adjustments based on their feedback.
What do they think of the way we’ve implemented the ability to embargo requests – does it make sense to them? Do they trust us to keep embargoed requests private? Are they able to navigate between different interfaces in a way that seems intuitive? mySociety designer Martin has been figuring out how to take the cognitive load off the user and give them just the information they need, when they need it.
We’re also returning to prototyping mode to work out how to implement new features, like the ability to send round robin requests to multiple authorities, in an effective and responsible way. The other half of our design team, Zarino, has been showing us that a slideshow in presentation mode can be an effective tool for demonstrating how users might interact with an interface.
As we continue to round out the feature set in the UK, we’re also cooking up plans in the Czech Republic so that later in the year we can present the tools to a new audience of journalists there and again, use their feedback to make the tools more flexible so that they can be used in different jurisdictions.
As you can see, there’s lots going on, and we’re all really excited to be finally getting some real life users in front of the tools that we’ve been thinking about, and working on, for so many weeks. Don’t forget to sign up to the mailing list if you’d like to keep up with Alaveteli Professional as it develops.
About six million people a year visit mySociety’s Freedom of Information website WhatDoTheyKnow.com; there are well over 100,000 registered users, and over 385,000 requests have been made via the service.
Of course, it’s fantastic that WhatDoTheyKnow is so well used, but the growth and popularity of the site brings its own challenges, not least the day-to-day admin that keeps the site running.
Many aspects of the site’s operation are run by volunteers, supported by mySociety’s staff and trustees — and due to the site’s success we’re looking to expand the volunteer team.
What does volunteering involve?
The work is pretty varied, but there are some frequent and recurring tasks:
Dealing appropriately with requests to remove material from the website
This is one interesting challenge which arises fairly often. Sometimes these requests are from public bodies who’ve released information they didn’t mean to; and they can also come from individuals and companies who are named in correspondence on the site.
These decisions are not always as black and white as you might expect. Some recent examples where we had to carefully consider the balance on both sides were:
- Material which Transport for London were concerned could be used to steal a tube train. We considered: was this a genuine risk? Was our publication really increasing the risk? Was the information already available elsewhere? What was the potential value of publishing the information to tube staff and their representatives, travellers and the wider public?
- Sainsbury’s were concerned that published material didn’t reflect their corporate policy on “workfare”. That may have been the case, but we asked ourselves whether that made it inappropriate to continue to publish the information that had been released. Additionally, where did the public interest lie? What legal risks were there arising from continued publication?
Responding promptly and accordingly to accidental releases
Thankfully, the frequency with which public bodies accidentally release personal information in bulk via Freedom of Information responses is decreasing, but the WhatDoTheyKnow team still have to act promptly when this does occur.
We often help users on both sides of the FOI process. For requesters, we can answer questions about FOI and how to use it, and we also work with the staff of public bodies who are at the receiving end of requests.
And all the rest
There’s always more that can be done to promote the service, draw attention to interesting correspondence on the site, and lobby for improvements to our access to information laws.
The wider team at mySociety help people around the world to establish and run their own online Freedom of Information services; and new features are being added to the UK site to make it more attractive to professional users such as journalists and campaign groups. Volunteers have the opportunity to get involved in these activities, helping steer the direction of new projects, based on their frontline experience of being a site administrator.
Keeping the database of thousands of public bodies up to date is another challenge, especially given the frequency of reorganisations in the UK’s public sector.
We work primarily by email, with regular video conferencing meetings, and occasionally meet up in person.
As a volunteer, you can decide how much time you put in, and what aspects of running the service you decide to take part in — but ideally we’re looking for people who can spare at least an hour or two, a couple of days a week.
We understand that people’s external commitments vary over time, and of course, there’s a flexible approach if a team member needs to step away for a stretch now and then.
What makes a good WhatDoTheyKnow volunteer?
There’s one characteristic that all the WhatDoTheyKnow volunteers have in common: a belief in the value of Freedom of Information, or, more widely, the expectation of transparency and accountability from the bodies which citizens fund.
As for practical skills: perhaps you’ve been involved in moderating discussions on the web, or have experience with access to information, defamation, or data protection law. Or perhaps you have, or would like to gain, experience dealing with “customers” by email.
Primarily we’re looking for people capable of making good judgements, and who can communicate clearly online.
Before joining the team, new volunteers will have to agree to follow our policies covering subjects such as security and data protection. That said, part of the role may be, if desired, taking a part in developing and refining these, and other, policies as the service grows and changes.
How to apply
If helping us run WhatDoTheyKnow sounds like the kind of thing you’d be interested in doing, then please do apply to join us.
We only have the capacity to bring on and train a few volunteers at a time, and it is important that those chosen to help administer the service are trustworthy and committed to its policies, direction and non-partisan stance. For these reasons, we are recruiting volunteers via a formal application process.
To apply please write to us before the 20th of March 2017, introducing yourself, and letting us know about any relevant interests or experience you have.
What do we offer in return?
As a volunteer, the main reward comes from the satisfaction of assisting users, making good decisions, and helping run what is fast becoming a key part of the country’s journalistic and democratic infrastructure.
Volunteers may be invited to mySociety events and meet-ups, providing a chance to take part in discussions about the future direction of the service and the organisation’s activities more generally. There have been a number of conferences held, where those running Freedom of Information sites around the world have got together to share experiences: one or more volunteers may be invited to join in, with travel expenses paid.
Other ways to help out
If volunteering to join the WhatDoTheyKnow team isn’t for you, perhaps there’s something on mySociety’s Get Involved page that is — or you could:
Image: MarkBuckawicki [CC0], via Wikimedia Commons
MapIt has had a bit of a refresh to bring the look into line with the rest of the mySociety projects. At the same time, we thought we’d take the opportunity to make it a bit easier for non-technical folk to understand what it offers, and to make the pricing a little less opaque.
You may not be familiar with MapIt, but all the same, if you’ve ever found your MP on TheyWorkForYou, written to your representatives on WriteToThem, or reported an issue through FixMyStreet, you’re a MapIt user!
That’s because MapIt does the heavy lifting in the background when you enter a postcode or location, matching that input to the boundaries it falls within (ward, constituency, borough, etc). It is, if you like, the geographic glue that holds mySociety services together.
Like most of mySociety’s software offerings, MapIt is available for others to use. So for example, the GOV.UK website uses it to put users in touch with the right council for a number of services, and Prostate Cancer UK uses it on their campaign site, using MapIt’s knowledge of CCG (Clinical Commissioning Group) region boundaries.
And you can use MapIt too: if your app or website needs to connect UK locations with areas like constituencies or counties, it will save you a lot of time and effort.
Pricing and payment is a lot slicker now: while it was previously managed manually, you can now purchase what you need online, quickly and without the need for human intervention. It’s also quite simple to see the pricing options laid out.
We hope that this will make it easier for people to make use of the service, and better understand what level of usage they need. But if you need to experiment, there’s a free ‘sandbox’ to play about with!
As ever, we’re happy to provide significant discounts for charity and non-profit projects: see more details on the licensing page.
If you have any questions or comments please do get in touch.
There’s a new piece of data on MapIt, and it wasn’t added by us. It’s tiny but useful, and it’s slightly esoteric, so bear with us and we’ll explain why it’s worth your attention.
Local Authority codes come from the government’s set of canonical registers. They may not look much, but they’re part of a drive to bring consistency across a wide range of data sets. That’s important, and we’ll try to explain why.
One name can refer to more than one thing
If you try to buy a train ticket to Gillingham in the UK, and you are lucky enough to be served by a conscientious member of staff, they will check whether you are going to the Gillingham in Kent (GIL), or the one in Dorset (GLM).
The names of the two towns might be identical, but their three-letter station codes differ, and quite right too — how, otherwise, would the railway systems be able to charge the right fare? And more importantly, how many people would set off confidently to their destination, but end up in the wrong county?
I mention this purely to illustrate the importance of authoritative, consistent data, the principle that is currently driving a government-wide initiative to ensure that there’s a single canonical code for prisons, schools, companies, and all kinds of other categories of places and organisations.
Of particular interest to us at mySociety? Local authorities. That’s because several of our services, from FixMyStreet to WriteToThem, rely on MapIt to connect the user to the correct council, based on their geographical position.
One thing can have more than one name
I live within the boundaries of Brighton and Hove City Council.
That’s its official name, but when talking or writing about my local authority, I’m much more likely to call it ‘Brighton’, ‘Brighton Council’, or at a push, ‘Brighton & Hove Council’. All of which is fine within everyday conversation, but which is an approach which could cause mayhem for the kind of data that digital systems need (“machine readable” data, which is consistent, structured and in a format which can be ‘understood’ by computer programs).
Registers of Open Data
The two examples above go some way towards explaining why the Department for Local Government & Communities, with Government Digital Services (GDS), are in the process of creating absolute standards, not just for councils but for every outpost of their diverse and extensive set of responsibilities, from the Food Standards Agency to the Foreign & Commonwealth Office, the Land Registry and beyond.
Where possible, these registers are published and shared as Open Data that anyone can use. It’s all part of GDS’ push towards ‘government as a platform’, and in keeping with the work being done towards providing Open Data throughout the organisation. Where possible these registers are openly available, and can be used by anyone building apps, websites and systems.
And now we come to those Local Authority codes that you can find on MapIt.
Anyone can contribute to Open Source code
Like most mySociety codebases, MapIt is Open Source.
That means that not only can anyone pick up the code and use it for their own purposes, for free, but that they’re also welcome to submit changes or extensions to the existing code.
And that’s just how GDS’ Sym Roe submitted the addition of the register.
What it all means for you
If you’re a developer, the addition of these codes means that you can use MapIt in your app or web service, and be absolutely sure that it will integrate with any other dataset that’s using the same codes. So, no more guessing whether our ‘Plymouth’ is the same as the ‘Plymouth’ in your database; the three-letter code tells you that it is.
Plus, these register codes identify a local authority as an organisation, or a legal entity, as opposed to setting out the boundary, so that’s an extra layer of information which we are glad to be able to include.
The Police Federation of England and Wales is the latest body to be added to WhatDoTheyKnow.
Thanks to the Policing and Crime Act 2017, which came into force on January 31, the Federation is now subject to Freedom of Information. That means that if you make a request for information which they hold, under most circumstances they must provide it.
These new responsibilities were announced by Theresa May back in 2014 when she was Home Secretary:
I will bring forward proposals to make the Police Federation – that is, the national organisation and all the regional branches – subject to the Freedom of Information Act.
I know that some of you will find these changes unpalatable. In particular, I know that some of you will find the Freedom of Information Act an unwelcome intrusion. But the Police Federation is an organisation created by statute, it serves a public function and the Normington Review demonstrated very clearly that it is an organisation in need of greater transparency and accountability. So it is a change that I believe needs to be made.
Whether it was found unpalatable or not — it happened. Accordingly, that’s now reflected on WhatDoTheyKnow, so if you have a burning question for the Federation, now is the time to ask.
It’s a little-known fact that FixMyStreet was originally called Neighbourhood Fix-It. Launching the site was a good idea, but changing that name may be the next best thing we ever did.
Ten years on, the site has processed over 900,000 reports, sending them to every local authority in the UK. In doing so, it helps citizens take an active part in keeping their own local communities clean, safe and functional. Meanwhile it ensures that you, the user, never have to give a second thought to which council needs to receive which type of report.
It’s been adopted as several councils’ primary fault-reporting interface on their own websites, from Bristol to Oxfordshire and even Zürich, and we’ve worked in partnership with these authorities to develop new features that make it as useful and simple to use as possible. Watch this space, as we’ll be talking a lot more about these soon.
FixMyStreet continues to surprise even us. Thanks to its remarkable flexibility, the codebase has also been used to underpin a number of other projects, including Collideoscope, where you can report cycling collisions and near misses, and the Channel 4 tie-in, the Empty Homes Spotter. We know there will be many more to come.
So, here’s to FixMyStreet. At heart, it’s a little site that matches a pin on a map with the body that’s responsible for that location. But when you consider what it’s achieved — getting communities fixed up, making council reporting interfaces more user-friendly, empowering people to take their first steps into local participation, even challenging corruption — well, we hope you’ll see why we’re proud of how far FixMyStreet has come.