mySociety’s EveryPolitician project aims to make data available on every politician in the world. It’s going well: we’re already sharing data on the politicians from nearly every country on the planet. That’s over 68,652 people and 2.9 million individual pieces of data, numbers which will be out of date almost as soon as you’ve read them. Naturally, the width and depth of that data varies from country to country, depending on the sources available — but that’s a topic for another blog post.
Today the EveryPolitician team would like to introduce you to its busiest member, who is blogging at EveryPolitician bot. A bot is an automated agent — a robot, no less, albeit one crafted entirely in software.
First, some background on why we need our little bot.
Because there’s so much to do
One of the obvious challenges of such a big mission is keeping on top of it all. We’re constantly adding and updating the data; it’s in no way a static dataset. Here are examples — by no means exhaustive — of circumstances that can lead to data changes:
- Legislatures change en masse, because of elections, etc.
We try to know when countries’ governments are due to change because that’s the kind of thing we’re interested in anyway (remember mySociety helps run websites for parliamentary monitoring organisations, such as Mzalendo in Kenya). But even anticipated changes are rarely straightforward, not least because there’s always a lag between a legislature changing and the data about its new members becoming available, especially from official national sources.
- Legislatures change en masse, unexpectedly
Not all sweeping changes are planned. There are coups and revolutions and other unscheduled or premature ends-of-term.
- Politicians retire
Or die, or change their names or titles, or switch party or faction.
- New parties emerge
Or the existing ones change their names, or form coalitions.
- Areas change
There are good reasons (better representation) and bad reasons (gerrymandering) why the areas in constituency-based systems often change. By way of a timely example, our UK readers probably know that the wards have changed for the forthcoming elections, and that mySociety built a handy tool that tells you what ward you’re in.
- Existing data gets refined
Played Gender Balance recently? Behind that is a dataset that keeps being updated (whenever there are new politicians) but which is itself a source of constantly-updating data for us.
- Someone in Russia updates the wikipedia page about a politician in Japan
Wikidata is the database underlying projects like Wikipedia, so by monitoring all the politicians we have that are also in there, we get a constant stream of updates. For example, within a few hours of someone adding it, we knew that the Russian transliteration of 安倍晋三’s name was Синдзо Абэ — that’s Shinzo Abe, in case you can’t read kanji or Cyrillic script. (If you’re wondering, whenever our sources conflict, we moderate in favour of local context.)
- New data sources become available
Our data comes from an ever-increasing number of sources, commonly more than one for any given legislature (the politicians’ twitter handles are often found in a different online place from their dates of birth, for example). We always welcome more contributions — if you think you’ve got new sources for the country you live in, please let us know.
- New old data becomes available
We collect historic data too — not just the politicians in the current term. For some countries we’ve already got data going back decades. Sources for data like this can sometimes be hard to find; slowly but surely new ones keeping turning up.
So, with all this sort of thing going on, it’s too much to expect a small team of humans to manage it all. Which is where our bot comes in.
We’ve automated many of our processes: scraping, collecting, checking changes, submitting them for inclusion — so the humans can concentrate on what they do best (which is understanding things, and making informed decisions). In technical terms, our bot handles most things in an event-driven way. It springs into action when triggered by a notification. Often that will be a webhook (for example, a scraper finishes getting data so it issues a webhook to let the bot know), although the bot also follows a schedule of regular tasks too. Computers are great for running repetitive tasks and making quantitative comparisons, and a lot of the work that needs to be done with our ever-changing data fits such a description.
The interconnectedness of all the different tasks the bot performs is complex. We originally thought we’d document that in one go — there’s a beautiful diagram waiting to be drawn, that’s for sure — but it soon became clear this was going to be a big job. Too big. Not only is the bot’s total activity complicated because there are a lot of interdependencies, but it’s always changing: the developers are frequently adding to the variety of tasks the bot is doing for us.
So in the end we realised we should just let the bot speak for itself, and describe task-by-task some of the things it does. Broken down like this it’s easier to follow.
We know not everybody will be interested, which is fine: the EveryPolitician data is useful for all sorts of people — journalists, researchers, parliamentary monitors, activists, parliamentarians themselves, and many more — and if you’re such a person you don’t need to know about how we’re making it happen. But if you’re technically-minded — and especially if you’re a developer who uses GitHub but hasn’t yet used the GitHub API as thoroughly as we’ve needed to, or are looking for ways to manage always-shifting data sets like ours — then we hope you’ll find what the bot says both informative and useful.
The bot is already a few days into blogging — its first post was “I am a busy bot”, but you can see all the others on its own Medium page. You can also follow it on twitter as @everypolitbot. Of course, its true home, where all the real work is done, is the everypoliticianbot account on GitHub.
Images: CC-BY-SA from the EveryPolitician bot’s very own scrapbook.
- Legislatures change en masse, because of elections, etc.
You may have seen the blanket press coverage last week: the London Legacy Development Corporation (LLDC), the publicly-funded authority which owns the Olympic Stadium, lost its recent tribunal and was ordered to publish its contract with West Ham football club.
This is a story which goes back to last August, when we first blogged that WhatDoTheyKnow user Richard Hunt had submitted a request for the contract via the site, on behalf of a group of Football Supporters’ Trusts.
In September, we updated the story as LLDC pushed back from publishing the full contract, citing ‘commercial confidentiality’. It seems the subsequent tribunal dismissed this as a valid reason to withhold the information — information which has now been pored over in detail by the nation’s media.
Many concluded that the authority have struck a poor deal on behalf of the general public; we particularly enjoyed a statement from Barry Hearn, former chairman of Leyton Orient, who reportedly stated, “My dog could have negotiated a better deal for the taxpayer.”
Whatever your opinion on the deal itself, we think it’s right that the information should be firmly in the public domain, so that people can clearly see the financial affairs of the authorities they pay for.
Richard Hunt, whose request kickstarted this whole affair, says that it represents a good result for football, too:
The effort to get the contract released under FOI was started by a football fan and then, as the LLDC resisted disclosure, mushroomed into a full scale campaign run by a coalition of football club Supporters Trusts.
It gained such wide support precisely because football fans are taxpayers too, and there was a widespread perception that one such club was receiving public funds to get a new stadium, whereas other clubs had funded new stadia themselves (or more accurately from the revenues earned from their fans ).
It was a rare example of football fans overcoming tribal divisions to work together, and is expected to be showcased at the Supporters Summit meeting organised by the Football Supporters Federation this coming July.
Well done to all involved! You can see the original Freedom of Information request here.
We’re helping our friends at Citizen Beta by sponsoring (and co-hosting) an international-themed event next Monday (25th April 2016, 7pm–10pm). There will be three presentations, one from OneWorld about their work overseas, one by our own Jen about some of our recent international projects, and then there’s the spotlight: a talk from the Kuala Lumpur-based Sinar Project.
If you’re in or near London, please do come: sign up on Citizen Beta’s attending.io page.
Citizen Beta is a roughly-monthly meetup for civic tech people. Next Monday’s event will follow the usual format of Q&As after each presentation with (deliberately) lots of meeting, mixing, and chatting too. It happens in Newspeak House in Shoreditch, and refreshments will be provided.
We’re very pleased to have been able to make this event happen. It was precipitated because Khairil from Sinar is coming over to our side of the world for mySociety’s forthcoming TICTeC 2016 conference in Barcelona. (If you haven’t already got a ticket to TICTeC then, sorry, you’re too late; but of course keep an eye on this blog because we’ll be sharing photos, videos and accounts from the event).
OneWorld, like mySociety, are based in the UK, but work all over the globe. Their projects often depend upon the classic civic tech components of web and mobile, but they’re also actively involved in addressing health and rights issues in the countries in which they operate. They have a wealth of experience from running projects in developing countries, especially when it comes to understanding both the capabilities and the limitations of the technologies they use.
The Sinar Project, which the aforementioned Khairil heads, is a civic tech group based in Kuala Lumpur. They first came to our attention because they were using two of our codebases (FixMyStreet and MapIt). We’ve been friends ever since, and it’s always been a delight when our international paths have crossed. In many ways Sinar perfectly epitomises the philosophy of civic tech/open source reuse that mySociety, and specifically Poplus, is so passionate about: when you have a small team of super-focused developers (shout-out to Sinar’s Motionman and Sweester!), you simply can’t afford to waste time reinventing the wheel.
On Monday evening Khairil will be showing some of the impressive ways they have been combining their tech skills and tools in a political environment which is considerably more hostile than that in the UK.
Oh, one last thing — sorry; we do realise that this is a London event. mySociety itself is a remote-working organisation, which means we’re spread all around the UK, so we know that not everything happens in London. But that’s where Citizen Beta is, so that’s where this particular meetup is happening.
It’s going to be great — so if you can come, please sign up. See you there!
Image credit: CC BY-NC-SA 2.0 by Horng Yih Wong of Khairil Yusof speaking at TEDxKL
Are you still in the same ward? Check whether your ward boundaries have changed here.
May 5 is election day
If you’re a UK citizen, you have an election in your near future. We can say that with confidence.
May 5 sees elections not only for the Scottish Parliament, the National Assembly of Wales and the Northern Ireland Assembly, but also for many local councils. Londoners will be picking their London Assembly representatives and their Mayor. As if all that isn’t enough, there are also Police and Crime Commissioner Elections.
Ward boundaries are changing
You might think you already know where to vote, and who’s standing for election in your area.
But both are dictated by which ward you live in — and that may not be the one you’re used to, thanks to ongoing changes in ward boundaries.
How do researchers accumulate knowledge about civic technology? So far, many of our most active discussions still happen at conferences and in the informal writing of blog posts. Individual organizations (like mySociety) are investing more heavily in the production of academic-quality research, so we can all see more clearly what happens when we try to use technology to bring people and governments together. But is this work yet diffusing into academic use?
I wrote about this question while researching the context of mySociety’s current work on US city-led civic tech. Although the term “e-government” seems to have been largely bypassed in current civic technology practice, it’s useful to see that “e-government” is still a very important concept. For those of us trying to see how governments approach the use of technology in connecting to citizens, research on e-government provides a useful perspective. At the same time, however, the perspectives of e-government and civic technology also diverge — and that too is important to note.
Why Civic Technologists Should Still Care About E-Gov
As a recovering academic, I spend more time than I should going down rabbit holes: reading articles which cite interesting studies, which I then go find and read, and then they cite interesting papers, and pretty soon way too much time has gone by. I’m currently researching the role of civic technology projects enacted by U.S cities for mySociety, so I’ve been actively looking for those civic tech rabbit holes.
There’s just one hitch: civic technology barely exists in the academic context. E-government, on the other hand, continues to have an active research program.
Searching “e-government” in Google Scholar returns 169,000 cited works—10,600 published since 2015 alone. Searching “civic technology,” meanwhile, nets a total of 185. (And one of the top results references technology in the Honda Civic.)
Read more at Civicist.
Last week, Ukrainian Freedom of Information site Dostup do Pravdy processed its 10,000th FOI request.
That’s pretty impressive, given that they launched just a couple of years ago, in 2014.
We offer hearty congratulations to the Dostup do Pravdy team. We’re also looking very closely at how they achieved this level of usage, because the site runs on our Alaveteli platform — and we’re keen to share the secrets of their success with the rest of the Alaveteli community.
So we called Yaroslav, one of the team, and asked him to outline the various factors that have helped boost the site’s popularity. We’ll be writing this up in more detail as part of a guide to marketing Alaveteli sites, but for now, here are the headline points.
Link with a news outlet
Dostup do Pravdy was set up in collaboration with Ukraine’s biggest online news outlet, and from the beginning they have employed a journalist to work solely on stories generated through Freedom of Information.
This has given them several great advantages: a ready-made audience for their most interesting requests; a channel through which to ensure that the general population knows about their rights in FOI; and professional expertise in pulling out which information was the most newsworthy.
Of course, no-one would choose to live through political upheaval, but there’s no doubt that Ukraine’s recent history made the populace all the more keen to access facts.
FOI proved a crucial tool in uncovering and publicising stories of corruption, such as the diversion of funds meant for the army, when high-up officials were coincidentally seen driving top-of-the-range BMWs.
Stories that grab the public’s imagination
Right now, Dostup do Pravdy are working on a campaign to find the owners of historic buildings which are falling into disrepair, a story which has captured the attention of the wider community.
Similarly, they’ve probed into figures on domestic violence cases, a story which got picked up by all the national media.
On the road
Ukraine is a relatively big country, with some regions where internet access is poor. The Dostup do Pravdy team are partway through a series of 15 grant-funded ‘roadshows’ in which they invite local activists to come and learn more about Freedom of Information, and train them in how to make requests.
These activists also help to spread the word amongst the wider community and local media. Where there is no access to the internet, they revert to the lower-tech FOI channels of phone and written letter.
The visits are also an opportunity to meet with officers from public authorities — the people on the receiving end of the FOI requests.
Employ an intern
There’s always more work than there is time to do it, when you’re a small team trying to make a big difference. Dostup do Pravdy were only able to find all the details they needed for their historic buildings project by employing an intern who could go through all the various registers to find crucial information.
Use social media
Dostup do Pravdy have seen great increases in visits to their site, both in terms of people browsing information, and those who go on to make an FOI request.
Alaveteli does allow for a certain amount of discussion of requests, via its annotations functionality, but Dostup do Pravdy also have almost 10,000 followers on Facebook, and it’s here that they’ve seen discussion flourish. It’s also a great platform for sharing their investigative stories, and publicising their events.
Users also come to Facebook to ask for assistance in making their requests, or following up those that have gone unanswered. Administrators encourage users to keep pushing for the information they require, and can point out where authorities are in breach of law, or point them in the right direction to get further help from the Institute of Media law, who can offer legal aid and advice.
So there you are: that’s the combination of factors that have led to success for Dostup do Pravdy. We wish them all the best as they charge towards their next milestone. Будьмо!
We share the belief, set out in the recent Connected Councils report from Nesta, that open standards are key to unlocking the potential of government as platform for local government.
We want to help local authorities understand the benefits of open standards, too. So to that end we’re holding a half-day workshop at Newspeak House London on Tuesday 19th April (the day before the Digital Government conference in London).
From our perspective this is important because when councils adopt new services they often miss the opportunity to create a genuinely open platform allowing councils, third sector and commercial organisations to work well together (I’ve written more on that in a post over on Medium).
We’re inviting local authority staff who are responsible for setting strategy for open data and digital standards, and we’ll have a handful of interesting speakers, roundtable discussions and a spot of lunch as well.
We’ve been fans of the Open311 standard for reporting non-emergency local issues online for some time. This open standard makes it easy for us to submit issues from FixMyStreet.com directly into a local authority’s case management system, and, just as importantly, report back when they have been resolved.
Open standards are a fundamental aspect of digital transformation for every local council. We want to do what we can to help extend the scope and use of these standards, and learn how we can better deliver services that make use of them.
We’d love to hear from you about your use of open standards in local government, to share experiences on how this can power service development and identify opportunities to extend the take up of standards. We’d also like to ensure that we are building services that you actually want to use to help your local residents.
If you or a colleague would like to attend, or if you know people in other councils who are interested in Open311, FixMyStreet and other open standards, then please request an invite on our Eventbrite page.
Image credit Deborah Fitchett https://flic.kr/p/7EyMVT
Undertaking client work through our commercial subsidiary mySociety Services has been a vital part of our identity, and it provides an important source of additional revenue to complement our core grant funding.
We’ve worked with numerous organisations that share our principles and focus on impact such as Médecins Sans Frontières, The Financial Conduct Authority, the NHS, and notably we produced the UK Parliamentary Digital Report which led to the establishment of the Parliamentary Digital Service.
This is good work, but it’s meant we’ve had to support two teams, two marketing efforts and often had to juggle priorities with our charitable work.
Our overall aim is to create impactful services that benefit as many people as possible. So rather than continue to spread ourselves too thinly, from now on we’re going to concentrate primarily on appropriate commercial services that sit alongside our three thematic areas of focus: Freedom of Information, Democracy and Better Cities.
We’re taking the first step today with the announcement of a new grant from the Google Digital News Initiative, for which we’re extremely grateful. We’ll be making use of the grant to develop a new toolset for journalists using Freedom of Information.
In the next few weeks we’ll share more details on what for the moment we’ve codenamed Alaveteli Professional. Our intention is that this toolset will sit alongside as a companion service to our free FOI platform Alaveteli.org, and should it become viable we may offer a version as a commercial service through mySociety Ltd.
In the interim we’ll be speaking to lots of users, especially journalists and campaigning organisations on their use of FOI. If you’re are interested in helping us shape this product, please get in touch with us at firstname.lastname@example.org and we’ll keep you up to date.
This new approach will mean we can better develop complementary commercial services that fully realise their potential and better support our charitable aims and objectives.
Well, it certainly all happened over the weekend: the resignation of one Secretary of State on Friday and the quick appointment of another by Saturday.
It all left a lot of people wondering just who this Stephen Crabb fellow was, and what he stood for.
Fortunately, there’s a very handy website where you can look up the details, debates and voting records of every MP — we refer, of course, to our very own TheyWorkForYou. Over the weekend, we saw the link to Crabb’s voting record shared across social media (and even good old traditional media; we were also mentioned on Radio 4’s Any Answers). Naturally, most interest was around Crabb’s voting habits when it comes to welfare and benefits.
The upshot of this was that TheyWorkForYou saw almost three times our normal traffic for a Saturday. Over the weekend, 30% of all page views were for Crabb’s profile or voting records. In contrast, just 1.83% thought to check out his predecessor’s record: yesterday’s news already, it seems.
So Stephen Crabb’s the new guy, and you may want to keep up to date with his contributions to Parliament. Sign up here and we’ll send you an email every time he speaks.
We’ve just released Alaveteli 0.24! Here are some of the highlights.
We’ve added better management for censor rules in the admin interface. Previously, only request and user censor rules could be managed; now you can manage rules for authorities and global rules that get applied to everything.
We’ve added support deleting incoming messages in bulk on a request page. This is useful if you’re experiencing spam to the holding pen. You can zap them all in a couple of clicks.
Facebook is a big driver of traffic to Alavetelis. We’ve added support for a specific opengraph image for pages when shared on Facebook. You’ll need to add a custom version for your site in your theme. Take a look at the upgrade notes for more info.
Long Term Support
Alaveteli 0.24 adds support for the most current versions of Debian and Ruby. Jessie has security support until May 2018 and Long Term Support until May 2020.
The full list of highlights and upgrade notes for this release is in the changelog.
Thanks again to everyone who’s contributed!