In case you hadn’t heard, yesterday Parliament debated whether Donald Trump should be barred entry to the UK.
This is a bit of an occasion, because the first petition has been signed by more people than any other in this Parliament. It has 573,971 signatures, and its title is “Block Donald J Trump from UK entry”. The second petition is titled “Don’t ban Trump from the United Kingdom”. That petition is curious. It has 42,898 signatures, but 30,000 signatures were removed because they were thought to be suspect and coming from one source.
Now, regular TheyWorkForYou readers know that parliamentary debates are often interesting, sometimes thought-provoking, and occasionally amusing. The Trump debate is a great example of all of those things.
But most people see the goings-on in Parliament as very dull. Today, you might want to do someone a favour, and point them towards this particular debate, which you can see in full here.
As always with TheyWorkForYou content, it’s easy to search, share or link to any individual section. And as if that’s not enough, this debate contains the only use of the word wazzock yet recorded in Parliament. Now that’s got to be worth a share.
There was some excitement here at mySociety this week, as the People’s Assembly website launched in South Africa. It’s the result of a year’s partnership with PMG and a good test of some of our newest collaborative software.
The site contains a vast amount of information, all available in the same place for the first time, and offering a simple way for South African citizens to keep an eye on what their representatives are doing. There are pages for each representative, Hansard and parliamentary Questions and Answers, records of members’ interests, and more.
Locating, processing and displaying this data was quite a challenge: it has been taken from a wide range of sources, and came in an even greater range of formats, including PDF documents, Word documents, Excel files, CSV files and sometimes just e-mailed lists of information.
But perhaps most significant is the site’s Representative Locator function. For the first time, South African citizens can now find out, with ease, who represents them – not as simple as it might seem at first.
The Proportional Representative system means that members of the National Assembly and National Council of Provinces are not directly elected from constituencies. Political parties are, however, funded to run constituency offices and to allocate representatives to those offices. We believe that this is the first time this data has been consolidated and presented as a simple search tool.
The software that runs the site
As you’ll know if you read our recent blog post about SayIt, our recent focus has been reaching out to provide software for civic or democratic-focused websites anywhere in the world.
The idea is that such groups no longer need worry about writing code from scratch, since we’ve already done it – and their energies can be better expended on gathering data or adjusting the software to work within the local governmental systems.
People’s Assembly is a great example of this. It utilises two underpinning pieces of technology:
Firstly, the Pombola platform, our software for running parliamentary monitoring websites.
If you’re reading this in the UK, you may be familiar with our own parliamentary monitoring site, TheyWorkForYou. Pombola provides several tools that make it easy to do much of what TheyWorkForYou does: it provides a structured database of the names and positions of those in power; it allows people to look up their elected representatives by inputting their location, and to isolate and see what a specific MP has contributed to discussions in Parliament’s committees and plenaries; albeit, in the case of Hansard, after a six-month delay necessitated by South Africa’s own protocols.
We first developed Pombola for Kenya’s Mzalendo.com, and it’s been re-used for ShineYourEye.org in Nigeria and Odekro.org in Ghana. It’s superb to see this re-use, as it’s exactly what we set out to acheive.
Secondly, People’s Assembly is the very first site to use SayIt, which is embedded as a Django app to power the Hansard, Questions and Committees content. SayIt is one of our Components, built under the Poplus project, and we’re truly delighted to see it in place, proving its worth and being used as we first envisaged.
Thanks are due
The main work on the People’s Assembly has been funded by the Indigo Trust, and the SayIt component work was funded by Google.org as part of the Poplus Project. We also wish to thank Geoff Kilpin, who helped greatly with the scrapers and templating.
As you may know, TheyWorkForYou hasn’t displayed proceedings from the Scottish Parliament for a couple of years – but we’re glad to say that we’ve now fixed that. You can read debates from the main chamber from the Official Report and sign up to alerts from the Scottish Parliament here – just as you can for the UK Parliament and the Northern Ireland Assembly.
For those who are interested in the ‘whys’, in January of 2011, the Scottish Parliament changed the way that they published the Official Report on their website. This change broke our scraper and parser – that is, the pieces of software that fetch content and turn it into structured data.
mySociety is a small organisation with many priorities, and, because it wasn’t a simple fix, we weren’t able to allocate resources to it. So massive thanks are due to our developer Mark, who made the necessary changes to our code in his own free time.
You can help
There’s still more work to be done to get TheyWorkForYou’s data for Scotland to be as complete as it was before they changed their website, such as restoring written answers. If you think you have the expertise to help with that or any of the other issues for TheyWorkForYou Scotland, then we’d love to hear from you. And there’s still the Welsh Assembly to work on too!
Photo by Shelley Bernstein (CC)
One of the key differences between the UK’s national parliament and its local governments is that Parliament produces a written record of what gets said – Hansard.
This practice – which has no actual legal power – still has a huge impact on successful functioning of Parliament. MPs share their own quotes, they quote things back to one-another, journalists cite questions and answers, and every day TheyWorkForYou sends tens of thousands of email alerts to people who want to know who said what yesterday in Parliament. Without freely available transcripts of Parliamentary debates, it is likely that Parliament would not be anything like as prominent an institution in British public life.
No Local Hansards
Councils, of course, are too poor to have transcribers, and so don’t produce transcripts. Plus, nobody wants to know what’s going on anyway. Those are the twin beliefs that ensure that verbatim transcripts are an exceptional rarity in the local government world.
At mySociety we think the time has come to actively challenge these beliefs. We are going to be building a set of technologies whose aim is to start making the production of written transcripts of local government meetings a normal practice.
We believe that being able to get sent some form of alert when a council meeting mentions your street is a gentle and psychologically realistic way of engaging regular people with the decisions being made in their local governments. We believe transcripts are worth producing because they show that local politics is actually carried out by humans.
The State of the Art Still Needs You
First, though – a reality check. No technology currently exists that can entirely remove human labour from the production of good quality transcripts of noisy, complicated public meetings. But technology is now at a point where it is possible to substantially collapse the energy and skills required to record, edit and publish transcripts of public meetings of all kinds.
We are planning to develop software that uses off-the-shelf voice recognition technologies to produce rough drafts of transcripts that can then be edited and published through a web browser. Our role will not be in working on the voice recognition itself, but rather on making the whole experience of setting out to record, transcribe and publish a speech or session as easy, fast and enjoyable as possible. And we will build tools to make browsing and sharing the data as nice as we know how. All this fits within our Components strategy.
But mySociety cannot ourselves go to all these meetings. And it appears exceptionally unlikely that councils will want to pay for official transcribers at this point in history. So what we’re asking today is for interest from individuals – inside or outside councils – willing to have a go at transcribing meetings as we develop the software.
It doesn’t have to be definitive to be valuable
Hansard is the record of pretty much everything that gets said in Parliament. This has led to the idea that if you don’t record everything said in every session, your project is a failure. But if Wikipedia has taught us anything, it is that starting small – producing little nuggets of value from the first day – is the right way to get started on hairy, ambitious projects. We’re not looking for people willing to give up their lives to transcribe endlessly and for free – we’re looking for people for whom having a transcript is useful to them anyway, people willing to transcribe at least partly out of self interest. We’re looking for these initial enthusiasts to start building up transcripts that slowly shift the idea of what ‘normal’ conduct in local government is.
Unlike Wikipedia we’re not really talking about a single mega database with community rules. Our current plans are to let you set up a database which you would own – just as you own your blog on Blogger or WordPress, perhaps with collaborators. Maybe you just want to record each annual address of the Lord Mayor – that’s fine. We just want to build something that suits many different people’s needs, and which lifts the veil on so much hidden decision making in this country.
Get in touch
The main purpose of this post is to tell people that mySociety is heading in this direction, and that we’d like you along for the ride. We won’t have a beta to play with for a good few months yet, but we are keen to hear from anyone who thinks they might be an early adopter, or who knows of other people who might want to be involved.
And we’re just as keen to hear from people inside councils as outside, although we know your hands are more tied. Wherever you sit – drop us a line and tell us what sort of use you might want to make of the new technology, and what sort of features you’d like to see. We’ll get back in touch when we’ve something to share.
Over 115,000 Freedom of Information requests.
Almost 225,000 FixMyStreet reports.
Close to 3,000 public transport problems.
Every word spoken in Parliament since 1935.
So, what would you like to know?
There’s no doubt about it, mySociety sites store a lot of data. And once you have that much data, you can start finding the answers to interesting questions. Questions like:
- Which public bodies receive the most FOI requests?
- Which county gets the most pothole reports?
- Which train routes are people complaining most about?
- Which MP has spoken for the longest cumulative time in the history of Parliament?
There are less obvious questions, too – how about:
- Which regions of the country are most likely to include bad language when submitting a form online?
- How many times does the Speaker have to interject, “Order, order!” in an average week?
- Which words are most spoken in Parliament, and which have only become popular in the last five years?
- What topics do people submit the most Freedom of Information requests about?
- Just how often does a UK citizen get so fed up about dog poop that they take action?
We reckon there are almost limitless stories in our data, waiting to be teased out. Some of them will be surprising, fascinating, or just plain funny. Some may even be potential front page news. So, we’ve invited journalists who have a particular interest in data, or indeed in any of the areas we work in, to come and have at it at our first ever mySociety Data Hackday.
Not a journalist?
Journalists aren’t the only ones with bright ideas, so if you’re reading this and there’s a burning question that springs to mind, leave a comment below. Given all these reams of data, what would you be looking for? We’ll add the best ideas to our list, and we’ll be reporting back on everything we find out.
Actually, I am a journalist!
There are still a few places, so if you’d like to attend, please drop us a line. Note: we will expect you to get stuck in! We will run the data, but you may be sifting through the results, looking for significant stories, and sharing your findings. Bring a laptop, and plenty of ideas.
If you can’t attend, but really wish you could, let us know what data you’d like us to run, and we’ll add it to the list.
ETA: Lanyrd page here.
Image credit: Johan Nilsson
When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.
Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:
- It is entirely responsively designed, so that it works on the simplest of mobile web browsers from day one.
- All the lessons we learned from storing political data wrongly have been baked into this site (i.e we can easily cope with people changing names, parties and jobs)
- Every organisation, position and place in the system is now a proper object in the database. So if you want to see all the politicians who went to Nairobi University, you can.
- There is lots of clear information on how parliament functions, what MPs and committees do, and so on.
- It synthesizes some very complex National Taxpayer’s Association data on missing or wasted money into a really clear ‘scorecard‘, turning large sums of money into numbers of teachers.
The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.
Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.
If you haven’t got a penny,
A ha’penny will do,
If you haven’t got a ha’penny,
Then God bless you.
We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?
mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.
Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.
What’s behind the door? A letter to Santa.
If you can fit them down the chimney, here’s what we’re dreaming of:
More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.
We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.
Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.
Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?
A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.
Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?
FixMyTransport buy-in – from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.
Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.
TheyWorkForYou has, until now, only covered things that have already happened, be that Commons main chamber debates since 1935, Public Bill committees back to 2000, or all debates in the modern Northern Ireland Assembly.
From today, we are taking the UK Parliament’s upcoming business calendar and feeding it into our database and search engine, which means some notable new features. Firstly, and most simply, you can browse what’s on today (or the next day Parliament is sitting), or 16th May. Secondly, you can easily search this data, to e.g. see if there will be something happening regarding Twickenham. And best of all, if you’re signed up for an email alert – see below for instructions – you’ll get an email about any matching future business along with the matching new Hansard data we already send. We currently send about 25,000 alerts a day, with over 65,000 email addresses signed up to over 111,000 alerts.
Mark originally wrote some code to scrape Parliament’s business papers, but this sadly proved too fragile, so we settled on Parliament’s calendar which covers most of the same information and more importantly has (mostly) machine-readable data. Duncan and I worked on this intermittently amidst our other activities, with Duncan concentrating on the importer and updating our search indexer (thanks as ever to Xapian) whilst I got on with adding and integrating the new data into the site.
I’ve also taken the opportunity to rejig the home page (and fix the long-standing bug with popular searches that meant it was nearly always Linda Gilroy MP!) to remove the confusingly dense amount of recent links, bring it more in line with the recently refreshed Scottish Parliament and Northern Ireland Assembly home pages, and provide more information to users who might not have any idea what the site covers.
Signing up for an email alert: If you want to receive an email alert on a particular person (MP, Lord, MLA or MSP), visit their page on TheyWorkForYou and follow the “Email me updates” link. If you would like alerts for a particular word or phrase, or anything else, simply do a search for what you’re after, then follow the email alert or RSS links to the right of the results page.
As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.
To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).
Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.
Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.
There don’t appear to be any central official lists of candidates in these elections. Amnesty.org.uk has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations 🙂
Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15” to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.
The House of Commons debate coverage on TheyWorkForYou has recently extended back from the 2001 general election to the 1935 general election, and our knowledge of MPs now extends back to the start of the 19th century. This means TheyWorkForYou now includes things such as Anthony Eden on the Suez Canal in 1956, saying “there was not foreknowledge that Israel would attack Egypt“; the debate the day after Bloody Sunday in 1972; Geoffrey Howe’s resignation statement in 1990; Neville Chamberlain on the eve and start of the second World War in 1939; and Winston Churchill‘s speeches to the House, such as We shall fight on the beaches and This was their finest hour in 1940. This and much, much more are available and searchable using our new improved advanced search, which allows you to filter by e.g. date range or person. We hope people enjoy researching this huge wealth of information (I certainly do), and add useful annotations to the text to help other people.
This would not have been possible without the original project by Parliament to digitise historical copies of Hansard and make them available, nor the internal Parliamentary project to clean up the data, match up speaker names, and so on. The project was kindly funded by the Ministry of Justice’s Innovation Fund, which also supported the creation of FixMyStreet and GroupsNearYou.