A key part of mySociety’s research agenda is understanding how Civic Technology is (or isn’t) helping under-represented groups in society access government services and their representation. In 2015 we released a report Who Benefits from Civic Technology, that explored variations in usage of Civic Tech in various countries and demographics. You can read or download it here.
In this blog post I’m going to talk a bit about how we’ve internally tried to apply our data to understanding the under-representation of women in politics and as users of our services, as well as some interesting things that external researchers have found using our data.
Our EveryPolitician dataset contains information on current (and in some cases historical) politicians for a large number of countries around the world. For a large number of representatives, this includes gender information.
However, a key problem of international comparisons of the representation of women is, as Miki Caul points out, that it “overlooks the fact that individual parties vary greatly in the proportion of women MPs within each nation”. Similarly, Lena Wängnerud argues “cross-country studies tend to miss variations between parties within a single system. Variations in the proportion of women to men are even greater across parties than across nations”.
Fortunately, this is exactly the kind of problem that an international dataset like EveryPolitician is well placed to examine – on Thursday we’ll be using a new mini-site to explore the gender and party information contained in EveryPolitician to give a sense of the international picture and the party-level differences within each country. Stay tuned! Or you can download the data yourself (there are APIs for Python, Ruby and R) and try and beat us to it.
TheyWorkForYou makes it easy to search through the history of what has been said in Parliament, and we make the data (based on the Hansard dataset but more consistently formatted) freely available to download. As essentially a download of a very large amount of text, getting insights from this dataset is a bit more complicated, but potentially very rewarding.
Jack Blumenau has a paper based on TheyWorkForYou data using language to analyse whether appointing female ministers changes how other female MPs participate in debates. Looking at “half a million Commons’ speeches between 1997 and 2017, [he demonstrates] that appointing a female minster increases the participation of women MPs in relevant debates by approximately one third over the level of female participation under male ministers” – and that “female MPs also became more influential in debates under the purview of female ministers […] female ministers respond in a systematically different fashion to the speeches of female MPs.” In this case, influence is a measure of whether the language an individual used is then taken up by others, and this kind of analysis shows how the TheyWorkForYou dataset can be used to demonstrate not just counts of how many women were in Parliament, but the substantive effects of women holding office on the political process.
As Myf talked about yesterday, TheyWorkForYou’s Commons content now extends back to 1918, and so includes every speech by a female MP ever made. We hope this is a useful resource for anyone interested in exploring the history of the representation of women in the UK and have plans for a small project in the upcoming months to show in a simple way how this data can be used (please sign up to our mailing list if you’re interested in hearing about this when it’s completed).
FixMyStreet and WriteToThem
Understanding the under-representation of women is important across our services. Where men and women are experiencing different issues and concerns, imbalances in access (or use of access) potentially lead to differences in resource allocation.
The majority of reports on FixMyStreet.com are reported by men – but to make things more complicated, it’s not just that women make fewer reports, but women report substantively different kinds of reports.
Reka Solymosi, Kate Bowers and Taku Fujiyama investigated FixMyStreet reports and found (by determining gender from names of problem reporters) that different kinds of reports are more likely to be reported by men and women – they suggest that at “first glance it appears that men are more likely to report in categories related to driving (potholes and road problems), whereas women report more in categories related to walking (parks, dead animals, dog fouling, litter)”.
If different kinds of reports are differently gendered, this complicates thinking about how to improve how women use the website – as potential users are having substantially different experiences of problems in the real world well before they interact with the site. We have to engage with the nuance of this kind of finding to understand how to redress issues of access to services.
We’re currently in the process of extending this kind of analysis to our other service. For WriteToThem, we’ve learned that while the majority of people using the service to write to MPs are male (around 60%), this picture is different depending on the level of government – for instance the gender balance for people writing to councils is pretty close to 50/50.
As part of this, we’re investigating whether having the same gender as their representative makes people more likely to make contact. This has some interesting preliminary findings, and we hope to have more to say about this towards the end of the year.
Our research in this area is ongoing, and we’re keen to help people use our data to investigate under-representation – especially where you have expertise or knowledge that we don’t. If you’d like to discuss potential uses of the data please get in touch, or sign up to our mailing list to hear about future research releases.
In case you hadn’t heard, yesterday Parliament debated whether Donald Trump should be barred entry to the UK.
This is a bit of an occasion, because the first petition has been signed by more people than any other in this Parliament. It has 573,971 signatures, and its title is “Block Donald J Trump from UK entry”. The second petition is titled “Don’t ban Trump from the United Kingdom”. That petition is curious. It has 42,898 signatures, but 30,000 signatures were removed because they were thought to be suspect and coming from one source.
Now, regular TheyWorkForYou readers know that parliamentary debates are often interesting, sometimes thought-provoking, and occasionally amusing. The Trump debate is a great example of all of those things.
But most people see the goings-on in Parliament as very dull. Today, you might want to do someone a favour, and point them towards this particular debate, which you can see in full here.
As always with TheyWorkForYou content, it’s easy to search, share or link to any individual section. And as if that’s not enough, this debate contains the only use of the word wazzock yet recorded in Parliament. Now that’s got to be worth a share.
There was some excitement here at mySociety this week, as the People’s Assembly website launched in South Africa. It’s the result of a year’s partnership with PMG and a good test of some of our newest collaborative software.
The site contains a vast amount of information, all available in the same place for the first time, and offering a simple way for South African citizens to keep an eye on what their representatives are doing. There are pages for each representative, Hansard and parliamentary Questions and Answers, records of members’ interests, and more.
Locating, processing and displaying this data was quite a challenge: it has been taken from a wide range of sources, and came in an even greater range of formats, including PDF documents, Word documents, Excel files, CSV files and sometimes just e-mailed lists of information.
But perhaps most significant is the site’s Representative Locator function. For the first time, South African citizens can now find out, with ease, who represents them – not as simple as it might seem at first.
The Proportional Representative system means that members of the National Assembly and National Council of Provinces are not directly elected from constituencies. Political parties are, however, funded to run constituency offices and to allocate representatives to those offices. We believe that this is the first time this data has been consolidated and presented as a simple search tool.
The software that runs the site
As you’ll know if you read our recent blog post about SayIt, our recent focus has been reaching out to provide software for civic or democratic-focused websites anywhere in the world.
The idea is that such groups no longer need worry about writing code from scratch, since we’ve already done it – and their energies can be better expended on gathering data or adjusting the software to work within the local governmental systems.
People’s Assembly is a great example of this. It utilises two underpinning pieces of technology:
Firstly, the Pombola platform, our software for running parliamentary monitoring websites.
If you’re reading this in the UK, you may be familiar with our own parliamentary monitoring site, TheyWorkForYou. Pombola provides several tools that make it easy to do much of what TheyWorkForYou does: it provides a structured database of the names and positions of those in power; it allows people to look up their elected representatives by inputting their location, and to isolate and see what a specific MP has contributed to discussions in Parliament’s committees and plenaries; albeit, in the case of Hansard, after a six-month delay necessitated by South Africa’s own protocols.
We first developed Pombola for Kenya’s Mzalendo.com, and it’s been re-used for ShineYourEye.org in Nigeria and Odekro.org in Ghana. It’s superb to see this re-use, as it’s exactly what we set out to acheive.
Secondly, People’s Assembly is the very first site to use SayIt, which is embedded as a Django app to power the Hansard, Questions and Committees content. SayIt is one of our Components, built under the Poplus project, and we’re truly delighted to see it in place, proving its worth and being used as we first envisaged.
Thanks are due
The main work on the People’s Assembly has been funded by the Indigo Trust, and the SayIt component work was funded by Google.org as part of the Poplus Project. We also wish to thank Geoff Kilpin, who helped greatly with the scrapers and templating.
As you may know, TheyWorkForYou hasn’t displayed proceedings from the Scottish Parliament for a couple of years – but we’re glad to say that we’ve now fixed that. You can read debates from the main chamber from the Official Report and sign up to alerts from the Scottish Parliament here – just as you can for the UK Parliament and the Northern Ireland Assembly.
For those who are interested in the ‘whys’, in January of 2011, the Scottish Parliament changed the way that they published the Official Report on their website. This change broke our scraper and parser – that is, the pieces of software that fetch content and turn it into structured data.
mySociety is a small organisation with many priorities, and, because it wasn’t a simple fix, we weren’t able to allocate resources to it. So massive thanks are due to our developer Mark, who made the necessary changes to our code in his own free time.
You can help
There’s still more work to be done to get TheyWorkForYou’s data for Scotland to be as complete as it was before they changed their website, such as restoring written answers. If you think you have the expertise to help with that or any of the other issues for TheyWorkForYou Scotland, then we’d love to hear from you. And there’s still the Welsh Assembly to work on too!
Photo by Shelley Bernstein (CC)
One of the key differences between the UK’s national parliament and its local governments is that Parliament produces a written record of what gets said – Hansard.
This practice – which has no actual legal power – still has a huge impact on successful functioning of Parliament. MPs share their own quotes, they quote things back to one-another, journalists cite questions and answers, and every day TheyWorkForYou sends tens of thousands of email alerts to people who want to know who said what yesterday in Parliament. Without freely available transcripts of Parliamentary debates, it is likely that Parliament would not be anything like as prominent an institution in British public life.
No Local Hansards
Councils, of course, are too poor to have transcribers, and so don’t produce transcripts. Plus, nobody wants to know what’s going on anyway. Those are the twin beliefs that ensure that verbatim transcripts are an exceptional rarity in the local government world.
At mySociety we think the time has come to actively challenge these beliefs. We are going to be building a set of technologies whose aim is to start making the production of written transcripts of local government meetings a normal practice.
We believe that being able to get sent some form of alert when a council meeting mentions your street is a gentle and psychologically realistic way of engaging regular people with the decisions being made in their local governments. We believe transcripts are worth producing because they show that local politics is actually carried out by humans.
The State of the Art Still Needs You
First, though – a reality check. No technology currently exists that can entirely remove human labour from the production of good quality transcripts of noisy, complicated public meetings. But technology is now at a point where it is possible to substantially collapse the energy and skills required to record, edit and publish transcripts of public meetings of all kinds.
We are planning to develop software that uses off-the-shelf voice recognition technologies to produce rough drafts of transcripts that can then be edited and published through a web browser. Our role will not be in working on the voice recognition itself, but rather on making the whole experience of setting out to record, transcribe and publish a speech or session as easy, fast and enjoyable as possible. And we will build tools to make browsing and sharing the data as nice as we know how. All this fits within our Components strategy.
But mySociety cannot ourselves go to all these meetings. And it appears exceptionally unlikely that councils will want to pay for official transcribers at this point in history. So what we’re asking today is for interest from individuals – inside or outside councils – willing to have a go at transcribing meetings as we develop the software.
It doesn’t have to be definitive to be valuable
Hansard is the record of pretty much everything that gets said in Parliament. This has led to the idea that if you don’t record everything said in every session, your project is a failure. But if Wikipedia has taught us anything, it is that starting small – producing little nuggets of value from the first day – is the right way to get started on hairy, ambitious projects. We’re not looking for people willing to give up their lives to transcribe endlessly and for free – we’re looking for people for whom having a transcript is useful to them anyway, people willing to transcribe at least partly out of self interest. We’re looking for these initial enthusiasts to start building up transcripts that slowly shift the idea of what ‘normal’ conduct in local government is.
Unlike Wikipedia we’re not really talking about a single mega database with community rules. Our current plans are to let you set up a database which you would own – just as you own your blog on Blogger or WordPress, perhaps with collaborators. Maybe you just want to record each annual address of the Lord Mayor – that’s fine. We just want to build something that suits many different people’s needs, and which lifts the veil on so much hidden decision making in this country.
Get in touch
The main purpose of this post is to tell people that mySociety is heading in this direction, and that we’d like you along for the ride. We won’t have a beta to play with for a good few months yet, but we are keen to hear from anyone who thinks they might be an early adopter, or who knows of other people who might want to be involved.
And we’re just as keen to hear from people inside councils as outside, although we know your hands are more tied. Wherever you sit – drop us a line and tell us what sort of use you might want to make of the new technology, and what sort of features you’d like to see. We’ll get back in touch when we’ve something to share.
Over 115,000 Freedom of Information requests.
Almost 225,000 FixMyStreet reports.
Close to 3,000 public transport problems.
Every word spoken in Parliament since 1935.
So, what would you like to know?
There’s no doubt about it, mySociety sites store a lot of data. And once you have that much data, you can start finding the answers to interesting questions. Questions like:
- Which public bodies receive the most FOI requests?
- Which county gets the most pothole reports?
- Which train routes are people complaining most about?
- Which MP has spoken for the longest cumulative time in the history of Parliament?
There are less obvious questions, too – how about:
- Which regions of the country are most likely to include bad language when submitting a form online?
- How many times does the Speaker have to interject, “Order, order!” in an average week?
- Which words are most spoken in Parliament, and which have only become popular in the last five years?
- What topics do people submit the most Freedom of Information requests about?
- Just how often does a UK citizen get so fed up about dog poop that they take action?
We reckon there are almost limitless stories in our data, waiting to be teased out. Some of them will be surprising, fascinating, or just plain funny. Some may even be potential front page news. So, we’ve invited journalists who have a particular interest in data, or indeed in any of the areas we work in, to come and have at it at our first ever mySociety Data Hackday.
Not a journalist?
Journalists aren’t the only ones with bright ideas, so if you’re reading this and there’s a burning question that springs to mind, leave a comment below. Given all these reams of data, what would you be looking for? We’ll add the best ideas to our list, and we’ll be reporting back on everything we find out.
Actually, I am a journalist!
There are still a few places, so if you’d like to attend, please drop us a line. Note: we will expect you to get stuck in! We will run the data, but you may be sifting through the results, looking for significant stories, and sharing your findings. Bring a laptop, and plenty of ideas.
If you can’t attend, but really wish you could, let us know what data you’d like us to run, and we’ll add it to the list.
ETA: Lanyrd page here.
Image credit: Johan Nilsson
When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.
Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:
- It is entirely responsively designed, so that it works on the simplest of mobile web browsers from day one.
- All the lessons we learned from storing political data wrongly have been baked into this site (i.e we can easily cope with people changing names, parties and jobs)
- Every organisation, position and place in the system is now a proper object in the database. So if you want to see all the politicians who went to Nairobi University, you can.
- There is lots of clear information on how parliament functions, what MPs and committees do, and so on.
- It synthesizes some very complex National Taxpayer’s Association data on missing or wasted money into a really clear ‘scorecard‘, turning large sums of money into numbers of teachers.
The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.
Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.
If you haven’t got a penny,
A ha’penny will do,
If you haven’t got a ha’penny,
Then God bless you.
We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?
mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.
Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.
What’s behind the door? A letter to Santa.
If you can fit them down the chimney, here’s what we’re dreaming of:
More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.
We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.
Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.
Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?
A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.
Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?
FixMyTransport buy-in – from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.
Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.
TheyWorkForYou has, until now, only covered things that have already happened, be that Commons main chamber debates since 1935, Public Bill committees back to 2000, or all debates in the modern Northern Ireland Assembly.
From today, we are taking the UK Parliament’s upcoming business calendar and feeding it into our database and search engine, which means some notable new features. Firstly, and most simply, you can browse what’s on today (or the next day Parliament is sitting), or 16th May. Secondly, you can easily search this data, to e.g. see if there will be something happening regarding Twickenham. And best of all, if you’re signed up for an email alert – see below for instructions – you’ll get an email about any matching future business along with the matching new Hansard data we already send. We currently send about 25,000 alerts a day, with over 65,000 email addresses signed up to over 111,000 alerts.
Mark originally wrote some code to scrape Parliament’s business papers, but this sadly proved too fragile, so we settled on Parliament’s calendar which covers most of the same information and more importantly has (mostly) machine-readable data. Duncan and I worked on this intermittently amidst our other activities, with Duncan concentrating on the importer and updating our search indexer (thanks as ever to Xapian) whilst I got on with adding and integrating the new data into the site.
I’ve also taken the opportunity to rejig the home page (and fix the long-standing bug with popular searches that meant it was nearly always Linda Gilroy MP!) to remove the confusingly dense amount of recent links, bring it more in line with the recently refreshed Scottish Parliament and Northern Ireland Assembly home pages, and provide more information to users who might not have any idea what the site covers.
Signing up for an email alert: If you want to receive an email alert on a particular person (MP, Lord, MLA or MSP), visit their page on TheyWorkForYou and follow the “Email me updates” link. If you would like alerts for a particular word or phrase, or anything else, simply do a search for what you’re after, then follow the email alert or RSS links to the right of the results page.
As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.
To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).
Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.
Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.
There don’t appear to be any central official lists of candidates in these elections. Amnesty.org.uk has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations 🙂
Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15” to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.