Scottish Parliamentary proceedings are now back on TheyWorkForYou.
Back in August 2014, the Scottish Parliament changed the way it published the Official Report of its debates.
TheyWorkForYou works by fetching data from various parliamentary sources—and in this case, unfortunately, the change at the Scottish Parliament end meant that our code no longer worked. We replaced our ‘debates’ section with an apologetic note.
Well, thanks to the Scottish Parliament kindly republishing the data in almost the format we used to use, we’ve managed to make some small tweaks and restore that content—including debates from the previously missing period. If you’re subscribed to alerts, you should have received an email digest with links to the backdated content (always supposing there was any that matched your chosen keywords).
And if you’re not subscribed to alerts? Now is a great time to rectify that. We’ll send you an email every time your chosen word or phrase is mentioned in Parliament, or every time your chosen representative speaks.
While we were doing this work, we also modified TheyWorkForYou so that it now pulls in ministerial data from the Scottish Parliament API. This is a welcome time-saver for us: previously we were creating a list manually from the official PDFs, while we can now automatically fetch it and reformat it into Popolo JSON, meaning it’s consistent with all our other data.
Thanks for your patience; we know that many people were awaiting this repair, and for longer than we would have liked. Enjoy!
Back in November 2013, we asked you what improvements you’d like to see on TheyWorkForYou.
One answer dominated: you wanted more information about how MPs vote.
Adding information on voting has been the single biggest project on the site since its launch, and has required several different phases of development. We announced each of these as it happened, but now that we’re at the end of this large piece of work, it seems like a good time for a complete overview.
So let’s take a look at exactly what it has involved—and, more importantly, what it means for you.
We’ll start with a rundown of features, then go into more detail about how they are created at the end of the post, for those who are interested.
What vote information means for you
1. You can easily see how your MP voted
Just how much do you know about how your MP voted on the stuff that matters? Most of us would have a hard time keeping up with every vote, simply because it isn’t information that’s widely publicised.
On TheyWorkForYou, you can see a run-down of how any MP has voted on key policies, by visiting their page on the site and clicking the ‘voting record’ tab (see image, above). We’ve created summaries of their stance on all kinds of matters, including the EU, same-sex marriage, NHS reform and a lot more.
Each of these summaries is compiled from every vote the MP has made on a motion that impacts on that policy.
You can click ‘show votes’ (see image above) to see the specific votes that go to make up any particular stance, and we’ve laid them all out in plain English so that it’s easy to grasp exactly what the issue is.
And from there you can click through to the website Public Whip, where you can explore votes in more detail, including lists of who voted for or against any given motion.
2. You can find out how strongly your MP feels
When we first presented voting information, we said that an MP had voted ‘strongly for’ or ‘moderately against’ certain policies, which led to quite a large postbag from people asking, “How can you vote strongly, surely you either vote for or against?”.
We wrote in the second half of this blog post about the wording changes we made to clarify the fact that these stances are calculated from a number of votes.
3. You can assess if your MP is a sheep or a lone wolf
We’ve pulled out all the votes which differ substantially from the way that the majority of each MP’s party voted. If your MP has voted against the flow, you’ll see something like this on their page:
Why do we highlight this type of vote? Because we think they’re a really good indication of where an MP feels strongly enough about something to risk sticking their neck out. It’s also a great way to check the truth when people say, “MPs? They’re all the same”.
4. You can understand the background to the votes
Generally speaking, there’s a debate before any vote takes place in Parliament, covering all the matters which may be topmost in MPs’ minds before they cast their lot.
Clicking on the ‘show full debate’ link from the topic pages (see image above) will give you the full context.
How we compile vote information
If that all seems nice and simple, well, great! That was our aim.
Putting it all together definitely wasn’t so simple, though. Voting information has never been previously presented all in one place in quite this way before—on TheyWorkForYou or anywhere else, to the best of our knowledge—so we had to figure out how to import the data and how best to display it.
As with much of our work, it’s a mixture of manual graft and automating whatever we can. Some things, like rewriting votes so that everyone can understand them, can’t be done by a computer. But many of our users are surprised to learn just how much of what we publish out is untouched by human hand.
Our Developer Struan, who did the most recent round of work on the voting records, said:
We get all our voting data from PublicWhip, a site set up by Francis Irving (once of mySociety) and Julian Todd. Public Whip takes the data we [TheyWorkForYou] produce from Hansard and extracts only the information on votes (or divisions in Parliamentary jargon) that take place in Parliament. It then allows you to look up how an MP or a Lord voted.
Let’s just think about that for a moment. We’re looking at a process where Parliament publishes Hansard, TheyWorkForYou scrapes the data and re-presents it, Public Whip extracts the voting information and presents that, and TheyWorkForYou takes that voting information back for its own voting pages. Simple…
One of the first things we did was to ‘translate’ the votes into plain English, so that it was very clear what was being voted for or against— and if you want to read more about that process, we talked about it in a blog post back in July 2014.
That allowed us to move to the next phase, as Struan explains:
Public Whip groups related votes together into policies, e.g renewing Trident, so you can see how an MP voted on the policy as a whole.
It does this by saying which way an MP would have to vote each of the divisions in the policy if they agreed with the policy. It then takes the MP’s votes on each division in the policy and assigns a score to it based on how they voted. These scores are then added up and compared to the score they would get if they always voted in agreement with the policy. The closer the MP’s score is to the score of an MP who always voted in agreement with the policy, the more they agree with the policy.
Thanks to Public Whip’s grouping, we were able to start compiling our MPs’ voting records along those same policy lines.
One of the most fiddly parts of the process was figuring out how to ensure that the information we present is a true, non-biased representation of the MP’s intentions. You might think that a vote is quite a simple matter – it’s either a yes or a no for a particular motion. But as soon as we started displaying votes within a policy, things got a bit trickier.
Some divisions in a policy can be marked as important and voting with the policy in those divisions is worth more points. This is to prevent voting in agreement on a set of minor votes, e.g “Parliament will commission a report on the future of Trident”, outweighing voting against something important, e.g. “Renew Trident”. It also reflects the way Parliament works, often with several smaller votes on parts of a bill and then a vote on the bill as a whole.
For clarity I should point out here that sometimes voting no in a division is a vote for the policy, e.g voting no in a “This house believes Trident should not be renewed” division would clearly be a vote for our example “Renew Trident” policy.
This approach also helps where one vote straddles several topics: for example, consider a vote against the Budget when the Budget contains many proposals including, say, the capping of VAT. It’s quite possible that an MP may be for the capping of VAT but broadly against several other motions covered by the Budget, and so decides to vote against it on balance. So long as we mark the Budget vote as a weak vote for the capping of VAT, its significance should be properly accounted for.
Where we don’t have enough information to show a stance, for example where an MP never voted on the topic, is too new to have had a chance to vote on the topic, or all their votes on the topic have been labelled as ‘weak’, we say so:
A final little subtlety is the difference between “Never voted” on a policy and votes where the MP was absent. If it says an MP has never voted on a policy that means they were elected after all the divisions in the policy took place so did not have a chance to vote on them. Absent means they could have voted in the divisions but did not.
Absent votes count towards your score but at half the rate of voting in agreement with the policy. This is so that an MP who votes in agreement with the policy in one division and then misses all the other divisions shows as agreeing with the policy rather than against as it would if no score was assigned to absent votes. That does currently mean that if they were always absent it shows, slightly unhelpfully, as “a mixture of for and against”.
It’s not an ideal system as it does produce some odd results occasionally but it mostly works.
To show where an MP has voted against the majority of their party, we have to figure out a similar score across the party as a whole.
This is exactly the same process, only we add up all the votes by all the MPs but the maths is pretty much the same.
All in a day’s work
As mentioned at the top of this post, vote information was our most-requested addition. And rightly so! Our MPs represent us, so naturally we want to see their track records, quickly and easily.
If you’re not an expert, you might not have known how to find this information before. And that’s essentially what TheyWorkForYou aims to do: make the workings of Parliament more accessible for everyone.
Parliamentary copyright images are reproduced with the permission of Parliament.
Last Sunday, several mySociety team members woke up just in time to see the sun rise over the Andes – not our standard view in the morning.
We were on our way to Santiago, Chile, as were delegates from 27 different countries, all headed to the inaugural Poplus Conference. This was a joint event organised by ourselves and our Chilean friends Fundación Ciudadano Inteligente.
Long-haul flights don’t feature high on our list of favourite activities – so what was so important that we all happily put in the 7,000 miles to Santiago?
Before we can answer that question, you need to know what Poplus is. In short, it’s a project to create and share open source code that can help civic organisations around the world. You can find out more here.
While legal structures and ruling regimes vary from place to place, the needs of citizens are broadly constant: they require information, empowerment, and transparency.
The digital age has provided opportunities here, and there are many organisations like mySociety and Ciudadano Inteligente around the world: groups that harness the power of the internet to open access to civic and democratic processes.
Underpinning Poplus is the belief that we can make great efficiencies if we share our code – that way, there’s no need for each of these groups to rewrite what is effectively the same piece of software. Poplus aims to encourage the creation of ‘Components’ – small pieces of software that can be used easily, by anyone across the world.
Poplus – the next step
Poplus is a project in its infancy, and this conference was the next step in its growth. It brought together civic coders, and organisations with an interest in the code they create.
The Poplus project was initially conceived by Ciudadano Inteligente and mySociety – but in order to thrive, it needs many members from all over the world to play a part. The conference allowed us to present the general idea behind Poplus, and to ask for help shaping and refining it. Once some underlying principles had been agreed, we could become a true federation.
There were also many opportunities to listen to organisations, about what they needed from Poplus software, and their experiences running civic websites in their own territories. By the close of the conference, we had thrashed out some broad agreements, and there was a lot of excitement about carrying on the work to create Poplus Components – and the community around the movement.
Defining a Poplus Component
One important task was to define exactly what makes a Poplus Component: the conference offered us a real opportunity to get input from many different perspectives, and come away with a ‘gold standard’.
It was very useful to have developers and end users in the same room, talking about the process of creating Components, and the experience of using them.
You can see what we came up with in the photograph (click to see it bigger). Among other things:
- Poplus Components are small pieces of software which provide functionality for civic or democratic websites.
- Each Component solves a single problem.
- They are built to work in any country, making minimal assumptions about location.
- They are open source and free for anyone to use.
- They slot into any website, and may also inter-operate.
Just the start
The Poplus Conference was a great opportunity to nail down everyone’s thoughts. Now we have agreed on our shared purpose, the real work will begin. The conference fostered communication and sense of community, and we’ll all be trying to keep that alive.
A huge thanks to everyone who attended and contributed – and especially to Ciudadano Inteligente for being such welcoming and generous hosts.
- For a quick run-through of the conference as a whole, see it on Storify
- Or for a more in-depth experience, check out the #PoplusCon hashtag
- Session notes are here.
- There are photos on Flickr (with more to follow)
At the beginning of this year we launched SayIt, our software for bringing transcript publication into the internet age.
In that post, we briefly mentioned that “we use a cut-down version of the Akoma Ntoso open standard for data import”.
Well, that’s easy enough to say, but what does it actually mean?
In a nutshell, if you want to upload transcripts to SayIt, they need to be in a format that SayIt can recognise. It can then transform them into the linked pages that make SayIt so useful.
Akoma Ntoso is a simple way of showing (for example) which bits of the data are names, which are speeches, etc. – and how they all relate to one another. At first glance, it’s not all that different from HTML, the basic language behind many websites.
But there are some differences. There are also some interesting ideas behind it, from how it began and where it got its name, to why the world needs another open standard. And what is an open standard, anyway?
We pinned down Flavio Zeni, one of the people behind Akoma Ntoso’s creation, and he very patiently answered all our questions, even the most basic ones.
And then, because it seemed silly not to, we put the whole conversation into SayIt. You can read it here.
Much of what we do here at mySociety relies on Open Data, so naturally we support Open Data Day. In case you haven’t come across this event before, here’s the low-down:
Open Data Day is a gathering of citizens in cities around the world to write applications, liberate data, create visualizations and publish analyses using open public data to show support for and encourage the adoption open data policies by the world’s local, regional and national governments.
If you’re planning on being a part of Open Data Day, you may find some of mySociety’s feeds, tools and APIs useful. This post attempts to put them all in one place. (more…)
Last week we asked what improvements you’d like to see on TheyWorkForYou. Thanks so much for all the comments on that post (do keep them coming). They’ve all been carefully documented on our development list.
Our standard way of working on a project like this is in ‘sprints’ – short periods of activity after which we can spend some time reflecting on what went well, and what could have gone better.
This system is great for ensuring that we don’t get involved in a large piece of work, only to realise that it doesn’t do what was intended, or hasn’t had the desired effect. So, for example, if we’ve added a new feature, we might be asking ourselves, ‘Is anyone using it?’, ‘Have there been any bug reports?’, and ‘Has it fulfilled our original aim?’. We’re striving to be as analytical and methodical as possible about these assessments, so part of the process has also been figuring out which types of metrics to collect, and how.
That said, what have we already done?
It’s easier to find a specific representative
Where previously our pages listing all MPs, all MSPs and all MLAs just contained one very long list of names that you had to search or scroll through, there’s now an A-Z navigation at the top. We also added the ability to find your own MP from this page.
Why? This is an example of a small usability tweak which should make a difference to a large number of people – not everyone knows how to search a web page with Ctrl+F. It’s also a fix that’s been on our to-do list for two years!
The addition of the ‘find your MP’ box helps to serve one of our core aims: to make democracy easy to understand for the uninitiated.
We’ve added ‘like’ and ‘follow’ buttons
We thought you might not notice these discreet additions to our page footers – but we’ve certainly seen an upturn on the rate at which people are ‘liking’ our Facebook page. Whereas Twitter – not so much. Maybe TheyWorkForYou users are just more Facebook-inclined?
Why? In part, this addition is for our own benefit – we welcome the opportunity that social media gives our users to spread the word. As a small organisation with no advertising budget, this kind of grass roots promotion is invaluable. Then, we are hoping that it will help us to understand our users. Clicking that ‘like’ button can be seen as a form of positive affirmation and enagement that it’s very hard to quantify by other means.
We are still considering the addition of buttons which would allow you to share specific debates with your social circles.
We have noted the comments on our last post which made it clear that some of our users do not welcome integration with social media. That’s fine – we’ll never do anything that excludes you from the core activities of the site, whether you use Facebook and Twitter or not – our intention is simply to provide the functionality for those who want it.
Those comments have been a useful reminder to us that we should continue to consult our users, because we can’t always predict what you might object to!
You can change your email address
If you have an account, now you can change your email address yourself.
Why? This was identified as a common request that often puzzled users, and took up support time on our side.
MPs’ pages will look better
You can’t see these yet, because they’re still in progress. Due to some quirks of the code in which the site was originally built, the new design for the MPs’ pages has taken longer to implement than we’d anticipated. But we’re getting there.
Why? MPs’ pages contain an awful lot of information, from voting history to recent appearances, and more. The redesign will help us present all this information more clearly, making the page just as easy to read on a mobile device as it is on a desktop, and simply bringing the (frankly, dated) pages a more current look.
Bullets are bullets
This is almost ridiculous, but we think it was worth attending to. In recent user tests, we noticed some confusion, caused by the fact that our bullet points were in the form of small squares – they were frequently mistaken for check boxes.
Why? Just to rid the world of that one small piece of frustration that occurs when you try to tick a box that is not, in fact, a checkbox.
As I say, we are still actively collecting and working on your feedback, so please do keep it coming. Comment below this post, or drop us a line on firstname.lastname@example.org. I’ll be reporting back after our next sprint.
Photo by William Warby (CC)
More and more people are starting to build websites to help people become more powerful in their civic and democratic lives. Some of these are on codebases that mySociety has created which is so great. There are some things which we would love to happen when you take our code and re-use it.
We want people using our code to keep it as up to date as they can, so that they gain the benefits of any changes made to the code by us or by other users. There are a few reasons for this:
You can co-brand the site without breaking anything.
Dave, one of our developers, explains how you do this. “So suppose, instead of calling it FixMyStreet you want to call it FixMyBorchester with a Borchester logo. Obviously this is a very real requirement, because people want to rebrand. One very feasible (but wrong! As you’ll see…) way of doing this is downloading the FixMyStreet code, finding the bit that paints the FixMyStreet logo and replacing it with the words <h1>FixMyBorchester</h1> and an image. This would work as far as the FixMyBorchester branding would appear on the site.
But if you then saved and committed your change to git and passed it back to us as a push request, we would reject it. This is for the obvious reason that if we didn’t, next time we deployed FixMyStreet in the UK it would have your logo on it.
However, say we suddenly discover there is a bug with FixMyStreet. For (a bizarre) example, if someone put the number 0 in instead of a postcode and the site returns a huge picture of a kitten. We love kittens, but that’s not what the site is trying to do. So, we make some fixes to the code that rejects zeros, commit it, update the repo, and it’s now there on the master branch. We write to everyone saying “really everyone, update to the latest (most up-to-date) place on the master branch” And you think, “yeah OK!” and you download the latest version.
If you just download it and copy it into place, you’re going to lose your FixMyBorchester changes, because there’s a more recent version of that file from us that hasn’t got them. If you did a “git pull” (which roughly means, “git! get me the latest version of master branch”) then git will refuse because there’s a conflict on that file.
So, instead of inserting your FixMyBorchester stuff over ours, which can’t work, you make a new directory in the right place called ‘FixMyBorchester‘, put your stuff in there and switch the FixMyStreet config — which knows this is something people want to do — to use that cobrand. Any templates FixMyStreet finds in there will now be used instead of ours. You can now safely update the codebase from our repo from time to time and FixMyStreet and git will never damage your templates, because they are in a place it doesn’t mess with.”
You can add new features
Dave continues. “Say when someone uses FixMyBorchester it’s essential that you have their twitter handle, because every time a problem is updated, FixMyBorchester direct-tweets them a kitten for fun. Right now there is no capacity to store a twitter handle for a user in FixMyStreet.
You simply add a column to the users table in your database and add some code for accepting that twitter handle when you register, and sending the kittens etc. That’s new code that isn’t in FixMyStreet at all. Sooner or later you’ll need to put at least one line into the main FixMyStreet program code to make this happen. As soon as you do that you have the same problem we had before, only this time it’s in code not in an HTML template.
What we would encourage you to do is put all your new code in a branch that we can look at, and maybe set it to run only if there’s a config setting that says USE_TWITTER=true. That way any implementation that doesn’t want to use twitter, which is — at this point — every other FixMyStreet installation in the world — won’t be affected by it. You send that to us as a pull request and a developer checks it’s not breaking anything, and is up to scratch in quality, and has good test coverage. Then we’ll accept it.
Even though currently nobody else in the world wants your twitter feature, it’s not breaking anything and it’s now in the repo so you can automatically update from our master when we change bits of our files, and the installation/overwrite/git-pull will work. Plus anyone that does decide they want this feature will now be able to enable it and use it.”
And all of this helps everyone using the code; you have a secure website that can be patched and updated each time we release something, other people have access to features you’ve built and vice versa. And overall, the project becomes more feature rich.
Please do make changes and push them back to the main codebase!
Image credit: US Coast Guard CC BY-NC-ND
We’ve got some exciting news at mySociety.
As you know we’ve been helping with the Kenyan Mzalendo website for a while now. And, we’ve been lucky enough to gather interest in Mzalendo‘s codebase from a number of other countries. These range from Ghana to South Africa, and even as far afield as Paraguay. It’s amazing and humbling for us, but we’ve recently realised one thing: Mzalendo has a wealth of history in Kenya, and an amazingly complex political association. It’s also the name of the website there. So we needed a new name that would allow the code to be changed without the change being associated with the original Kenyan Mzalendo.
Enter Pombola. This is the new name for the codebase which powers TheyWorkForYou, Mzalendo, Odekro, ShineYourEye, work-in-progress Kuvakazim and other parliamentary monitoring websites across Africa and the world.
You may ask, “Why Pombola? What does it mean?”
Well, it is a pretty easy word to remember. And no one else is using it (possibly because we created it!). The word is a mix of the initials PMO (Parliamentary Monitoring Organisation) and Tombola.
“A Tombola?” I hear you cry in surprise. Well, in a Tombola, you are making a choice with no information at all – just selecting a ticket and hoping that you get lucky. A Pombola site aims to be the opposite – you’ll get as much information as possible about your elected representatives, so that when you make a choice in future, you’ll have all the facts.
If you’re interested in using the code you’ll find the repository here on github, along with some documentation.
If you’re not technical but still want to use this then please contact me and we can discuss what we can do!
And remember, this may monitor parliaments now, but you could use it for anything (*)
(*) Disclaimer: please only use for good. Girl with Balloons from Courtney Air map from OpenFlights.org
It was great to see everyone at the hacknight on Wednesday. This time, ideas ranged from using the FixMyStreet Platform for reporting of Trading Standards transgressions, to installing a Mzalendo for Greece. We hope that these ideas will flourish, and many others will arise at our future hacknights.
Talking of which, don’t forget to reserve a place if you’d like to come along. Hacknights are weekly. All the details are here.
Today, we are using the phrase “Alaveteli upgrade” rather a lot – and not just because it’s such a great tongue-twister. It’s also a notable milestone for our open-source community.
Alaveteli is the software that underlies WhatDoTheyKnow, our Freedom of Information website. The code can also be deployed by people in other countries who wish to set up a similar site. If you’re a ‘front-end user’, someone who just uses WhatDoTheyKnow to file or read FOI requests, this upgrade will go unnoticed… assuming all goes well at our end, that is. But if you’re a developer who’d like to use the platform in your own country, it makes several things easier for you.
Alaveteli will now be using the Rails 3 series – the series we were previously relying on, 2, has become obsolete. One benefit is that we’re fully supported by the core Rails team for security patches. But, more significant to our aim of sharing our software with organisations around the world, it makes Alaveteli easier to use and easier to contribute to. It’s more straightforward to install, dependencies are up-to-date, code is clearer, and there’s good test coverage – all things that will really help developers get their sites up and running without a problem.
Rails cognoscenti will be aware that series 4.0 is imminent – and that we’ve only upgraded to 3.1 when 3.2 is available. We will be upgrading further in due course – it seemed sensible to progress in smaller steps. But meanwhile, we’re happy with this upgrade! The bulk of the work was done by Henare Degan and Matthew Landauer of the Open Australia Foundation, as volunteers – and we are immensely grateful to them. Thanks, guys.
Image credit: Sashi Manek (cc)