This is a problem we have been warning about for some time. Islington Council were fined £70,000 for a similar incident in 2012. In light of this fresh incident we again urge all public authorities to take care when preparing data for release.
As with the Islington incident, the information was in parts of an Excel spreadsheet that were not immediately visible. It was automatically published on 14th November when Hackney Council sent it in response to a Freedom of Information request, as part of the normal operation of the WhatDoTheyKnow website. All requests sent via the website make it clear that this will happen.
This particular breach involved a new kind of hidden information we hadn’t seen before – the released spreadsheet had previously been linked to another spreadsheet containing the private information, and the private information had been cached in the “Named Range” data in the released spreadsheet.
Although it was not straightforward to access the information directly using Excel, it was directly visible using other Windows programs such as Notepad. It had also been indexed by Google and some of it was displayed in their search previews.
The breach was first hit upon by one of the data subjects searching for their own name. When they contacted us on 25th November to ask about this, one of our volunteers, Richard, realised what had happened. He immediately hid the information from public view and notified the council.
We did not receive any substantive response from the council and therefore contacted them again on 3rd December. The council had investigated the original report but not understood the problem, and were in fact preparing to send a new copy of the information to the WhatDoTheyKnow site, which would have caused the breach to be repeated.
We reiterated what we had found and advised them to consult with IT experts within their organisation. The next day, 4th December, we sent them a further notification of what had happened, copying the Information Commissioner’s Office (ICO). As far as we are aware, this was the first time the ICO was informed of the breach.
From our point of view it is very disappointing that these incidents are still happening. Freedom of Information requests made via WhatDoTheyKnow are a small fraction of all requests, so it is very likely that this kind of error happens many more times in private responses to requesters, without the public authority ever becoming aware.
Our earlier blog post has several tips for avoiding this problem. These tips include using CSV format to release spreadsheets, and checking that file sizes are consistent with the intended release. Either of these approaches would have averted this particular breach.
We would also urge the ICO to do as much as possible to educate authorities about this issue.
Every general election there are a load of projects that all need the same thing – a nicely formatted, accurate list of the candidates who are standing at the election.
Loads of people need this data – journalists, app builders, campaigners, Wikipedians, everyone.
But the government doesn’t actually publish the lists until right before the election, and when it does the data isn’t the least bit suitable for modern use (think unstructured PDFs and worse). It’s way too little and way too late.
YourNextMP.com is a totally free, open database of candidates, that is made partly from screen scraping and partly from volunteer contributions from people who think that having a single good quality list is a sane idea. It publishes the open data gathered both through a nice clean website, and through a nice modern API. Soon it’ll also provide csv export,too. And it means we can have nice shared identifiers for candidates, meaning greater potential connectivity between election-related journalism, tools, sites and projects run by different people and organisations.
The builders of YourNextMP have also taken steps to ensure accuracy and deter abuse, most strikingly by forcing all new data to be sourced, and keeping nice public logs of all the changes (and who made them).
To be clear, YourNextMP is not a mySociety project. We are just very happy to endorse the idea, and to supply one of our open source tools (PopIt) to help store and share the data in useful ways. Plus some of us have been chipping in in our spare time, for instance by adding data.
How can you help?
There are two main ways:
1) Add data! The main thing needed today, 146 days before the election, is the most basic data on who is known to be standing, today. We think that YourNextMP is probably already the most up to date candidate list out there, despite being very much unfinished.
Additional data, about candidates’ Facebook pages, birth dates and so on, isn’t such a high priority right now. You can help by looking up your constituency on the site, or choosing a random constituency, and just using your best Googling/telephoning skills to find out who’s definitely standing this time.
If you want to chat to other people who are doing the same thing, use the #yournextmp hashtag.
Don’t feel you have to stop when you’ve filled in your own constituency – there are plenty more to complete.
2) Spread the word that a single, high quality, free and shared database of candidates is just A Good Thing that people should support.
Who loves time-wasting? Nobody! What is YourNextMP if not an anti time-wasting project? Nothing! So, please, if you’re planning an election-related project, tell people that YourNextMP is a good idea, and consider letting them use your logo on their site, as a sign of good will.
And if you see someone in your office about to pay for a proprietary database of candidates, why not suggest they give the money to YourNextMP instead?
NB: No agencies please.
Party Conference season is upon us again, and, with it, a new set of fine promises and rhetorical flourishes, as each party’s top dogs take the podium. But what happens to those pledges, vows and forecasts once the banners are taken down and the party faithful turn for home?
Cast your mind back to November 2013, and you may recall that there was bit of a fuss about the fact that the Conservative party had removed old speeches from their website.
Not just that, but they’d also effectively erased them from the places where you can commonly find retired internet content… unless you really know where to look.
Was it a sinister rewriting of history, or a simple spring clean of elderly content? Well, that depends who you believe – but here at mySociety, we do think that you should be able to hold political parties to account for promises they made in the past.
Not only that, but we happen to have a splendid tool for publishing the spoken word: SayIt.
So we thought we’d track down that missing content and put it online for anyone to search and browse. And because we are a wholly non-partisan organisation, we did the same for Labour.
Note: we’re not intending to update these collections regularly – it’s a one-off initiative, designed to fill a gap in the public archive. And within the confines of this project, we’ve only published Labour and Conservative speeches.
On the other hand, if you’re interested in setting up similar sites for the other parties, or even taking over these ones, SayIt is very simple for anyone to use: just get in touch.Image by Klaus Riesner (CC)
An analysis, with code and data, of which Commons votes would have had different results, if Scottish MPs’ votes hadn’t been counted since 1997.
By Richard Taylor and Anna Powell-Smith.
PublicWhip is a wonderful thing. Founded and still run by independent volunteers, it contains the results of every House of Commons vote since 1997, scraped from the official web pages and presented as simple structured data. Here at mySociety, we’ve used it to power TheyWorkForYou for many years.
Most recently, it helped our staffer Richard create the new voting analyses on TheyWorkForYou’s MP pages. Want a quick, simple summary of your MP’s voting history on same-sex marriage or climate change, or on any of 62 other major issues? You’ll now find the answer on your MP’s TheyWorkForYou page, all based on PublicWhip data.
But here’s the most exciting thing about PublicWhip. If you know how to get around its slightly forbidding exterior, it contains a treasure-trove of data on MPs’ voting patterns, all structured, openly-licensed and ready for anyone to analyse.
A data challenge
Recently, while discussing the upcoming Scottish referendum, Richard posed a question to Anna: could PublicWhip data tell us which House of Commons votes would have had different results, if Scottish MPs’ votes hadn’t been counted?
This is interesting because if the Scottish people vote “yes” to independence on September 18th, we may see (probably not as soon as 2015, but perhaps soon thereafter) a House of Commons without Scottish MPs. No-one really knows how such a Parliament would be different.
While it was widely reported that that Scottish MPs’ votes carried the decision to introduce student tuition fees and foundation hospitals in England, those were just two high-profile votes. To our knowledge, no-one has published a comprehensive analysis of all votes that were carried by the Scottish MPs.
Anna chose to accept Richard’s challenge, and to use PublicWhip data to carry out this analysis. You can see all their code, and the data they produced, on GitHub.
The headline finding is that only 21 votes (out of nearly 5000 since 1997) would have gone differently if Scottish MP’s votes hadn’t been counted. This surprised Anna, who expected more.
Secondly, if there’s any visible pattern, it’s that English MPs seem to have a stronger civil-libertarian bent than their Scottish counterparts. High-profile votes on 42-day detention, “glorifying terrorism”, allowing the Lord Chancellor to suspend inquests, and on control orders: according to Anna’s analysis, all would have gone differently if Scottish MPs had not been in the chamber.
Other than that – Anna comments – the key finding is perhaps the absence of any other strong trend.
Here is the full list of votes that would have gone differently – click on the date to see the full vote details on PublicWhip. If Scottish MPs hadn’t been in the chamber:
- 5 Sep 2014 The majority of MPs would have voted to send the Affordable Homes Bill to a Select Committee rather than a Public Bill Committee.
- 29 August 2013 The majority of MPs would have voted to agree that a strong humanitarian response to the use of chemical weapons in Syria was required from the international community, and that it may, if necessary, require military action. (You may remember that David Cameron called MPs back from their summer break to vote on this, and MPs rejected the motion.)
- 29 Jan 2013 The majority of MPs would have voted against postponing a review of the boundaries of parliamentary constituencies until 2018 and against delaying a review of the effect of reducing the number of MPs.
- 31 Oct 2012 The majority of MPs would have voted against calling on the UK Government to seek a real-terms cut in the European Union budget.
- 24 Apr 2012 The majority of MPs would have voted to require products containing halal and kosher meat to be labelled as such.
- 24 Feb 2010 The majority of MPs would have voted for restrictions on the amount of carbon dioxide electricity generation plants are permitted to emit.
- 9 Nov 2009 The majority of MPs would have voted against allowing the Lord Chancellor (a minister) to suspend an inquest and replace it with an inquiry and against allowing the use of intercepted communications evidence in inquests.
- 8 Dec 2008 The majority of MPs would have voted to immediately starting the proceedings of a committee of MPs to investigate the House of Commons procedures in light of the seizure by the police of material belonging to Damian Green MP.
- 12 Nov 2008 The majority of MPs would have voted to require membership of new regional select committees to be determined taking account of the proportion of members of each party representing constituencies in the relevant region and for at least one member from each of the three largest parties to be on each committee.
- 11 Jun 2008 The majority of MPs would have voted against extending the period of police detention without making any criminal charges of terrorist suspects from 28 days to 42 days.
- 2 Jun 2008 The majority of MPs would have voted to require the National Policy Statement to contain policies which contribute to the mitigation of, and adaptation to, climate change.
- 15 Mar 2006 The majority of MPs would have voted against a proposed timetable for the Parliamentary consideration of the Education and Inspections Bill.
- 2 Nov 2005 The majority of MPs would have voted against making glorifying the commission or preparation of acts of terrorism an offence.
- 2 Nov 2005 The majority of MPs would have voted to make the offence of Encouragement of Terrorism only apply to cases where an individual intended their actions to encourage terrorism.
- 28 Feb 2005 The majority of MPs would have voted to give a greater role to the courts in relation to the imposition of control orders.
- 22 Apr 2004 The majority of MPs would have voted against installing a security screen separating the public gallery from the House of Commons Chamber.
- 31 Mar 2004 The majority of MPs would have voted against the introduction of variable university tuition fees (top-up fees) of up to £3,000 per year in place of the previous fixed fee of £1,250 per year.
- 27 Jan 2004 The majority of MPs would have voted against allowing university tuition fees to increase from £1,125 per year to up to £3,000 per year, and against making other changes to higher education funding and regulation arrangements.
- 19 Nov 2003 The majority of MPs would have voted against introducing NHS foundation trusts, bodies with a degree of financial and managerial independence from the Department of Health.
- 4 Feb 2003 The majority of MPs would have voted for an 80% elected House of Lords.
- 29 Oct 2002 The majority of MPs would have voted against starting sittings of the House of Commons on Tuesdays at 11.30am rather than 2.30pm.
In the 1997-2001 Parliament, Anna’s code found no votes that would have had different results.
IMPORTANT DISCLAIMER! We can’t conclude that all of the above would necessarily have become law if Scottish MPs had not been in the chamber. Bills don’t become law until they have passed through the House of Lords – not to mention the many other forces of history that would have acted differently.
Get the code and the data
You can see the code used for this analysis, and the full datasets, on GitHub. You can adapt it yourself if you want to do your own analyses.
This analysis is the work of one volunteer: we welcome any corrections. Like PublicWhip itself, the whole point is that it is out in open for anyone to analyse and improve.
Image by Catherine Bebbington. Parliamentary copyright image reproduced with the permission of Parliament.
We are recruiting.
As so often with positions at mySociety, it’s a job that’s a bit different from the norm. For the right person, it’s going to be a fantastic opportunity.
If you know someone who fits the bill, please do forward them the link to this job description: http://mysocietyltd.theresumator.com/apply/KJmhA1/Head-Of-Research.html.
They might also like to see our page on what it’s like working at mySociety.
Applications close on June 30th, so time is of the essence.
NO RECRUITERS OR AGENCIES PLEASEImage: Jason Samfield (CC)
We’ve just published the WriteToThem responsiveness league table for 2013. Check your MP’s performance here – just enter your postcode.
League table? What’s that?
Our website WriteToThem.com allows anyone to send a message to their elected representatives.
If you’ve ever done this, you’ll know that two weeks later, we email you to ask whether or not your representative replied.
The information we obtain from this questionnaire is important to us: it helps us check that WriteToThem remains an effective way to contact politicians. But, when it’s analysed further, there are interesting results to be found.
WriteToThem launched in 2005. Until 2008, we published an annual ‘league table’, ranking MPs by responsiveness. We did this because we believe that it is a fundamental part of an MP’s duty to respond to their constituents’ messages; we wanted to recognise the best performers, and highlight the ones who were falling below expectations.
We haven’t run this data since 2008 – mainly because we’re a very busy organisation with a wide range of priorities.
But our users frequently ask for the latest stats, and to that end we’ve now run the 2013 data. Take a look at it here.
A big WriteToThem gold star to some MPs
The people of Romsey and Southampton North should rest easy. Their Conservative MP Caroline Nokes is on the case. Top of our league table, she replied to 96% of messages sent through WriteToThem.
Other good performers include Conservatives John Glen MP for Salisbury, and Justin Tomlinson representing North Swindon. Gloria De Piero, Labour MP for Ashfield, comes in at 4th position. Check your MP’s performance here.
And ‘could do better, see me’ to others
Mansfield residents may feel like nobody’s listening; their representative Alan Meale (Labour) comes bottom of the rankings, having replied to a sole message in 2013.
Other low responders were Khalid Mahmood (Labour), representing Birmingham Perry Barr; Kenneth Clarke (Conservative) for Rushcliffe; and Tom Blenkinsop (Labour) in Middlesbrough South and East Cleveland. Check your MP’s performance here.
Not just MPs
WriteToThem isn’t just for contacting your MP. You can also use it to write to Lords, councillors, MEPs and members of the assemblies of Wales, Scotland and Northern Ireland.
Running this data also allows us to make broad comparisons across all of these bodies – see our figures here.
The Welsh Assembly comes out looking fairly respectable, with a 70% response rate, while the House of Lords (who, it must be noted, do not have an obligation to respond to correspondence) slink in at 27%.
We’ve also sliced the data so you can see which political parties perform best and worst overall. Guess who comes top?
Data and methodology
- Our figures are based on our follow-up questionnaire, and of course, not all users respond to it. This data is based on 58,573 responses; you can see more about the data below.
- Letters sent via WriteToThem represent less than 1% of the entire parliamentary postbag, so this has to be taken as a sample rather than giving the full picture across the board.
- WriteToThem is not the only way that people can contact their representatives. For all we know, those poor performers may be responding perfectly adequately to messages sent by other channels – although we do make it as simple as we can for them to reply to WriteToThem users, and it’s our belief that the channel of communication should not make any difference.
We know, too, that some messages don’t require an answer. We would not expect to see a 100% response rate, and, by the way, we are considering altering our questionnaire so that it includes the option “I didn’t get a reply, but my message didn’t need one”.
- It’s also important to note that this league table is not a ‘laziness’ ranking. MPs do many other things besides reply to their constituents’ letters. Poor responders may be incredibly active in their constituency, or in Westminster debates. So it’s what it says it is – a responsiveness league table, no more, no less.
- WriteToThem sent 96,396 messages to MPs in the year 2013 and 103,965 to other elected representatives.
- 58,573 people answered our feedback survey about communicating with their MP.
The survey asked whether people had had a reply (not just an acknowledgement) from their representative.
People were surveyed initially after 2 weeks, and if they didn’t answer, were surveyed again after 3 weeks.
Because of this, and because of the way different people interpret the survey, you should interpret the figures with some caution.
We did not include any MP who received fewer than 20 messages in 2013, as the sample numbers are too small to be indicative. See the bottom of our league table for the MPs affected: here you may also see which MPs do not accept correspondence sent via WriteToThem.
Before preparing this table, we contacted the lowest performers to ensure that we had the right email addresses for them.
In the cases of Caroline Flint (9 out of 63 positive responses to our survey), Stephen Dorrell (18 out of 94) and Tom Watson (4 out of 42), we were informed that while the addresses were monitored, there were better ones to use – these are now in place on WriteToThem.
In the case of Alasdair McDonnell (10 out of 57), we were informed that we had the correct address. Jack Lopresti (4 out of 70) and Stephen Williams (53 out of 267) did not respond.Image credit: Barry (CC)
The right conference, held at the right time and attended by people with common problems, can sometimes give birth to whole new organisations. I was at OpenTech when the Open Rights Group was born, and on a grander scale the Red Cross and the UN both featured conferences at catalytic moments in their early history.
Last week in Santiago, Chile, a conference took place that felt like exactly such a moment – PoplusCon. People from 27 countries spent two days talking about their shared goals and desires, and from it the skeleton of a new federation – the Poplus federation – started to take shape.
Not everyone at the conference worked on identical projects, or had identical skills. Some people were specialists in tracking suspicious relationships (‘This guy’s brother-in-law gets all the contracts’), others were big into training journalists how to use FOI, others specialised in making important datasets more accessible to members of the public, others still were journalists, skilled at constructing stories. But one theme emerged pretty quickly – people wanted better, easier, more reliable ways of sharing knowledge and sharing technology, so that they could all save time, effort and money.
What could a new federation do for you?
And so that is how the conversation turned to the idea of founding a new federation – an organisation that could serve the needs of many different groups without being run or owned by any one of them. In a brainstorm session about what people wanted from a new federation, the following ideas were raised:
- Running events to facilitate more sharing of ideas and tech
- Publishing stories about successful and unsuccessful projects, especially where those stories need to cross language barriers to spread
- Vetting and endorsing data standards
- Access to a community of peers (for sharing experience, encouragement, tips and tricks etc)
- Resources for projects that are running short
- Help and advice on making projects sustainable
- Certification of what counts as a Poplus Component
- Where groups face common challenges, perhaps coordinate advocacy
- Organisation of mentorship, exchanges and placements
This wish list is clearly far more than a nascent organisation could arrange in the near future, but there was some informal voting and the top priorities fairly quickly emerged. People really wanted access to their peers, and to the stories that they tell. And there was a strong wish to see Poplus Components become more official, and better explained.
Getting Real – Getting Involved
But a list is just a list without people willing to make it real. And so without doubt the most awesome thing that took place at PoplusCon was that eight people immediately volunteered to form a committee that would bring Poplus into being, representing half a dozen countries in different parts of the world.
This committee, which is completely open for anyone to join, will be meeting a couple of times in the next few weeks to agree on a plan for the first 12 months of the Poplus federation. It will work out how the new-born federation should govern itself, and what the first things that this entirely volunteer-run group should be doing. It’s an exciting, fragile moment and I’ve not seen anything like it in my ten-odd years working in this field. There’s no boss, no leader, just some people trying to build something of shared value.
Right now there are no rules, no barriers to entry, no bureaucracy. In fact there’s nothing but some hope, enthusiasm and some shared dreams of a stronger community of individuals and organisations.
I hope that if you read this and think that Poplus sounds cool, that you’ll consider joining the committee too. All you have to do is join the mailing list and ask where and when to show up. If you come to online committee meetings a couple of times, you’re de facto one of the people who runs Poplus. What happens next is – quite literally – down to you.
You may have heard that a widespread security problem – ‘Heartbleed’ – has been found that affects a large proportion of all websites on the Internet.
Here is one of the many explanations about the nature of the problem.
Members of the mySociety team have reviewed our potential exposure to the vulnerability.
We have no indication that our sites have been attacked, or that any information has been stolen, but the nature of the vulnerability would make an attack difficult to detect, and we prefer to be reasonably cautious.
What does this mean for you? The advice from around the web has been for people to change passwords, especially on sites they use that contain a lot of very important information (e.g. your email account).
We think the risk that passwords have been compromised is low, but as changing passwords occasionally is always a good idea anyway, now might be a good time.
For those of you interested in the technical detail of our response, we have:
- Upgraded the SSL software
- Installed new SSL certificates based on a new private key
- Revoked the old SSL certificates
- Replaced the secrets used for security purposes in the affected sites
- Removed active sessions on affected sites, so that users will need to log in again
- Required that users with administrative access to affected sites reset their passwords
- Required that staff users reset their passwords
- Notified affected commercial clients so that they can take appropriate action
A few months ago we won a contract from Parliament to review its digital service provision (brief advertorial – we can do this kind of work for your organisation too).
Today Parliament has published that review. Here are a few comments:
- It’s great news that the Management Boards in Parliament have agreed to implement the two recommendations contained in the report. Reviews are one thing, actions another.
- It’s great that Parliament chose to publish the review at all. They didn’t have to, but they chose to without any prodding. Big thumbs up.
- We interviewed a lot of parliamentary staff (dozens and dozens of people). They’re a fantastically dedicated, interesting bunch working under often absurd pressures, and we think Parliament overall would probably have a better reputation if they had as much visibility as the elected members. Time for a docusoap, maybe?
- The review contains only two recommendations, even though there were hundreds of good ideas floating around. The reason for such extreme minimalism was to ensure that there was no ambiguity whatsoever about what we believe to be the essential reforms*. Once those reforms are enacted, the ground will be much more fertile for specific digital projects.
- My colleagues Ben Nickolls, Dave Whiteland and Mike Thompson did the majority of the real work on this review, conducting interviews and analysing data. My thanks to them for a job well done.
Parliament is encouraging public feedback on the review. Let them know what you think via NewDigitalService@parliament.uk
* If you want a digital review filled to the brim with lots of recommendations, try this 25 point action plan from the US federal government instead. Just remember that it was published roughly three years before this.
Photo by Greg Dunlap (CC)