On the 21st of February 2012 Alex Skene, representing mySociety’s Freedom of Information website WhatDoTheyKnow, appeared in front of the UK Parliament’s Justice Select Committee. The MPs on the committee were holding an evidence session as part of their post-legislative scrutiny of the Freedom of Information Act.

Video of the session can be viewed online via ParliamentLive.TV and the BBC’s Democracy Live. A transcript of the session will become available via TheyWorkForYou, typically these take a week or two to be produced.

Prior to the session WhatDoTheyKnow had submitted written evidence to the review making three main points:

• The scope of the act should be extended to cover a wider range of public bodies.
• Time limits should be introduced for public interest tests and internal reviews.
• There is a need for more proactive publication of information, and a culture of openness and transparency needs to continue to be nurtured and extended within the UK’s public sector

The committee appeared genuinely interested in finding out how FOI has performed to-date and how it can be improved.

Supercharging FOI

Alex told the committee that FOI enables evidence based policy making and empowers citizens; he said the WhatDoTheyKnow.com website supercharges the provisions of the FOI Act making it easier for people to take advantage of the right to access information which it gives them.

Ghosts

Elfyn Llwyd MP raised the question of vexatious and frivolous requests through the medium of ghosts. Asked if requests about ghosts could ever be justified Alex told MPs that it was hard to draw the line between acceptable and unacceptable requests. He noted that one council had spent public money on an exorcism, so in that case there would be information held and an FOI request justified. He questioned if requests on ghosts were to be deemed unacceptable, what other areas might be excluded. UFOs? The MoD for a long period did have an office collating UFO reports, again there was public spending, and recorded information held, in this area. Homeopathy was also highlighted, that’s about as real as ghosts or UFOs, but again FOI requests about it must surely be permitted as significant amounts of taxpayers money are spent on it.

Maurice Frankel, the director of the Campaign for Freedom of Information, who was giving evidence alongside WhatDoTheyKnow took a stronger line. He described those who made FOI requests about ghosts as “idiots”; but also accepted it was hard, and undesirable, to try and outlaw requests on certain subjects. He added that such requests did not generally cost large amounts of money to deal with.

Time Limits

MPs on the committee appeared sympathetic to calls from the representatives of WhatDoTheyKnow and the Campaign for Freedom of Information to introduce stricter time limits. The need for time limits was brought into focus during the discussion of the time limits for prosecutions under S.77 of the Act (Offence of altering etc. records with intent to prevent disclosure), very few requests have gone through a response, and internal review, and the Information Commissioner within the time limit for launching a prosecution. An MP suggested making offences under S.77 triable in either a magistrates or a crown court so as to extend the time period while retaining consistency with the rest of the justice system.

Fees

When asked to comment on the idea of introducing fees for all FOI requests Alex said such proposals would be “devastating” and would deter many from making requests. Alex noted that the public had paid for the information in question already, via general taxation, and ought be able to access it.

Exempting Universities

When asked to comment on lobbying from universities to be exempted from FOI, Alex robustly defended their inclusion in the act, pointing to their role in controlling access to professions and awarding degrees. Maurice Frankel and Alex noted the universities’ argument that they were being funded by a decreasing fraction of public money wasn’t really relevant, as that is not the basis on which bodies are deemed to be covered by the Act.

Extending Coverage of FOI

The reach of FOI into commercial organisations carrying out work on behalf of public bodies was briefly discussed however notably there was little further discussion of extending the coverage of FOI, perhaps suggesting this may be a dedicated subject for future evidence session. This session was been described as the committee’s first, suggesting there will be more. At least one of these will presumably hear from the Information Commissioner.

The written evidence we submitted can be read on page 81 of the compendium of submitted evidence (PDF).

When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.

These days dozens of such sites exist around the world. But today sees the launch of a rather-special new transparency site: Mzalendo, covering the Parliament of Kenya.

Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:

• It is entirely responsively designed, so that it works on the simplest of mobile web browsers from day one.
• All the lessons we learned from storing political data wrongly have been baked into this site (i.e we can easily cope with people changing names, parties and jobs)
• Every organisation, position and place in the system is now a proper object in the database. So if you want to see all the politicians who went to Nairobi University, you can.
• There is lots of clear information on how parliament functions, what MPs and committees do, and so on.
• It synthesizes some very complex National Taxpayer’s Association data on missing or wasted money into a really clear ‘scorecard‘, turning large sums of money into numbers of teachers.

The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.

Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.

December 23rd

If you haven’t got a penny,

A ha’penny will do,

If you haven’t got a ha’penny,

Then God bless you.

We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?

mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.

Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.

We hope you’ll continue to follow us on Twitter, Facebook, or Google+ – see our Contacts page to find individual projects’ social media links.

December 22nd

What’s behind the door? A letter to Santa.

Dear Santa,

We think we’ve been pretty good this year. We’ve tried to keep our local neighbourhood clean, help with problems, and aid those in need, so we’re hoping there are a few presents coming our way.

If you can fit them down the chimney, here’s what we’re dreaming of:

More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.

We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.

Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.

Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?

A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.

Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?

FixMyTransport buy-in – from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.

Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.

(more…)

One of the most popular features on TheyWorkForYou is the plain English, non-judgemental vote analyses on MP pages that say things like “voted strongly against introducing a smoking ban“.  We call these ‘policies’, and they are authored by skilled people using the volunteer run PublicWhip website.

Making each one of these policies is a painstaking task, requiring good knowledge of how the Parliamentary voting system works, good writing skills, patience, and the strength of character not to let your own views about the issues cloud the analysis. It is of utmost importance to both mySociety and our users that these policies are fair and trustworthy.

Earlier this year we started to update the process by which we made new policies to make it even more rigorous, which we wrote about here. Marcus Fergusson and Stephen Young came onboard and did sterling work, but they have now moved on to greater things, and so we’re looking to recruit two to three new people to do this job. Uber volunteer Richard Taylor has been helping out recently, but this is really a job for two or more people.

You might very well ask ‘why two people, given the work is part time?’. The answer is that we really want every new policy to be cross-checked by two different people every time it is added or amended. This is to help eliminate possible mistakes, and prevent any unconscious biases.

We pay for this work on a piece work basis – £160 a time for a combination of one new policy authored, and one other policy double-checked. This money comes mainly from people making small donations, which I think helps keep everyone focussed on how important it is to get these right. We hope to add about two new policies a month, once the new team is up to speed.

If you’d like to be considered, please email hello@mysociety.org with ‘mspolicies’ in the subject line. Applications close 22nd November 2010.

We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

• Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
• As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
• From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
• Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
• As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
• Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

A number of non-departmental government bodies / quangos have been named as being up for abolition, merger, privatisation or absorption into parent departments, as part of the Coalition Government’s Spending Review, due this autumn. This has been widely dubbed in the press as a “bonfire of the quangos“.  The list of quangos up for review is still being compiled by the government, and there have been a number of clarifications, amendments and retractions as further details come to light.

The Telegraph has obtained and published today a leaked list of 177 quangos up for abolition, plus a further 200 that are still being reviewed.

This is a great opportunity to highlight that mySociety’s Freedom of Information site WhatDoTheyKnow covers nearly all of these little-known bodies that spend public money (we currently have just over 3,800 public authorities listed on the site).  Given their impending doom, there is little time left to find out what they spent public funds on, as only their most important records will be transferred to the National Archives or successor bodies for permanent storage.  The remainder are likely to be shredded, or deleted, as only “records identified as valuable for future administrative need” are kept.

You can see our annotated list of the Telegraph’s list here – our volunteers have added links to most of the bodies’ pages on WhatDoTheyKnow, so you can more easily make your final FOI requests to them…

Please send any missing contact details to the WhatDoTheyKnow team.

As you may know, TheyWorkForYou are conducting a survey of candidates for Parliament.

Quite a few people have been asking how we worked out the questions. There are two parts to this, one local and one national.

Local questions

We used the power of volunteers.

Thousands of DemocracyClub members were asked to suggest local issues in there area. These were then edited by other volunteers, to have consistent grammar, and be worded as statements to agree/disagree with, and filtered to remove national issues. The full criteria and examples are available.

You can view the issues for any constituency on the DemocracyClub site. They are in the “local questions” tab.

We’ve ended up with local issues for about 85% of constituencies. They’re really interesting and high quality, and quite unique for a national survey.

Thank you to all the volunteers who helped make this happen!

National questions

This was hard, because we felt that asking more than 15 questions would make the survey too long. We also wanted to be sure it was non-partisan.

We convened a panel of judges, either from mySociety/Democracy Club or with professional experience in policy, and from across the political spectrum. They were:

• James Crabtree, chair of judges, trustee of mySociety, journalist for Prospect magazine
• Tim Green, Democracy club developer, Physics student, Cambridge University.
• Michael Hallsworth, senior researcher, Institute for Government.
• Will Davies, sociologist at University of Oxford, has worked for left of centre policy think tanks such as IPPR and Demos.
• Andrew Tucker, researcher at Birkbeck, worked for Liberal Democrats from 1996-2000.
• Robert McIlveen, research fellow, Environment and Energy unit at Policy Exchange, did PhD on Conservative party election strategy.

They met at the offices of the Institute for Government, and had a 3 hour judging session on 29th March 2010. They were asked to think of 8-15 questions, with multiple choice answers, which could usefully be answered both by members of the public and prospective candidates for national office.

To ensure maximum transparency, the discussions of the judges were recorded. You can download the recordings in two parts: part 1, part 2 (2 hours, 20 mins total).

Details of the broad framework the judges operated under are given by the chair of judges, James Crabtree, a trustee of mySociety, in the opening to the recordings.

Please do ask any questions in the comments below.

The following is a message that we’d like to see emailed around within political parties of all stripes. If you work for a party, or know anyone who does, please send it along:

———-

Hi there,

TheyWorkForYou.com has sent online surveys to nearly 3000 candidates across the UK, including most of your party’s candidates. If you don’t know it, TheyWorkForYou is probably the largest politician transparency website in the UK, with about 3m visitors last year.

The survey we’ve sent is a rigorously neutral attempt to clarify candidates positions on many of the biggest issues at the election. It is also a long-term document – the data that comes from candidate responses will be viewed millions of times between now and the general election after this one. It also contains both local and national questions.

There are 6000+ volunteers now nagging non-responsive candidates. You can help your party improve its responsiveness rating, here, by passing on the word that TheyWorkForYou’s survey is not push-polling, not single issue, not short-termist.

Please help us by passing on the message that TheyWorkForYou will be one of the main ways that new MPs from all parties (and none) will be scrutinised and neither we nor new MPs want to start our relationship with a “refused to go on the record” badge on their pages.

If you are a candidate, and you want to do the survey, check your email for TheyWorkForYou (no spaces). If you don’t have it, drop a mail to developers@democracyclub.org.uk and it’ll be sent along shortly.

many thanks,

The staff and volunteers at TheyWorkForYou and Democracy Club

In January last year, at our yearly staff and volunteers retreat, we decided that TheyWorkForYou should do something special for the general election. We decided that we wanted to gather information on where every candidate in every seat stood on what most people would think were the biggest issues, not just nationally but locally too.

Our reasons for setting this ambitious goal were two fold. First, we thought that pinning people down to a survey that didn’t reward rhetorical flourishes would help the electorate cut through the spin that accompanies all elections. But even more important was to increase our ability to hold new MPs to account: we want users of TheyWorkForYou in the future to be able to see how Parliamentary voting records align with campaign statements.

This meant doing quite a lot of quite difficult things:

1. Working out who all the candidates are (thousands of them)
2. Working out how to contact them.
3. Gathering thousands of local issues from every corner of the country, and quality assuring them.
4. Developing a balanced set of national issues.
5. Sending the candidates surveys,  and chasing them up.

The Volunteer Army

This has turned out to be a massive operation, requiring  the creation of the independent Democracy Club set up by the amazing new volunteers Seb Bacon and Tim Green,  and an entire candidate database site YourNextMP, built by another new volunteer Edmund von der Burg.  Eventually we managed to get at least one local issue in over 80% of constituencies, aided by nearly 6000 new volunteers spread from Lands End to John O’Groats. There’s at least one volunteer in every constituency in Great Britain, and in all but three in Northern Ireland. Volunteers have done more than just submit issues, they’ve played our duck house game  to help gather thousands of email addresses, phone numbers, and postal addresses.

The Survey

What we ended up with is a candidate survey that is different for every constituency – 650 different surveys, in short. The survey always contains the same 15 national issues (chosen by a politically balanced panel held at the Institute for Government)  and then anything between zero and ten local issues. We’ve seen everything from cockle protection to subsidies for ferries raised – over 3000 local issues were submitted, before being painstakingly moderated, twice, by uber-volunteers checking for for spelling, grammar, obvious bias and straightforward interestingness (it isn’t really worth asking candidates if they are in favour of Good Things and against Bad Things).

In the last couple of days we’ve started to send out the first surveys – we’ve just passed 1000 emails, and there are at least 2000 still to be sent.

The Output

We’re aiming to release the data we are gathering on candidates positions on 30th April. We’ll build a nice interface to explore it, but we also hope that others will do something with what we are expecting to be quite a valuable dataset.

The Pressure

Candidates are busy people, so how do we get their attention? Happily, some candidates are choosing to answer the survey just because TheyWorkForYou has a well know brand in the political world, but this has limits.

The answer is that we are going to ask Democracy Club, and it’s army of volunteers to help. We’ll shortly roll out a tool that will tell volunteers which of their candidates haven’t taken the opportunity to go on the record , and provide a range of ways for them to push for their candidates to fill it in.

It would be a lie to say we’re confident we’ll get every last candidate. But we are confident we can make sure that no candidate can claim they didn’t see, or didn’t know it was important to their constituents. And every extra voice we have makes that more likely.

Join Democracy Club today

When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.

Peter Collingbourne got fed up with this, and luckily for us, he can code too. He forked our source code repository, and made a nice patch in his own copy of it.

He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.

Now if you go to an Outlook attachment on WhatDoTheyKnow,such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).

Thanks Peter!

If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.