In January last year, at our yearly staff and volunteers retreat, we decided that TheyWorkForYou should do something special for the general election. We decided that we wanted to gather information on where every candidate in every seat stood on what most people would think were the biggest issues, not just nationally but locally too.

Our reasons for setting this ambitious goal were two fold. First, we thought that pinning people down to a survey that didn’t reward rhetorical flourishes would help the electorate cut through the spin that accompanies all elections. But even more important was to increase our ability to hold new MPs to account: we want users of TheyWorkForYou in the future to be able to see how Parliamentary voting records align with campaign statements.

This meant doing quite a lot of quite difficult things:

1. Working out who all the candidates are (thousands of them)
2. Working out how to contact them.
3. Gathering thousands of local issues from every corner of the country, and quality assuring them.
4. Developing a balanced set of national issues.
5. Sending the candidates surveys,  and chasing them up.

The Volunteer Army

This has turned out to be a massive operation, requiring  the creation of the independent Democracy Club set up by the amazing new volunteers Seb Bacon and Tim Green,  and an entire candidate database site YourNextMP, built by another new volunteer Edmund von der Burg.  Eventually we managed to get at least one local issue in over 80% of constituencies, aided by nearly 6000 new volunteers spread from Lands End to John O’Groats. There’s at least one volunteer in every constituency in Great Britain, and in all but three in Northern Ireland. Volunteers have done more than just submit issues, they’ve played our duck house game  to help gather thousands of email addresses, phone numbers, and postal addresses.

The Survey

What we ended up with is a candidate survey that is different for every constituency – 650 different surveys, in short. The survey always contains the same 15 national issues (chosen by a politically balanced panel held at the Institute for Government)  and then anything between zero and ten local issues. We’ve seen everything from cockle protection to subsidies for ferries raised – over 3000 local issues were submitted, before being painstakingly moderated, twice, by uber-volunteers checking for for spelling, grammar, obvious bias and straightforward interestingness (it isn’t really worth asking candidates if they are in favour of Good Things and against Bad Things).

In the last couple of days we’ve started to send out the first surveys – we’ve just passed 1000 emails, and there are at least 2000 still to be sent.

The Output

We’re aiming to release the data we are gathering on candidates positions on 30th April. We’ll build a nice interface to explore it, but we also hope that others will do something with what we are expecting to be quite a valuable dataset.

The Pressure

Candidates are busy people, so how do we get their attention? Happily, some candidates are choosing to answer the survey just because TheyWorkForYou has a well know brand in the political world, but this has limits.

The answer is that we are going to ask Democracy Club, and it’s army of volunteers to help. We’ll shortly roll out a tool that will tell volunteers which of their candidates haven’t taken the opportunity to go on the record , and provide a range of ways for them to push for their candidates to fill it in.

It would be a lie to say we’re confident we’ll get every last candidate. But we are confident we can make sure that no candidate can claim they didn’t see, or didn’t know it was important to their constituents. And every extra voice we have makes that more likely.

Join Democracy Club today

We know from our inboxes that there are people all over the world who would love to start sites like TheyWorkForYou.com, FixMyStreet.com, or WhatDoTheyKnow.com in their own countries. Building and running these sites is hard, though, and takes time, money, and love. Until now we haven’t been able to do much for these keen correspondents beyond sharing our ideas, sharing our code, and wishing people the very best of luck. We’re happy to say that for at least some of these people, things are about to change for the better.

derived from http://commons.wikimedia.org/wiki/File:Eastern-Europe-map2.svg

If you live in Central or Eastern Europe, we’re now in a position to help you get effective democracy and transparency websites built. mySociety have teamed up with the Open Society Institute (OSI) and together we are now looking for determined people with great ideas for new digital transparency and accountability services in their countries.

Over the next few months we are running a Call for Proposals, similar to the one we recently ran in the UK. The big difference is that this time we’re not looking for projects that we will build. We’re looking for projects you want to build, but that for lack of funds or lack of the right skills, you can’t get started yourself.

Each month the Open Society Institute and mySociety will work closely together to select a series of projects to fund and mentor. Crucially, the call isn’t solely for existing NGOs: the process is absolutely open to submissions from individuals or groups with no prior direct experience of working in the transparency and accountability sector, but who have a good idea that addresses a problem they see in their country. We will, however, look more favourably on applicants with access to the advanced programming skills required to build sites like this.

The criteria are simple, though demanding:

1. The projects have to generate some kind of meaningful transparency, accountability, or democratic empowerment of another kind.
2. The projects must seize the unique benefits that the Internet brings with it, such as scalability, two way communication, easy data analysis and so on.

More details are available over at our new CEE site, but even if you don’t live in one of the eligible countries please help us spread the word about this exciting new opportunity!

On Saturday John Cross and Richard Taylor, two volunteers who work on mySociety’s freedom of information website WhatDoTheyKnow.com, gave a workshop on FOI to a meeting of activists from Republic, an organisation which campaigns for an elected head of state in the UK.

mySociety and WhatDoTheyKnow are non-partisan and don’t get involved in campaigning except in specific areas relating to openness and transparency. That said, members of the WhatDoTheyKnow team are be happy to consider invitations from any groups wishing to hold a workshop discussing freedom of information.

Many of those present at Saturday’s event were active campaigners on a wide range of subjects ranging from human rights to fair trade as well as having an interest in constitutional reform. The FOI workshop was oversubscribed with the majority of those present at the event deciding to attend the session. Unlike a previous workshop held at OpenTech where most attendees had made an FOI request themselves prior to the event, at this workshop all but one had not done so.

The Royals and FOI
Given the audience, the status of the royals with respect to FOI was particularly pertinent. The FOI act exempts information if it relates to: “communications with Her Majesty, with other members of the Royal Family or with the Royal Household, or the conferring by the Crown of any honour or dignity”. This exemption does not apply though if it is determined that it is in the public interest for the information to be released. The requirement for this public interest test is under threat as the Prime Minister has been moving to strengthen the restrictions on releasing information related to the Royal family. On the 10th of June 2009 in a speech to Parliament on Constitutional Renewal Gordon Brown said:

…we have considered the need to strengthen protection for particularly sensitive material, and there will be protection of royal family and Cabinet papers as part of strictly limited exemptions.

Following that speech BBC journalist Martin Rosenbaum obtained a statement from the Ministry of Justice clarifying that in practice what Gordon Brown’s words meant was:

… the relevant exemption in the Freedom of Information Act will be made absolute for information relating to communications with the Royal Household that is less than 20 years’ old.

In FOI jargon an “absolute exemption” is one not subject to a public interest test.

Even with the law as it stands it is not easy to obtain information on how the royals are, or are attempting to, influence government. For example John Cross has asked the Ministry of Justice to supply him with copies of correspondence they had received from the Queen and Prince of Wales. They rejected his request on the grounds that the public interest in non-disclosure exceeded the public interested in disclosure; as well as suggesting exemptions relating to “information provided in confidence” and “personal information” also applied.

The Royal Household’s position on FOI
The Royal Household is not subject to the freedom of information act; though it has made a statement on the subject saying:

Despite its exemption from the FOI Acts, the Royal Household’s policy is to provide information as freely as possible in other areas, and to account openly for its use of public money.

WhatDoTheyKnow’s policy is to include such organisations which have indicated they are willing to voluntarily comply with the act to the site. While we list The Royal Household, at the time of writing no-one has yet used the facility to request information.

Using WhatDoTheyKnow for Campaigning
While we stress the importance of keeping freedom of information requests focused, FOI is a powerful tool for campaigners. We were asked if it would be possible for a group like Republic to set up an account on WhatDoTheyKnow for their campaign? The answer to this is: “Yes! – WhatDoTheyKnow wants to encourage groups to use the site”. The information commissioner has confirmed that it is acceptable to use the name of a “corporate body” when making a FOI request, that’s a broad term which encompasses many organisations, groups and charities.

Republic themselves use FOI extensively and often generate major national news stories as a result of responses to their requests. They want to be able to either offer journalists exclusive stories or write a press release based on information released. They can’t do this if the story gets out first via WhatDoTheyKnow so would be interested in an ability to make requests initially in private. mySociety and WhatDoTheyKnow have been considering an option for journalists to be able to make hidden requests via the site. Such a feature could potentially generate an income stream for the site as well as encourage a greater proportion of FOI requests to be made via it. Once the article had been published then the FOI correspondence could be opened up to the public providing access to the source material backing up the story.

As well as meeting those who use, or might want to use, the site to make requests WhatDoTheyKnow also wants to engage positively with public authorities; we see them as important users of our service too. Developer Francis Irving represented the site at the FOI Live conference for information professionals in June and will be speaking at the Freedom of Information Scotland conference in December.

Image from jaygoldman

Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites

I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.

Identifying and categorising these types of site turns out to be surprisingly useful.  First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.

Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.

1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.

2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou in NZ, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch,  Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.

3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.

4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.

5. Government spending – where the big money goes. In the US the dominant site is FedSpending.org, and in the UK we have ukpublicspending.co.uk.

6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and  Statute Law Database. Much of the legal database field, however, remains essentially private.

7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet‘s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.

8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places,  the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others.  mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.

9. Political document archives – This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.

10. Bulk data – Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.

——

Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.

Earlier this week someone browsing mySociety’s freedom of information website WhatDoTheyKnow.com contacted us reporting they couldn’t open a PDF document Southampton University had sent in reply to a request asking about the amount of printer credit purchased at the institution. The user suspected the file might be corrupt.

We investigated and determined the attachment is in fact in a fancy PDF format which cannot be read by many PDF readers. Those applications which are able to open the document present a cover page inviting the reader to agree to an “intellectual property rights notice”. The terms of the notice forbid “use” of the “material” without “the written permission of the university”. The intent appears to be to ensure only those agreeing to the terms are given access to the document’s content.

The Freedom of Information Act requires authorities to release information whether or not the person making the request wants to enter into an agreement with respect to its use; so one could argue the university are not properly discharging their responsibilities. In any case their stance is silly because anyone who wants to can request their own copy of the information. All they’re doing is creating more work for themselves.

What were they thinking?
An article John Ozimek of The Register has written in response to the correspondence reports the university explaining their actions by saying:

The Freedom of Information Act gives applicants the right to have information held by public authorities communicated to them, not to documents. Applicants are entitled to use that information so long as they do not breach the intellectual property rights of the public authority: taking the information and using it in a document of their own is acceptable, making use of a document which contains not only the information requested but copyright material is not.

One problem with this was pointedly highlighted by a comment on the Register article saying: Ask yourself, “how do I prove my facts if I can’t publish the evidence?”

WhatDoTheyKnow routinely publishes information released under FOI despite a wide variety of copyright and other legal notices and disclaimers suggesting we shouldn’t. (Read the policy on copyright.) This has been the first relatively elaborate technological attempt to circumvent WhatDoTheyKnow’s efforts to make the responses to FOI requests easily accessible to everyone.

Actions
WhatDoTheyKnow’s staff, volunteers, and users have responded to the university’s actions in a number of ways:

• Those making requests to the university are now warned what it is up to, and are made aware that section 11 of the freedom of information act allows them to specify the format in which they would like to receive their response.
• The original requestors have asked for the information to be released in a more accessible format. (One is citing the disability discrimination act).
• The text of the responses, along with more accessible PDF versions have been placed on WhatDoTheyKnow.
• FOI requests have been made asking the university about advice and discussions relating to the way responses are marked to warn requestors of the university’s rights under copyright and intellectual property law as well as for details of the software used to create the protected files along with the policy decisions made to use it.

They’ve got form
This is not the first time the freedom of information team at Southampton University have been inventive in their use of PDFs; back in May they responded to a request via a password protected PDF version of a compliments slip which they attached to an email.

As a new edition has just been released, and I’ve had to tweak the parser to cope with the new highlighting, it’s a good time to write a brief article on TheyWorkForYou’s handling of the House of Commons Register of Members’ Financial Interests (Register of Members’ Interests as was before the current edition). Way back in the day, a scraper/parser was written (by either Julian or Francis) that monitors the Register pages on www.parliament.uk for new editions, and downloads and broadly parses the HTML into machine-readable data. The XML produced can be found at http://ukparse.kforge.net/parldata/scrapedxml/regmem/ – TheyWorkForYou then pulls in this XML into its database, and makes the latest data available on every MP’s page.

However, as it’s been scraping/parsing the Register since 2000, we can do more than that. Each MP’s page contains a link to a page giving the history of their entry in the Register – when things were added, removed, or changed. You can also view the differences between one edition of the Register and the next, or view a particular edition in a prettier form than the official site. There’s a central page containing everything Register-related at http://www.theyworkforyou.com/regmem/

As the decisive voting day dawns, mySociety has eight full or partial endorsements of our 3 Principles from possible candidates for Speaker of the Commons. We hope you take a look at what they said in full on their TheyWorkForYou pages. Just five possible candidates didn’t reply in writing, or at all – those absentees being Patrick Cormack, Sylvia Heal, Margaret Beckett, Parmijit Dhanda and Ann Widdecombe (who did phone, but doesn’t seem to have followed up with email). Interestingly, the five non-respondants included both candidates whose offices don’t accept email (Beckett and Cormack).

The last time MPs voted for a speaker, the one candidate who didn’t show at the hustings went on to win. Let’s hope MPs learned the lesson of voting for a candidate who isn’t willing to stand up and be counted when it comes to the issues that make their job so critical…

‘Thumbs up’ by Carf (cc)

A few days ago mySociety asked the known possible candidates for Speaker to endorse 3 principles relating to making Parliament more transparent on the Internet.

We’ve now had endorsements which you can read on the individual pages of Sir George Young, Sir Menzies Campbell, Frank Field, Tony Wright and Sir Alan Beith , which until Parmijit Dhanda declared this morning, represented endorsement by 50% of the possible field. We also just recieved a typically frank and interrogative phonecall from Ann Widdecombe, who will be writing a formal response soon.

So, come on, John Bercow, Alan Haselhurst, Patrick Cormack, Sylvia Heal, and Chris Mullin. What’s holding up your replies? The days counter on your pages is telling the world how quick you are to respond…

Update: 11 June – John Bercow has now endorsed, and we’ve written to Margaret Beckett and Parmjit Dhanda, who’ve just declared their candidacies.

Update 2: Chris Mullin has told us he is ‘not a candidate’.

Update 3: Sir Alan Haselhurst has also endorsed.

Update 4 – Speaker Election Day: And Sir Michael Lord endorses too.

In less than 24 hours we’ve seen the first reply to our emails asking possible Speaker candidates to endorse our three principles. It is from Sir George Young – we’re looking forward to seeing the responses from the others that we wrote to.

Sir George is broadly supportive, which is great, and we’ve printed his reply in full on his own TheyWorkForYou MP page.

In the mean time, please do write to your MP and ask them to ensure that whoever they vote for, it is a candidate who has endorsed mySociety’s three simple principles, You really can have an impact on this issue: MPs are desperate to be seen to be acting for their constituents right now.

NB mySociety is strictly non-partisan and non-party aligned. We want all candidates from all parties to endorse these principles, and we have ensured that none of the wording of the principles leans towards any particular party or set of beliefs not connected to transparency in the modern age.

Speaker’s Chair (Parliamentary Copyright)

mySociety has today emailed (and in one case, posted) a set of 3 Principles which we believe it is important that all candidates for Speaker endorse, before the election of a new Speaker by MPs.

1. Voters have the right to know in detail about the money that is spent to support MPs and run Parliament, and in similar detail how the decisions to spend that money are settled upon.

2. Bills being considered must be published online in a much better way than they are now, as the Free Our Bills campaign has been suggesting for some time.

3. The Internet is not a threat to a renewal in our democracy, it is one of its best hopes. Parliament should appoint a senior officer with direct working experience of the power of the Internet who reports directly to the Speaker, and who will help Parliament adapt to a new era of transparency and effectiveness.

We will be posting the status of requests on the likely candidates web pages where we expect large numbers of people to see them before the vote in late June. We have also taken the unusual step of allowing possible candidates to leave a statement of up to 150 words on the principles.

(NB no candidates have actually declared at this stage, so we are starting with the BBC’s list of possibles)

Rationale

mySociety helped lead the campaign back in January to prevent the last ditch attempts to conceal MPs’ expenses. We did so not because, like the newspapers, we wanted to revel in embarrassment and scandal, but because we believe that in the Internet age, the only way for our democracy and government to thrive is if they are open and connected to the net as the rest of us expect them to be. The dramatic events seen in Parliament in recent days vindicate the view that secrecy breeds poor policies and seeds untrustworthy behaviour in the weaker willed.

Furthermore, more than a simple attitude of openness is required of the new Speaker: the public needs a genuine will to push for technological reform using the power of the Internet that will take both open-mindedness and a willing to tread on toes, especially in some parts of the unelected establishment.

Case in point: Over the last two years we have been trying to persuade Parliament to acknowledge that the way it publishes its Bills online is hopelessly inadequate for the Internet age. The campaign has faltered, despite multi partly endorsement from 140 MPs and a campaign membership of thousands. To see why, just take a look at this colourful and error-crammed internal email that we uncovered using the Freedom of Information Act, published for the first time today.

The new Speaker will have a tough job on their hands to overcome resistance of this kind. The best thing we can do is help the new Speaker, whoever they are, assume their new job with a clear mandate from the public, as well as from members.

Act!

That is why, as a final part of this call, we are asking you, our community, to write to your MP today to let them know that you expect them to vote for a candidate that has endorsed the principles above. Your voices to your own constituency MPs can resonate in a way that no blog post or newspaper article ever can. Go to it.