On Saturday John Cross and Richard Taylor, two volunteers who work on mySociety’s freedom of information website WhatDoTheyKnow.com, gave a workshop on FOI to a meeting of activists from Republic, an organisation which campaigns for an elected head of state in the UK.

mySociety and WhatDoTheyKnow are non-partisan and don’t get involved in campaigning except in specific areas relating to openness and transparency. That said, members of the WhatDoTheyKnow team are be happy to consider invitations from any groups wishing to hold a workshop discussing freedom of information.

Many of those present at Saturday’s event were active campaigners on a wide range of subjects ranging from human rights to fair trade as well as having an interest in constitutional reform. The FOI workshop was oversubscribed with the majority of those present at the event deciding to attend the session. Unlike a previous workshop held at OpenTech where most attendees had made an FOI request themselves prior to the event, at this workshop all but one had not done so.

The Royals and FOI
Given the audience, the status of the royals with respect to FOI was particularly pertinent. The FOI act exempts information if it relates to: “communications with Her Majesty, with other members of the Royal Family or with the Royal Household, or the conferring by the Crown of any honour or dignity”. This exemption does not apply though if it is determined that it is in the public interest for the information to be released. The requirement for this public interest test is under threat as the Prime Minister has been moving to strengthen the restrictions on releasing information related to the Royal family. On the 10th of June 2009 in a speech to Parliament on Constitutional Renewal Gordon Brown said:

…we have considered the need to strengthen protection for particularly sensitive material, and there will be protection of royal family and Cabinet papers as part of strictly limited exemptions.

Following that speech BBC journalist Martin Rosenbaum obtained a statement from the Ministry of Justice clarifying that in practice what Gordon Brown’s words meant was:

… the relevant exemption in the Freedom of Information Act will be made absolute for information relating to communications with the Royal Household that is less than 20 years’ old.

In FOI jargon an “absolute exemption” is one not subject to a public interest test.

Even with the law as it stands it is not easy to obtain information on how the royals are, or are attempting to, influence government. For example John Cross has asked the Ministry of Justice to supply him with copies of correspondence they had received from the Queen and Prince of Wales. They rejected his request on the grounds that the public interest in non-disclosure exceeded the public interested in disclosure; as well as suggesting exemptions relating to “information provided in confidence” and “personal information” also applied.

The Royal Household’s position on FOI
The Royal Household is not subject to the freedom of information act; though it has made a statement on the subject saying:

Despite its exemption from the FOI Acts, the Royal Household’s policy is to provide information as freely as possible in other areas, and to account openly for its use of public money.

WhatDoTheyKnow’s policy is to include such organisations which have indicated they are willing to voluntarily comply with the act to the site. While we list The Royal Household, at the time of writing no-one has yet used the facility to request information.

Using WhatDoTheyKnow for Campaigning
While we stress the importance of keeping freedom of information requests focused, FOI is a powerful tool for campaigners. We were asked if it would be possible for a group like Republic to set up an account on WhatDoTheyKnow for their campaign? The answer to this is: “Yes! – WhatDoTheyKnow wants to encourage groups to use the site”. The information commissioner has confirmed that it is acceptable to use the name of a “corporate body” when making a FOI request, that’s a broad term which encompasses many organisations, groups and charities.

Republic themselves use FOI extensively and often generate major national news stories as a result of responses to their requests. They want to be able to either offer journalists exclusive stories or write a press release based on information released. They can’t do this if the story gets out first via WhatDoTheyKnow so would be interested in an ability to make requests initially in private. mySociety and WhatDoTheyKnow have been considering an option for journalists to be able to make hidden requests via the site. Such a feature could potentially generate an income stream for the site as well as encourage a greater proportion of FOI requests to be made via it. Once the article had been published then the FOI correspondence could be opened up to the public providing access to the source material backing up the story.

As well as meeting those who use, or might want to use, the site to make requests WhatDoTheyKnow also wants to engage positively with public authorities; we see them as important users of our service too. Developer Francis Irving represented the site at the FOI Live conference for information professionals in June and will be speaking at the Freedom of Information Scotland conference in December.

Image from jaygoldman

Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites

I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.

Identifying and categorising these types of site turns out to be surprisingly useful.  First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.

Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.

1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.

2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou in NZ, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch,  Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.

3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.

4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.

5. Government spending – where the big money goes. In the US the dominant site is FedSpending.org, and in the UK we have ukpublicspending.co.uk.

6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and  Statute Law Database. Much of the legal database field, however, remains essentially private.

7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet‘s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.

8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places,  the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others.  mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.

9. Political document archives – This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.

10. Bulk data – Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.

——

Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.

Earlier this week someone browsing mySociety’s freedom of information website WhatDoTheyKnow.com contacted us reporting they couldn’t open a PDF document Southampton University had sent in reply to a request asking about the amount of printer credit purchased at the institution. The user suspected the file might be corrupt.

We investigated and determined the attachment is in fact in a fancy PDF format which cannot be read by many PDF readers. Those applications which are able to open the document present a cover page inviting the reader to agree to an “intellectual property rights notice”. The terms of the notice forbid “use” of the “material” without “the written permission of the university”. The intent appears to be to ensure only those agreeing to the terms are given access to the document’s content.

The Freedom of Information Act requires authorities to release information whether or not the person making the request wants to enter into an agreement with respect to its use; so one could argue the university are not properly discharging their responsibilities. In any case their stance is silly because anyone who wants to can request their own copy of the information. All they’re doing is creating more work for themselves.

What were they thinking?
An article John Ozimek of The Register has written in response to the correspondence reports the university explaining their actions by saying:

The Freedom of Information Act gives applicants the right to have information held by public authorities communicated to them, not to documents. Applicants are entitled to use that information so long as they do not breach the intellectual property rights of the public authority: taking the information and using it in a document of their own is acceptable, making use of a document which contains not only the information requested but copyright material is not.

One problem with this was pointedly highlighted by a comment on the Register article saying: Ask yourself, “how do I prove my facts if I can’t publish the evidence?”

WhatDoTheyKnow routinely publishes information released under FOI despite a wide variety of copyright and other legal notices and disclaimers suggesting we shouldn’t. (Read the policy on copyright.) This has been the first relatively elaborate technological attempt to circumvent WhatDoTheyKnow’s efforts to make the responses to FOI requests easily accessible to everyone.

Actions
WhatDoTheyKnow’s staff, volunteers, and users have responded to the university’s actions in a number of ways:

• Those making requests to the university are now warned what it is up to, and are made aware that section 11 of the freedom of information act allows them to specify the format in which they would like to receive their response.
• The original requestors have asked for the information to be released in a more accessible format. (One is citing the disability discrimination act).
• The text of the responses, along with more accessible PDF versions have been placed on WhatDoTheyKnow.
• FOI requests have been made asking the university about advice and discussions relating to the way responses are marked to warn requestors of the university’s rights under copyright and intellectual property law as well as for details of the software used to create the protected files along with the policy decisions made to use it.

They’ve got form
This is not the first time the freedom of information team at Southampton University have been inventive in their use of PDFs; back in May they responded to a request via a password protected PDF version of a compliments slip which they attached to an email.

As a new edition has just been released, and I’ve had to tweak the parser to cope with the new highlighting, it’s a good time to write a brief article on TheyWorkForYou’s handling of the House of Commons Register of Members’ Financial Interests (Register of Members’ Interests as was before the current edition). Way back in the day, a scraper/parser was written (by either Julian or Francis) that monitors the Register pages on www.parliament.uk for new editions, and downloads and broadly parses the HTML into machine-readable data. The XML produced can be found at http://ukparse.kforge.net/parldata/scrapedxml/regmem/ – TheyWorkForYou then pulls in this XML into its database, and makes the latest data available on every MP’s page.

However, as it’s been scraping/parsing the Register since 2000, we can do more than that. Each MP’s page contains a link to a page giving the history of their entry in the Register – when things were added, removed, or changed. You can also view the differences between one edition of the Register and the next, or view a particular edition in a prettier form than the official site. There’s a central page containing everything Register-related at http://www.theyworkforyou.com/regmem/

As the decisive voting day dawns, mySociety has eight full or partial endorsements of our 3 Principles from possible candidates for Speaker of the Commons. We hope you take a look at what they said in full on their TheyWorkForYou pages. Just five possible candidates didn’t reply in writing, or at all – those absentees being Patrick Cormack, Sylvia Heal, Margaret Beckett, Parmijit Dhanda and Ann Widdecombe (who did phone, but doesn’t seem to have followed up with email). Interestingly, the five non-respondants included both candidates whose offices don’t accept email (Beckett and Cormack).

The last time MPs voted for a speaker, the one candidate who didn’t show at the hustings went on to win. Let’s hope MPs learned the lesson of voting for a candidate who isn’t willing to stand up and be counted when it comes to the issues that make their job so critical…

‘Thumbs up’ by Carf (cc)

A few days ago mySociety asked the known possible candidates for Speaker to endorse 3 principles relating to making Parliament more transparent on the Internet.

We’ve now had endorsements which you can read on the individual pages of Sir George Young, Sir Menzies Campbell, Frank Field, Tony Wright and Sir Alan Beith , which until Parmijit Dhanda declared this morning, represented endorsement by 50% of the possible field. We also just recieved a typically frank and interrogative phonecall from Ann Widdecombe, who will be writing a formal response soon.

So, come on, John Bercow, Alan Haselhurst, Patrick Cormack, Sylvia Heal, and Chris Mullin. What’s holding up your replies? The days counter on your pages is telling the world how quick you are to respond…

Update: 11 June – John Bercow has now endorsed, and we’ve written to Margaret Beckett and Parmjit Dhanda, who’ve just declared their candidacies.

Update 2: Chris Mullin has told us he is ‘not a candidate’.

Update 3: Sir Alan Haselhurst has also endorsed.

Update 4 – Speaker Election Day: And Sir Michael Lord endorses too.

In less than 24 hours we’ve seen the first reply to our emails asking possible Speaker candidates to endorse our three principles. It is from Sir George Young – we’re looking forward to seeing the responses from the others that we wrote to.

Sir George is broadly supportive, which is great, and we’ve printed his reply in full on his own TheyWorkForYou MP page.

In the mean time, please do write to your MP and ask them to ensure that whoever they vote for, it is a candidate who has endorsed mySociety’s three simple principles, You really can have an impact on this issue: MPs are desperate to be seen to be acting for their constituents right now.

NB mySociety is strictly non-partisan and non-party aligned. We want all candidates from all parties to endorse these principles, and we have ensured that none of the wording of the principles leans towards any particular party or set of beliefs not connected to transparency in the modern age.

Speaker’s Chair (Parliamentary Copyright)

mySociety has today emailed (and in one case, posted) a set of 3 Principles which we believe it is important that all candidates for Speaker endorse, before the election of a new Speaker by MPs.

1. Voters have the right to know in detail about the money that is spent to support MPs and run Parliament, and in similar detail how the decisions to spend that money are settled upon.

2. Bills being considered must be published online in a much better way than they are now, as the Free Our Bills campaign has been suggesting for some time.

3. The Internet is not a threat to a renewal in our democracy, it is one of its best hopes. Parliament should appoint a senior officer with direct working experience of the power of the Internet who reports directly to the Speaker, and who will help Parliament adapt to a new era of transparency and effectiveness.

We will be posting the status of requests on the likely candidates web pages where we expect large numbers of people to see them before the vote in late June. We have also taken the unusual step of allowing possible candidates to leave a statement of up to 150 words on the principles.

(NB no candidates have actually declared at this stage, so we are starting with the BBC’s list of possibles)

Rationale

mySociety helped lead the campaign back in January to prevent the last ditch attempts to conceal MPs’ expenses. We did so not because, like the newspapers, we wanted to revel in embarrassment and scandal, but because we believe that in the Internet age, the only way for our democracy and government to thrive is if they are open and connected to the net as the rest of us expect them to be. The dramatic events seen in Parliament in recent days vindicate the view that secrecy breeds poor policies and seeds untrustworthy behaviour in the weaker willed.

Furthermore, more than a simple attitude of openness is required of the new Speaker: the public needs a genuine will to push for technological reform using the power of the Internet that will take both open-mindedness and a willing to tread on toes, especially in some parts of the unelected establishment.

Case in point: Over the last two years we have been trying to persuade Parliament to acknowledge that the way it publishes its Bills online is hopelessly inadequate for the Internet age. The campaign has faltered, despite multi partly endorsement from 140 MPs and a campaign membership of thousands. To see why, just take a look at this colourful and error-crammed internal email that we uncovered using the Freedom of Information Act, published for the first time today.

The new Speaker will have a tough job on their hands to overcome resistance of this kind. The best thing we can do is help the new Speaker, whoever they are, assume their new job with a clear mandate from the public, as well as from members.

Act!

That is why, as a final part of this call, we are asking you, our community, to write to your MP today to let them know that you expect them to vote for a candidate that has endorsed the principles above. Your voices to your own constituency MPs can resonate in a way that no blog post or newspaper article ever can. Go to it.

You may have seen a few months ago that mySociety led the campaign to stop the Freedom of Information Act being changed to conceal MPs expenses. And we won, which was nice.

Given the wall-to-wall revelations about taxpayer funded moats and bathplugs, and the new wave of  resignations and repayments,  we want to exercise a little accountability by reminding readers of the arguments that were used to conceal this information twice in the last two years. These helpful examples should assist mySociety’s friends in keeping an eye out for similar dubious logic in the future.

First, in 2007, a concealment bill was tabled by a backbencher, but which oddly made it all the way to the Lords before failing (it would normally have been struck down by the government). The argument used then was that private mail sent by constituents to MPs would end up in the hands of unscrupulous characters, even though there was already another law to prevent this, and even though hardly anyone appears to have complained to the Information Commissioner about what the proposing MP described as a ‘vexed problem‘.

Then, back in January this year, a different and bold explanation was given: none. Instead, a strange pretence was played out in Parliament, in which the fact that MPs were being given the opportunity to vote to overrule a court-mandated order to publish was simply not mentioned: Watch the video of the Leader explaining what’s going to be voted for – any idea why people might be laughing as she stands up? Can you spot where she explains why the court needs overruling? The only explanation I could find anywhere for this reversal of openness was an anonymous quote in the Guardian.

So here we are after both attempts to hide expenses were defeated, watching as the rules around expenses change substantially and as MPs reach deep into their own pockets: all things that would not have taken place if either of the above proposals had passed. Simply the fact that the rules are changing and that the leaders of parties are apologising must make it clear that the excuses and non-excuses given above were, even if unintentionally, blocking better government.

Despite the obvious pain for MPs this week, and the fact that the whole act is doubtless being cursed across Westminster, we must shout from the hilltops that this week is a great success for the Freedom of Information Act, and a clear justification for why it is worth having on our books.

Bad policies that both wasted money and eroded public trust are being swept away, and it is entirely down to the  Act and its supporters. More Freedom of Information will mean more such improvements, and people of good will should support its defence and its extension.

Note: When can the rest of us have the data, please?

UPDATE: If you approve of what we did this week, and what to help make sure we can still do it in the future, please pledge to support us: http://www.pledgebank.com/supportmysociety

The vote on concealing MPs’ expenses has been cancelled by the government!

In other words – we won!

This is a huge victory not just for transparency, it’s a bellwether for a change in the way politics works. There’s no such thing as a good day to bury bad news any more, the Internet has seen to that.

Over 7000 people joined a Facebook group, they sent thousands of emails to over 90% of all MPs. Hundreds of thousands of people found out about the story by visiting TheyWorkForYou to find something they wanted to know, reading an email alert, or simply discovered what was going on whilst checking their Facebook or Twitter pages. Almost all of this happened, from nowhere, within 48 hours, putting enough pressure on Parliament to force change.

Make no mistake. This is new, and it reflects the fact that the Internet generation expects information to be made available, and they expect to be able to make up their own minds, not be spoon fed the views of others. This campaign was always about more than receipts, it was about changing the direction of travel, away from secrecy and towards openness.

Today we stopped moving in the wrong direction. Tomorrow we start moving the right way. Sign up to our news mailing list (box on the right) to get updates on what mySociety gets up to.