WhatDoTheyKnow

WhatDoTheyKnow has been built to help you get information out of government departments and agencies. It’s your right under the Freedom of Information Act to request information from any public body – and they have to respond. So we built a site to make it easy to do so.

Just visit, pick a department, type a request, and we’ll handle the rest.

Even better, WhatDoTheyKnow is an archive of requests and responses made by other people, so you can search for information other people have found, or even set up email or RSS alerts to get notified when something comes in that you’re interested in.

Want to keep up with our latest news? You can read recent blog posts about WhatDoTheyKnow below, follow us on Twitter or like us on Facebook.

MPs are about to review the first five years of the operation of the Freedom of Information Act 2000. We’d like to encourage users of mySociety’s Freedom of Information website WhatDoTheyKnow.com to share their views and experiences with the MPs who are to carry out the review.

The review is being conducted by the House of Commons’ Justice Select Committee.

The committee is currently inviting people to make submissions to it. The deadline for submissions is Friday 3 February 2012.

A memorandum from the Ministry of Justice has been prepared to brief the committee, that document notes, in paragraph 67:

Very little research has been published detailing the views of requesters of information.

Particularly in-light of this we thought it would be worthwhile alerting our users to this review; if we could encourage our users to make submissions to the committee that might help ensure they receive balanced evidence: from outside, as well as within, the public sector.

While the committee is interested in any comments on the act’s operation, specific questions the committee has asked for comment on are:

• Does the Freedom of Information Act work effectively?
• What are the strengths and weaknesses of the Freedom of Information Act?
• Is the Freedom of Information Act operating in the way that it was intended to?

Responses can be emailed to: justicecommemo@parliament.uk

Details of how responses should be formatted and technical details relating to submission are available on the webpage announcing the call for submissions.

Some time in the middle of last night, our Freedom of Information site WhatDoTheyKnow.com was used to send its 100,000th FOI request. It was a simple one, made to the Queen Victoria Hospital NHS Foundation Trust.

WhatDoTheyKnow was launched in February 2008, with these aims: to make it easy to file a FOI request, and to keep a public archive of the requests and (more importantly) the responses received from public bodies. The Freedom of Information Act had been in force since 2005, but we wanted to make it fully accessible to people who were not journalists, lobbyists or professional operatives – it is a law that gives us all a right, not just those experts.

At base, mySociety is about giving people power to people who don’t believe that they have any way of affecting the world around them. Giving practical access to the right enshrined in this Act was and is a meaningful way of advancing that goal.

Then, thanks to a flash of inspiration from our late colleague Chris, we saw a great opportunity to increase the value created by the existence of the Act: we built a system that published the entire exchange of messages between users and public bodies online.

We believe that because of this decision to publish all exchanges with public bodies, WhatDoTheyKnow represents a very unusual phenomenon: a third-party web site that takes an existing piece of legislation and makes it better value for money for the taxpayer. Public money was already being spent answering FOI, but by running WhatDoTheyKnow we could magnify the value generated by each request by making it public, without requiring anyone who worked in a public sector to retrain, buy a new computer system or spend any new money.

And this theory turned out to be right. For every request made on the site, around twenty people come to read materials contained on WhatDoTheyKnow. The multiplier is remarkable, and one of the things that we think is most worth celebrating about this site.

WhatDoTheyKnow’s success is only possible because of a team of fantastically dedicated volunteers. These loyal enthusiasts have helped countless users, and do a simply amazing amount of maintenance work to keep the site friendly, helpful and effective. They are astonishingly talented, principled and knowledgeable, and mySociety owes them a debt of gratitude it will never really be able to pay back.

However, to give them a bit of the credit they deserve, and to highlight some of the countless uses of WhatDoTheyknow, we asked them to pick out some notable requests from the last four years.

Helen “The use of the site by campaign groups like the Campaign for Better Transport to find out about bus subsidy cuts as part of their save our buses campaign.”

John “There was the accidental release of how tax is applied to the Royal Family – which resulted in a Daily Mail front page story.”

Alex “This request about the Warmfront boiler installation scheme has a significant number of annotations. What makes it different is that the user patiently persisted with her original FOI requests, and then has carried on by continuing to help loads more people with details of how to complain and lobby for help and general warm encouragement.”

WhatDoTheyKnow is one of mySociety’s most visited sites, with one and a half million unique visitors in 2011. Like our other projects, it was built as an open source project. Thanks to the Open Society Foundation, we are in the process of making it much easier to re-deploy around the world, under the brand name ‘Alaveteli’.  As we speak, there are sites based on our code in places as far apart as New Zealand, Kosovo, Brazil, and the EU, and we’re looking forward to helping people from around the world create more grandchild sites in the years ahead.

Our 100,000 request milestone comes at an interesting time for the Freedom of Information Act. It’s currently under scrutiny by the Justice Select Committee, who are investigating whether it works effectively and in the way that it was intended.

As you might expect, at mySociety, we’re passionate about the right to information. We’ll be submitting evidence to the Justice Select Committee to show just how vital FOI is to good government and a good society. If FOI has touched your life, you might want to do the same.

December 23rd

Santa's Chocolate Coin Mint by Johnathan_W

If you haven’t got a penny,

A ha’penny will do,

If you haven’t got a ha’penny,

Then God bless you.

We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?

mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.

Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.

We hope you’ll continue to follow us on Twitter, Facebook, or Google+ – see our Contacts page to find individual projects’ social media links.

December 22nd

What’s behind the door? A letter to Santa.

Dear Santa,

We think we’ve been pretty good this year. We’ve tried to keep our local neighbourhood clean, help with problems, and aid those in need, so we’re hoping there are a few presents coming our way.

If you can fit them down the chimney, here’s what we’re dreaming of:

More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.

We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.

Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.

Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?

A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.

Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?

FixMyTransport buy-in - from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.

Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.

December 21st

What’s behind the window? 10 red-faced novice joggers.

It’s not long now until you’ll be making your new year’s resolutions. But will motivation drop off by February? Time to acquaint yourself with one of mySociety’s clever little projects: Hassleme.

Hassleme sends you reminders to do whatever it is you want to do, whether that’s to go for a run, tell someone you love them, or write another chapter of your blockbuster novel. Think of it as benign nagging.

Yes, you could set up your Google calendar to do just the same, but here’s the clever bit – Hassleme sends reminders at “semi-unpredictable intervals” . You can set a rough time period, such as every three days or every year – but you’ll never know precisely when that reminder will drop into your inbox.

You can even make a joint resolution, as a family, perhaps, or even in the office. Input multiple email addresses and we’ll randomise who gets each reminder – ideal for allocating tasks fairly.

Or use it to send a message to yourself ten years hence. Here are some examples from people who have done just that.

December 20th

What’s behind the door? Santa’s little helpers.

mySociety runs some pretty ambitious projects. There’s TheyWorkForYou, which publishes all parliamentary activity since 1935, as well as representatives’ voting records. Then there’s WhatDoTheyKnow, which has sent, and archived, over 30,000 freedom of information requests.

FixMyStreet maps all of Great Britain and sends your reports to the correct council contact. And now we also run FixMyTransport, with its details of over 300,000 public transport routes and stops.

None of these projects runs itself. mySociety’s core team only consists of a few people, so we rely on dedicated volunteers to help us manage the day-to-day maintenance of these sites. Our volunteers have been key to forging a community around each site, and to helping us understand exactly what we want the sites to be.

For example, our FixMyTransport volunteers (aka Anoraks) spend a lot of time leaving helpful comments on users’ problems, often before the operators can get around to answering themselves. Leading by example, they’re making FixMyTransport into a friendly and useful community, encouraging other users to make very constructive contributions, too.

The TheyWorkForYou volunteer team spent quite a bit of time analysing voting records earlier this year, allowing us to add more policy lines to each MP’s page, and providing a snapshot of their affiliations.

And, although WhatDoTheyKnow has been around for three years, the team still find themselves actively debating site policy.

We’re always delighted to welcome new volunteers. If you’re interested, drop us a line at hello@mysociety.org, or come along to one of our pub-meets. There’s one tomorrow! See the Dec 16th advent calendar entry, below, or watch this blog for details of the next one.

December 19th

What’s behind the door? A little donkey.

If you’re using public transport this Christmas, make sure you pack all the essentials: good food, presents – and the web address for FixMyTransport.com.

We hope you have a smooth journey, but if not FixMyTransport will allow you to report overcrowding, delays, or freezing cold carriages – and all on-the-go, if you have a smartphone.

Christmas is for giving, so share that URL with family, friends, and even your fellow passengers, should you find yourself in a coach or train that’s going nowhere. The power to contact the nation’s transport operators directly may just be the greatest gift you’ll ever give.

Well, ok, maybe that’s putting it a bit strongly, but when we see new bus stops being installed, new ticket machines, and longer trains being commissioned, we do start to hear angels sing.

Start your report here, or click on issues near you to see what’s irking passengers in your area. Transport all running smoothly? Lucky you – but the recent issues page is always an interesting read.

December 16th

What’s behind the door? A cup of good cheer.

Our last pub-meet of the year will be the usual chance to come and have a chat with the mySociety team and volunteers. Reindeer antlers and Santa hats are optional, but welcome. Mulled wine may be in evidence. Mince pies could well be found on the premises.

If you’ve been wanting to ask more about any of our projects, to find out about volunteering – or if you would just like a chat and a drink with friendly people – please do pop by.

When? This Wednesday, the 21st of December, from about 6pm and into the evening.

Where? We’ll be at the Prince Arthur, near Euston station in London (map). One or more of us will be wearing a mySociety hooded top, to help you identify us.

One of our New Year’s resolutions is to have meet-ups in places other than London, so if you live outside the capital, watch this space.

Spread the word Because we’re one of those new-fangled digital-type organisations, we encourage use of a hashtag: #mySocial. And you can let us know you’re coming by dropping us a tweet on @mySociety.

December 15th

What’s behind the door? A half-dead Christmas tree.

Christmas comes but once a year… and in its wake, the inevitable slew of dumped Christmas trees and uncollected bins.

In Swindon last year, household bins weren’t collected for three weeks. In Canterbury, a puzzled American tourist mistook the dead fir trees on every street for some kind of crazy English tradition.

Perhaps worse (certainly when it comes to timing), Midnight Mass was made considerably less pleasant for this church-goer in Appledore when he stepped in some dog poop.

We know councils are doing their best to clear things up in the new year, up and down the country – but if those browning Christmas trees, overflowing bins and bottle-littered streets are getting you down, don’t forget FixMyStreet.com.

December 14th

What’s behind the door? A steaming Christmas pudding.

TheyWorkForYou.com keeps a complete record of parliamentary business as far back as 1935. So not only does it  help  you stay up to date with the latest business in Parliament, it also acts as a fascinating, searchable archive.

Consider, for example:

• Why was an American actress refused permission to act in the pantomime Mother Goose? (More details in this 1936 newspaper)
• Just two months after WW2 broke out, which German goods were found on British shelves?
• How many Christmas cards did Tony Blair send in 2004?
• If church bells could only be rung as a signal of invasion, would ringing them on Xmas day cause confusion?
• Was Aberdeen worse hit than the rest of the country when it came to making Christmas puddings?
• Who played Father Christmas at the Westminster party this year?

You can search for any word or phrase on TheyWorkForYou.com. Click on ‘more options’, and you can also restrict the dates you search within.

December 13th

What’s behind the door? An icy pothole.

Does it count as bleak mid-winter yet? After the mild start to the season, in some parts of the country it still feels as if the really cold weather is yet to come.

And yet, the freeze won’t be long in coming. Uneven pavements and potholes turn from a mild inconvenience to a real hazard in the ice – and you will certainly have already noticed if your streetlights aren’t coming on, now that the dark evenings are here.

So here’s for one last big push on our Fix Before the Freeze campaign. Make sure you report all those pesky potholes, uneven pavements, and broken street lights before the snow and ice get here in earnest, and help make your local community a safer place this winter.

December 12th

What’s behind the door? An angelic host, complete with shiny halos.

Our website Pledgebank has been used for some good causes around Christmas time. It’s based on the simple idea of promising that you will do something if other people promise to, too. It’s an effective way of taking an action and multiplying its impact.

In previous years, we’ve seen a pledge to visit people who may be alone at Christmas, and another to walk for an hour on Christmas day, among others.

If you’ve got plans this Christmas – say, donating to charity, giving gifts to the poor, or even organising a party,  Pledgebank could be the tool that tips the balance and helps you get the people-power you need.

Pledgebank isn’t just for individuals: Barnet council have been innovative in their usage of the Pledgebank software for the good of their community. Check out how they are using it to arrange a collection of gifts for the needy, and gritting.

December 9th

What’s behind the door? Frosty the headless snowman.

FixMyStreet is our website for reporting problems such as potholes or broken streetlights, but last January, one user in Brighton and Hove wanted to express his outrage about something else.

Unfortunately, the council have rather less control over the kicking down of snowmen. Much as we sympathise with the frustrated anonymous reporter, we can’t really blame the council for not responding to this particular complaint.

Meanwhile, in Midlothian, we see nature doing the fixing but the council apparently taking the credit, much to our user’s displeasure.

If your neighbourhood suffers from uncleared snow, by all means use FixMyStreet.com to report it this year. If you feel the gritting could have been better, report it. If your snowman suffers an injury, however, maybe keep it to yourself.

December 8th

What’s behind the door? A boring old bauble again.

What is a “Christmas Tree bill”?

A search through Hansard reveals that this is a commonly-used term in Parliament, and it refers to a bill which, as it passes through its various stages, has all sorts of “baubles” hung on it – that is to say, small, unrelated issues which are added to the main legislation.

The term apparently originated in the States, but has become commonplace in UK parliamentary discourse – and indeed provides an opportunity for some florid extemporising, as David Burrowes, Private Secretary, demonstrated recently in a debate about knife crime:

“As we look forward to Christmas and see today the Third Reading of a criminal justice Bill, I am reminded of previous Government Bills that ended up as Christmas tree Bills with baubles being hung on them at any given opportunity as they went through Parliament. I am sure that as this Bill goes to the other place, Ministers will want to ensure that further baubles are not hung on it in the form of extra pieces of law that take the fancy of noble Lords, as well as any little elves.”

Did you know that you can subscribe to any word or phrase on TheyWorkForYou? It’s very handy for making sure you know whenever your pet topic is debated. Set up your alert here.

December 7th

What’s behind the door? A kindly Santa Claus

Our website WriteToThem.com allows you to contact your elected representatives – even if you don’t know who they are.

When you input your postcode, you’re given a list of your local councillors, MPs, MEPs and anyone else who represents you in any of our governmental bodies. The site then allows you to contact them directly.

That’s all very well, but what about the highest administration of them all – the one who decides if you’ve been naughty or nice? Sadly, WriteToThem.com does not cover Lapland, but we do have a helpful page providing Santa’s postal address in full.

Meanwhile, it’s just a thought – but you might find that putting your wishlist in front of your local representatives actually has more effect than a letter sent up the chimney, especially if it concerns your civic or community rights. Start here.

December 6th

What’s behind the door? A fizzling, blinking neon light

Our parents always told us that if decorations weren’t removed by Twelfth Night, terrible things would happen – but it seems that some councils are not so superstitious. Users of our website FixMyStreet reveal the occasionally erratic handling of this tradition.

7th of January was already too late for a resident of Durham. How would he have felt had he lived in Thatcham, where decorations were still up on the 18th of January?

It gets worse. In Birmingham, one lonely decoration was spotted on the 31st of January. In Consett, not only were the decorations taken down after Epiphany had passed, but they had been on 24 hours a day for the entire Xmas period. In the village of Cark, the Christmas tree was blocking access to a car park in early February. But we think Bournemouth takes the prize, with a Christmas decoration reported as still being in place on the 15th of March.

People are always complaining that Christmas starts too early – and now it seems it’s also dragging on too late. If you’d like to report council decorations that have outstayed their welcome, don’t forget FixMyStreet.com this January.

The 5th of January, in fact, if you’d like to adhere to Twelfth Night superstition. We’ll be looking out for the spike in users on that day.

December 5th

What’s behind the door? An inn, with no vacancies over the Christmas period

It’s more than 2,000 years since a heavily pregnant Mary was told there was no room at the inn. With zoning restrictions a thing of the far-distant future, an empty stable was repurposed for her use, and… well, you know the rest.

Today, if there’s an empty stable (or, more likely, a house) near you, Channel 4 want to know about it. They are broadcasting the first in their Great British Property Scandal series tonight, examining the causes behind homelessness.

Key to the campaign is the fact that there are over a million empty properties in the UK, while two million families need a home. On their site you’ll find an empty property spotter tool, which allows you to report any vacant buildings to your council. There’s also an app.

Those tools have been built by a crack team of mySociety developers, drawing on our extensive experience of mashing up postcode and constituency data, and sending reports off to the right council contacts. If you’re wondering where we honed such skills, look no further than FixMyStreet, WriteToThem, and TheyWorkForYou, among other mySociety projects.

Not everyone knows that mySociety are available for contracting. All revenue from our commercial activities goes towards funding our not-for-profit projects. It’d really make our Christmas special if you were to spread the word, next time you hear of someone in need of innovative and really rather well-priced development work.

December 2nd

What’s behind the door? Ten Lords a-tweeting

Why is a Christmas card better than a tweet? It turns out not to be the start of a bad joke…

As Roger Gale MP revealed in a debate on the use of electronic devices (including mobile phones) in the Chamber of the House of Commons, “multi-tasking and a dual use of time” means that in the six weeks before Christmas “committee tables will suddenly be piled with Christmas cards being signed while Members are also participating in Committee business”.

Gale’s point is that such behaviour is excusable, but that having MPs updating their Twitter and Facebook statuses in the Chamber would be a bridge too far. What do you reckon? Personally we’d rather have a stream of useful comment, accessible from our phones or desktop computers, than a hastily-signed Christmas card.

Whether you’re a social media junkie, or agree that such things are unwelcome in the workplace, the entire debate is worth a read – along with hundreds of thousands of other speeches and statements from Lords and MPs, available on mySociety’s TheyWorkForYou.com.

December 1st

Children everywhere open the first door of their Advent calendars today – and we’re digging deep into the mySociety vat of Christmas spirit and presenting our very own countdown to the 25th. Didn’t think a civic and democratic charity had much in common with Christmas? Well, we’re here to prove otherwise.

Between now and the 25th, we’ll be updating this post each weekday with a Christmassy nugget from our archives. Enjoy them, and here’s hoping that Santa brings you whatever your heart desires, whether it’s the reply to that FOI request you put in on WhatDoTheyKnow.com, or the improved bus service you asked for on FixMyTransport.com.

What’s behind the door? A string of flashing lights

As Christmas lights go on in towns and cities across the country, your inner Scrooge might be prompted to ask just how much they’re costing the public purse.

Never fear, Bah Humbuggers, for this is a topic that has been thoroughly explored by the users of our Freedom of Information request website WhatDoTheyKnow.com. See, for example, how Manchester cannily bartered for free celebrity appearances last year, while Lewisham puts importance on low-energy lightbulbs.

You can also check Westminster, Lewes, and Cardiff’s costs – and plenty more besides. We think that Leeds has the highest expenditure mentioned, at £477,600 for this year, but leave us a comment if you find a higher one.

Don’t forget that if you want to know how much your own council spent on Christmas decorations – or indeed anything else – you have the right to submit an  FOI request. Just remember to check that the information isn’t already available online before you do.

The Ministry of Justice have just published their latest quarterly statistics on the handling of Freedom of Information requests by central government bodies.  We’ve crunched the numbers to compare them to the requests made using WhatDoTheyKnow.com

The graph shows our share of FOI requests sent to central Departments of State jumped to 14.6% in the 1st quarter of 2011.

This time round, the top 3 departments were:

1. Home Office (which includes the UK Border Agency, CRB & Identity & Passport Service) – 254 requests out of 866 – 29%
2. Department for Education – 81 requests out of 328 – 25%
3. Department for Communities and Local Government – 59 requests out of 250 – 24%

Many of the WhatDoTheyKnow users contacting the Home Office & UK Border Agency are trying to find out information about their own immigration case.  We regularly receive emails from applicants asking for help, as they have often been waiting months (or even years in some cases) for an official update to their case, often with the UKBA holding on to identity documents or passport.  Applicants then feel they have to resort to making FOI requests. Many of these are auto-replied by this standard FAQ, and applicants don’t receive a personal answer.  The large 29% share of all Home Office requests suggests that the normal contact methods to keep people updated aren’t working or even that their service is simply struggling with demand.  It’s also likely that they don’t consider these types of requests as formal FOI requests, so it is worth noting that we are likely to be slightly overstating the percentage share figures.

Free schools were a popular topic for the Department of Education – 9 out of 81 requests were on this subject, and nearly all were refused on the basis that information would be published at some unspecified date in the future.

To understand the limitations of the data analysis, please see here.

One interesting trend that has been consistently seen is that FOI requests are more frequent in odd-numbered quarters compared to even ones – if you have any ideas why this may be the case, please add them to the comments!

Member of the National Secular Society Robert Christian used mySociety’s Freedom of Information site, WhatDoTheyKnow to ask all 227 English NHS “provider” Trusts about how much they spend on chaplaincy.

On the 28th of February 2011 the results of his research were published in an article on the National Secular Society website (full report [PDF]). He found that £29m of NHS funds were used to pay chaplains in 2009/10 and also observed a wide variation in the amount, as a fraction of total spend, that specific trusts were spending on chaplaincy.

The publication of the research prompted a number of articles in the UK media. eg. (Daily Mail, The Independent, The Mirror).

Mr Christian has commented:

“To have identified the right FOI contact for every provider NHS Trust in England would have been daunting if not impossible. I doubt that my study would ever have got off the ground without WDTK. I particularly valued the way that the site tracks which Trust has and has not yet responded. I liked the capability to thank each FOI lead after they had responded.”

The fact that making requests via WhatDoTheyKnow allowed Mr Christian to cite the source of his raw data was important to him. He added:

“The transparency of the raw data is, I think, one of the main strengths of the WDTK website for three reasons. First, I was able to hyperlink every piece of data back to its source – and that meant that it was easy for colleagues from the NSS to check the accuracy of the data (with so many Trusts a transcription error was always a possibility). Second, it ensured that if anyone had wanted to challenge the accuracy of the data they could be directed to see that the study was simply quoting the Trusts’ own information. Third, it means that the data is there for future reference to see if there are any changes over the coming years.”

mySociety and WhatDoTheyKnow are non-partisan and don’t get involved in campaigning except in specific areas relating to openness and transparency. We take no view on issues such as how much, if anything, the NHS ought be paying for chaplaincy. However we welcome campaign groups making use of our services.

Bulk Requests

WhatDoTheyKnow currently has around 2-4 “bulk requests” per month made via its site. At the moment we don’t provide any mechanism to make bulk requests automatically. We are considering adding such a system, for requests which have been sanity checked by the WhatDoTheyKnow team. The provision of such a system would probably be associated with a mechanism for preventing other “bulk requests” from being made without the site administrators’ explicit approval.

Making the requests is only a small part of the work involved in a study such as that carried out by Mr Christian. Chasing public bodies for responses, as well as collating and analysing the information released is likely to be much more time consuming than submitting the requests themselves. This is something Mr Christian agrees with, stating:

“If enquirers are not prepared to individually contact each organisation to ask the question, I would doubt their commitment to retrieve and analyse the information (as that is actually a much bigger task)”.

Clearly any facility for enabling requests to be made in bulk will have to incorporate safeguards to ensure responsible use.

Whereas Mr Christian has been happy to conduct his research in public, and still been able to generate media coverage following publication, we are aware that many campaign groups, and others such as journalists, like to make Freedom of Information requests in private.

Mr Christian has commented on the issue of “scoops” and the effect of conducting his research in public:

“The question of ‘scoops’ is an issue for journalists and in fact this problem did happen in this case. Someone appears to have trawled the WDTK know site and noticed what I was doing. A short piece was run by the Daily Express before we completed and published the study. So clearly this might be an issue. But the risk of a spoiler being run will tend to be low when the number of organisations being contacted is large. This is because the amount of work needed to collate and analyse the data is enormous and so casual trawling will show only that a question is being asked – not what the conclusions are.”

In order to get as great a fraction of the total number of FOI responses available on WhatDoTheyKnow we have also been considering an option for making requests in private, for a fee. The idea would be that once the findings were published then the FOI response could be opened up to the public providing access to the source material backing up the story.

Any views on our ideas for the future and on the way WhatDoTheyKnow has been used for this, and similar, research would be welcome in the comments below.

The Government is currently proposing to reform the UK’s defamation laws. The WhatDoTheyKnow.com team has responded to the consultation on a Draft Defamation Bill currently being run by the Ministry of Justice.

The bill proposes extending and clarifying the list of types of material subject to “privilege” ie. which can be published without fear of being sued for defamation. “Matter published by or on the authority of a government or legislature anywhere in the world” is already covered but we have been advised that might not extend to all Freedom of Information responses; if it does or not is something which is yet to be tested in court.

We are asking for the law to be clarified and for “privilege” to be extended to a fair and accurate report or summary of, copy of or extract from material released by public bodies. The proposed new provision would enable the republication of Freedom of Information responses from public bodies without fear of libel action. Such a provision would clearly be of value to services such as WhatDoTheyKnow.com. It would also allow campaigners, journalists and others working with such material more freedom from legal threats and uncertainty; as such this addition would appear to be in-line with the coalition’s stated aims of their amendments to libel law.

We would rather see Parliament explicitly clarify the law rather than see a journalist, campaigner or website operator be subjected to an expensive and time consuming legal action.

Additional suggestions

We have also suggested:

• That the principle that any governmental body should be open to uninhibited public criticism, and therefore should not be able to use or threaten use defamation law to quash critics ought be extended to apply to all public bodies and those, such as contractors, acting on behalf of public bodies.
• That merely pointing to defamatory material, through the provision of a weblink, ought not in itsself be actionable where there is no express endorsement of the defamatory material along with the link.

The full consultation response can be read online at FOIWiki

Earlier today the Department for Education, which is headed by Education Secretary Michael Gove, wrote to WhatDoTheyKnow to let us know that the main email address they use to receive FOI requests is to be phased out. They would prefer the public to make their FOI requests via the contact form on their own website instead or even by post. We believe that this approach is contrary to the spirit of the law and principles of Freedom of Information.

The message we received stated:

We changed the way that people contact our department last year, encouraging customers to go to our website to find what they are looking for and submit an enquiry via our contact us page (www.education.gov.uk/contactus) if they could not locate information.

The [main FOI] mailbox that your system points to ([email]) will eventually be phased out and I would be grateful if you could advise customers using your website to refer to www.education.gov.uk/contactus if they need to contact the Department.

We certainly agree that people should check whether the information they are looking for is already available before submitting a FOI request — and indeed we already prompt all users of WhatDoTheyKnow to do so, not just for the Department of Education, but for every public authority we list.

When requests are submitted through WhatDoTheyKnow responses are automatically published ensuring a lot more information ends up online and publicly accessible than when submitted privately. If the Department for Education wants to reduce the amount of correspondence it gets in relation to already published material it should be encouraging people to make their FOI requests via WhatDoTheyKnow. Already, based on Ministry of Justice statistics, we calculate around 10% of all Freedom of Information requests to the Department of Education are made via our service.

We have asked the department to let us know which alternative email address they would prefer us to forward FOI requests to, and we await their reply. We are happy to use whichever email address is easiest for a public body.

We have also made clear that we will continue to offer our users the ability to make requests to the Department of Education via our site and will not be removing that facility and directing people to the department’s contact form as we were asked. Forms often include unnecessary mandatory fields that the FOI legislation does not require (in the DfE’s case they ask what kind of a requester you are, making you specifically type in “prefer not to say” into an “Other” box if you want to opt out).

The law rejects the idea that public bodies are allowed to erect artificial barriers like this, and we have noted that a FOI request is valid regardless of which email address or member of staff within an organisation it is sent to.

I recently found these requests by James Muldoon covering FOI statistics for the London Boroughs for 2009.  As we regularly carry out analysis of WhatDoTheyKnow’s percentage share of FOI requests to central Government Departments of State, I thought it would make for an interesting comparison to do the same for the 33 Metropolitan borough councils, plus the City of London.

Below is a graph of the market share for WDTK.

Overall, the share for 2009 was 8.1%.  During the year, the share did fluctuate quite a bit, and the requests on WhatDoTheyKnow were significantly lower in the 2nd quarter for some reason.

Q1: Jan-Mar 2009 – 9.4%
Q2: Apr-Jun 2009 – 5.1%
Q3: Jul-Sep 2009 – 9.5%
Q4: Oct-Dec 2009 – 8.3%

The City of Westminster has a much higher number of FOI requests compared to the other boroughs, mostly apparently due to a large motorcyclist parking campaign/protest.  73% of all requests made to Westminster via WhatDoTheyKnow in 2009 contained the words “parking”, “motorcycle” or “Verrus” (203 out of 278).

I will soon start looking for FOI statistics for Local Authorities outside London, either on WDTK, or via their disclosure logs.  The Ministry of Justice encourages Local Authorities to regularly publish statistics on their FOI data.

Data caveats

• Brent - excluded from totals & comparison as the underlying FOI request is still outstanding.  The ICO is apparently investigating.
• Camden -  Q1-2009 data excluded from totals & comparison due to partial refusal to the FOI request by Camden (FOI Act Section 12, costs of complying too high)
• Southwark - excluded from totals & comparison.  They said in their FOI response:  “due to a serious malfunction of our reporting database we have no access to the data stored centrally”.  The data has been re-requested by James to see if the malfunction has been fixed.

Today’s Sunday Times carries an article on very high salaries paid to some of those working in the “publicly funded arts world”. The article reports Antonio Pappano, the Royal Opera House’s Music Director, is paid more than £630,000 a year and is given four months a year off to carry out a second job as music director of a Rome orchestra.

While the Sunday Times’ paywall means we don’t have a direct link to their article; it appears to be based on much the same information as an article published a few days earlier by The Arts Desk.

The Sunday Times article states the Government has “expressed surprise at the sums paid” and Ed Vaizey the Culture Minister is quoted as saying:

“There really must be full transparency for all publicly funded arts bodies”.

There is also a statement from the Arts Council expressing a similar, though more limited, sentiment:

“Anybody in receipt of significant public money should be transparent about their core funding costs”.

The Arts Council, the main body which distributes public funding to the arts, is subject to the Freedom of Information Act. The Arts Council is listed on mySociety’s Freedom of Information website WhatDoTheyKnow.com which enables people to easily make requests for information in public. While the Arts Council is responsible for handing out the money, it does not necessarily know the details of how the recipient organisations spend it. The bodies which receive funds are not themselves yet subject to freedom of information law, irrespective of how much public money they receive or how dependent they are on that subsidy.

While it may take the Minister some time to legislate to ensure “full transparency for all publicly funded arts bodies” we are happy to add such bodies to our site on request right now, so our users can ask them, in public, about their activities.

As of today the following organisations are now listed on our site:

• The Royal Opera House
• The National Theatre
• English National Opera
• The Southbank Centre
• Birmingham Royal Ballet
• City of Birmingham Symphony Orchestra
• London Symphony Orchestra

We use the WhatDoTheyKnow site to actively campaign for expansion of Freedom of Information to cover more public organisations. We list a number of bodies not formally subject to FOI some of which are present on the grounds they are substantially publicly funded.

For some time we have listed the British Board of Film Classification, a key arts regulatory body which is not subject to freedom of information law, and the British Film Institute; the latter two bodies are funded by the DCMS directly so Minister Ed Vaizey may well be able to get them to voluntarily comply with FOI legislation first thing on Monday morning.

A particular set of arts funding bodies which some of our users have made us aware they would like to see subject to the act are the UK Screen Agencies (eg. Film Agency Wales) which distribute public funds to the film culture sector.

Please contact the WhatDoTheyKnow team if you have any suggestions for further bodies which you would like to see us list on our site.

A number of non-departmental government bodies / quangos have been named as being up for abolition, merger, privatisation or absorption into parent departments, as part of the Coalition Government’s Spending Review, due this autumn. This has been widely dubbed in the press as a “bonfire of the quangos“.  The list of quangos up for review is still being compiled by the government, and there have been a number of clarifications, amendments and retractions as further details come to light.

The Telegraph has obtained and published today a leaked list of 177 quangos up for abolition, plus a further 200 that are still being reviewed.

This is a great opportunity to highlight that mySociety’s Freedom of Information site WhatDoTheyKnow covers nearly all of these little-known bodies that spend public money (we currently have just over 3,800 public authorities listed on the site).  Given their impending doom, there is little time left to find out what they spent public funds on, as only their most important records will be transferred to the National Archives or successor bodies for permanent storage.  The remainder are likely to be shredded, or deleted, as only “records identified as valuable for future administrative need” are kept.

You can see our annotated list of the Telegraph’s list here - our volunteers have added links to most of the bodies’ pages on WhatDoTheyKnow, so you can more easily make your final FOI requests to them…

Please send any missing contact details to the WhatDoTheyKnow team.

In January 2007 the Information Commissioner’s Office disclosed via FOI 28 “line to take” (LTT) documents (pdf).  These are topic-based documents, produced by their Policy Team, and they are used internally by their case workers as how they interpret and apply the various sections of the Freedom of Information Act 2000 and the Environmental Information Regulations 2004.

They are mostly based on Information Tribunal rulings which have set a legal precedent, or relevant ICO decision notices, but for some topics they are based on the ICO’s own interpretations of the FOI Act, indicating how they’re likely to handle the subject matter of these complaints.

For someone looking to appeal a particular FOI decision made by a public authority (eg a refusal to provide information due to an exemption in the FOI Act), these LTTs are a very useful addition to the FOI guidance already produced by the Ministry of Justice or the ICO.  They can be used to verify whether the authority’s stance is appropriate, and allow the requester to point the authority toward how the ICO sees that particular topic, and therefore challenge a FOI decision.

Although I’m a volunteer for WhatDoTheyKnow, I occasionally make FOI requests in a personal capacity. As I only recently found the above LTTs from 2007, I therefore asked the ICO for an updated list of LTTs.  These have now been provided, and there is now a big library of 177 LTTs.  They sent them via 6 large PDF files, so I have made them available in a more structured, searchable and (hopefully) easier to use manner via FOIwiki.com which is maintained by some of the WDTK volunteers:

http://foiwiki.com/foiwiki/index.php/LTT

In addition to the master list of LTTs available via the link above, they can also be accessed in context with their associated FOI Act exemption page, eg Section 41: Information Provided In Confidence

Two items of note from the LTTs.

• They may not always be 100% up to date. E.g LTT47 (last updated in 2007) says “it should be noted that the Time for Compliance Regulations do not extend to schools in Northern Ireland” – this was fixed via legislation passed last year (SI 2009/1369)
• There are a few LTTs which set out that the ICO disagree with a particular Information Tribunal ruling, and that they would approach the complaint in a different way, e.g LTT119 or LTT168. These could present a problem with making a relevant FOI complaint

I hope you find them useful!  All comments or suggestions welcomed.

Have you used WhatDoTheyKnow to make a Freedom of Information request?

If so, you can now add your photograph to the site, and some text on your user page about what you’re researching. This can include links to your blog, campaign page or twitter feed.

To add this to your profile, first log into WhatDoTheyKnow, and go to your user page by choosing “my requests”.

There are then links to add a profile photo and/or set some text about you, and what you’re using FOI for.

I’d go and do it while I remember – it will help you and others find and understand each other, hopefully leading to that little bit more collaborative research!

The Ministry of Justice recently released the latest statistics on freedom of information implementation in central government for the first quarter of 2010.  We can use this data to roughly calculate the share of FOI requests made via mySocety’s Freedom of Information website WhatDoTheyKnow.com.

According to the Ministry of Justice 6,857 requests were made to “Departments of State” between January and March 2010, of which 707 were made via WhatDoTheyKnow – a 10% share.  Defra and DCSF had the highest percentage of WhatDoTheyKnow requests in the first quarter, at 23% and 24% respectively.  We have made the departmental breakdown of the statistics for the past two years available as a Google Spreadsheet.

Precise calculations of the percentage share are not possible, due to the way each department defines a valid request.  For example, some FOI requests to the UK Border Agency (included in Home Office data) or the FCO made via WhatDoTheyKnow would not have been counted as FOI requests, but instead as “routine requests for information”, falling outside the scope of data collection for the statistics produced by the Ministry of Justice.  In addition, requests for data readily available (i.e. exempt under Section 21 of the Freedom of Information Act) are not counted either by the MoJ, but will be included in the WhatDoTheyKnow statistics.  Overall, these form a minority of requests made by WhatDoTheyKnow, so the percentage share we quote is fairly accurate.

The graph shows that the total numbers and percentage of requests have been fairly consistent over the past year, around 10-13%.  Share of requests increased significantly during 2008 as the site grew in popularity following its launch.

The large increase in total FOI requests between Q4 2008 and Q1 2009 may partly be due to the increased popularity of WhatDoTheyKnow and also increased visibility of Freedom of Information to the general public through more mentions of FOI law in the media.  Key stories in the press included Jack Straw’s veto against disclosure of the Iraq War Cabinet Minutes and Parliament’s aborted attempt to hide details of MPs’ Expenses. Requests made via WhatDoTheyKnow accounted for around 20% of the total increase in FOI requests.

It’s worth bearing in mind that the above graph only covers the 22 central Departments.  WhatDoTheyKnow.com lists over 3,750 authorities including local government councils, police forces, NHS Trusts, non-ministerial departments and many more, and also covers Scotland’s public authorities. There is no national monitoring of how many FOI requests are made to such bodies, or how well they perform when responding to requests.

In total, 6,565 requests were made via WDTK in the first quarter, 87% of which were to non-central Departments.  It’s unlikely that the site will ever get full coverage as the majority of FOI requests originate from companies carrying out market research, journalists, political parties and charities, most of whom prefer to make requests in private in order to maintain exclusivity over the released data.  WhatDoTheyKnow would like to include these companies in future, possibly by offering a managed FOI service on a fee basis, including an embargo period prior to the released information being made fully available on the internet – more information is available from the WhatDoTheyKnow team.

Articles based on previous statistics releases:

• Fraction of FOI Requests Made via WhatDoTheyKnow.com Increasing Fast – October 2009
• What Percentage of FOI Requests are made via WhatDoTheyKnow – July 2009

On Monday the 12th of April 2010 Parliament was formally dissolved in advance of the forthcoming general election. This has prompted some interesting, and rather bizarre, responses to the Freedom of Information requests which have been made to the House of Commons and House of Lords since dissolution. Each such request made via mySociety’s Freedom of Information website WhatDoTheyKnow.com has received a standard reply stating:

When Parliament has been dissolved there is no ‘House of Commons/Lords’ for the purposes of Part 1 of Schedule 1 to the 2000 Act, and there is therefore no ‘public authority’ to which the 20 day deadline under section 10 of the 2000 Act is capable of applying. The time limits do not, therefore, apply during the period of Dissolution.

The effect of the 2000 Act, including its time limits, resumes when the new House of Commons/Lords first meets.

Requests which were due to receive a response during the period parliament is dissolved have had their “clocks” stopped, with notices saying:

As your request was received before the House dissolved, the 20 working day time limit of your request will be split, ceasing on 12 April 2010 and resuming on 18 May 2010 when the new Parliament first meets.

As mySociety’s Freedom of Information website WhatDoTheyKnow automatically publishes requests, correspondence and responses online it’s not just the requestors who can see those responses, anyone can.

During the 2005 election according to the UK FOI blog Parliament placed a notice on its website saying it had consulted with the Information Commissioner and agreed the procedure for extending the time limit for a response.

Clearly the Houses of Parliament still have staff employed and people are still acknowledging the FOI requests. While both houses have stopped meeting, the institutions behind them must surely still be operating, and to claim they have ceased to exist is bordering on the utterly ludicrous.

Are the staff who would otherwise be in a position to respond to requests for things like viewer statistics for the Parliament Live TV stream, content of the Commons’ Intranet or cost of the Parliament Education Service not at their desks at the moment? If they are who’s employing them? Who’s paying them? While it is presumably a busy time for those staff preparing for a new intake of MPs; you might think that without MPs and Lords around it may be a quite time for many staff who might want to use the opportunity to catch up with correspondence like FOI requests. Perhaps in the midst of all this rather British oddness we should be happy that at least the parliament website hasn’t been turned off at this time of peak interest in the nation about parliament and our democratic system.

WhatDoTheyKnow already has to be aware of public holidays and follows some rather complex rules when it comes to calculating deadlines for responses however we have decided against updating the system to deal with this new and unexpected situation. We don’t think it is right that the institution of Parliament should consider itsself not to exist during an election period. We still be marking FOI requests as “late” when the twenty working day statutory time limit has expired regardless of the odd stance being taken by Parliamentary officials.

When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.

Peter Collingbourne got fed up with this, and luckily for us, he can code too. He forked our source code repository, and made a nice patch in his own copy of it.

He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.

Now if you go to an Outlook attachment on WhatDoTheyKnow,
such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).

Thanks Peter!

If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.

It seems Transport for London (TfL) really didn’t want anyone to be able to get hold of their internal guidelines describing how they deal with congestion charge appeals. Now though, thanks to mySociety’s freedom of information site WhatDoTheyKnow anyone wanting to find out in what circumstances TfL will cancel a congestion charge penalty can read the document detailing TfL’s criteria themselves.

I believe there are basic principles involved here; we all ought be able to find out what the law is, both “in statue” and “in practice”; it is right that we can find out the detailed rules which are being applied to the application of the congestion charge in London. I think that this request, enabling that, has been an excellent use of the Freedom of Information Act; it also shows how using the access to information laws can redress the balance of power between the citizen and the state.

The Road User Charging (Enforcement and Adjudication) (London) Regulations 2001 (the law) describes six circumstances in which someone served with a Penalty Charge Notice relating to the London Congestion Charge can make representations against it. However in practice TfL are in-fact allowing representations on a much wider basis. Now the internal document is available it can be seen that even the detailed: “Helping you with your Congestion Charging Penalty Charge Notice” information sheet offered by TfL does not contain the full details of what is and is not accepted as a valid reason for not paying the congestion charge.

Members of the public can now find out that TfL is happy to waive the congestion charge for those who are travelling “to register a death or travelling to hospital due to death of relative”, as long as proof of the death is provided, but not for someone who is “travelling to attend a funeral”. The document also suggests TfL won’t waive the charge if your vehicle has a breakdown and is recovered while the charge is operational; but the charge will be waived if, as a result of being clamped by a local authority, you have to collect your car from within the zone. Reassurance that TfL won’t pursue you for your congestion charge after you’re dead (assuming someone proves you’re really dead to TfL) is also provided.

There is a wealth of detail in the document, including revelations that foreign military vehicles are subject to the charge, but UK ones are not; along with details of circumstances where people will be given a second chance ie. their first appeal will be allowed but second and subsequent ones will not.

The public availability of this document may well make TfL’s life easier; some people may no longer bother making appeals in circumstances where they know they’ll be rejected and others will be able to phrase their appeal letters in such a way that it makes it easy for TfL staff to assess them against their criteria and accept them.

Request Details
Transport for London (TfL) is responsible for the London Congestion Charge. A document entitled: “Criteria for dealing with Representations and Appeals” describes the procedure TfL staff use to determine if someone’s appeal against a congestion charge penalty will be be accepted or rejected. This document was the subject of a Freedom of Information request made January 2008 which was initially refused. The argument TfL made against disclosure was that releasing the document would prejudice the exercise of TFL’s functions; Freedom of Information Act exemptions under S.30 (Investigations and proceedings conducted by public authorities) and S.31 (Law enforcement) were claimed.

On appeal the information commissioner’s office issued a decision notice saying it agreed with TfL that “the public interest in maintaining the exemption outweighs the public interest in disclosure”. The individual who had requested the document was persistent, and took his case to the information tribunal; there the information commissioner’s decision was overturned and TfL were ordered to release the document. The tribunal ordered the document be released to the original requestor by the 23rd of December 2009. A WhatDoTheyKnow.com user had made a separate request for the same document on the 8th of December 2009, and received it on the 7th of January 2010.

TfL still have not placed the released document on their disclosure log, which is perhaps an indication they’re still not too keen on the fact they’ve been compelled to release it.

Members of the team running mySociety’s freedom of information website WhatDoTheyKnow.com also campaign for improvements to freedom of information law.

Volunteer John Cross has been drawing his MP’s attention to a major loophole in the UK’s Freedom of Information Act which means that a company wholly owned by one local authority is subject to the act but a company owned by two local authorities is not. John’s MP, Peter Bottomley, has been convinced that this anomaly in the law does not make sense and has submitted an Early Day Motion calling for the loophole to be closed. The EDM also highlights the fact that currently a company owned 95% or even 99.5% by a single public authority is not subject to the provisions of the act, as only companies owned 100% by a single authority are currently covered.

The text of the motion states:

“That this House notes that section 6 of the Freedom of Information Act 2000 with certain exceptions makes companies wholly owned by the Crown or by a single public authority subject to the Act; further notes that a company wholly owned by two or more public authorities or 95 per cent. owned by a single public authority will be outside the scope of the Freedom of Information Act 2000; and calls for the closure of this loophole and for companies owned 90 per cent. or more by any number of public authorities to be subject to the Freedom of Information Act 2000.”

The motion is currently open for other MPs to sign-up to and show their support. If you would like to help increase the number of public bodies that are covered by Freedom of Information legislation please consider writing to your MP, asking them to add their name to the current signatories.

There are many situations where public authorities working together or even setting up jointly owned companies is commendable. Such arrangements can lead to savings though economies of scale and avoid duplication; we may see more such companies set up as a response to economic pressures. What is problematic though is the loss of accountability which currently occurs when public bodies come together and set up these companies without requiring them to follow the highest standards of openness, transparency and accountability.

Examples of Companies Which Would Be Made Subject to FOI if the Loophole was Closed:

• Connexions Nottinghamshire Limited – provides support services to young people and is jointly owned by Nottingham City Council and Nottinghamshire County Council.
• Coventry and Solihull Waste Disposal Company – owned two thirds by Coventry City Council and one third by Solihull MBC
• G-Mex Limited – Through its ownership of Destination Manchester Ltd and investment in Modesole Ltd, Manchester City Council has a 95% shareholding in G-Mex Limited
• Higher Education Statistics Agency (HESA) – this company is the official agency for the collection, analysis and dissemination of quantitative information about higher education.
• Manchester Airport PLC – the Manchester Airport Group is owned by the ten local authorities of Greater Manchester
• The Russell Group – is owned by, and represents 20 of the UK’s largest universities, the company’s aims as set out in documents filed on incorporation included “to influence and make representations to stakeholders and legislators”

Many housing associations, purchasing consortia, representative bodies and urban development companies are among the organisations which would be required to operate in a more transparent manner should this loophole be closed.

Many MPs and Lords use the Freedom of Information Act to obtain information from public authorities despite the fact they are able to table parliamentary questions. Occasionally they make their requests via mySociety’s freedom of information website WhatDoTheyKnow.com which ensures both the request, and its response, are freely available online. Surprisingly the freedom of information route can result in the release of more, and better quality, information than a written Parliamentary Question.

For example on the 12th of November 2009 Eleanor Laing the Conservative Shadow Minister for Justice submitted the following written Parliamentary Question:

To ask the Secretary of State for Health how many staff in his Department were employed on the management of freedom of information requests submitted to his Department in each year since 2005; and how much his Department spent on the management of such requests in each such year.

The response contained the number of staff per year as requested but with respect to the spending the parliamentary response stated: “The information requested on expenditure could be provided only at disproportionate cost.”

A very similar request for information had been made many months previously, in July, by WhatDoTheyKnow.com user and FOI campaigner Heather Brooke. The response to the FOI request contained more information, and more precise information, than Eleanor Laing had obtained via her parliamentary question. When the request was made via WhatDoTheyKnow how much staff substantially involved in answering requests were paid was disclosed, in detail.

While the costs of complying with a particular request are capped by regulations under the Freedom of Information Act, data on total costs of FOI compliance such as that released by this request allows the average costs of dealing with a request to be calculated.

MPs using WhatDoTheyKnow

• Stephen Crabb MP
• Lindsay Roy MP
• Tom Watson MP

Do let us know in the comments if you’ve spotted any more!

Today we have a strange story about a department that appears to think that it has a duty not to release information under FOI if it makes people angry.

It all starts in January 2009 the Department for Children, Schools and Families (DCSF) appointed an expert by the name of Graham Badman to conduct a review of elective home education in England. It probably goes without saying that this is an issue far from our concerns, and an issue that mySociety has no views on – what makes us interested is the process that followed.

Shortly after the publication of the report, Elaine Walton, a user of mySociety’s freedom of information website WhatDoTheyKnow.com requested copies of communications between the Department for Children, Schools and Families and Nektus Ltd. the company through which it appears Mr Badman was paid for his work.

According to email replies to Ms Walton, the DCSF located two relevant invoices which show how much money was paid, but refused to disclose them.  Strangely, though, they were not refused on grounds of commercial confidentiality, but rather on something more unusual. Here are the exemptions they cited:

• Section 40 – Personal Information
• Section 38 – Health and safety

Health and Safety? A little investigation reveals more.

When Ms Walton appealed against this decision, an internal review was carried out within the DSCF.  The internal review’s findings stated that Mr Badman was likely become a victim of harassment if certain personal details were made public, hence a health and safety concern, and hence no publication of these invoices. Fair enough – nobody would be in favour of revealing private, sensitive information that would endanger anyone’s life or family, especially in the presence of a known threat.  But take a look at this:

“That the Department had initially been drafting a response that included the release of invoices with only personal data redacted. But before the draft was complete it was apparent that there was a campaign of harassment and vilification against Graham Badman and other individuals/organisations that had contributed to the Report. In the light of this, at the weekly review meeting of FOI cases, it was considered that the balance of public interest might have shifted towards withholding.”

What is very curious here is the admission that the department had been thinking of releasing the invoices with personal data hidden (ie no home address, bank details etc).  But then because of a campaign of harassment, it was decided that they wouldn’t publish anything at all. So not just no personal information, but no dates, no amounts of money, nothing.

What is so unease-making about this FOI decision is that it appears to be saying that departments may conceal information on how much public money has been spent on something because releasing that information will make some angry people even angrier. Surely this can’t be right – if it were every budget would be conducted in complete secrecy. We would encourage the Information Commissioner’s Office to take a look.

mySociety’s Freedom of Information website WhatDoTheyKnow is designed to appear simple and straightforward to users. That appearance belies the fact that behind the scenes a significant amount of effort goes into making sure both those making freedom of information requests and those answering them have a positive experience of the site. While the site is almost entirely automated sometimes human involvement is necessary. This article highlights those key “edge cases” which are dealt with by the staff and volunteers who make up the WhatDoTheyKnow team.

In the last year 15,233 freedom of information requests have been made via WhatDoTheyKnow.

Message Delivery
444 messages on 360 requests (2.3%) had to be manually placed on the correct request as a result of authorities not sending replies to the email address given. The errors are introduced as authorities apparently manually transcribe email addresses from incoming email into correspondence management systems. There have been suggestions some may even print out and scan-in emails into such systems. WhatDoTheyKnow’s code has been improved in light of experience, common errors are now detected automatically and in many cases the system suggests which request the message was intended to be directed to.

In terms of outgoing messages just 52 (0.3%) requests over the course of the year were marked as receiving an error message in response and users marked 94 (0.6%) as requiring administrator attention. These are generally either transient errors which simply require a message to be resent or prompt us to check and update the contact details we hold for a particular organisation. Regularly there are problems with authority’s spam filters and we have to encourage them to change the way their filters are set up to allow messages from WhatDoTheyKnow.com through.

Gone Postal
119 (0.8%) requests were at some point marked as “Handled by Post”. In many of these cases users eventually persuaded authorities to release the information in electronic form. Where information is supplied outside the site users can add annotations describing the information released, then can link to copies of the data they have posted online, or as has been done in respect of 14 requests (0.1% of the total, 11% of those handled by post) they can supply the information to WhatDoTheyKnow to upload manually. When the site was being designed there was a worry that authorities would reply to many requests by post. This has not occurred, in part perhaps because the freedom of information act contains a provision (section 11) requiring the requestor’s preferred means of communication to be used where it is reasonable. A requestor using an @whatdotheyknow email address is clearly expressing a preference for a reply to be made electronically via the site.

Libel
One of the major challenges facing the site is keeping it operating in the face of the UK’s libel laws. Unlike in other countries, such as the US, we cannot publish statements on our users’ behalf without taking the risk of being sued for libel ourselves. Even simply republishing FOI responses from public authorities is not without risk in the UK. While we don’t actively police the site a lot of administrator time is taken up dealing with cases where potentially libelous or defamatory comments have been brought to our attention. Cases can be very complicated and involve a great deal of correspondence. mySociety is lucky to have the services of a specialist internet and technology barrister with expertise in libel who provides his services free of charge. We try and act in such a way as to maximise transparency while ensuring that the existence of WhatDoTheyKnow and mySociety are not threatened by legal risks.

In the last year there have been only seven significant cases where requests have been hidden from public view on the site due to concerns relating to potential libel and defamation. Three of those cases have involved groups of twenty or so requests made by the same one or two users. While actual number of requests we have had to hide is around 70 (0.4% of the total) even this small fraction overstates the situation due to the repetition of the same potentially libellous accusations and comments in different requests. In all cases we have kept as much information up on the site as possible. Our policy with respect to all requests to remove information from the site is that we only take down information in exceptional circumstances; generally only when the law requires us to do so.

Personal Information
Sometimes people accidentally post personal information to the site; for example they make a request which is not a Freedom of Information request but a subject access request under the Data Protection Act. We are happy to remove such requests. On occasion we get requests from both our users and public sector employees asking us to remove their names from the site. As we are trying to build up a FOI archive we are very reluctant to remove information from the site, our policy is only to remove names in exceptional circumstances. Often information, such as an out of office reply, which a public body or civil servant considers irrelevant and asks to be removed is in fact critical to the correspondence thread and timeline of a response.

Copyright and Control of Information Released
The fact information is subject to copyright and restrictions on re-use does not exempt it from disclosure under the Freedom of Information Act (though there is a closely related exemption relating to “commercial interest”). Occasionally public bodies will offer to reply to a request, but in order to deter wider dissemination of the material they will refuse to reply via WhatDoTheyKnow.com. Southampton University have released information in protected PDF documents and the House of Commons has refused to release information via WhatDoTheyKnow.com which it has said it would be prepared to send to an individual directly.

Mantaining and Expanding The List of Authorities
WhatDoTheyKnow lists around 3,000 public authorities, there is a regular turnover of changes in contact details. Our coverage, while large, is not comprehensive so we have requests to add bodies such as parish councils, schools, and doctors surgeries which we have not yet attempted to add in a systemic manner based on official sources of information.

We have also had to carefully consider what we do when for handling the various situations where an authority becomes defunct and its responsibilities are taken over by another body for example as a result of reorganisations of local government and the creation and merging of government departments.

Providing Advice and Assistance
The team at WhatDoTheyKnow.com often provide advice to users. We encourage users to keep their requests focused so as to reduce the chance of any problems due to libel or requests being classed vexatious. On occasion we suggest appropriate authorities for users to direct requests to, provide advice to those unhappy with the response to their request, and answer a broad range of other queries as they arise such as if particular bodies are subject to the act or not. Increasingly we link to authority’s publication schemes which are intended to let people know what information an authority has and how it can be accessed.

Spam
Lastly, like all websites which allow people to post content online WhatDoTheyKnow.com occasionally suffers from spam in various forms. Most is dealt with automatically but some has to be removed by hand. With spam, like the other aspects of running the site, the site’s code and processes are constantly being developed and improved to reduce the fraction of cases requiring any manual intervention.

This article was prompted in part by a team in New Zealand considering launching their own version on the site asking us what’s involved.

Statistics were recently released on the performance of UK central government departments with respect to their handling of freedom of information requests. The latest figures are for the second quarter of 2009. We have been able to use these to calculate the fraction of all requests which are made via mySociety’s freedom of information website WhatDoTheyKnow.com.

• 13.1% of all FOI requests to “Departments of State” in the second quarter of 2009 were made via WhatDoTheyKnow.com. In absolute terms this was 753 out of 5769 requests; this is up from 8.5% in the first quarter of 2009.
• 32.3% of FOI requests to the Home Office (which includes the UKBA and the IPS) were made via WhatDoTheyKnow in the second quarter of 2009. In absolute terms this was 206 out of 638 requests.
• The latest figures show that in twelve of the UK’s twenty-one Departments of State more than 10% of FOI requests were made via WhatDoTheyKnow.

What these statistics mean is that an ever increasing fraction of the information released in response to freedom of information requests is being archived and made publicly available by WhatDoTheyKnow.com. Hopefully this will reduce the number of duplicate requests being submitted and ensure the information released is made available to the widest possible audience which in-turn should increase the chances it is acted on.

Only forty-three central government bodies have their freedom of information performance monitored centrally. This is a tiny fraction of the three thousand or so bodies currently listed by WhatDoTheyKnow.

Raw Data

• Freedom of Information Act: statistics on implementation in central government – UK Ministry of Justice
• Detailed statistics on WhatDoTheyKnow’s share of requests on a per department basis.

On Saturday John Cross and Richard Taylor, two volunteers who work on mySociety’s freedom of information website WhatDoTheyKnow.com, gave a workshop on FOI to a meeting of activists from Republic, an organisation which campaigns for an elected head of state in the UK.

mySociety and WhatDoTheyKnow are non-partisan and don’t get involved in campaigning except in specific areas relating to openness and transparency. That said, members of the WhatDoTheyKnow team are be happy to consider invitations from any groups wishing to hold a workshop discussing freedom of information.

Many of those present at Saturday’s event were active campaigners on a wide range of subjects ranging from human rights to fair trade as well as having an interest in constitutional reform. The FOI workshop was oversubscribed with the majority of those present at the event deciding to attend the session. Unlike a previous workshop held at OpenTech where most attendees had made an FOI request themselves prior to the event, at this workshop all but one had not done so.

The Royals and FOI
Given the audience, the status of the royals with respect to FOI was particularly pertinent. The FOI act exempts information if it relates to: “communications with Her Majesty, with other members of the Royal Family or with the Royal Household, or the conferring by the Crown of any honour or dignity”. This exemption does not apply though if it is determined that it is in the public interest for the information to be released. The requirement for this public interest test is under threat as the Prime Minister has been moving to strengthen the restrictions on releasing information related to the Royal family. On the 10th of June 2009 in a speech to Parliament on Constitutional Renewal Gordon Brown said:

…we have considered the need to strengthen protection for particularly sensitive material, and there will be protection of royal family and Cabinet papers as part of strictly limited exemptions.

Following that speech BBC journalist Martin Rosenbaum obtained a statement from the Ministry of Justice clarifying that in practice what Gordon Brown’s words meant was:

… the relevant exemption in the Freedom of Information Act will be made absolute for information relating to communications with the Royal Household that is less than 20 years’ old.

In FOI jargon an “absolute exemption” is one not subject to a public interest test.

Even with the law as it stands it is not easy to obtain information on how the royals are, or are attempting to, influence government. For example John Cross has asked the Ministry of Justice to supply him with copies of correspondence they had received from the Queen and Prince of Wales. They rejected his request on the grounds that the public interest in non-disclosure exceeded the public interested in disclosure; as well as suggesting exemptions relating to “information provided in confidence” and “personal information” also applied.

The Royal Household’s position on FOI
The Royal Household is not subject to the freedom of information act; though it has made a statement on the subject saying:

Despite its exemption from the FOI Acts, the Royal Household’s policy is to provide information as freely as possible in other areas, and to account openly for its use of public money.

WhatDoTheyKnow’s policy is to include such organisations which have indicated they are willing to voluntarily comply with the act to the site. While we list The Royal Household, at the time of writing no-one has yet used the facility to request information.

Using WhatDoTheyKnow for Campaigning
While we stress the importance of keeping freedom of information requests focused, FOI is a powerful tool for campaigners. We were asked if it would be possible for a group like Republic to set up an account on WhatDoTheyKnow for their campaign? The answer to this is: “Yes! – WhatDoTheyKnow wants to encourage groups to use the site”. The information commissioner has confirmed that it is acceptable to use the name of a “corporate body” when making a FOI request, that’s a broad term which encompasses many organisations, groups and charities.

Republic themselves use FOI extensively and often generate major national news stories as a result of responses to their requests. They want to be able to either offer journalists exclusive stories or write a press release based on information released. They can’t do this if the story gets out first via WhatDoTheyKnow so would be interested in an ability to make requests initially in private. mySociety and WhatDoTheyKnow have been considering an option for journalists to be able to make hidden requests via the site. Such a feature could potentially generate an income stream for the site as well as encourage a greater proportion of FOI requests to be made via it. Once the article had been published then the FOI correspondence could be opened up to the public providing access to the source material backing up the story.

As well as meeting those who use, or might want to use, the site to make requests WhatDoTheyKnow also wants to engage positively with public authorities; we see them as important users of our service too. Developer Francis Irving represented the site at the FOI Live conference for information professionals in June and will be speaking at the Freedom of Information Scotland conference in December.

Last week a user of mySociety’s Freedom of Information website WhatDoTheyKnow.com made a request for the release of the results of research into pollutants and urban greenspace in London which had been carried out by The Forestry Commission. Despite this work having been led by the government department responsible for the UK’s woodlands, carried out in collaboration with UK universities, and largely funded by public money distributed via the Engineering and Physical Sciences Research Council the results of the research were not freely accessible. The user was referred to an academic paper entitled An integrated tool to assess the role of new planting in PM10 capture and the human health benefits: A case study in London which has been published in the October 2009 edition of Elsevier Ltd’s Environmental Pollution journal. The publishing company are currently offering a A PDF version of the publication for $31.50 via their website.

Exemptions Applicable to Research
In terms of the freedom of information act there are a number of provisions which can be used to exempt the output of publicly funded scientific research:

• Section 22 of the act excludes “Information intended for future publication”, a large fraction of research cumulates in the publication of an academic paper so comes into this category.
• Section 21 excludes “Information accessible to applicant by other means.” This means that once research work has been published a requestor can merely be directed to the publication. Section 21(2)a of the act makes clear “information may be [considered] reasonably accessible to the applicant even though it is accessible only on payment.”

With the above exemptions in mind it might well be possible to phrase requests in such a way that they don’t apply. For example I have had some limited success in relation to a request for a research protocol.

First Come First Served?
Our user was offered a hard copy of the publication; the reason this request was drawn to our attention was that the team to was contacted to help the two parties to get in touch directly. I suspect the reason that an electronic copy of the document was not supplied via WhatDoTheyKnow may have been related to a concern over breach of copyright on the research results which has probably been transferred or licensed to the publisher. While one individual may have obtained a copy of the information, it is still not accessible to everyone. Tony Hutchings, the Forestry Commission’s Head of Land Regeneration and Urban Greenspace, who led the research told me: “We have prints of the paper which we could supply you with”. How many printed copies he has to distribute and what happens when he runs out is not clear.

Open Access Publishing
Ideally the results of publicly funded scientific research ought be published in an unrestricted format in open access journals. The UK government, is moving towards such a stance but at a painfully slow pace. I asked the author of the research why he had taken the decision not to publish in a more accessible journal. He responded by saying:

The Research Councils (as do many funders from both private industry and public bodies) assess the quality of the research undertaken by the impact factors of the papers produced. … To my knowledge there are unfortunately few open access journals with high impact factors.

The EPSRC who funded his research have a Policy on Access to Research Outputs which states: “knowledge derived from publicly-funded research must be made available and accessible for public use”. When I asked them for a comment on this particular case Dr Sue Smart their Head of Performance and Evaluation responded saying: “Tony Hutchings is mistaken in his assertion that we use journal impact factors in assessing the quality of research”, but she also ruled his offer of a paper copy of the research article was: “in keeping with the principles of the RCUK (Research Councils UK) position statement [on access to research outputs].”

Like the other volunteers who help out with WhatDoTheyKnow.com I use the site for my own activism and campaign independently for more openness and transparency in a range of areas. I have written an extended article on my own website on the subject of open access publishing where I have included more details of the responses from the research council and researcher quoted above.

WhatDoTheyKnow keeps growing and growing, sucking people in from Google as its archive of maybe 8.5% of Freedom of Information requests gets more and more detailed.

(Graph of number of FOI requests made using WhatDoTheyKnow over time; click for larger version)

There’s round about 8Gb of unfettered Government data in the core database, plus a whole bunch more for indexing and caching. For comparison, TheyWorkForYou (which now goes back to 1935) has 12Gb. And it’s catching up on traffic also – WhatDoTheyKnow has about half the number of visitors as TheyWorkForYou.

Unfortunately, this new found traffic has led to performance problems. You might have seen errors when using WhatDoTheyKnow in the last week or two. This post is firstly an apology for that. Thank you for your patience. Hopefully it is fixed now – do let us know if you get problems still. And secondly it is some techy stuff about debugging such problems in Ruby on Rails…

When WhatDoTheyKnow started failing, we did the obvious things to start with – moving the database to a separate server, and moving some other services off the same server, to give WDTK more room to breathe. It still kept breaking.

None of my server monitoring tools shed any very clear light as to the problem. I upgraded to the latest version of Passenger, the best Rails deployment tool I’ve seen yet. It’s pretty good, but still not mature enough for my liking. I was still getting the same problems with it, but reporting tools like passenger-memory-stats were really helpful.

Eventually I worked out that it was to do with memory use of the Rails processes. Individual ones would leap up to 1Gb, and never drop back down. If several did, the server (with 4Gb of RAM) would start swapping and grind to a halt. The world of Ruby and Rails memory monitoring software is patchwork at best, and in the end I found the simplest tools the most useful. Here’s some:

• I found some Rails processes were getting jammed, and not dieing even when I restarted Apache. I think in the end this was due to the Passenger spawning method, and our use of the Xapian Ruby module. Running Passenger in RailsSpawnMethod conservative mode made things much more robust.
• Monit, which in a previous life had a job holding up vital structural pillars of buildings with duct tape, makes you feel dirty. Actually it is really useful. Given I couldn’t quickly fix the problem, Monit let me at least reduce the suffering for people trying to use the site meanwhile. Here’s the rule I used, which gives Apache a kick every time server memory use is too high. It was firing every 5 or 10 minutes…
  

  check system localhost
      if memory > 3500 MB then exec "/usr/sbin/apache2ctl graceful"

• I found memory_profiler on a blog. It helps you find the kind of memory leak where you unintentionally continue to reference an object you don’t use any more. With a specialist subject of string objects. This led to a fix to do with declaring static arrays in classes vs. modules, which I still don’t really understand. But it wasn’t the cause of the big 1Gb memory munching, there were no large enough leaks of this sort.
• The record_memory function in WDTK’s application controller came from another blog. It’s handy as it shows you how much of the system memory in the Ruby process each request causes an increase by. With caveats, this was the best way for me to identify the most damaging requests (search results, and certain public body pages). And it also brought focus on the actual problem – the peak memory use during a request. That’s really important, because Ruby’s memory manager never returns memory to the operating system… The Gb leaps in memory use were because of temporary memory used during certain requests, which the Ruby memory manager then never frees later.
• I made a bunch of functions culminating in allocated_string_size_around_gc. This was really useful in use with the “just add lots of print statements and fiddle” school of debugging. Not everyone’s favourite school, but if your test code can’t catch it, one I often end up using (it gets really involved rarely enough that it doesn’t seem worth setting up an interactive debugger). It led me to various peak memory savings, such as calling “text.gsub!” rather than “text = text.gsub” while removing (email addresses and private information) from FOI request responses, which help quite a bit when dealing with multi-megabyte attachments.
• Finally, I used the overlooked debugging tool, and the one you should never rely on, being common sense. That is, common sense informed by days of careful use of all the other tools. In order to quickly show text extracts when searching, WDTK stores the extracted attachment text in the database. A few of these attachments are quite large, and led to 50Mb fields, often several of which were being loaded and processed in one page request. That this would cause a high peak of memory use all became just obvious to me some time yesterday. I checked that that was the case, and this morning, I changed it to use the full text for indexing, but to at most keep 1Mb for use in snippets. So sometimes now you won’t get a good search extract for queries, but it is rare, and it will at least still return the right result.

I’ve more work to do, I think there are quite a few other quick wins, all of which are making the site faster too. I’m quite happy that WhatDoTheyKnow also has a bunch more test code as a result of all this.

On the other hand, what a disappointing disaster for open source languages beginning with P/R (as opposed to J). Yes, the help and tools were just about there to work it out, but would seem primitive if you’d used say Java’s Memory Analyzer. Indeed somebody over on StackOverflow suggested running your site in JRuby and using exactly that tool…

Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites

I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.

Identifying and categorising these types of site turns out to be surprisingly useful.  First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.

Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.

1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.

2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou.co.nz, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch,  Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.

3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.

4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.

5. Government spending – where the big money goes. In the US the dominant site is FedSpending.org, and in the UK we have ukpublicspending.co.uk.

6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and  Statute Law Database. Much of the legal database field, however, remains essentially private.

7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet‘s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.

8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places,  the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others.  mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.

9. Political document archives - This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.

10. Bulk data - Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.

——

Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.

Earlier this week someone browsing mySociety’s freedom of information website WhatDoTheyKnow.com contacted us reporting they couldn’t open a PDF document Southampton University had sent in reply to a request asking about the amount of printer credit purchased at the institution. The user suspected the file might be corrupt.

We investigated and determined the attachment is in fact in a fancy PDF format which cannot be read by many PDF readers. Those applications which are able to open the document present a cover page inviting the reader to agree to an “intellectual property rights notice”. The terms of the notice forbid “use” of the “material” without “the written permission of the university”. The intent appears to be to ensure only those agreeing to the terms are given access to the document’s content.

The Freedom of Information Act requires authorities to release information whether or not the person making the request wants to enter into an agreement with respect to its use; so one could argue the university are not properly discharging their responsibilities. In any case their stance is silly because anyone who wants to can request their own copy of the information. All they’re doing is creating more work for themselves.

What were they thinking?
An article John Ozimek of The Register has written in response to the correspondence reports the university explaining their actions by saying:

The Freedom of Information Act gives applicants the right to have information held by public authorities communicated to them, not to documents. Applicants are entitled to use that information so long as they do not breach the intellectual property rights of the public authority: taking the information and using it in a document of their own is acceptable, making use of a document which contains not only the information requested but copyright material is not.

One problem with this was pointedly highlighted by a comment on the Register article saying: Ask yourself, “how do I prove my facts if I can’t publish the evidence?”

WhatDoTheyKnow routinely publishes information released under FOI despite a wide variety of copyright and other legal notices and disclaimers suggesting we shouldn’t. (Read the policy on copyright.) This has been the first relatively elaborate technological attempt to circumvent WhatDoTheyKnow’s efforts to make the responses to FOI requests easily accessible to everyone.

Actions
WhatDoTheyKnow’s staff, volunteers, and users have responded to the university’s actions in a number of ways:

• Those making requests to the university are now warned what it is up to, and are made aware that section 11 of the freedom of information act allows them to specify the format in which they would like to receive their response.
• The original requestors have asked for the information to be released in a more accessible format. (One is citing the disability discrimination act).
• The text of the responses, along with more accessible PDF versions have been placed on WhatDoTheyKnow.
• FOI requests have been made asking the university about advice and discussions relating to the way responses are marked to warn requestors of the university’s rights under copyright and intellectual property law as well as for details of the software used to create the protected files along with the policy decisions made to use it.

They’ve got form
This is not the first time the freedom of information team at Southampton University have been inventive in their use of PDFs; back in May they responded to a request via a password protected PDF version of a compliments slip which they attached to an email.

Go straight to the 2009 Call for Proposals

Do you have a ‘mySocietyish’ idea that you’d like to see become reality? Is there something radical about the sites we already run that should change? Do you have any smart ideas about helping more people to benefit from the services we already offer? Or would you just like to read and comment on ideas submitted by other people?

If the answer to any of these questions is yes, then we’d like you to submit your idea to our 2009 Call for Proposals (built for us by Richard Pope). We’ve run these twice before in 2003 and 2006, resulting in the launch and success of sites such as WriteToThem and WhatDoTheyKnow.

Just as on previous occasions we’ll pick a winner and some runners up, but also just as before we can only promise to do our best – we don’t have the resources to solemnly promise to build the winner, whatever it might be.

What We’re Looking For (or, an insight into the mySociety mindset)

The characteristics of the winning and runner up ideas are highly likely to include one or more of the following factors. Don’t try and include all of them, that’d be silly :)

1. They have to involve the Internet. We don’t do clay modelling.
2. They will capitalise on one or more things that the Internet does really well, better than offline or other forms. WhatDoTheyKnow, for example, seizes on the fact that email can be simultaneously published and rerouted – a simple but critical insight.
3. They will either be a whole new website idea, or a smart and impactful modification of something we already do.
4. They will be ideas that have clear social, civic or democratic benefits that are really easy to explain to the least political person you know, even if the technology behind them is fiendishly complicated.
5. They will have some characteristic that will widely spread the word that the service exists, or that other mySociety services exist.
6. They will offer brilliant value for money, even if they’re expensive to build in the first instance.
7. They will be genuinely new ideas
8. They won’t contain the phrase ‘social media’

We might well change these guidelines a bit as the first responses come in. The call will stay open until September 15th, and we’ll hope to announce the winners in early October.

So what are you waiting for – check out the 2009 Call for Proposals

The volunteer team behind our Freedom of Information (FOI) site
WhatDoTheyKnow.com, has used statistics released by the Ministry of Justice to discover the proportion of all FOI requests being made via the site.

They found that in the first quarter of 2009, 8.5% of all requests made to central government departments were made using WhatDoTheyKnow. In absolute terms that was 514 of 6019 requests.

The breakdown by department is given in the below table. Notably, one in five FOI requests to the Home Office (122 of 643) were made via WhatDoTheyKnow.

Department	Total FOI Requests	Requests Via WDTK	WDTK Share
Home Office	643	122	19.0%
Department for Children, Schools and Families	217	27	12.4%
Department for Environment, Food and Rural Affairs	131	16	12.2%
Department of Health	423	40	9.5%
Ministry of Defence	758	70	9.2%
Foreign and Commonwealth Office	281	25	8.9%
Communities and Local Government	204	18	8.8%
Cabinet Office	274	23	8.4%
Department for Transport	586	41	7.0%
Department for Work and Pensions	533	34	6.4%
HM Treasury	446	27	6.1%
Department for Business, Enterprise and Regulatory Reform	216	12	5.6%
Department of Energy and Climate Change	55	3	5.5%
Department for Innovation, Universities and Skills	74	4	5.4%
Attorney General’s Office	19	1	5.3%
Ministry of Justice	757	38	5.0%
Department for International Development	80	4	5.0%
Scotland Office	58	2	3.4%
Department for Culture, Media and Sport	176	6	3.4%
Northern Ireland Office	65	1	1.5%
Export Credits Guarantee Department	9	0	0.0%
Wales Office	14	0	0.0%
Q1 2009 Totals	6019	514	8.5%

WDTK = WhatDoTheyKnow; Source for total FOI request statistics : Statistics for Q1 2009 (released on the 25th of June 2009); Extended table covering all monitored bodies available.

The Ministry of Justice only monitors, and provides statistics on, 44 bodies’ compliance with the Freedom of Information Act; WhatDoTheyKnow currently lists 2910. We cover a wide range of local bodies including Primary Care Trusts, Local Councils and the Police. There is no national monitoring of how many FOI requests are made to such bodies, or how well they perform when responding to requests.

If you want to see such performance statistics, please help categorise more of the responses made via the site. It can be quite addictive!

Thanks to Richard Taylor for doing this research – see his blog post for some more details, including some information about Scotland.

Is there something part of the government is doing that you’d like to investigate? Find out everything from MPs’ expenses, to the length of allotment waiting lists, to whether your council’s Guy Fawkes bonfire is properly checked for hedgehogs.

mySociety are running a practical workshop on Freedom of Information at OpenTech on 4th July.

The workshop will help you make your first Freedom of Information request, including working out what to request, where to request it from and what exactly to write.

If you’re an old hand, you can get and give tips on how to take requests further.

We’ve got a fantastic team of Freedom of Information (FOI) experts to kick things off and answer hard questions.

• Heather Brooke used FOI to cause the frurore over MPs’ expenses.
• Francis Davey is a lawyer with a specialist knowledge in FOI.
• Elena Egawhary is a freelance journalist, currently working and using FOI for Panorama.
• John Cross, Alex Skene and Richard Taylor are volunteers who run and improve WhatDoTheyKnow, and all use it for their own activism.

Bring a laptop if you have one. Internet will be provided for the workshop only, so we can scour Government websites, and make requests on mySociety’s WhatDoTheyKnow.com website.

As usual, the rest of OpenTech is brimming with great talks, and will be full of interesting geeky wonks and wonky geeks. Book your place here so you can go to them and to the workshop. Hurry, it’s nearly sold out.

Super WhatDoTheyKnow volunteer John Cross has made an interesting petition about Freedom of Information and publicly owned companies

“We the undersigned petition the Prime Minister to support a change to the law to make companies owned two thirds or more by public authorities subject to the Freedom of Information Act 2000.”

The petition goes on to explain (in more details at the bottom right of the petition page) that the situation is quite comical at the moment. If a company is owned by one local authority, then it is subject to FOI, but if it is jointly owned by two then it isn’t. This makes little sense, and it is also very important, as private companies owned by authorities often do important work.

Sign the petition.

Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.

We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.

I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.

• TheyWorkForYou opens up access to conventional, representational democracy, between and during elections. In 10 years time, I asked for Parliament to publish all information about its work in a structured way, as hinted at in our Free Our Bills campaign. So it is much easier for everyone to help make new laws better.
• FixMyStreet is local control of the things people care about, a very practical democracy. In 10 years time I’d like to see all councils running their internal systems (planning, tree preservation orders… everything that isn’t about individuals) in public, so everyone can see and be reassured about what is being done, why and where.
• WhatDoTheyKnow shows the deep interest that there is by the public in the functioning of all areas of government. In 10 years time, I’d like to see document management systems in wide use by public authorities that publish all documents by default. Only if overridden for national security or data protection reasons would they be hidden.

Charles Armstrong, from the Circus Foundation, has written up the workshop.

Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort – Charles is going to work on it for another time.

Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).

We sometimes have incorrect or out of date addresses for sending Freedom of Information requests to. Now anyone can check our addresses. Click “view FOI email address” on the page for any authority, and enter two of those squiggly words to prove you are not a robot.

If you are using WhatDoTheyKnow, and suspect problems with a request, please do check the address we are using is correct. If you are from an authority, or work closely or know a particular authority, please also check the address.

The Register has an article today Parliament’s take on Freedom of Information which describes an FOI request I made using WhatDoTheyKnow, and the House of Commons’ refusal to respond to it because the response would be automatically republished.

Hopefully the House will choose to waive copyright on the document, and send it soon – I still haven’t seen a good reason why they could or should not.

(Also, I haven’t changed my name to Francis Stirling, hopefully The Register will correct it soon!)

Public authorities have now sent back 100 Excel files in response to FOI requests on WhatDoTheyKnow.

The nice thing is, that if somebody bothered to use a spreadsheet, it must contain useful, factual, numerical data across either time or space. Everything from job advert expenditure in Kings Lynn council, to school budgets in the Western Isles.

Have a mine.

P.S. Don’t forget to click “Track things matching ‘filetype:xls’ by email” to be emailed when there are new spreadsheets to look at :)

I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.

mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.

Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages – causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).

Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.

(the autumn photo thanks to Nico Cavallotto)

It’s the first full working day for the new facility to annotate Freedom of Information (FOI) requests on WhatDoTheyKnow, and people have been hard at it.

Mr Ormerod points out that private information isn’t necessarily so private if someone has died, so perhaps the exemption the MOD used shouldn’t apply.

Trevor R Nunn has posted three annotations (e.g. this one) to show that his three FOI requests are being treated as one. The annotations facility is great for handling edge cases like this, which don’t happen often enough to be worth explicitly adding to the code, but need some mention.

And finally Edward Betts has processed the list of post boxes retrieved by FOI into a more structured data format, and posted up a link to it. Exactly the kind of collaboration I love to see!

And that’s just this morning!

Francis has been furiously adding new features to our Freedom of Information website WhatDoTheyKnow ever since it launched earlier this year. He’s just added one of the most important missing features, the ability to leave annotations or comments on FOI requests.

This is especially useful for providing plain English summaries of what information in a response was actually interesting, or to discuss refusals to supply information and what to do with them. To add one just go to a request page and scroll to the bottom, just like adding a comment on a blog post.

So, whether you’ve made a request in the past, or you’re just interested in helping out, get annotating.

A WhatDoTheyKnow user Tom Taylor has posted a cool query to the Royal Mail – he wants a list of where all the postboxes in the UK are (presumably so he can build a ‘find your nearest post box’ web site).

After some delay Colin Young of the Royal Mail responded with a list in a PDF file. However, whilst the list is pretty long, it only contains the postcode location of each postbox, not an actual coordinate that can be plotted on a map. So neither he, nor anyone else, can build a postbox finder service.

Just think about that for a second. The Post Office doesn’t know where its Post Boxes are. Whoda thunk? Good use of WhatDoTheyKnow.com, Tom!

One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.

So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.

I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline – something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.

The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.

If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.

We’re busy as bees, lots of things happening, increasingly many of which are commercial, and we can’t talk about until they’re released.

Commercial? But you’re a charity! Yes – but just as Oxfam have a trading subsidiary company which runs the second hand clothes shops, we have a trading subsidiary company that sells services relating to the websites that we make (structural details here).

Everything from other small charities to large media companies are buying our services – which range from customised versions of FixMyStreet, through to strategic consulatancy. If you’ve got something that you think we might be able to help with, email Karl. He’s easier to talk to than us geeks.

Meanwhile we’re cracking on with our free services for the public, which are increasingly funded by this commercial work.

TheyWorkForYou recently launched a Scottish version, thanks to volunteer Mark Longair, and Matthew. More goodies in store as the Free Our Bills campaign unfolds. We’ve started a sprint to get a photo for every MP’s page. If you work for or are an MP or have copyright of a photo of one that we’re missing, then email it to us.

WhatDoTheyKnow is getting lots of polishing – the new site design that Tommy has been working on is nearly ready. Today I just turned on lots of new email alerts and RSS feeds, so you can get emailed, for example, when a new request is filed to a particular public body, or when a request is successful.

Our super ace volunteers have been busy adding public authorties to the site, and we now have 1153 in total. We’re getting a steady trickle of good requests (pretty graph) coming in. Blogs such as Blind man’s buff and confirm or deny are sorting the wheat from the chaff. Do blog about and link to any interesting requests that you see!

Other things in the works are a much needed revamp of www.mysociety.org, some interesting things on GroupsNearYou, and no doubt squillions of other things. I’ll let Matthew post up anything I’ve missed :)

We’re going to be adding lots of features and major design improvements to our Freedom of Information site WhatDoTheyKnow.com in the next few weeks. One thing we want to add is little explanatory videos helping describe how to make the best request possible.

Today we’re launching the hastily named WhatDoTheyKnow Video Challenge. We want you to make short videos (max 2 mins) in which you explain in a clear and friendly way how to file successful FOI requests. We’re not expecting Hollywood production values, just a friendly face and a good explanation would do fine. If you can do funny, splendid.

Instructions:
1. Record your vid
2. Upload to your video hosting venue of choice
3. Post the link as a comment to this post

We’ll send out a coveted mySociety hoodie to anyone who makes anything we’d seriously consider using (unless bazillions of people enter, of course – do you want to bankrupt us?). We don’t sell our coveted mySociety hoodies, they’re only for people who’ve done something useful for mySociety so they’ll mark you out as a pillar of the community the first time you walk down the street.

There’s a lot left to do, but Francis Irving’s brilliant new mySociety Freedom of Information site is now live. You can file requests to central government departments (most of the them), and browse what other people have been requesting (already fascinating). It doesn’t have a name yet, nor any slick design, nor half the features we want it to have, but it works and it gets things done.

And dammit, people, that’s what mySociety’s all about. Can we explain it any better?

This week has been quite bitty. I’ve been doing more work on the Freedom of Information site, have been getting into the swing of Ruby on Rails. Once you’ve learnt its conventions, it is quite (but not super) nice.

As far as languages are concerned, Ruby seems identical in all interesting respects to Python. It’s like learning Spanish and Italian. Both are super languages. Ruby has nice conventions like exclamation marks at the end of function names to indicate they alter the object, rather than return the value (e.g. .reverse!). But then Python has a cleaner syntax for function parameters. It is swings and roundabouts.

Rails has lots of ways of doing things which we already have our own ways of doing for other sites. The advantage of relearning them, is that other people know them too. So Louise was able to easily download and run the FOI site, and make some patches to it. Which would have been much harder if it was done like our other sites. Making development easier is vital – for a long time I’ve wanted a web-based cleverly forking web application development wiki. But while I dream about that, Rails packaging everything you need to run the app in a standard way in one directory that quite a few people know how to use, helps.

Other things… I’ve been helping Richard set up GroupsNearYou on our live servers, it should be ready for you to play with soon. It looks super nice, and is easy to use. I’ve had some work to do with recruitment. And catching up on general customer support email for TheyWorkForYou and PledgeBank. I’ve also been updating the systems administration documentation on our internal wiki, so others can work out how to run our servers.