Help us to make more
useful things.
Donate to mySociety
Would you like to work with the team that built all these sites? We’re recruiting.
Public authorities have now sent back 100 Excel files in response to FOI requests on WhatDoTheyKnow.
The nice thing is, that if somebody bothered to use a spreadsheet, it must contain useful, factual, numerical data across either time or space. Everything from job advert expenditure in Kings Lynn council, to school budgets in the Western Isles.
Have a mine.
P.S. Don’t forget to click “Track things matching ‘filetype:xls’ by email” to be emailed when there are new spreadsheets to look at :)
mySociety has today emailed (and in one case, posted) a set of 3 Principles which we believe it is important that all candidates for Speaker endorse, before the election of a new Speaker by MPs.
2. Bills being considered must be published online in a much better way than they are now, as the Free Our Bills campaign has been suggesting for some time.
3. The Internet is not a threat to a renewal in our democracy, it is one of its best hopes. Parliament should appoint a senior officer with direct working experience of the power of the Internet who reports directly to the Speaker, and who will help Parliament adapt to a new era of transparency and effectiveness.
We will be posting the status of requests on the likely candidates web pages where we expect large numbers of people to see them before the vote in late June. We have also taken the unusual step of allowing possible candidates to leave a statement of up to 150 words on the principles.
(NB no candidates have actually declared at this stage, so we are starting with the BBC’s list of possibles)
Rationale
mySociety helped lead the campaign back in January to prevent the last ditch attempts to conceal MPs’ expenses. We did so not because, like the newspapers, we wanted to revel in embarrassment and scandal, but because we believe that in the Internet age, the only way for our democracy and government to thrive is if they are open and connected to the net as the rest of us expect them to be. The dramatic events seen in Parliament in recent days vindicate the view that secrecy breeds poor policies and seeds untrustworthy behaviour in the weaker willed.
Furthermore, more than a simple attitude of openness is required of the new Speaker: the public needs a genuine will to push for technological reform using the power of the Internet that will take both open-mindedness and a willing to tread on toes, especially in some parts of the unelected establishment.
Case in point: Over the last two years we have been trying to persuade Parliament to acknowledge that the way it publishes its Bills online is hopelessly inadequate for the Internet age. The campaign has faltered, despite multi partly endorsement from 140 MPs and a campaign membership of thousands. To see why, just take a look at this colourful and error-crammed internal email that we uncovered using the Freedom of Information Act, published for the first time today.
The new Speaker will have a tough job on their hands to overcome resistance of this kind. The best thing we can do is help the new Speaker, whoever they are, assume their new job with a clear mandate from the public, as well as from members.
Act!
That is why, as a final part of this call, we are asking you, our community, to write to your MP today to let them know that you expect them to vote for a candidate that has endorsed the principles above. Your voices to your own constituency MPs can resonate in a way that no blog post or newspaper article ever can. Go to it.
In less than 24 hours we’ve seen the first reply to our emails asking possible Speaker candidates to endorse our three principles. It is from Sir George Young – we’re looking forward to seeing the responses from the others that we wrote to.
Sir George is broadly supportive, which is great, and we’ve printed his reply in full on his own TheyWorkForYou MP page.
In the mean time, please do write to your MP and ask them to ensure that whoever they vote for, it is a candidate who has endorsed mySociety’s three simple principles, You really can have an impact on this issue: MPs are desperate to be seen to be acting for their constituents right now.
NB mySociety is strictly non-partisan and non-party aligned. We want all candidates from all parties to endorse these principles, and we have ensured that none of the wording of the principles leans towards any particular party or set of beliefs not connected to transparency in the modern age.
One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.
So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.
I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline – something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.
The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.
If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.
Santa's Chocolate Coin Mint by Johnathan_W
If you haven’t got a penny,
A ha’penny will do,
If you haven’t got a ha’penny,
Then God bless you.
We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?
mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.
Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.
We hope you’ll continue to follow us on Twitter, Facebook, or Google+ – see our Contacts page to find individual projects’ social media links.
Santa Watching by LadyDragonflyCC
What’s behind the door? A letter to Santa.
Dear Santa,
We think we’ve been pretty good this year. We’ve tried to keep our local neighbourhood clean, help with problems, and aid those in need, so we’re hoping there are a few presents coming our way.
If you can fit them down the chimney, here’s what we’re dreaming of:
More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.
We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.
Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.
Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?
A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.
Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?
FixMyTransport buy-in - from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.
Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.
The epic task of manually matching each of the 42,019 video clips of MPs was started way, way back, ooh, about 12 whole weeks ago. Two days ago the Number 1 rated volunteer timestamper in our league table, Abi Broom, completed the last clip in our database, bringing her personal tally to 8,543 clips.
Abi Broom, No1 timestamping league table champ
Last night we went out and met with Abi and Robert Whittakker, one of the other super-timestampers who had done over 2,000 himself.
As a result of their efforts, and those of hundreds of other volunteers, we have put all the video that we have of the House of Commons sitting over the last year online, next to the text of the debates. The many thousands of people per day who visit TheyWorkForYou can, as a direct consequence of this work, now see video of most of the debates for the last year. When people embed clips on their own sites, that’ll also be thanks in part to our volunteers.
Robert Whittaker, down in 5th place with a measly 2047 clips to his name
When Parliament starts again in the Autumn there’ll be another 300-400 clips a day to do, but we have a feeling the only problem doing them will be who gets to them first.
In the meantime, we’ll soon be working on another game-like toy to help create more data. Hint – it might have something to do with GroupsNearYou.
It’s the first full working day for the new facility to annotate Freedom of Information (FOI) requests on WhatDoTheyKnow, and people have been hard at it.
Mr Ormerod points out that private information isn’t necessarily so private if someone has died, so perhaps the exemption the MOD used shouldn’t apply.
Trevor R Nunn has posted three annotations (e.g. this one) to show that his three FOI requests are being treated as one. The annotations facility is great for handling edge cases like this, which don’t happen often enough to be worth explicitly adding to the code, but need some mention.
And finally Edward Betts has processed the list of post boxes retrieved by FOI into a more structured data format, and posted up a link to it. Exactly the kind of collaboration I love to see!
And that’s just this morning!
Screenshot from Brief Encounters
FixMyTransport is the most challenging project mySociety has ever tried to build. It’s so ambitious that we’re taking the unusual move of breaking off part of the problem and stress-testing it in the form of the new mini-site Brief Encounters, which has gone live today. It was built by Louise Crow, or Crowbot, as we know her, with design support from Dave Whiteland.
Brief Encounters is not, as the name might suggest, mySociety’s long awaited attempt at a dating site. Instead it’s a place where people can share whimsical stories about unusual things that happened them them, or other people, on public transport. We hope you’ll have a go, read some examples and then contribute your own.
You might be thinking that a whimsical story site doesn’t sound very mySocietyish – and you’d be right. Brief Encounters is actually a technology test-bed to help us crack a new design and data problem: how do you make it as easy as possible for users to pinpoint a specific bus stop, or train route, or a ferry port, as easily as possible? There are over 300,000 such beasties, and nobody has ever really tried to build an interface that makes it easy to find each one quickly and reliably.
So, what we want from you, dear readers, is three fold. We want:
For those of you tech minded, the project is built in Ruby and uses the NaPTAN dataset of stations, bus stops and ferry terminals, the National Public Transport Gazetteer database of towns and settlements in the UK, and the National Public Transport Data Repository of sample public transport journeys, from 2008. The first two datasets are free of charge, and the third one mySociety pays for.
Lastly, kudos must go to the hyper-imaginative Nicky Getgood who suggested we collect stories on FixMyTransport, as well as problem reports.
A lot has been happening with PledgeBank in the U.S. over the past few months, but an exciting podcast just went up that’s a great synthesis of some of that work. Idealist just launched a podcast featuring PledgeBank, and also telling the story of one of our favorite pledge campaigns. You can hear the podcast here.
You can also see more from the Bakul Foundation on their new website. Good things are happening, my friends…please help spread the word about PledgeBank to those folks you know who want to do incredible things like what the Bakul Foundation has done!
They could perhaps have picked a better day, as it was quite serious – at the stroke of midnight on the 1st of April, 37 district councils and 7 county councils in England ceased to exist, replaced by 9 new unitary authorities. This means people in Durham, Northumberland, Cornwall, Shropshire, Wiltshire, Chesire, and Bedfordshire only have one principal local authority to deal with now. The Wikipedia article on the changes has more information on the background to this change.
Obviously this meant some work for WriteToThem and FixMyStreet, both of which require up-to-date local council information. Our database of voting areas, MaPit, has “generations”, so we can keep old areas around for various historical purposes. So firstly, I created a new generation and updated all the areas that weren’t affected to the new generation. Next, six of the new unitary authorities (all the counties except Cheshire and Bedfordshire, plus Bedford) share their boundaries and wards with the coterminous councils they’re replacing, so for them it was a simple matter of updating those councils to be unitary authorities.
That left Bedfordshire and Cheshire. I created areas for the three new councils (Cheshire West and Chester, Cheshire East, and Central Bedfordshire), and transferred across the relevant wards from the old county councils – basically a manual process of working out the list of correct ward IDs. april2009-update.sql has the gory SQL details if you care.
WriteToThem was now dealt with, but FixMyStreet needed a little more work. The councils that no longer existed had understandably disappeared from the all reports table, so I had to modify the function that fetches the list of councils to optionally return historical areas so they could be included. And lastly, FixMyStreet needs a way of mapping a point on a map to the relevant council. For this, it needs to know the area covered by a council, which was missing for the new authorities I’d manually created. Thankfully, each of the three new authorities are made up of the areas of either 2 or 3 district councils (e.g. Cheshire East is the area covered by Congleton, Macclesfield, and Crewe and Nantwich), so I just had to write a script that stuck those areas together to create the area of the new council. april2009-construct-new.pl. It all seems to work, and I’m sure our users will be in touch if it doesn’t :)
So goodbye to Alnwick, Bedfordshire, Berwick-upon-Tweed, Blyth Valley, Bridgnorth, Caradon, Carrick, Castle Morpeth, Cheshire, Chester, Chester-le-Street, Congleton, Crewe and Nantwich, Derwentside, Durham City, Easington, Ellesmere Port and Neston, Kennet, Kerrier, Macclesfield, Mid Bedfordshire, North Cornwall, North Shropshire, North Wiltshire, Oswestry, Penwith, Restormel, Salisbury (which is getting a new town council), Sedgefield, Shrewsbury and Atcham, South Bedfordshire, South Shropshire, Teesdale, Tynedale, Vale Royal, Wansbeck, Wear Valley, and West Wiltshire. RIP.
Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.
We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.
I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.
Charles Armstrong, from the Circus Foundation, has written up the workshop.
Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort – Charles is going to work on it for another time.
Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).
Wow! The video timestamping site on TheyWorkForYou has been live for just under ten days, but you’ve already managed to timestamp almost 14,000 clips – that’s almost 40% of our entire video archive!
Thank you for all the good work – and especially to our top timestampers. David Jones, Alex Hazell and Lee Maguire are currently the top three in the overall rankings, but there are five more people who have timestamped more than 500 clips, another seventeen people who have done 100 clips or more, and more than 100 people who’ve done anything from 1 to 100 clips. And of course, there’s also a fair few anonymous people who haven’t yet registered, so their individual contributions to the “anonymous” total of 3349 clips are not recorded on the league tables. Remember, we’ll be handing out prizes to the top timestampers, so get registered before you timestamp your next video!
We’re starting to collect a list of notable clips that we can use to compile a “best of parliament” video gallery – if you would like to nominate a particular speech, please leave a comment below or send an email to team@theyworkforyou.com – just tell us the name of the MP speaking, and the URL of the page where this speech appears on theyworkforyou.com. We’ll put the best of them together and publish a list later this summer.
We’re busy as bees, lots of things happening, increasingly many of which are commercial, and we can’t talk about until they’re released.
Commercial? But you’re a charity! Yes – but just as Oxfam have a trading subsidiary company which runs the second hand clothes shops, we have a trading subsidiary company that sells services relating to the websites that we make (structural details here).
Everything from other small charities to large media companies are buying our services – which range from customised versions of FixMyStreet, through to strategic consulatancy. If you’ve got something that you think we might be able to help with, email Karl. He’s easier to talk to than us geeks.
Meanwhile we’re cracking on with our free services for the public, which are increasingly funded by this commercial work.
TheyWorkForYou recently launched a Scottish version, thanks to volunteer Mark Longair, and Matthew. More goodies in store as the Free Our Bills campaign unfolds. We’ve started a sprint to get a photo for every MP’s page. If you work for or are an MP or have copyright of a photo of one that we’re missing, then email it to us.
WhatDoTheyKnow is getting lots of polishing – the new site design that Tommy has been working on is nearly ready. Today I just turned on lots of new email alerts and RSS feeds, so you can get emailed, for example, when a new request is filed to a particular public body, or when a request is successful.
Our super ace volunteers have been busy adding public authorties to the site, and we now have 1153 in total. We’re getting a steady trickle of good requests (pretty graph) coming in. Blogs such as Blind man’s buff and confirm or deny are sorting the wheat from the chaff. Do blog about and link to any interesting requests that you see!
Other things in the works are a much needed revamp of www.mysociety.org, some interesting things on GroupsNearYou, and no doubt squillions of other things. I’ll let Matthew post up anything I’ve missed :)
mySociety’s Freedom of Information website WhatDoTheyKnow is designed to appear simple and straightforward to users. That appearance belies the fact that behind the scenes a significant amount of effort goes into making sure both those making freedom of information requests and those answering them have a positive experience of the site. While the site is almost entirely automated sometimes human involvement is necessary. This article highlights those key “edge cases” which are dealt with by the staff and volunteers who make up the WhatDoTheyKnow team.
In the last year 15,233 freedom of information requests have been made via WhatDoTheyKnow.
Message Delivery
444 messages on 360 requests (2.3%) had to be manually placed on the correct request as a result of authorities not sending replies to the email address given. The errors are introduced as authorities apparently manually transcribe email addresses from incoming email into correspondence management systems. There have been suggestions some may even print out and scan-in emails into such systems. WhatDoTheyKnow’s code has been improved in light of experience, common errors are now detected automatically and in many cases the system suggests which request the message was intended to be directed to.
In terms of outgoing messages just 52 (0.3%) requests over the course of the year were marked as receiving an error message in response and users marked 94 (0.6%) as requiring administrator attention. These are generally either transient errors which simply require a message to be resent or prompt us to check and update the contact details we hold for a particular organisation. Regularly there are problems with authority’s spam filters and we have to encourage them to change the way their filters are set up to allow messages from WhatDoTheyKnow.com through.
Gone Postal
119 (0.8%) requests were at some point marked as “Handled by Post”. In many of these cases users eventually persuaded authorities to release the information in electronic form. Where information is supplied outside the site users can add annotations describing the information released, then can link to copies of the data they have posted online, or as has been done in respect of 14 requests (0.1% of the total, 11% of those handled by post) they can supply the information to WhatDoTheyKnow to upload manually. When the site was being designed there was a worry that authorities would reply to many requests by post. This has not occurred, in part perhaps because the freedom of information act contains a provision (section 11) requiring the requestor’s preferred means of communication to be used where it is reasonable. A requestor using an @whatdotheyknow email address is clearly expressing a preference for a reply to be made electronically via the site.
Libel
One of the major challenges facing the site is keeping it operating in the face of the UK’s libel laws. Unlike in other countries, such as the US, we cannot publish statements on our users’ behalf without taking the risk of being sued for libel ourselves. Even simply republishing FOI responses from public authorities is not without risk in the UK. While we don’t actively police the site a lot of administrator time is taken up dealing with cases where potentially libelous or defamatory comments have been brought to our attention. Cases can be very complicated and involve a great deal of correspondence. mySociety is lucky to have the services of a specialist internet and technology barrister with expertise in libel who provides his services free of charge. We try and act in such a way as to maximise transparency while ensuring that the existence of WhatDoTheyKnow and mySociety are not threatened by legal risks.
In the last year there have been only seven significant cases where requests have been hidden from public view on the site due to concerns relating to potential libel and defamation. Three of those cases have involved groups of twenty or so requests made by the same one or two users. While actual number of requests we have had to hide is around 70 (0.4% of the total) even this small fraction overstates the situation due to the repetition of the same potentially libellous accusations and comments in different requests. In all cases we have kept as much information up on the site as possible. Our policy with respect to all requests to remove information from the site is that we only take down information in exceptional circumstances; generally only when the law requires us to do so.
Personal Information
Sometimes people accidentally post personal information to the site; for example they make a request which is not a Freedom of Information request but a subject access request under the Data Protection Act. We are happy to remove such requests. On occasion we get requests from both our users and public sector employees asking us to remove their names from the site. As we are trying to build up a FOI archive we are very reluctant to remove information from the site, our policy is only to remove names in exceptional circumstances. Often information, such as an out of office reply, which a public body or civil servant considers irrelevant and asks to be removed is in fact critical to the correspondence thread and timeline of a response.
Copyright and Control of Information Released
The fact information is subject to copyright and restrictions on re-use does not exempt it from disclosure under the Freedom of Information Act (though there is a closely related exemption relating to “commercial interest”). Occasionally public bodies will offer to reply to a request, but in order to deter wider dissemination of the material they will refuse to reply via WhatDoTheyKnow.com. Southampton University have released information in protected PDF documents and the House of Commons has refused to release information via WhatDoTheyKnow.com which it has said it would be prepared to send to an individual directly.
Mantaining and Expanding The List of Authorities
WhatDoTheyKnow lists around 3,000 public authorities, there is a regular turnover of changes in contact details. Our coverage, while large, is not comprehensive so we have requests to add bodies such as parish councils, schools, and doctors surgeries which we have not yet attempted to add in a systemic manner based on official sources of information.
We have also had to carefully consider what we do when for handling the various situations where an authority becomes defunct and its responsibilities are taken over by another body for example as a result of reorganisations of local government and the creation and merging of government departments.
Providing Advice and Assistance
The team at WhatDoTheyKnow.com often provide advice to users. We encourage users to keep their requests focused so as to reduce the chance of any problems due to libel or requests being classed vexatious. On occasion we suggest appropriate authorities for users to direct requests to, provide advice to those unhappy with the response to their request, and answer a broad range of other queries as they arise such as if particular bodies are subject to the act or not. Increasingly we link to authority’s publication schemes which are intended to let people know what information an authority has and how it can be accessed.
Spam
Lastly, like all websites which allow people to post content online WhatDoTheyKnow.com occasionally suffers from spam in various forms. Most is dealt with automatically but some has to be removed by hand. With spam, like the other aspects of running the site, the site’s code and processes are constantly being developed and improved to reduce the fraction of cases requiring any manual intervention.
This article was prompted in part by a team in New Zealand considering launching their own version on the site asking us what’s involved.
UPDATE: If you approve of what we did this week, and what to help make sure we can still do it in the future, please pledge to support us: http://www.pledgebank.com/supportmysociety
The vote on concealing MPs’ expenses has been cancelled by the government!
In other words – we won!
This is a huge victory not just for transparency, it’s a bellwether for a change in the way politics works. There’s no such thing as a good day to bury bad news any more, the Internet has seen to that.
Over 7000 people joined a Facebook group, they sent thousands of emails to over 90% of all MPs. Hundreds of thousands of people found out about the story by visiting TheyWorkForYou to find something they wanted to know, reading an email alert, or simply discovered what was going on whilst checking their Facebook or Twitter pages. Almost all of this happened, from nowhere, within 48 hours, putting enough pressure on Parliament to force change.
Make no mistake. This is new, and it reflects the fact that the Internet generation expects information to be made available, and they expect to be able to make up their own minds, not be spoon fed the views of others. This campaign was always about more than receipts, it was about changing the direction of travel, away from secrecy and towards openness.
Today we stopped moving in the wrong direction. Tomorrow we start moving the right way. Sign up to our news mailing list (box on the right) to get updates on what mySociety gets up to.
A number of non-departmental government bodies / quangos have been named as being up for abolition, merger, privatisation or absorption into parent departments, as part of the Coalition Government’s Spending Review, due this autumn. This has been widely dubbed in the press as a “bonfire of the quangos“. The list of quangos up for review is still being compiled by the government, and there have been a number of clarifications, amendments and retractions as further details come to light.
The Telegraph has obtained and published today a leaked list of 177 quangos up for abolition, plus a further 200 that are still being reviewed.
This is a great opportunity to highlight that mySociety’s Freedom of Information site WhatDoTheyKnow covers nearly all of these little-known bodies that spend public money (we currently have just over 3,800 public authorities listed on the site). Given their impending doom, there is little time left to find out what they spent public funds on, as only their most important records will be transferred to the National Archives or successor bodies for permanent storage. The remainder are likely to be shredded, or deleted, as only “records identified as valuable for future administrative need” are kept.
You can see our annotated list of the Telegraph’s list here - our volunteers have added links to most of the bodies’ pages on WhatDoTheyKnow, so you can more easily make your final FOI requests to them…
Please send any missing contact details to the WhatDoTheyKnow team.
mySociety has been asked to do some work on an online petition system for Number 10 Downing street. mySociety is delighted to have been asked to bring our experience to the task, and aims to develop a new tool as easy to use, popular and trustworthy as our other non-partisan democractic web sites. In particular we are pleased to have jointly designed a system in which moderation is extremely transparent, and which we hope can be used as an exemplar of good practice elsewhere.
Francis and Matthew have just started coding, and we plan to have a public beta running later in the year. All the code that is generated by mySociety (and which isn’t specific to the No10 site, like logos) will of course be available under open source licences, and we will be delighted to talk to other organisations who are interested in petition systems themselves.
We sometimes have incorrect or out of date addresses for sending Freedom of Information requests to. Now anyone can check our addresses. Click “view FOI email address” on the page for any authority, and enter two of those squiggly words to prove you are not a robot.
If you are using WhatDoTheyKnow, and suspect problems with a request, please do check the address we are using is correct. If you are from an authority, or work closely or know a particular authority, please also check the address.
Earlier today the Department for Education, which is headed by Education Secretary Michael Gove, wrote to WhatDoTheyKnow to let us know that the main email address they use to receive FOI requests is to be phased out. They would prefer the public to make their FOI requests via the contact form on their own website instead or even by post. We believe that this approach is contrary to the spirit of the law and principles of Freedom of Information.
The message we received stated:
We changed the way that people contact our department last year, encouraging customers to go to our website to find what they are looking for and submit an enquiry via our contact us page (www.education.gov.uk/contactus) if they could not locate information.
The [main FOI] mailbox that your system points to ([email]) will eventually be phased out and I would be grateful if you could advise customers using your website to refer to www.education.gov.uk/contactus if they need to contact the Department.
We certainly agree that people should check whether the information they are looking for is already available before submitting a FOI request — and indeed we already prompt all users of WhatDoTheyKnow to do so, not just for the Department of Education, but for every public authority we list.
When requests are submitted through WhatDoTheyKnow responses are automatically published ensuring a lot more information ends up online and publicly accessible than when submitted privately. If the Department for Education wants to reduce the amount of correspondence it gets in relation to already published material it should be encouraging people to make their FOI requests via WhatDoTheyKnow. Already, based on Ministry of Justice statistics, we calculate around 10% of all Freedom of Information requests to the Department of Education are made via our service.
We have asked the department to let us know which alternative email address they would prefer us to forward FOI requests to, and we await their reply. We are happy to use whichever email address is easiest for a public body.
We have also made clear that we will continue to offer our users the ability to make requests to the Department of Education via our site and will not be removing that facility and directing people to the department’s contact form as we were asked. Forms often include unnecessary mandatory fields that the FOI legislation does not require (in the DfE’s case they ask what kind of a requester you are, making you specifically type in “prefer not to say” into an “Other” box if you want to opt out).
The law rejects the idea that public bodies are allowed to erect artificial barriers like this, and we have noted that a FOI request is valid regardless of which email address or member of staff within an organisation it is sent to.
Over a thousand Ebbsfleet United supporters have used PledgeBank to raise enough money to buy a striker Michael Gash from Cambridge City.
This is an excellent example of why you should never pre-determine exactly how people are going to use your site!
On Twitter about 15 minutes ago, @greenerleith asked: “Has anyone worked out how to display the most recent #fixmystreet reports on a local map widget that can be embedded? #hyperlocal”
Like this? :)
It’s very simple to do:
The latest reports from FixMyStreet, superimposed on a Google Map, embedded in your blog. Hope that’s helpful.
Last week at the SustainIT eWell-Being Awards, we picked up an award for FixMyStreet. The judges said it was “[a]n excellent example of an independent website which empowers the general public in their dealings with their local council. It is a relatively simple application, yet highly effective and replicable.” One example the accompanying Independent supplement mentioned was “a community in Great Yarmouth which joined forces through FixMyStreet to clear their local unused railway track. The site made possible a dialogue between community members and the council’s community development worker, who organised a “clear up” day where locals could get involved with rectifying the situation, with tools, insurance and even a barbeque provided.” It’s great to see that sort of thing happening on the site, and also great to be recognised in this way.
In a spirit of celebration (though more to celebrate the endorsements the campaign has received), TheyWorkForYou now covers the Scottish Parliament – see the TheyWorkForYou news for more information.
You may already be aware of our website FixMyStreet.com, which helps you report common street problems – such as potholes and uneven pavements – to the relevant local council. This year, we thought we’d give people a gentle nudge before winter comes.
Many of the 1,000 issues which the site deals with every week are of the sort which are far better seen to before the big freeze. Potholes only worsen with the frost, and no-one wants a dodgy streetlight once the long dark nights are here.
Let’s get our local communities as safe as they can be, before the cold weather hits.
Click on each thumbnail to be taken to the actual-size resource, then right click or ctrl+click to save a copy to your hard drive.
A4 sheet of fliers to print out:
Poster to print out:
Badge for your blog or website (165×165 pixels):
(If you’d like a larger image, feel free to save the one at the top of this post).
HTML for inserting the badge onto your site without downloading – just copy and paste the below into your HTML editor:
<a href=”http://www.mysociety.org/?p=4790″ title=”Find out more about Fix Before the Freeze from FixMyStreet.com”><Img alt=”Fix Before the Freeze – report those dangerous potholes and broken streetlights before winter hits” src=”http://www.mysociety.org/wp/wp-content/uploads/2011/11/fixbeforethefreezebadge165.gif”></a>
Image by John Blackbourn, used under the Creative Commons licence, with thanks
In the nine days our Fix Before the Freeze campaign has been running, there’s been a 47% increase in reports on FixMyStreet.com. Thank you to everyone who has spread the word or remembered to use the site to get something fixed.
As you may remember, the campaign encourages you to report problems such as broken streetlights or potholes before winter comes. It’s great to see this start to happen, and we hope you’ll experience the benefits once the cold weather takes grip. Hey, you might even find that the warm glow of community spirit cuts a few quid from your fuel bills…
Meanwhile, we’re sure there are still plenty of pavements, roads and amenities that could do with a patch-up before winter. So if there’s a gap on a notice board near you, don’t forget our print-outs and resources here. How about printing out a few and leaving them in your local library, cafe, or community centre?
Image by Waldo Pepper, used under the Creative Commons licence, with thanks.
FixMyStreet, is, as its name suggests, a system that reports street problems to the relevant local council. But at heart, it’s a problem-reporting system that could be adapted for a multitude of different uses.
For example, with just a few modifications, large institutions such as hospitals could use it for everyone – staff, visitors, patients – to report maintenance issues. Same for universities, especially those spread over large campuses. Supermarket chains could adapt it so that people could report abandoned trolleys – in fact we’ve been admiring an Aussie site that’s way ahead of us on that idea.
We’ve been enjoying thinking of new possible uses, from the practical to the frankly rather ridiculous, but we’re also keen to hear any ideas we might not have thought of. Is there an area in your life – personal or professional – that would be made much easier if you had an easy way to report it on-the-go? What challenges do you see, and why hasn’t it been done before? Ideas below, please.
Ever got a problem fixed by reporting it on FixMyStreet? Written to your representative via WriteToThem? Here’s an opportunity to pay the favour forward to someone stranded on a wet Wednesday by the non-arrival of the number seven bus.
Bus Stop by Myles Noton (cc)
We’ve reached the point in FixMyTransport development where we can start asking for your help. We need to fill in the information we’ll use to report people’s transport problems to the companies that run bus and train routes. If you have five minutes to spare, please spend them adding a contact email address or two for your local bus companies to this spreadsheet:
…then you can bask in the glory of a karmic balance restored*.
* Will also work if you accidentally ran over a kitten on your way to work this morning.
Image by PMC FirstPix
Just a week after WhatDoTheyKnow’s big, round number, FixMyStreet also passed a significant milestone.
200,000 reports have been sent through FixMyStreet since its launch in February 2007. It currently sends an average of 250+ messages about potholes, broken streetlights, and other problems to local authorities each day. So far this month, we’ve processed just over 5,000 reports.
Those reports are the work of over 87,000 people, 52% of whom had never before reported an issue to the council. That statistic is important to us: we aim to make it easy to access civic rights, especially for people doing so for the first time.
FixMyStreet.com is a site with a simple premise, and it hasn’t changed greatly since 2007 – though it is currently undergoing a facelift, bringing it more in line with today’s design expectations. Last year we introduced user accounts and zoomable maps, along with a few tweaks here and there.
Like other mySociety projects, FixMyStreet is, of course, built on open code, so that it can be replicated by anyone with a little technical knowledge. The FixMyStreet interface is already up and running in Norway, and soon, the Philippines will see trials of their own version – proving that the model can work in very different infrastructures. Meanwhile, the basic FixMyStreet concept has been replicated in Brazil, New Zealand, and South Korea. Here in the UK, some councils have bought FixMyStreet to embed into their own websites.
FixMyStreet sends reports to the council, and also publishes them online – so each report is read by many people. This simple system helps them find out more about their local community, and what the council are doing to get things fixed.
Uneven paving stones and malfunctioning pelican crossings may not be the stuff of high drama, but against expectations, FixMyStreet does make for fascinating reading sometimes. Take a look at this page if you’d like to see some of the more unusual reports. And if you’d like some insight into some of the issues our developers deal with, you might like to read Matthew Somerville’s solution to the dog poo problem. It’s all glamour at the cutting edge of FixMyStreet.
I’m very excited to announce that the iPhone app for FixMyStreet is now live and available for download on the App Store (link opens the App Store in iTunes). You can now record a problem when out and about with your iPhone, using its camera and GPS, ready for checking and submitting to the council. Hopefully people will find this useful! :)
We’ve had reports that our FixMyStreet iPhone app is crashing on iPhone 3.0, and so have withdrawn it from the App Store until we are able to find out what’s wrong and fix it. I’m afraid I don’t know when that will be, as it’s all rather busy at present – if anyone has the skills and would like to volunteer to help, the code is available and should just import into XCode. I can supply some crash logs too.
FixMyStreet has a lot of RSS feeds. There’s one for every one-tier council (170), one for every ward of every one-tier council (another 5,044), two for every two-tier (county and district) council (544), and two for every ward of every two-tier council (20,296) – two per two-tier council because you might want either problems reported to one council of a two-tier set-up in particular, or all reports within the council’s boundary.
Then there’s an RSS feed every 162m across Great Britain in a big grid, returning all reports within a radius of that point, the radius by default being automatically determined by that point’s population density, but customisable to any distance if preferred. That’s, at a very rough approximation assuming Great Britain is a rectangle around its extremities, which it’s not, 19 million RSS feeds, lots of which will admittedly be very similar. :)
Every single one of those feeds can be subscribed to by email instead if that’s preferable to you, and are all accessible through a simple interface at http://www.fixmystreet.com/alert.
However, none of these RSS feeds was suitable for the person who emailed from a Neighbourhood Watch site and said that all they had was a postcode and they wanted to display a feed of reports from FixMyStreet. Given you could obviously look up a FixMyStreet map by postcode, it did seem odd that I hadn’t used the same code for the RSS feeds. Shortly thereafter, this anomaly was fixed, and if you now go to a URL of the form http://www.fixmystreet.com/rss/pc/postcode you will be redirected to the appropriate local reports feed for that postcode (I could say that adds another 1.7 million RSS feeds to our lot, but given they’re only redirects, that’s not strictly true). And after a couple more emails, I also added pubDate fields to the feeds which should make displaying in date order easier.
It’s great to see our RSS feeds being used by other sites – other examples I’ve recently come across include Brent Council integrating FixMyStreet into their mapping portal (select Streets, then FixMyStreet), or the Albert Square and St Stephen’s Association listing the most recent Stockwell problems in their blog sidebar. If you’ve seen any notable examples, do leave them in the comments.
Residents turn to web in lane fight describes a set of problems reported on FixMyStreet.
A NARROW mountain lane has been damaged and turned into an “international playground” for 4x4s and satnav-guided lorries, angry villagers claimed yesterday.
The interesting thing is the claim that making the reports public really helped pressurise the authority into fixing them
one [resident] claimed the local authority had now been “embarrassed” into action by the complaints on fixmystreet.com
Delayed trains by Glenn Scott, used under the Creative Commons licence, with thanks.
It’s been a while since we updated you on the progress of our next major project, FixMyTransport, but we’re still working hard behind the scenes. As you may recall, FixMyTransport will deal with public transport problems – delayed trains, vandalised stations, overcrowded buses, you name it. It’ll put problems in the public arena, while also reporting them directly to the relevant transport operator. Read more about the project here.
Things are going to get exciting very soon. As launch date approaches, we’ll be starting a closed beta (mid July), rapidly followed by a full open public beta launch (end of July). During the closed beta we want to get as much feedback as possible from future users of the site, as well as pressure groups, transport operators, and anyone else who has anything valuable to contribute.
If you would like to be invited to beta test, and weren’t one of our alpha testers, please email us on team@fixmytransport.com. Alpha testers will, of course, be invited to test again.
We got extremely useful feedback from our alpha testers, and a wealth of crowdsourced data from our community. Thanks to their efforts we now have contact details for the operators of about 50% of the routes in the UK. However, this leaves a lot of operators where we don’t know how to get in touch.
We really need your help to get them! If you can spare a few minutes, visit our spreadsheet and see if you can fill in any of the missing details.
The more contact details we can get hold of, the better experience FixMyTransport will offer to our users. As well as publishing passengers’ reports on the site, FixMyTransport sends them directly to the operators too, helping to get the issue fixed.
So, we especially need the email addresses for operators’ customer services departments. Finding these may be as simple as visiting the operators’ websites, or it may require a bit of sleuth-work on your part. If advanced Googling gets you nowhere, we’ve found that simply phoning head office can get results.
Incidentally, the main operators are near the top of the sheet – those are the ones that will benefit the most users, although obviously the nearer completion we get, the better.
You’ll notice that the spreadsheet now includes a non-obligatory column for your name: this is to offer a small incentive. If you want to, tag your entries and at the end we’ll be offering goodies to the top contributors. Depending on your preference, this might be one of our highly sought-after mySociety hooded tops (they’re snuggly!), or a chance to become more involved in the project.
Those who helped in the first iteration, please note that although this sheet looks different, your details have been retained and indeed have been extremely useful as we build the site. Also - if you have already been a major contributor during our previous rounds of testing and data collecting, please holler so that we can give you proper credit.
Not long now… we hope you’re as excited as we are.
Everyone at mySociety is quite bubbling with excitement at the news that we’re today officially launching FixMyTransport.com , mySociety’s first new core charitable website since WhatDoTheyKnow launched in 2008. We’ve never before launched a site that took so much work to build, or that contained so much data.
What is it for?
FixMyTransport has two goals – one in your face, and the other more subtle.
The first goal, as the site’s name suggests, is to help people get common public transport problems resolved. We’re talking broken ticket machines, gates that should be open and stations without stair-free access. We’ll help by dramatically lowering the barrier to working out who’s responsible, and getting a problem report sent to them – a task that would have been impossible without the help of volunteers who gathered a huge number of operator email addresses for us. Consequently the service works everywhere in Great Britain, our database has over 300,000 stops and routes for train, tube, tram, bus, coach and ferry.
The second goal – the subtle one – is to see if it is possible to use the internet to coax non-activist, non-political people into their first taste of micro-activism. Whilst the site intentionally doesn’t contain any language about campaigning or democracy, we encourage and provide tools to facilitate the gathering of supporters, the emailing of local media, the posting of photos of problems, and the general application of pressure where it is needed. We also make problem reports and correspondence between operators and users public, which we have frequently seen create positive pressure when used on sister sites FixMyStreet and WhatDoTheyKnow.
Who made it?
FixMyTransport was largely built by one remarkable coder – Louise Crow, who started as a volunteer and who is now one of our longest serving core developers. She spent 18 months coding the site almost entirely by herself, wrestling with truly tortuous data problems and collaborating with Birmingham’s fantastic SuperCool design to make it look lovely (you should hire them, they’re great). She also tolerated my ‘aspirational scattergun’ school of project management with remarkable good humour. She really is the king of transport coding.
Credit must also go to mySociety core dev Dave Whiteland, who made the Facebook integration work, despite not having an account himself!
Why is it dedicated to Angie Martin?
Angie Martin was a mySociety coder for an all-too-brief period before she succumbed to cancer at a devastatingly early age. We’re dedicating this site to her in remembrance of a great, self taught perl monger who should still be here.
We’ll be posting further blog posts about the development process, the data challenges, and the overall project philosophy. In the mean time, please keep arms and legs inside the carriage – FixMyTransport is just about to depart.
Image by Yi-tao 'Timo' Lee, used with thanks.
Just like the famous Yellow Pages ad, FixMyTransport is not only there for the nasty things in life.
Quite late in FixMyTransport’s development, we realised that praise is a form of feedback and that this channel ought to be open to our users, too – that’s why on every page for a stop or a route, you’ll see a stonking great button inviting you to ‘say something nice’ – here’s an example: a station manager giving the type of good service that gladdens the heart.
There’s also a little love for Pulborough station, nice words for Brighton’s well-informed station staff, an example of a very accommodating bus driver, and a comment that makes me want to take the 37 bus to Clapham Junction right now.
We haven’t really stressed this side of FixMyTransport, so we’re really glad to see people finding it for themselves and putting it to good use. If you encounter particularly excellent service on your next journey, and you want to spread the love too, just remember the big blue button. We’re sure the operators will be very glad to hear from you.
Statistics were recently released on the performance of UK central government departments with respect to their handling of freedom of information requests. The latest figures are for the second quarter of 2009. We have been able to use these to calculate the fraction of all requests which are made via mySociety’s freedom of information website WhatDoTheyKnow.com.
What these statistics mean is that an ever increasing fraction of the information released in response to freedom of information requests is being archived and made publicly available by WhatDoTheyKnow.com. Hopefully this will reduce the number of duplicate requests being submitted and ensure the information released is made available to the widest possible audience which in-turn should increase the chances it is acted on.
Only forty-three central government bodies have their freedom of information performance monitored centrally. This is a tiny fraction of the three thousand or so bodies currently listed by WhatDoTheyKnow.
Super WhatDoTheyKnow volunteer John Cross has made an interesting petition about Freedom of Information and publicly owned companies
“We the undersigned petition the Prime Minister to support a change to the law to make companies owned two thirds or more by public authorities subject to the Freedom of Information Act 2000.”
The petition goes on to explain (in more details at the bottom right of the petition page) that the situation is quite comical at the moment. If a company is owned by one local authority, then it is subject to FOI, but if it is jointly owned by two then it isn’t. This makes little sense, and it is also very important, as private companies owned by authorities often do important work.
On Saturday John Cross and Richard Taylor, two volunteers who work on mySociety’s freedom of information website WhatDoTheyKnow.com, gave a workshop on FOI to a meeting of activists from Republic, an organisation which campaigns for an elected head of state in the UK.
mySociety and WhatDoTheyKnow are non-partisan and don’t get involved in campaigning except in specific areas relating to openness and transparency. That said, members of the WhatDoTheyKnow team are be happy to consider invitations from any groups wishing to hold a workshop discussing freedom of information.
Many of those present at Saturday’s event were active campaigners on a wide range of subjects ranging from human rights to fair trade as well as having an interest in constitutional reform. The FOI workshop was oversubscribed with the majority of those present at the event deciding to attend the session. Unlike a previous workshop held at OpenTech where most attendees had made an FOI request themselves prior to the event, at this workshop all but one had not done so.
The Royals and FOI
Given the audience, the status of the royals with respect to FOI was particularly pertinent. The FOI act exempts information if it relates to: “communications with Her Majesty, with other members of the Royal Family or with the Royal Household, or the conferring by the Crown of any honour or dignity”. This exemption does not apply though if it is determined that it is in the public interest for the information to be released. The requirement for this public interest test is under threat as the Prime Minister has been moving to strengthen the restrictions on releasing information related to the Royal family. On the 10th of June 2009 in a speech to Parliament on Constitutional Renewal Gordon Brown said:
…we have considered the need to strengthen protection for particularly sensitive material, and there will be protection of royal family and Cabinet papers as part of strictly limited exemptions.
Following that speech BBC journalist Martin Rosenbaum obtained a statement from the Ministry of Justice clarifying that in practice what Gordon Brown’s words meant was:
… the relevant exemption in the Freedom of Information Act will be made absolute for information relating to communications with the Royal Household that is less than 20 years’ old.
In FOI jargon an “absolute exemption” is one not subject to a public interest test.
Even with the law as it stands it is not easy to obtain information on how the royals are, or are attempting to, influence government. For example John Cross has asked the Ministry of Justice to supply him with copies of correspondence they had received from the Queen and Prince of Wales. They rejected his request on the grounds that the public interest in non-disclosure exceeded the public interested in disclosure; as well as suggesting exemptions relating to “information provided in confidence” and “personal information” also applied.
The Royal Household’s position on FOI
The Royal Household is not subject to the freedom of information act; though it has made a statement on the subject saying:
Despite its exemption from the FOI Acts, the Royal Household’s policy is to provide information as freely as possible in other areas, and to account openly for its use of public money.
WhatDoTheyKnow’s policy is to include such organisations which have indicated they are willing to voluntarily comply with the act to the site. While we list The Royal Household, at the time of writing no-one has yet used the facility to request information.
Using WhatDoTheyKnow for Campaigning
While we stress the importance of keeping freedom of information requests focused, FOI is a powerful tool for campaigners. We were asked if it would be possible for a group like Republic to set up an account on WhatDoTheyKnow for their campaign? The answer to this is: “Yes! – WhatDoTheyKnow wants to encourage groups to use the site”. The information commissioner has confirmed that it is acceptable to use the name of a “corporate body” when making a FOI request, that’s a broad term which encompasses many organisations, groups and charities.
Republic themselves use FOI extensively and often generate major national news stories as a result of responses to their requests. They want to be able to either offer journalists exclusive stories or write a press release based on information released. They can’t do this if the story gets out first via WhatDoTheyKnow so would be interested in an ability to make requests initially in private. mySociety and WhatDoTheyKnow have been considering an option for journalists to be able to make hidden requests via the site. Such a feature could potentially generate an income stream for the site as well as encourage a greater proportion of FOI requests to be made via it. Once the article had been published then the FOI correspondence could be opened up to the public providing access to the source material backing up the story.
As well as meeting those who use, or might want to use, the site to make requests WhatDoTheyKnow also wants to engage positively with public authorities; we see them as important users of our service too. Developer Francis Irving represented the site at the FOI Live conference for information professionals in June and will be speaking at the Freedom of Information Scotland conference in December.
This week has been quite bitty. I’ve been doing more work on the Freedom of Information site, have been getting into the swing of Ruby on Rails. Once you’ve learnt its conventions, it is quite (but not super) nice.
As far as languages are concerned, Ruby seems identical in all interesting respects to Python. It’s like learning Spanish and Italian. Both are super languages. Ruby has nice conventions like exclamation marks at the end of function names to indicate they alter the object, rather than return the value (e.g. .reverse!). But then Python has a cleaner syntax for function parameters. It is swings and roundabouts.
Rails has lots of ways of doing things which we already have our own ways of doing for other sites. The advantage of relearning them, is that other people know them too. So Louise was able to easily download and run the FOI site, and make some patches to it. Which would have been much harder if it was done like our other sites. Making development easier is vital – for a long time I’ve wanted a web-based cleverly forking web application development wiki. But while I dream about that, Rails packaging everything you need to run the app in a standard way in one directory that quite a few people know how to use, helps.
Other things… I’ve been helping Richard set up GroupsNearYou on our live servers, it should be ready for you to play with soon. It looks super nice, and is easy to use. I’ve had some work to do with recruitment. And catching up on general customer support email for TheyWorkForYou and PledgeBank. I’ve also been updating the systems administration documentation on our internal wiki, so others can work out how to run our servers.
I haven’t posted here for a while – in late May / early June I was on holiday in China for a month. After 18 months of hectic mySociety with only a break for a week, I needed it!
Since I got back, I’ve been busy on the no. 10 petitions site, which in the tag-team coding way that we do, Matthew has now taken over for a bit. And I’m doing something I’m hesitatingly calling “geocascading pledges”.
I’m not quite sure what to call it yet. It lets you make one pledge, but whose target counts separately in different towns. When someone signs up, they also choose a place. Rather than having to make a separate pledge for everywhere in the world, you can just make one.
Here’s an example pledge, which is also real (more on that later), so do sign it if you like. And then post your thoughts in the comments – it needs a bit of polishing yet.
On Wednesday this week, mySociety’s Tom and Paul were in Southampton, competing in the Geovation finals.
Geovation is an initiative coordinated by Ordnance Survey which gives out funding to projects that help “communities address their unmet needs through the application of geographic data, skills and expertise”. When we discovered that the theme this time was “How can we improve transport in Britain?” we knew we had to enter.
As many of you will know, mySociety has been working for some time on FixMyTransport, a project for reporting problems with public transport. Taking much of what we’ve learned from FixMyStreet, we are, in the trademark mySociety way, building a website that will make the process easy, whilst hiding all the complexities out of sight.
FixMyTransport is well under way, and we’re hoping to launch shortly. But with Geovation funding, we hoped to be able to roll out an accompanying mobile application.
This is incredibly important because, after all, the best time to make a transport report is immediately you experience the problem.
mySociety has, of course, always been into maps and geodata – we use them in what we hope are fun and innovative ways across many of our sites, including (obviously) Mapumental, and (less obviously) TheyWorkForYou and WriteToThem. We’re also rather fond of public transport.
We also really enjoyed meeting the other contestants, particularly Cyclestreets whose project looks like it will be one to watch.
At the end of the day, we were delighted to learn that we had been awarded £27,000 to develop a simple, intuitive, cross-platform mobile application for FixMyTransport. We can’t wait to get started. We really believe it’s going to be of real benefit to public transport users across the UK (and possibly further, given the open-source nature of all our work).
If you’d like to stay up to date with FixMyTransport as we build and launch it, you might want to be one of the very first to “like” our Facebook page or follow us on Twitter.
So, I’ve just had a shower and I’m waiting for Matthew and Tom to turn up. As time goes on, mySociety seems to get more geographically disparate, and I look forward to meeting my coworkers. Then for 1pm we’ll be heading to CB2 for the mySociety developers meeting. Feel free to come along any time afternoon or evening, whatever your skills or interest in mySociety.
I haven’t posted on here for ages, since October. I’ve been away on holiday quite a lot, and when I’ve not been away I’ve been busy, partly with systems administration. We’ve set up lots of servers in the last month for the E Petitions site. When you go from 3 servers to 7 servers, there’s another step change in sorting out systems administration tools. For example, I had to change the monitoring script so every server wouldn’t monitor every other. And I had to work out the quirks and bugs in the system we have for storing config files for different classes of server in CVS. Because we only had one class of server before.
I’ve also had to learn lots about server monitoring and load balancing. Things have slowed down a bit now, to maybe 10 hits per second. But a few weeks ago the road pricing petition was often getting 50 hits per second. I’ve never worked on a site with that level of traffic before. You find all the bugs in your code, all the missing indices in PostgreSQL, all the badly tweaked FastCGI parameters. I’ve been sucking knowledge off Chris like a sponge, so tools like strace and vmstat begin to become instinctive. Seemingly nobody offers a book or a course which teaches this stuff well – every server setup is different, everyone knows different ways to tune and profile. But maybe you can tell me different in the comments.
Louise has been busily working away on lots of things. Amongst that is a major change to WriteToThem, to let you write to all the members in a multi-member constituency in one go. The last day or two, she’s been installing the WriteToThem test code on one of our servers, when it has only run on my laptop before. This will be fantastic – hopefully can get Matthew to be bolder about making changes to WriteToThem, if he has a test script he can easily run (getting Matthew to be bold isn’t normally a problem, but he seems mildly less bold when it comes to the WriteToThem queue daemon).
Tom and I have also been busy on a second travel maps report for the DfT. More on that soon. Lots of running screen scraping jobs on TransportDirect which take days. On the subject of Tom, he seems to have got expert at “stacking meetings” next to each other. In one day last week he had 7 meetings!
The Hansard Society have just published a report entitled MPs Online: Connecting with Constituents. I’m only going to talk about one part, the part that mentions the mySociety project WriteToThem in a section on MPs’ use of email.
We’re surprised and disappointed to see our methodology for collecting data on how well MPs respond to constituency mail being called “unreliable”, especially from a paper that makes a number of simple mistakes of its own in just a few lines on one page.
The Hansard Society, to the best of my knowledge, never got in touch with us to request any clarification or ask about our data or methodology, which we would have been more than happy to supply.
Today we have a strange story about a department that appears to think that it has a duty not to release information under FOI if it makes people angry.
It all starts in January 2009 the Department for Children, Schools and Families (DCSF) appointed an expert by the name of Graham Badman to conduct a review of elective home education in England. It probably goes without saying that this is an issue far from our concerns, and an issue that mySociety has no views on – what makes us interested is the process that followed.
Shortly after the publication of the report, Elaine Walton, a user of mySociety’s freedom of information website WhatDoTheyKnow.com requested copies of communications between the Department for Children, Schools and Families and Nektus Ltd. the company through which it appears Mr Badman was paid for his work.
According to email replies to Ms Walton, the DCSF located two relevant invoices which show how much money was paid, but refused to disclose them. Strangely, though, they were not refused on grounds of commercial confidentiality, but rather on something more unusual. Here are the exemptions they cited:
Health and Safety? A little investigation reveals more.
When Ms Walton appealed against this decision, an internal review was carried out within the DSCF. The internal review’s findings stated that Mr Badman was likely become a victim of harassment if certain personal details were made public, hence a health and safety concern, and hence no publication of these invoices. Fair enough – nobody would be in favour of revealing private, sensitive information that would endanger anyone’s life or family, especially in the presence of a known threat. But take a look at this:
“That the Department had initially been drafting a response that included the release of invoices with only personal data redacted. But before the draft was complete it was apparent that there was a campaign of harassment and vilification against Graham Badman and other individuals/organisations that had contributed to the Report. In the light of this, at the weekly review meeting of FOI cases, it was considered that the balance of public interest might have shifted towards withholding.”
What is very curious here is the admission that the department had been thinking of releasing the invoices with personal data hidden (ie no home address, bank details etc). But then because of a campaign of harassment, it was decided that they wouldn’t publish anything at all. So not just no personal information, but no dates, no amounts of money, nothing.
What is so unease-making about this FOI decision is that it appears to be saying that departments may conceal information on how much public money has been spent on something because releasing that information will make some angry people even angrier. Surely this can’t be right – if it were every budget would be conducted in complete secrecy. We would encourage the Information Commissioner’s Office to take a look.
A few Christmases ago Chris Lightfoot and Etienne Pollard built a little throwaway website called HassleMe.co.uk as a one-day coding challenge. The purposes of HassleMe was to let people nag themselves into doing things that needed doing by arranging to be sent automated emails.
The genius of HassleMe lay in the timing of the responses. Instead of it mailing users exactly 5 days or 2 weeks after they subscribed, it mails people approximately when they ask. As a consequence you never know when the mail’s going to arrive, making it strangely more effective as a nag.
After Chris died the site sat running on his server for a couple of years, and mySociety didn’t have access to it. Last week Francis migrated the site to our own machines with the help of Pete Stevens from Mythic Beasts. When Francis looked at the database he saw that there were 16,000 hassles that users had agreed could be made public. However, the feature to show them was never built.
Now it has been added on this page, and makes for a fascinating, often foul mouthed insight into what it is that people need to motivate themselves to do. There are thousands of examples, each stranger than the next, so just hit refresh for more. And please do paste and favourites as comments to this post.
PS And as always, if you like any of mySociety’s services, or even just find the comments amusing, please donate – we’re a charity!
Members of the team running mySociety’s freedom of information website WhatDoTheyKnow.com also campaign for improvements to freedom of information law.
Volunteer John Cross has been drawing his MP’s attention to a major loophole in the UK’s Freedom of Information Act which means that a company wholly owned by one local authority is subject to the act but a company owned by two local authorities is not. John’s MP, Peter Bottomley, has been convinced that this anomaly in the law does not make sense and has submitted an Early Day Motion calling for the loophole to be closed. The EDM also highlights the fact that currently a company owned 95% or even 99.5% by a single public authority is not subject to the provisions of the act, as only companies owned 100% by a single authority are currently covered.
The text of the motion states:
“That this House notes that section 6 of the Freedom of Information Act 2000 with certain exceptions makes companies wholly owned by the Crown or by a single public authority subject to the Act; further notes that a company wholly owned by two or more public authorities or 95 per cent. owned by a single public authority will be outside the scope of the Freedom of Information Act 2000; and calls for the closure of this loophole and for companies owned 90 per cent. or more by any number of public authorities to be subject to the Freedom of Information Act 2000.”
The motion is currently open for other MPs to sign-up to and show their support. If you would like to help increase the number of public bodies that are covered by Freedom of Information legislation please consider writing to your MP, asking them to add their name to the current signatories.
There are many situations where public authorities working together or even setting up jointly owned companies is commendable. Such arrangements can lead to savings though economies of scale and avoid duplication; we may see more such companies set up as a response to economic pressures. What is problematic though is the loss of accountability which currently occurs when public bodies come together and set up these companies without requiring them to follow the highest standards of openness, transparency and accountability.
Examples of Companies Which Would Be Made Subject to FOI if the Loophole was Closed:
Many housing associations, purchasing consortia, representative bodies and urban development companies are among the organisations which would be required to operate in a more transparent manner should this loophole be closed.
Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?
I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. :-)
Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.
The highlighting works the other way – as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! :) ).
Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.
Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse :-)
As you may know, TheyWorkForYou are conducting a survey of candidates for Parliament. You can view the questions for your constituency on this list.
Quite a few people have been asking how we worked out the questions. There are two parts to this, one local and one national.
Local questions
We used the power of volunteers.
Thousands of DemocracyClub members were asked to suggest local issues in there area. These were then edited by other volunteers, to have consistent grammar, and be worded as statements to agree/disagree with, and filtered to remove national issues. The full criteria and examples are available.
You can view the issues for any constituency on the DemocracyClub site. They are in the “local questions” tab. e.g. For Liverpool, Riverside (where I live) you would go here.
We’ve ended up with local issues for about 85% of constituencies. They’re really interesting and high quality, and quite unique for a national survey.
Thank you to all the volunteers who helped make this happen!
National questions
This was hard, because we felt that asking more than 15 questions would make the survey too long. We also wanted to be sure it was non-partisan.
We convened a panel of judges, either from mySociety/Democracy Club or with professional experience in policy, and from across the political spectrum. They were:
They met at the offices of the Institute for Government, and had a 3 hour judging session on 29th March 2010. They were asked to think of 8-15 questions, with multiple choice answers, which could usefully be answered both by members of the public and prospective candidates for national office.
To ensure maximum transparency, the discussions of the judges were recorded. You can download the recordings in two parts: part 1, part 2 (2 hours, 20 mins total).
Details of the broad framework the judges operated under are given by the chair of judges, James Crabtree, a trustee of mySociety, in the opening to the recordings.
Please do ask any questions in the comments below.
Here is a diagram of how the backend of Mapumental works. Take it in the spirit that Chris Lightfoot set when he made a similar diagram for the No. 10 petitions site – although many such diagrams are useless, hopefully this one contains useful information.
If you haven’t seen Mapumental yet, first take a look at the video, and sign up for the private beta.
(Click on the diagram for a large version)
Below, I’ve explained what the main components are, and some interesting things about them.
Everything can, at least in theory, run on lots of servers. Currently we are only actually using one server for web requests, because of problems with HAProxy. We’re runnning isodaemons on two different servers.
Basic web application – it started out as raw Python, but the more Matthew hacks on it the more Django libraries he pulls in. Soon it’ll be indistinguishable from a Django app. When someone enters a new postcode, it adds it to the work queue in the PostgreSQL database, then refreshes waiting for the job to be finished. Then it displays the flash application (made by Stamen), set up to load the appropriate tile layers.
Tile server and cache – This uses the Python-based TileCache, calling Geospatial Data Abstraction Library (GDAL) to help render the tiles from points. It was originally written by Stamen, and expanded by mySociety. GDAL isn’t perfect, it doesn’t have fancy enough algorithms for my liking. e.g. Using a median rather than a weighted mean.
Isodaemons – These are controlled by a Python script, but the bulk of the code is custom written in C++. Slightly crazily, this can find the quickest route by public transport for each of 300,000 journeys from every station in the UK to a particular station, arriving at a particular time, in 10 to 30 seconds.
I had no idea how to do this, but luckily I live in Cambridge, UK. It’s a city fit to bursting with computer scientists. Many of the jobs are dull, and need little computing, never mind science – like writing interface layers for SQL server. So if you have a real interesting problem it’s easy to get help!
The universal advice was to use Dijkstra’s algorithm, which needed a bit of adaptation to work efficiently over space-time, rather than just space. Normally it is used for planning routes round a map, but public transport isn’t like that, you have to arrive in time for each particular train, so time affects what journeys you can take.
I originally wrote it in Python, which was not only too slow, but used up far far too much RAM. It could never have loaded the whole dataset in. However, the old Python code is still run by the test script, to double check the C++ code against. It is also still used to make the binary timetable files, see below.
Travel times, 1 binary file / postcode – I briefly attempted to insert 300,000 rows into PostgreSQL for each postcode looked up, but it was obvious it wasn’t going to scale. Going back to basics, it now just saves the time taken to travel to each station in a simple binary file – two bytes for each station, 600k in total. The tile server then does random access lookups into that file, as it renders each tile. It only needs to look up the values for the stations it knows are on/near the tile.
There’s various other bits:
Thanks to everyone who helped make Mapumental, we couldn’t have done it without lots of clever people.
I realise the above is a sketchy overview, so please ask questions in the comments, and I’ll do my best to answer them.
So, there are now over 600 petitions in the petitions system, and we’re getting a steady stream of appeals from our users to add categories.
I’m posting to ask how you all think we should handle this. It seems to me that there are a few options:
More than just thinking about the overall philsophy I’d also appreciate thoughts on design. When you come to the homepage, how should the category system be presented to you? Tricky stuff, and I’d really appreciate your thoughts.
The two days leading up to election day are a hugely important time for less politically-obsessive voters. The parties know that a lot of people are only starting to seriously think how to vote today and tomorrow, and TheyWorkForYou saw its biggest spike ever the day before the election, way back in 2005.
This means it’s a super-important time to get trustworthy, non-partisan information in front of as many people as possible. And you can help by doing the following simple things:
1. Go to your constituency page on the TheyWorkForYou Election Quiz and take a good look at the answers. Is there anything surprising in the answers? Has anyone failed to respond who really shouldn’t? Is there anything funny in the responses? Make a couple of notes about what you think are the most interesting findings.
2. If you know the name of your local papers or radio stations, try to Google for the email or phone number of the news desk. If you don’t know the names, try sticking the name of your nearest town into a media database like this, to get a phone number or email address.
3. If possible, you should start your pitch by phoning rather than emailing. If you get a phone number for a news desk, give them a bell and say that you’re a volunteer from “The country’s largest non-partisan election information project”, and ask for the email of a specific person who might be interested in a story about what local candidates are saying.
4. Once you have an email address of a specific journalist, compose a locally specific email for them, along the following lines:
“Hi X,
I’m a resident of Z constituency, and this election I’ve been one of 6000 volunteers helping to build an unprecedented project to get candidates across the country to go on the record, in conjunction with the website TheyWorkForYou.com. It’s a strictly non-partisan project, aimed at giving voters a really clear, spin-free view of what their candidates stand for. I’d really appreciate it if you could give it some coverage before election day.
In my constituency, N candidates have completed our survey. From this we can see some quite interesting things, namely:
* Candidate A thinks…
* Candidate B thinks…
Would you be so kind as to print a story encouraging people to check our their candidates via TheyWorkForYou.com, and mentioning some of the highlights I’ve included?
all the best,
Your name, email, phone”
5. An hour after you send the email through, give the journalist a call back to see if they need any more help.
6. If you do this, please leave us a comment on this post so we know who’s had a go!
Thank you for helping spread some non-partisan information this election time, and enjoy the election…
Transit Distress by Eric Parker - used with thanks under the Creative Commons licence
FixMyTransport was launched a month ago. It is now well on its way to listing 1,000 individual complaints, suggestions and requests to the public transport operators of Great Britain.
As the sample size grows, we’re able to see just what provokes the country’s mild-mannered passengers into action. There are the diurnal irritations – the leaky station roof, constant announcements, smelly trains; there is the discomfort of overcrowding and overheated buses.
All of which are important, of course. And in this, FixMyTransport is achieving its aim of allowing people to make their reports to the operators, while at the same time creating small bunches of people who read those reports and think, ‘Hey, me too!’.
But FixMyTransport is not just for the little gripes. It’s uncovering some pretty big issues, too. Prime among these is the issue of accessibility: reports have come in of buses driving away rather than let a wheelchair user on; a disabled passenger who has surmised that it’s easier to invite friends to come to him rather than try to navigate London’s public transport system; a station from which those with restricted mobility can only travel in one direction.
The big question for us is, what happens now? Are these reports making anything better? In some cases, yes.
There is the train that will now stop at intermediate stations, and the pedestrian crossings that are no longer blocked by buses. Little things that’ll make a big difference to the people who reported them. But the pay-off is not always so immediate. Bigger issues are obviously not going to be fixed overnight. And some problems won’t be fixed, for a multitude of reasons – they don’t fit in with the operators’ plans, or they’re not budgeted for, or they just aren’t seen as sufficiently important.
How is FixMyTransport going to crack those? Well, it was set up so that you can show your operator that there is demand, that budgets need to be massaged, or that plans should change. If you’ve used the site, you may well have been on the receiving end of a comment from one of the team, nudging you to spread the word of your campaign, among friends, family, and fellow-passengers.
The fact that people can sign up to your page helps make FixMyTransport different from just contacting the operator directly. We also reckon it’ll make a difference when it comes to getting changes. Consider, for example, the campaign to get increased cycle parking at Cambridge station – with 176 supporters (and still growing every day), our biggest yet. It’s been picked up by local press, talked about on Twitter – and eventually, National Express won’t be able to ignore the public demonstration of a palpable need.
Well, that’s the plan. We know it’s early days, and that FixMyTransport represents a massive sea change for some operators who are not used to interacting openly and online.
If you’ve written an impassioned, well-reasoned request, gathered supporters and spread the word far and wide, and still hit a brick wall, we have other suggestions. FixMyTransport allows us to get you writing to your local councillor, to the local paper, or to relevant groups like Passenger Focus, Transport for All, and the Campaign for Better Transport. These groups have been bashing away at the big problems like accessibility for far longer than we have, and it makes sense to tap into their expertise.
We know that for some issues, it’ll be a long game – just as it’ll be a long game trying to get every operator fully signed-up to the notion of transparent online interaction. But we’ll keep trying, and we hope you will too.
In January 2007 the Information Commissioner’s Office disclosed via FOI 28 “line to take” (LTT) documents (pdf). These are topic-based documents, produced by their Policy Team, and they are used internally by their case workers as how they interpret and apply the various sections of the Freedom of Information Act 2000 and the Environmental Information Regulations 2004.
They are mostly based on Information Tribunal rulings which have set a legal precedent, or relevant ICO decision notices, but for some topics they are based on the ICO’s own interpretations of the FOI Act, indicating how they’re likely to handle the subject matter of these complaints.
For someone looking to appeal a particular FOI decision made by a public authority (eg a refusal to provide information due to an exemption in the FOI Act), these LTTs are a very useful addition to the FOI guidance already produced by the Ministry of Justice or the ICO. They can be used to verify whether the authority’s stance is appropriate, and allow the requester to point the authority toward how the ICO sees that particular topic, and therefore challenge a FOI decision.
Although I’m a volunteer for WhatDoTheyKnow, I occasionally make FOI requests in a personal capacity. As I only recently found the above LTTs from 2007, I therefore asked the ICO for an updated list of LTTs. These have now been provided, and there is now a big library of 177 LTTs. They sent them via 6 large PDF files, so I have made them available in a more structured, searchable and (hopefully) easier to use manner via FOIwiki.com which is maintained by some of the WDTK volunteers:
http://foiwiki.com/foiwiki/index.php/LTT
In addition to the master list of LTTs available via the link above, they can also be accessed in context with their associated FOI Act exemption page, eg Section 41: Information Provided In Confidence
Two items of note from the LTTs.
I hope you find them useful! All comments or suggestions welcomed.
TheyWorkForYou now finds whenever an old version of Hansard is referenced (which they do by date and column number, e.g. Official Report, 29 February 2008, column 1425) and turns the citation into a link to a search for the speeches in that column on that date. This only really became feasible when we moved server, upgraded Xapian, and added date and column number metadata (among others), allowing much more advanced and focussed searching – the advanced search form gives some ideas. Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech, much like our video matching (nearly 80% of our archive done!). :)
Kudos to Google and Yahoo! for spotting this change within a couple of days, as they’re now so busy crawling everything for changes that they’re slowing the whole website down… ;-)
Matthew and I have been sitting next to each other today looking at the outputs of his lovely new custom built conversion tracking system, designed to ensure that the optimal number of users who just come to one of our services as a one off get signed up to something else longer lasting.
I’ve been banging on for ages about how government should seize on cross selling people who’ve just finished using one online service into using another of a more democratic nature, so it seems worth spelling out some of the lessons.
First, there’s some interesting data from the last few weeks, since our newest conversion tracking infrastructure has been running in its nice new format.
One of the adverts randomly served to users of WriteToThem (after they’ve finished sending their letter) encourages them to sign up to TheyWorkForYou email alerts – the service people use to get emailed whenever their MP speaks in Parliament. The advert features a slogan of encouragement, and a pre-populated email form containing the user’s email, and a ‘Subscribe me’ button. This advert was shown to 2328 users last month, of whom 676 became TheyWorkForYou email subscribers, which is a pretty cool 29.04% conversion rate. However, we also showed another advert for the same service, to the same WriteToThem users, which also had the same button and text, but which hid the form (and their address). That was shown to 2216 users of whom 390 signed up, a more modest 17.6%. So the impact of simply showing an email box with the users email address in it, versus hiding it, was worth 10% more users. Why? Go figure!
So now we’ve canned the advert that hides the address form, and instead we’re comparing two different adverts both of which feature the pre-populated signup form, but which use different words. It’s probably too early to judge, but the new ad appears to have a very similar conversion rate suggesting it might be hard to squeeze many more subscribers out of this page. We’ll keep trying though!
Another thing we learned of interest was that monthly subscribers to email alerts on TheyWorkForYou were down year on year in the month before we added this new advertising & conversion tracking system, even though the total number of visitors were clearly up on the same month last year. This appears to suggest that two things are happening. First RSS is catching on, so some users who would previously have got email alerts are subscribing to RSS feeds instead. Second, it suggests that the TheyWorkForYou user audience might have been getting more saturated with regulars – proportionally fewer new users coming (although more visitors in absolute terms) so fewer people signing up to get alerts. The cross marketing and conversion tracking seems to have reversed that trend, which is awesome.
We also advertise several different services to people who just finish signing up to get email alerts on TheyWorkForYou itself. We’ve just noticed that a full 25% of people shown the advert to sign up for HearFromYourMP proceed to sign up. We’ve therefore just decided to dump other adverts shown on TheyWorkForYou (such as advertisements for other sorts of TheyWorkFor you email alert) and concentrate on just cross selling HearFromYourMP. A back of the envelope calculation suggests that by just advertising this one site from the completion page we should get an extra 10,000 subscribers to HearFromYourMP this year on top of the organic growth. Not bad for a few minutes analysis, and a number likely to make a fair few more MPs post messages to their patiently waiting constituents.
One last interesting thing (at least to me) is how some more demanding services are a much harder sell than others to users. So asking people to make new groups on GroupsNearYou.com or report a problem in a street on FixMyStreet tend to result in more traditional online marketing scale conversion rates of 0.1% to 2%. Still worth doing, and so we compare different versions of those ads too, to try and eke up those rates for these sites that arguably have more tangible, direct impacts on people and communities.
It will be a challenge for mySociety’s future to work out how to trade off impact against scale of service use – are 10 HearFromYourMP subscribers worth one pothole that doesn’t get fixed? Answers on a postcard…
One of the most popular features on TheyWorkForYou is the plain English, non-judgemental vote analyses on MP pages that say things like “voted strongly against introducing a smoking ban“. We call these ‘policies’, and they are authored by skilled people using the volunteer run PublicWhip website.
Making each one of these policies is a painstaking task, requiring good knowledge of how the Parliamentary voting system works, good writing skills, patience, and the strength of character not to let your own views about the issues cloud the analysis. It is of utmost importance to both mySociety and our users that these policies are fair and trustworthy.
Earlier this year we started to update the process by which we made new policies to make it even more rigorous, which we wrote about here. Marcus Fergusson and Stephen Young came onboard and did sterling work, but they have now moved on to greater things, and so we’re looking to recruit two to three new people to do this job. Uber volunteer Richard Taylor has been helping out recently, but this is really a job for two or more people.
You might very well ask ‘why two people, given the work is part time?’. The answer is that we really want every new policy to be cross-checked by two different people every time it is added or amended. This is to help eliminate possible mistakes, and prevent any unconscious biases.
We pay for this work on a piece work basis – £160 a time for a combination of one new policy authored, and one other policy double-checked. This money comes mainly from people making small donations, which I think helps keep everyone focussed on how important it is to get these right. We hope to add about two new policies a month, once the new team is up to speed.
If you’d like to be considered, please email hello@mysociety.org with ‘mspolicies’ in the subject line. Applications close 22nd November 2010.
PledgeBank is mySociety’s second project. The purpose of PledgeBank is to get people past a barrier which strikes down endless good plans before they can are carried out – the fear of acting alone. It allows anyone to say “I’ll do X if other people also do X”, for example “I’ll write to my councillor if 5 other people on my street do the same”. However, there is no scale to big or too small, it could equally be used to say “I’ll start recycling if 10,000 other people in Britain also start”.
Pledgebank developement will start soon, and we are now looking for partner organisations, large and small, who would like to use Pledgebank when it launches. If you run a charity or other organisations, or if you know anyone who does, please do email us.
Ian has used PledgeBank to start a residents association for a new block of flats in Cambridge, UK.
We’re going to be adding lots of features and major design improvements to our Freedom of Information site WhatDoTheyKnow.com in the next few weeks. One thing we want to add is little explanatory videos helping describe how to make the best request possible.
Today we’re launching the hastily named WhatDoTheyKnow Video Challenge. We want you to make short videos (max 2 mins) in which you explain in a clear and friendly way how to file successful FOI requests. We’re not expecting Hollywood production values, just a friendly face and a good explanation would do fine. If you can do funny, splendid.
Instructions:
1. Record your vid
2. Upload to your video hosting venue of choice
3. Post the link as a comment to this post
We’ll send out a coveted mySociety hoodie to anyone who makes anything we’d seriously consider using (unless bazillions of people enter, of course – do you want to bankrupt us?). We don’t sell our coveted mySociety hoodies, they’re only for people who’ve done something useful for mySociety so they’ll mark you out as a pillar of the community the first time you walk down the street.
It’s high time we updated you on Mapumental, our journey-time mapping project. For those who may not remember, Mapumental is based on a simple idea: to visualise transit times, by public transport, from or to any postcode in Great Britain.
It all began in 2006, when the Department for Transport approached us to see what we might do with public transport data; in 2009 we won an investment loan from Channel 4 and Screen West Midlands which enabled us to build a beta tool – you might have played with it. If not, go on, have a go. It’s fun!
It’s been quite a long journey to where we are today. Unlike many mySociety projects, funding for Mapumental’s development came from a commercial investment loan, with a condition that we set it up as a business. For that reason, it’s not enough that it’s beautiful and useful – we need to find ways for it to be profitable, too. All revenues are set to come back to fund our not-for-profit activities.
We could tell from very early on in the project that Mapumental would be a sought-after tool for all sorts of purposes, from business to personal use. For example, you can see commute times at a glance, so it’s great for house-hunters and job-seekers. Consequently, it’s also great for the property and recruitment industries.
“Your maps look amazing, such a great way of representing what could be really boring data, but isn’t.” – A jobseeker
We can see loads of other possibilities too – like urban planning. This sort of analysis would have been far more expensive in the past; with Mapumental, planners can see at a glance how accessible a new development would be by public transport. Its potential uses are wide-ranging, answering questions for businesses, organisations, charities, and public facilities – especially those wanting to maximise accessibility or encourage use of greener transport options.
“The maps are a fantastic, a great tool and should be used for every planning application. I will be using Mapumental for all of our projects!” – Lee Taylor, Veridis Design
We’ve recently refined a product that’s pared down from the dynamic maps you may remember from that beta tool: static maps. These are simple, non-interactive maps which show transit time in bands. They’re flexible in that they can be generated for any postcode, with any maximum travel time, and depict travel at any given time of day.
We can provide a one-off map for personal use, or batches of many thousands of maps – as we have done for estate agents Foxtons, who now have a Mapumental map on every property listing.
As we generate more and more maps for different uses, showing different parts of the country, we’re really enjoying digging out all sorts of surprising facts – like how it’s quicker to travel from Watford to Westminster than it is from some parts of Harringay. Or how Cardiff University students might sensibly live at all points east as far as Newport, but will be stymied for transport in the west if they live anywhere other than Barry or Bridgend.
In fact, our very favourite use so far has come from an individual who centred his map around his home postcode. He tells us he has printed it off and put it up by the front door, so that on his way out of the house, he can find a new and surprising destination for day-trips.
Find out more on the Mapumental website – and please do spread the word among friends and colleagues who might benefit from a Mapumental map.
Today’s Sunday Times carries an article on very high salaries paid to some of those working in the “publicly funded arts world”. The article reports Antonio Pappano, the Royal Opera House’s Music Director, is paid more than £630,000 a year and is given four months a year off to carry out a second job as music director of a Rome orchestra.
While the Sunday Times’ paywall means we don’t have a direct link to their article; it appears to be based on much the same information as an article published a few days earlier by The Arts Desk.
The Sunday Times article states the Government has “expressed surprise at the sums paid” and Ed Vaizey the Culture Minister is quoted as saying:
“There really must be full transparency for all publicly funded arts bodies”.
There is also a statement from the Arts Council expressing a similar, though more limited, sentiment:
“Anybody in receipt of significant public money should be transparent about their core funding costs”.
The Arts Council, the main body which distributes public funding to the arts, is subject to the Freedom of Information Act. The Arts Council is listed on mySociety’s Freedom of Information website WhatDoTheyKnow.com which enables people to easily make requests for information in public. While the Arts Council is responsible for handing out the money, it does not necessarily know the details of how the recipient organisations spend it. The bodies which receive funds are not themselves yet subject to freedom of information law, irrespective of how much public money they receive or how dependent they are on that subsidy.
While it may take the Minister some time to legislate to ensure “full transparency for all publicly funded arts bodies” we are happy to add such bodies to our site on request right now, so our users can ask them, in public, about their activities.
As of today the following organisations are now listed on our site:
We use the WhatDoTheyKnow site to actively campaign for expansion of Freedom of Information to cover more public organisations. We list a number of bodies not formally subject to FOI some of which are present on the grounds they are substantially publicly funded.
For some time we have listed the British Board of Film Classification, a key arts regulatory body which is not subject to freedom of information law, and the British Film Institute; the latter two bodies are funded by the DCMS directly so Minister Ed Vaizey may well be able to get them to voluntarily comply with FOI legislation first thing on Monday morning.
A particular set of arts funding bodies which some of our users have made us aware they would like to see subject to the act are the UK Screen Agencies (eg. Film Agency Wales) which distribute public funds to the film culture sector.
Please contact the WhatDoTheyKnow team if you have any suggestions for further bodies which you would like to see us list on our site.
We’ve been shown or seen a few responses from MPs, after people wrote to them, saying that they are worried about their addresses being made public. If this is their main barrier to voting No on Thursday, they have nothing to worry about: they voted to exclude their residential addresses (and expenses on security, and future/regular travel) from the Freedom of Information Act in July 2008.
In fact, one of the reasons it is costing so much money to collate and edit these expenses is staff have been going through and making sure precisely that such information would not be released.
It is sad that MPs don’t seem to know what the law is, and I hope someone will stand up in the debate on Thursday and make this point.
A few moments ago the team rolled out changes to our biggest and best known site, TheyWorkForYou.com meaning that every visitor to any page of the site will be greeted with a call to arms on the issue of some MPs voting this Thursday to conceal their expenses. And after the vote, we’ll be prominently publishing who voted which way – there should be a couple of million visitors at least before the next election.
Our explicit goal is to have a lot of constituents from around the country let their MPs know they won’t be impressed with a ‘yes’ vote or an abstention (the same thing in this case), and to build our Facebook group to the point where the mainstream media starts to take notice of this Net driven discontent.
Please do everything you can to get as many people as possible writing to their MPs and joining that Facebook group. We’re doing our bit – please do yours. Together we can stop the encouraging trend of more openness in our Parliament scrunching into reverse.
MPs are about to review the first five years of the operation of the Freedom of Information Act 2000. We’d like to encourage users of mySociety’s Freedom of Information website WhatDoTheyKnow.com to share their views and experiences with the MPs who are to carry out the review.
The review is being conducted by the House of Commons’ Justice Select Committee.
The committee is currently inviting people to make submissions to it. The deadline for submissions is Friday 3 February 2012.
A memorandum from the Ministry of Justice has been prepared to brief the committee, that document notes, in paragraph 67:
Very little research has been published detailing the views of requesters of information.
Particularly in-light of this we thought it would be worthwhile alerting our users to this review; if we could encourage our users to make submissions to the committee that might help ensure they receive balanced evidence: from outside, as well as within, the public sector.
While the committee is interested in any comments on the act’s operation, specific questions the committee has asked for comment on are:
Responses can be emailed to: justicecommemo@parliament.uk
Details of how responses should be formatted and technical details relating to submission are available on the webpage announcing the call for submissions.
Which are yours?
Bakul-Library – it’s just awesome, that they used PledgeBank, that they got a 1000 people, and that they’re going to make a new library in India because of it.
deathpenalty – Not because it was effective, and not because I agree with it. But because I disagree with it, and because of the reaction it got from left-wingers not used to seeing other points of view. PledgeBank is for anyone.
walkout – PledgeBank offline. These school students got together to march against war (whatever or yippeee), but just LOOK at how many text message sign ups there were. Tom saw in London an illegally flyposted flyer.
First1000 – I mean, they may be crazy libertarians, but ONE THOUSAND PEOPLE are going to move house. OK loads won’t move. But hundreds will.
tunepiano – It’s about fun and joy, not just politics and worthiness (bit of a worthy example to make that point I know, but it is a piano!)
Google have launched a new UK politics site at google.co.uk/politics. mySociety were delighted to be asked to build a TheyWorkForYou widget for this site. There’s no doubt that this sort of modular re-purposing of our information is going to happen a lot more in the future, and it’s great to start out with the best of possible partners.
Do you have a ‘mySocietyish’ idea that you’d like to see become reality? Is there something radical about the sites we already run that should change? Do you have any smart ideas about helping more people to benefit from the services we already offer? Or would you just like to read and comment on ideas submitted by other people?
If the answer to any of these questions is yes, then we’d like you to submit your idea to our 2009 Call for Proposals (built for us by Richard Pope). We’ve run these twice before in 2003 and 2006, resulting in the launch and success of sites such as WriteToThem and WhatDoTheyKnow.
Just as on previous occasions we’ll pick a winner and some runners up, but also just as before we can only promise to do our best – we don’t have the resources to solemnly promise to build the winner, whatever it might be.
What We’re Looking For (or, an insight into the mySociety mindset)
The characteristics of the winning and runner up ideas are highly likely to include one or more of the following factors. Don’t try and include all of them, that’d be silly :)
We might well change these guidelines a bit as the first responses come in. The call will stay open until September 15th, and we’ll hope to announce the winners in early October.
So what are you waiting for – check out the 2009 Call for Proposals
Making Parliament clearer; mySociety announces a new process for monitoring MP's votes. (Image: Monet)
mySociety is lucky enough to have a number of small donors who give us monthly donations, normally ranging from £5-£20 (if you like our work and want to support us, please do join them!). Today we’re announcing a change to TheyWorkForYou which is supported by these donations.
One of the most popular features on TheyWorkForYou is the vote analysis – the bit that tells you that your MP “voted strongly against introducing a smoking ban” and so on. These voting analyses cut through a massive wall of parliamentary opacity whilst still allowing visitors to examine the details first hand. Despite each analysis resulting in just a single line on TheyWorkForYou, each one is rather time-consuming to construct, and TheyWorkForYou has not updated them as much as our users deserve.
Thanks to our small donors we’ve now been able to commission two part time researchers, Marcus Fergusson and Stephen Young, to help add new vote analyses more regularly. We’re pleased to say that we’ve just rolled out the first new policies, covering issues relating to schools, inquests and the House of Lords. We aim to add a couple of new vote analyses a month for the foreseeable future.
We take the business of authoring analyses that are scrupulously fair and neutrally worded extremely seriously. To this end we have replaced our previously ad-hoc approach with a newly instituted process designed to ensure the maximum rigour and balance, and to ensure we focus on issues which MPs thought were important even if they were not so well covered by the media.
TheyWorkForYou’s analysis of MP’s voting positions relies on The Public Whip, a project run by Julian Todd which tracks which way MPs vote.
The new process for analysing MP’s positions works like this:
We hope you find these analyses useful. Thanks to Richard Taylor for his divisions list and help with this post.
There’s a lot left to do, but Francis Irving’s brilliant new mySociety Freedom of Information site is now live. You can file requests to central government departments (most of the them), and browse what other people have been requesting (already fascinating). It doesn’t have a name yet, nor any slick design, nor half the features we want it to have, but it works and it gets things done.
And dammit, people, that’s what mySociety’s all about. Can we explain it any better?
FixMyTransport probably wouldn't have helped Mr Robinson (who survived)
I must admit that I’m pretty happy to announce mySociety’s plans to build our first major new non-comercial website since WhatDoTheyKnow.com launched in 2008. Late in 2010 we plan to launch FixMyTransport , a site focussed on connecting and empowering people who share transport problems of different kinds. The fantabulous Louise Crow will be lead developer.
Crucially, we at mySociety are under no illusions that it is an order of magnitude more difficult to get a new ticket machine in your station than it is to get your local council to fill a pothole (FixMyStreet surveys report 2371 problems fixed in the last month alone). The difficulty of achieving even minor changes to transport services and infrastructure is why we are simultaneously announcing our plan to build FixMyTransport on top of a major new back end system called Project Fosbury.
Project Fosbury is about helping people get over difficult obstacles. It is a modular platform for breaking down a complicated civic task into pieces which can then be allocated to one or more people. So someone asking their council to change the timing on some traffic lights might be allocated the tasks of:
Each task will ultimately be carried out entirely within a joined up infrastructure, each module being built to mySociety’s habitually stringent rule that “it must be easy and satisfying to someone who’s never engaged politically before”. We will work to create incentive structures, peer pressure, and hopefully a sense of fun. There will be a single public home page for each mini campaign, showing recent activities on the site, as well as integrating with external social media. We hope to repeat the FixMyStreet phenomena where some ‘insoluble’ problems suddenly become soluble once they’re in the public domain.
Now for the credit where it is due. mySociety’s sysadmin Keith Garrett suggested FixMyTransport back in January 2008.
The actual mechanics of breaking the problem into pieces (the idea that became Project Fosbury) came from a wide discussion at our retreat, with excellent suggestions coming particularly from Richard Pope. But the more general idea that the Internet hasn’t yet produced a really good system for bringing people together to solve everyday problems (as opposed to chat, or win the US presidency) came from numerous Call for Proposals submissions, including ones from Mark W, Rob Shorrock, Peter Silverman, Mahmood Choudhury and more.
mySociety will be building this site using money donated by people like you, profits from commercial projects, and any specific funding we can raise around it. If you know of anyone or any organisation that you think might like to support FixMyTransport or Project Fosbury, please do get in touch.
Heard the song ‘We Built this City (on rock and roll)’ by Starship? No? Not to worry, I am just trying to draw parallels with how mySociety.org is organised and managed. You may not have realised, but a good sized chunk of the work that we do is actually carried out by volunteers, that includes everything from translating a single page to full website development
In other words, volunteers make our organisation tick and thought it about time that we shed some light on who they are and what they do. First out being subject to closer scrutiny is Tim Morley who looks after the everyday running of PledgeBank.com.
Tim has been volunteering for mySociety.org since 2005. Having heard about us through an article in the Guardian, he started out by translating content to Esperanto, and has through the years progressed to his current role.
Being a trained primary school language teacher, Tim estimates that he spends anything from five minutes to 3-4 hours a day on his volunteering work depending on how much there is to do. Task varies from helping users with technical queries to help out organising events.
Challenge was an initial motivating factor in deciding to start volunteering. He could also see the benefit of PledgeBank.com as a tool for the Esperanto speaking community, to help and encourage people to organise happenings in what is a very widely- and thinly-spread group. Three years later what keeps him going are the people involved with mySociety.org, the fact that he’s impressed with other things that we are doing and is proud to be associated with our organisation. Making a contribution and taking PledgeBank.com forward still feels important.
If you are interested in volunteering for mySociety.org don’t hesitate to get in touch. As can be seen above, Tim is a highly involved volunteer. But all contributions to the running of our organisation are appreciated so don’t let him put you off. :-)) Further info on some of our other volunteers is in the pipeline if you are looking for inspiration on how you can help out.
Last night was the annual New Statesman New Media Awards, held in Westminster Abbey’s College Gardens. mySociety were finalists in two categories, Modernising Government and Contribution to Civic Society, with both Number 10 petitions and FixMyStreet nominated in both. Also, two other projects we host, PlanningAlerts and The Government Says, were both finalists in the Information & Openness category.
It was a lovely evening, seeing some people I haven’t seen for some time and meeting new people too. We ended up winning in both our categories – the Number 10 petitions site in Modernising Government, and FixMyStreet in Contribution to Civic Society, which is obviously fantastic for everyone involved. The judges were impressed at the open source nature of the petitions site, and the “deceptive simplicity” of FixMyStreet. This is now the third year in a row we’ve won the Civic Society award – TheyWorkForYou won in 2005, and WriteToThem in 2006, so we’re obviously doing something right. :)
It’s a shame that Chris could not be with us, but his mother did attend to see the projects he worked on recognised.
Thanks and congratulations to all the other winners and finalists.
If you’ve visited our parliamentary site TheyWorkForYou.com, you’ll have noticed that on each MP’s page there is a short summary of his or her voting record on various key issues.
These issues have always been carefully chosen to give a simple but neutral top-line view of each MP’s voting activity. Judging by Twitter, they’re a fairly popular part of the site, too.
There’s way, way more tedious complexity behind producing these little summaries than you might think, and due to a lack of appropriately skilled people in our team over the last year we had let our vote analyses get a bit behind the times. If you’re really interested you can read about why authoring these things in such a scrupulously balanced way is so time consuming here.
We’re posting today to tell you that we have recruited a pair of excellent new part-time voting analysts, David and Ambreen, and they have recently produced the first of a new generation of voting summaries.
The first shows how each MP has voted on increasing the rate of VAT, and second on the recent changes to university tuition fees. We have also increased the number of votes which feed into the EU integration policy to bring it more up to date.
To see this new data, just pop along to TheyWorkForYou’s home page, stick in your postcode, and check out your own MPs’ page. Then, if you want to be made aware as soon as we’ve published the next analyses, please follow our new TheyWorkForYou Twitter account.
Lastly, I just want to say thank you to the vote analysts Ambreen and David, to senior developer Matthew and to uber-volunteer Richard Taylor for kicking this vital part of TheyWorkForYou back into top gear.
Image by European Parliament.
Image from jaygoldman
Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites
I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.
Identifying and categorising these types of site turns out to be surprisingly useful. First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.
Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.
1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.
2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou.co.nz, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch, Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.
3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.
4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.
5. Government spending – where the big money goes. In the US the dominant site is FedSpending.org, and in the UK we have ukpublicspending.co.uk.
6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and Statute Law Database. Much of the legal database field, however, remains essentially private.
7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet‘s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.
8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places, the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others. mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.
9. Political document archives - This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.
10. Bulk data - Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.
——
Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.
I’m very pleased to announce that the petitions system we’ve built for 10 Downing Street has gone live today.
I’m very grateful for the hard and often inspired work put into this by Chris Lightfoot and Matthew Somerville, as well as the civil servants who have helped to build a petitions system which I believe is in a real class of its own.
The most notable features are:
1. Petitions are accepted and published, regardless of the political slant of the petition. However, if they break the Ts&Cs (a petition that doesn’t actually ask for any action, for example) then they are put on a special rejected petitions page: they don’t just vanish. We think this transparency feature is probably unique.
2. The site is being launched in beta, and will change over time. This might seem too commonplace to note for many of you, but it reflects a willingness to see a public IT service evolve in response to users, not simply fulfil a contract agreed in advance. mySociety exists partly to spread good practice in the public sector, and we think this is a nice example of that in action.
3. The code, including Chris’s amazing high-load optimised engine, is all open source.
Any questions? Come into our chat channel at www.irc.mysociety.org or mail us at team@mysociety.org.
Francis has been furiously adding new features to our Freedom of Information website WhatDoTheyKnow ever since it launched earlier this year. He’s just added one of the most important missing features, the ability to leave annotations or comments on FOI requests.
This is especially useful for providing plain English summaries of what information in a response was actually interesting, or to discuss refusals to supply information and what to do with them. To add one just go to a request page and scroll to the bottom, just like adding a comment on a blog post.
So, whether you’ve made a request in the past, or you’re just interested in helping out, get annotating.
I’ve had the good fortune to talk with a few folks over the past several days who have echoed many of the things that we’ve been learning at PledgeBank over the past several months and years, and I thought I’d share…
First, I chatted yesterday with Jason Dick, who blogs at A Small Change. The blog focuses on nonprofit fundraising, and in our conversation we chatted about the tendency of nonprofits to be more than a bit behind the curve when it comes to adoption of new ideas/technologies…but that’s old news. We also chatted about the organizations that have decided to take a bit of a chance on online fundraising, and that have done some incredible things. We’ve seen the same thing happen at PledgeBank — a small organization (or a not-yet-formed organization) trusts its supporters enough to put some modicum of responsibility in their hands, and gets a tremendous response.
I also chatted yesterday with Peter Dietz of Social Actions and the impressive lot behind the Social Actions Mashup (selected as a finalist in the NetSquared competition, btw). There was great conversation about the value in aggregating information, what the nonprofit sector can gain from the syndication of social actions, and how we can all work together to create more ways for more people to do more good. Really.
Finally, I had the pleasure of reconnecting with an old friend, Susannah Cowden, who is now working with Be the Change. As an organization that’s still forming and growing, there’s significant pressure to both be different and be innovative — I hear the same struggle from a lot of organizations in the U.S. While there are many folks who have every intention of trusting their supporters to act on their behalf and equipping them with the tools to do so, there are very few who actually do it.
I wanted to capture these conversations because these themes are not at all unique. We all know that nonprofits move slowly when it comes to technology. We all know that those willing to take a chance are, more often than not, rewarded. And we all know that there’s a delicate balance between controlling your organization’s message (especially if your tax status is 501(c)3 in the U.S.) and empowering your organization’s members.
What I think these conversations illuminated for me is the need for someone to cut through all of these lessons and to find ways to make real stuff happen, with real impact. That’s what mySociety is about: giving people “simple, tangible benefits in the civic and community aspects of their lives.” And (I hope) that’s what PledgeBank enables. Holler (heather at pledgebank dot com) if you’re interested in working together to create a larger platform for collective community action.
We’re delighted to announce that mySociety has been generously awarded $100,000 by Omidyar Network, a mission-based investment group committed to enabling individual self-empowerment. The purpose of the investment, our largest to date from a philanthropic source, is to spread the use of PledgeBank.com to a host of community and grassroots groups in the US.
Omidyar Network invested in PledgeBank because it encourages collective effort by enabling users to rally the support of others through their own commitment to take action.
From starting new organizations, to giving blood, to volunteering to mentor others, Pledgebank has enabled people to do things they wouldn’t have done otherwise. Just this morning we were sent these pictures of a protest in Brazil that was organised using PledgeBank.
What has been missing to date, though, has been any resource to spread the use of the tool to traditionally offline groups. Whether these are schools, community groups, NGOs, churches or neighbourhood watch groups, many could benefit from PledgeBank, but few have had the opportunity to hear about it. Therefore, PledgeBank is looking to hire an outreach coordinator who will travel around the US meeting groups, listening to their goals, and explaining how PledgeBank can help them, for free.
mySociety will be recruiting the outreach coordinator in the New Year. If you’d like to get in touch, perhaps to pre-book an appointment to meet or talk with the outreach coordinator, or because you know someone who could do the job well, please drop us a line to hello@mysociety.org.
When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.
Peter Collingbourne got fed up with this, and luckily for us, he can code too. He forked our source code repository, and made a nice patch in his own copy of it.
He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.
Now if you go to an Outlook attachment on WhatDoTheyKnow,
such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).
Thanks Peter!
If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.
Current Birmingham parliamentary boundaries
Birmingham parliamentary boundaries at the next election
Parliamentary boundary changes appear to be a source of confusion to many people and organisations. The facts are quite simple – parliamentary boundary changes, proposed by the various Boundary Commissions, do not take effect until the next general election. Until then, your MP remains whoever they have been, no matter what literature you may get through your letter box, or what anyone may tell you.
As one example, take Birmingham City Council. Their page on constituencies and wards correctly states that Birmingham is divided into eleven parliamentary constituencies, but then goes on to list only ten – they are listing the new constituencies which do not yet exist, as Birmingham is losing one constituency at the next election. It appears that they have organised themselves along the new boundaries in advance – which is fine, but this doesn’t affect current Parliamentary representation, and so they should explain this clearly, as otherwise members of the public get confused (and blame us for giving them the “wrong” MP, when we haven’t done so). As you can see from the maps above (which highlight Birmingham, Hall Green), the constituencies will be changing their boundaries quite a bit, and we have had reports of people receiving letters from candidates in the next election who are MPs of different neighbouring constituencies, simply referring to themselves as an MP, which is a great source of confusion.
St Josephs Avenue is just below the Royal Orthopaedic Hospital
An inhabitant of St Josephs Avenue, Birmingham (behind the Royal Orthopaedic Hospital), which is currently within the Selly Oak parliamentary constituency (red), and the Northfield ward of Birmingham City Council (green), would, on looking at Birmingham City Council’s website, assume they’re in a parliamentary constituency called Northfield. Northfield is currently the constituency to the west of Selly Oak; at the next election, its boundary with Selly Oak will change to the blue line, at which point St Josephs Avenue will be in the Northfield constituency. But not until then.
Map of Streatham constituency at next election
Current Streatham parliamentary boundary
As another example (chosen purely as it has come up in user support), the Labour candidate for Streatham has a page about the constituency – obviously you would expect a candidate to be talking about the future constituency, but would it hurt to add some explanation that Streatham is currently a slightly different shape?
Boundaries of different things are all independent – if a ward boundary moves due to some local issue, the corresponding Parliamentary boundary does not necessarily change with it (probably not, in fact). So when Birmingham changed its ward boundaries back in 2003, they became out of sync with the Parliamentary constituencies. From the next election, things will be more in sync as the new Parliamentary boundaries are based on more recent ward boundaries, but this will again separate over time. All we can do is always clearly explain the current situation, and ask that others do the same.
One of mySociety’s most below-the-radar projects is GroupsNearYou.com , a project to build a web service that eventually will allow other websites, such as FixMyStreet, to tell their users “Look! There’s a local email list here. Why not join it and discuss what you can do to stop those phoneboxes being smashed up?”"
However, in order for GroupsNearYou to become a useful web service for mySociety and the rest of the geospatial Internet, it really needs a good pile of pre-existing groups adding from across the globe. To help with this process Richard Pope has built a little game, rather in the spirit of our video timestamping game. To play it involves trying to identify which Yahoo Groups (and soon others, like Google Groups) cover which areas on the ground.
Reasons we think you should have a play include:
Thanks Richard!
Some lovely new volunteers have been working their socks off to add the Scottish Parliament to TheyWorkForYou. Yay!
Unfortunately, the server that TheyWorkForYou sits on is almost full, so we can’t launch their hard work. Boo!
TheyWorkForYou isn’t an externally funded project, and we need funding from other sources to keep it growing and improving. So if the season has filled you with generosity of spirit, why not drop us a few pennies to pay for some upgrades? Any extra beyond what we need will go into the general pot to keep mySociety running and the developers from starving.
Unfortunately, I’ve had to disable the PledgeBank Facebook application. It used to let you sign and share pledges from within Facebook.
Facebook recently changed their platform (again!), breaking our code for sending success/failure messages. Obviously, it is no good signing up to a pledge if you don’t get informed when it succeeds.
I tried to fix it, but couldn’t work out how to do so quickly. We don’t have the time and money at the moment to chase after this, so I’ve disabled the application entirely. Links to PledgeBank pages on Facebook now redirect to pledgebank.com.
Hopefully it’ll be back one day – do send us emails if you miss it (or money if you have a large pledge that really needs it!). I think there may be a better solution with a simpler interface – the current application tried too hard to reimplement all of PledgeBank within Facebook. And besides, we should be supporting OpenSocial now it exists. It’s an open standard, Facebook isn’t.
Technical details: We used infinite session keys to send notifications from cron jobs. Quite reasonably, this no longer works. However, I couldn’t find out what to use instead. I think Facebook should respect backwards compatibility of its APIs a lot more, and if it breaks it they should give clear instructions about what to use instead. This does put me off ever wanting to develop anything on their platform again.
Street party photo by garryknight (flickr)
Being strictly non-partisan mySociety has no official view on Wills & Kate, but we are unashamedly Pro People Having Parties. And recently we’ve been able to work on a project with Barnet council that has helped us make more of them.
Most councils want people to be able to have a street party if they want – I mean, who’s against a party? But closing a street has costs associated with it, and there’s no point in spending that money if the ‘Street Party Committee’ is actually just one person, and the party isn’t actually going to happen.
Tackling this particular problem seemed ideally suited to PledgeBank, which exists solely to make sure there are enough people signed up to make a particular activity worthwhile.
So after some custom hackery, here’s what happens if you live in Barnet and apply to run a street party for the Royal Wedding. First, you visit this page and give your details. Then the council makes a pledge like this one, and then emails it back to the applicant. All the pledges are of the same form, and read:
“Barnet Council will arrange free public liability insurance for a street party in [Your Street name] but only if 3 or more households will get involved.”
It is then the applicant’s task to get another couple of people (or more) to sign the pledge. Once the signers exceed the threshold, the council believes the party is bona fide and starts work. Simple.
And it works! There are 24 parties currently listed that have passed the threshold, so that’s 24 streets that are already good to go. There are another 27 that may succeed or fail, depending on their organiser’s motivation.
Strangely, though, our invitations haven’t arrived yet, but, you know… they probably got lost in the post (sniff).
So, PledgeBank launched today. Already, you can sign pledges to have trees planted to compensate for your CO2 emissions, help clean up the banks of the River Taff, support local retailers, and one or two, err, more politically controversial ones. So, take a look, and sign — or, better, create — some pledges!
If you enter your postcode on TheyWorkForYou and it’s Scottish or Northern Irish, you’re now presented with your MSPs and MLAs as well as your MP, which makes sense given the site covers their Parliament and Assembly respectively. :-) You also get an extra tab in the navigation linking through to Your MSPs or MLAs. In order to do this, I needed a quick way of determining if a postcode was Northern Irish or Scottish. Northern Ireland was easy, as all postcodes there begin with BT. I assumed Scotland was also easy, which turned out to be true apart from the TD postcode area that straddled the border like a mail-sorting Niagara Falls. After some very dull investigation, I eventually worked out that e.g. most of TD15 is in England, but (amongst others) TD15 1X* is in Scotland, except for TD15 1XX which is apparently back in England. The final result was the postcode_is_scottish() function in postcode.inc, which (hopefully) correctly determines if a given postcode is Scottish or not – perhaps someone else will find it useful.
My house mate just said that his friend, who is at sixth form college, just had a PSHE (personal, social and health education) lesson in which they studied the website TheyWorkForYou.com.
Apparently it is good and I should go to it.
As a new edition has just been released, and I’ve had to tweak the parser to cope with the new highlighting, it’s a good time to write a brief article on TheyWorkForYou’s handling of the House of Commons Register of Members’ Financial Interests (Register of Members’ Interests as was before the current edition). Way back in the day, a scraper/parser was written (by either Julian or Francis) that monitors the Register pages on www.parliament.uk for new editions, and downloads and broadly parses the HTML into machine-readable data. The XML produced can be found at http://ukparse.kforge.net/parldata/scrapedxml/regmem/ – TheyWorkForYou then pulls in this XML into its database, and makes the latest data available on every MP’s page.
However, as it’s been scraping/parsing the Register since 2000, we can do more than that. Each MP’s page contains a link to a page giving the history of their entry in the Register – when things were added, removed, or changed. You can also view the differences between one edition of the Register and the next, or view a particular edition in a prettier form than the official site. There’s a central page containing everything Register-related at http://www.theyworkforyou.com/regmem/
I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.
mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.
Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages – causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).
Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.
(the autumn photo thanks to Nico Cavallotto)
A number of people report dog fouling through FixMyStreet, using slightly more… colloquial language. A number of councils have strict obscenity filters, blocking anything containing swearing. As I’m a pragmatist and not that interested in campaigning against councils blocking legitimate emails from their citizens (feel free!), FixMyStreet simply changes any “dog shit” reference to “dog poo”. This works well for everyone.
Recently, the infamous Intellectual Property Manager from Portakabin™ Limited got in touch to complain about a couple of reports on FixMyStreet containing the words “portacabin” or “portaloo”. Again, as a pragmatist, I’m not really interested in whether users using trade marks or trade mark variants in a generic way on a problem report actually constitutes trade mark infringment (actually, I’d guess not), I just want legal people to go away and not waste our precious resources. So from now on, any report containing portakabin or similar will become [portable cabin], and portaloo will become [portable loo].
For anyone who’s interested, this is accomplished through a simple regular expression, that looks for porta followed by 0 or more spaces, then cabin, kabin, or loo, and sticks “ble” in the middle.
Last month saw the launch of not one but two websites asking the public to report empty properties to the relevant council. First off the blocks was mySociety’s ReportEmptyHomes.com, commissioned by the Empty Homes Agency, followed shortly afterwards by EveryHomeCounts.info from a group of eight councils in Surrey and Hampshire. Since mySociety claims to want to show the public sector how to use the internet properly, I thought it might be interesting to compare the two sites, at least from a user’s perspective.
I’m going to imagine I was walking down, say, Fosters Lane in Knaphill, Surrey, and I noticed that the house on the corner next to the chip shop was in a state of disrepair.* I snapped a picture on my mobile phone, and I want to send it to the council to see if they can do something about it.
[*I ought to just add that this is entirely fictional. I've never been to Knaphill, I've no idea whether there's a chippy on Fosters Lane, and even if there is, the house next to it probably belongs to a lovely couple. Please don't go taking pictures of their house for the council.]
So, first up: EveryHomeCounts.info. Clicking the big red 
REPORTING AN EMPTY PROPERTY button takes me to a page of text telling me why the council might like people to report empty properties, although presumably if I’ve got as far as finding the website and clicking the big red button, I’m already convinced of the case. At the bottom of the text I’m invited to “click here” to report an empty property.
On the next page I’m asked for… a whole load of personal information. I want to tell you about an empty house; do I really need to declare my title, first name, surname, house name, house number, street, locality, town, county, postcode, country, telephone number, and email address before doing so? Well, as it turns out, no — they only insist on an email address (although the single letter “f” was accepted as a valid email address).
On to page four, and I’m finally asked for the address of the house. I suppose “house on the corner next to the chippy, Queens Road, Knaphill” would probably be enough for the council to identify it. But then — get this — they want me to tell them which borough council might be responsible for this address, so that the report can be sent to the right place! Unless I happen to live in that street, how would I know? Even if I could have an educated guess, it might be near a boundary, or just over a boundary… Leaving the field blank isn’t allowed, and there’s no option that says “I’m not sure, sorry” — I’m told in red ink that I must specify a council if I want to continue filing my report.
Finally, I reach a screen that says at the top, “Thank you. You have reached the end of this form. blah blah” The second paragraph says, “What will happen next? The council will process your form. You will receive an email blah blah.” So, I pat myself on the back, turn off the computer and go for a walk. Except that if I’d scrolled down the page, I would have seen “submit” button, along with the “review” and “cancel” buttons. My form hasn’t been submitted, and I’ve wasted half an hour filling in a form that’s been thrown away.
Now, what would have happened if I’d gone to ReportEmptyHomes.com instead?
The top of the front page asks me for a postcode, street name or area. I enter “Fosters Lane, Knaphill” and hit enter. This brings up an Ordnance Survey map with Fosters Lane in the middle of it, and I click on the offending property. The text on the page immediately changes and tells me that this problem falls in the area of Woking Borough Council, and I’m asked for a description of the property, a photo if I’ve got one to upload, my name, email and phone number.
Having filled in the information and clicked “submit”, I’m told to go off and check my emails, where I’ll find a confirmation link to click. This finalises the report.
So, how do the two sites compare? The mySociety site certainly gets the user through the process quicker, and offers maps and photos to boot. It helps the user greatly by taking responsibility for finding the right council, and does so for the whole country too, not just for a couple of counties in the south. On the down side, one could question why it’s so important to verify the user’s email address before filing the report; waiting for a confirmation link by email adds an extra hurdle which will probably trip at least some users, so why do it?
Also, EveryHomeCounts.info isn’t just for filing reports about empty homes; it contains information on buying, selling, owning and letting them too, providing ways for local people to perhaps make use of empty properties without enlisting the council’s help at all, which can only be a good thing.
To be fair, the councils concerned should be applauded for taking the initiative to launch this service, and I hope it proves to be a worthwhile use of council tax money. It’s great to see public bodies using the internet in innovative ways to try to make concrete improvements in people’s immediate environment. It appears though that mySociety have shown that it can be done better, and for the whole country, and probably more cheaply to boot.
Member of the National Secular Society Robert Christian used mySociety’s Freedom of Information site, WhatDoTheyKnow to ask all 227 English NHS “provider” Trusts about how much they spend on chaplaincy.
On the 28th of February 2011 the results of his research were published in an article on the National Secular Society website (full report [PDF]). He found that £29m of NHS funds were used to pay chaplains in 2009/10 and also observed a wide variation in the amount, as a fraction of total spend, that specific trusts were spending on chaplaincy.
The publication of the research prompted a number of articles in the UK media. eg. (Daily Mail, The Independent, The Mirror).
Mr Christian has commented:
“To have identified the right FOI contact for every provider NHS Trust in England would have been daunting if not impossible. I doubt that my study would ever have got off the ground without WDTK. I particularly valued the way that the site tracks which Trust has and has not yet responded. I liked the capability to thank each FOI lead after they had responded.”
The fact that making requests via WhatDoTheyKnow allowed Mr Christian to cite the source of his raw data was important to him. He added:
“The transparency of the raw data is, I think, one of the main strengths of the WDTK website for three reasons. First, I was able to hyperlink every piece of data back to its source – and that meant that it was easy for colleagues from the NSS to check the accuracy of the data (with so many Trusts a transcription error was always a possibility). Second, it ensured that if anyone had wanted to challenge the accuracy of the data they could be directed to see that the study was simply quoting the Trusts’ own information. Third, it means that the data is there for future reference to see if there are any changes over the coming years.”
mySociety and WhatDoTheyKnow are non-partisan and don’t get involved in campaigning except in specific areas relating to openness and transparency. We take no view on issues such as how much, if anything, the NHS ought be paying for chaplaincy. However we welcome campaign groups making use of our services.
WhatDoTheyKnow currently has around 2-4 “bulk requests” per month made via its site. At the moment we don’t provide any mechanism to make bulk requests automatically. We are considering adding such a system, for requests which have been sanity checked by the WhatDoTheyKnow team. The provision of such a system would probably be associated with a mechanism for preventing other “bulk requests” from being made without the site administrators’ explicit approval.
Making the requests is only a small part of the work involved in a study such as that carried out by Mr Christian. Chasing public bodies for responses, as well as collating and analysing the information released is likely to be much more time consuming than submitting the requests themselves. This is something Mr Christian agrees with, stating:
“If enquirers are not prepared to individually contact each organisation to ask the question, I would doubt their commitment to retrieve and analyse the information (as that is actually a much bigger task)”.
Clearly any facility for enabling requests to be made in bulk will have to incorporate safeguards to ensure responsible use.
Whereas Mr Christian has been happy to conduct his research in public, and still been able to generate media coverage following publication, we are aware that many campaign groups, and others such as journalists, like to make Freedom of Information requests in private.
Mr Christian has commented on the issue of “scoops” and the effect of conducting his research in public:
“The question of ‘scoops’ is an issue for journalists and in fact this problem did happen in this case. Someone appears to have trawled the WDTK know site and noticed what I was doing. A short piece was run by the Daily Express before we completed and published the study. So clearly this might be an issue. But the risk of a spoiler being run will tend to be low when the number of organisations being contacted is large. This is because the amount of work needed to collate and analyse the data is enormous and so casual trawling will show only that a question is being asked – not what the conclusions are.”
In order to get as great a fraction of the total number of FOI responses available on WhatDoTheyKnow we have also been considering an option for making requests in private, for a fee. The idea would be that once the findings were published then the FOI response could be opened up to the public providing access to the source material backing up the story.
Any views on our ideas for the future and on the way WhatDoTheyKnow has been used for this, and similar, research would be welcome in the comments below.
Say hello to GroupsNearYou.com
The Short Geeky Explanation
GroupsNearYou.com is an entirely user generated API-queryable database of the location and nature of local online communities, irrespective of the platform they are hosted on. A piece of the programmable web, in short, with local community building focus. Check this lovely example of how easily the syndicated community information can be layered onto a map, for example.
The Business Problem it Solves
Do you run a site that tells people stuff about their local area? Do you suspect that there might be quite a lot of internet enabled community activity going on in local areas that you’d like to tell your users about? Use our feeds.
The Social Problem It Solves:
There’s a proven real world social value to people belonging to very local email lists and other forms of local online community. However there is no eBay or Craiglist or other market dominant player in the local online community world, instead there’s a myriad of google groups, yahoo groups, Facebook & other YASN groups, extremely old school CCed email lists, online forums and so on. As a consequence of not having one big simple place to go to find and join local groups (many of which are not even on the web for Google to find) far fewer people ever find out about and join their local online groups. GroupsNearYou.com is about getting more people to join groups, groups that are not hosted by us, and (hopefully) mainly discovering them via uses of our syndicated info on sites that aren’t run by us. It’s a piece of pure internet infrastructure, with a positive social bent.
Who did it?
Astonishingly, the project was almost entirely built by a volunteer, Richard Pope assisted in design by another brilliant volunteer Denise Wilton. Their only reward is a highly sought after mySociety Hoodie, plus the love and gratitude of all our users.
Richard has been an amazingly dedicated volunteer for mySociety and on his own projects for over two years, and deserves the reputation he is rapidly gaining as one of the world’s truly great civic minded web innovators. The project was funded by the UK’s Government’s Ministry of Justice who have been trying to run experiments of different kinds in the realm of electronic democracy. Their money will go to help improve and grow the site, rather than building it, which is a very interesting funding model in its own right. The several hundred groups already in the system are mainly added by users of WriteToThem.com
Next steps
Almost all the groups listed in the database are in the UK at the moment, and they’re all from users of our other sites. We’re interested in working with anyone who runs sites that might want to either take information out of it, or put information into it (Hello email list/social network providers!)
Anyway, it’s dead simple really, just a little brick in the internet wall, albeit one that I hope will help a few more people meet their neighbours and improve their communities.
It is with great sadness that I must report the death of Chris Lightfoot, mySociety’s first developer and a good friend to all of us. He was found by friends at his flat on February 11th. The main announcement can be read in this post on his blog.
Chris was perhaps the pre-eminent example so far of what polymath means in the Internet age. His contributions to the world are more than just a formidable legacy of computer code of the very highest quality, for mySociety and many others. They also include substantial contributions to applied statistics, geographic information systems, economics and a range of public policy issues from identity cards to speed cameras.
Everything Chris did in these fields combined an incredulity-inducing array of technical and analytical skills with a wickedly funny, savage turn of phrase. To understand what a remarkable intellectual outlier he was, simply sift through his blog and marvel at the quantity of primary research and original coding that went into it. Documenting and exploring his work would provide material for many years of research, and yet all this was accomplished by the age of 28.
Within mySociety he was involved right from the start through the development of WriteToThem, HearFromYourMP and PledgeBank, building some amazing underpinning geographic and political web services like Gaze, MaPit and DaDem. These components make all our sites work and make a raft other tools and sites possible in the future.
For the last three or four months he was working at another employer, Media Molecule four days a week, but still helped the full time staff with the petitions work. The last major thing he built for us was the system that serves up the maps for Neighbourhood Fix-It, a site which was only just soft launched before he died, but of which he was apparently fond for its WriteToThem-like habit of getting simple things done that mattered to normal people.
Building mySociety’s major sites involved mighty team efforts, something which can obscure even huge invididual talent. So perhaps the sort of work for which Chris will be be most remembered is his wonderfully individualistic, virtuoso forays into scholastic areas in which he had no formal training. He wandered into differing disciplines, made a mark, and wandered on again like a giant that had no idea he’d just trodden on a village. The political survey work he did both hugely illuminates our understanding of our own political world, whilst raising the question “how come none of the professional political analysts thought of this?” And his travel-time maps should make everyone in government wonder if they’re sitting on information which could be reused to such amazing, potentially life changing effect.
Chris’ intellect and appetite for knowledge was surpassed by only one aspect of his character: his integrity. If you’ve ever wondered why WriteToThem goes to such lengths to protect users’ data it is largely because of his rock solid belief in the dignity and social indispensibility of privacy. Chris was an energetic campaigner in this field, notably for No2ID, who have posted a tribute.
It doesn’t stretch the truth an inch to say that with his death the whole of the UK’s citizenry, not just his family, friends and colleagues, will be worse off. Rest in peace, Chris.
Have you used WhatDoTheyKnow to make a Freedom of Information request?
If so, you can now add your photograph to the site, and some text on your user page about what you’re researching. This can include links to your blog, campaign page or twitter feed.
To add this to your profile, first log into WhatDoTheyKnow, and go to your user page by choosing “my requests”.
There are then links to add a profile photo and/or set some text about you, and what you’re using FOI for.
I’d go and do it while I remember – it will help you and others find and understand each other, hopefully leading to that little bit more collaborative research!
Paywall met by those trying to access the research article via the publisher's website.
Exemptions Applicable to Research
In terms of the freedom of information act there are a number of provisions which can be used to exempt the output of publicly funded scientific research:
With the above exemptions in mind it might well be possible to phrase requests in such a way that they don’t apply. For example I have had some limited success in relation to a request for a research protocol.
First Come First Served?
Our user was offered a hard copy of the publication; the reason this request was drawn to our attention was that the team to was contacted to help the two parties to get in touch directly. I suspect the reason that an electronic copy of the document was not supplied via WhatDoTheyKnow may have been related to a concern over breach of copyright on the research results which has probably been transferred or licensed to the publisher. While one individual may have obtained a copy of the information, it is still not accessible to everyone. Tony Hutchings, the Forestry Commission’s Head of Land Regeneration and Urban Greenspace, who led the research told me: “We have prints of the paper which we could supply you with”. How many printed copies he has to distribute and what happens when he runs out is not clear.
Open Access Publishing
Ideally the results of publicly funded scientific research ought be published in an unrestricted format in open access journals. The UK government, is moving towards such a stance but at a painfully slow pace. I asked the author of the research why he had taken the decision not to publish in a more accessible journal. He responded by saying:
The Research Councils (as do many funders from both private industry and public bodies) assess the quality of the research undertaken by the impact factors of the papers produced. … To my knowledge there are unfortunately few open access journals with high impact factors.
The EPSRC who funded his research have a Policy on Access to Research Outputs which states: “knowledge derived from publicly-funded research must be made available and accessible for public use”. When I asked them for a comment on this particular case Dr Sue Smart their Head of Performance and Evaluation responded saying: “Tony Hutchings is mistaken in his assertion that we use journal impact factors in assessing the quality of research”, but she also ruled his offer of a paper copy of the research article was: “in keeping with the principles of the RCUK (Research Councils UK) position statement [on access to research outputs].”
Like the other volunteers who help out with WhatDoTheyKnow.com I use the site for my own activism and campaign independently for more openness and transparency in a range of areas. I have written an extended article on my own website on the subject of open access publishing where I have included more details of the responses from the research council and researcher quoted above.
The “um’s” and “uh’s” on the interview are embarrassing, to say the least, but PledgeBank just got its first podcast coverage. Check out my NetSquared interview here. I’m hoping this gets us a bit of exposure in the U.S.
And I’ll do better next time with all the incidental sounds… :-)
Is there something part of the government is doing that you’d like to investigate? Find out everything from MPs’ expenses, to the length of allotment waiting lists, to whether your council’s Guy Fawkes bonfire is properly checked for hedgehogs.
mySociety are running a practical workshop on Freedom of Information at OpenTech on 4th July.
The workshop will help you make your first Freedom of Information request, including working out what to request, where to request it from and what exactly to write.
If you’re an old hand, you can get and give tips on how to take requests further.
We’ve got a fantastic team of Freedom of Information (FOI) experts to kick things off and answer hard questions.
Bring a laptop if you have one. Internet will be provided for the workshop only, so we can scour Government websites, and make requests on mySociety’s WhatDoTheyKnow.com website.
As usual, the rest of OpenTech is brimming with great talks, and will be full of interesting geeky wonks and wonky geeks. Book your place here so you can go to them and to the workshop. Hurry, it’s nearly sold out.
I recently found these requests by James Muldoon covering FOI statistics for the London Boroughs for 2009. As we regularly carry out analysis of WhatDoTheyKnow’s percentage share of FOI requests to central Government Departments of State, I thought it would make for an interesting comparison to do the same for the 33 Metropolitan borough councils, plus the City of London.
Below is a graph of the market share for WDTK.
Overall, the share for 2009 was 8.1%. During the year, the share did fluctuate quite a bit, and the requests on WhatDoTheyKnow were significantly lower in the 2nd quarter for some reason.
Q1: Jan-Mar 2009 – 9.4%
Q2: Apr-Jun 2009 – 5.1%
Q3: Jul-Sep 2009 – 9.5%
Q4: Oct-Dec 2009 – 8.3%
The City of Westminster has a much higher number of FOI requests compared to the other boroughs, mostly apparently due to a large motorcyclist parking campaign/protest. 73% of all requests made to Westminster via WhatDoTheyKnow in 2009 contained the words “parking”, “motorcycle” or “Verrus” (203 out of 278).
I will soon start looking for FOI statistics for Local Authorities outside London, either on WDTK, or via their disclosure logs. The Ministry of Justice encourages Local Authorities to regularly publish statistics on their FOI data.
Data caveats
Earlier this week someone browsing mySociety’s freedom of information website WhatDoTheyKnow.com contacted us reporting they couldn’t open a PDF document Southampton University had sent in reply to a request asking about the amount of printer credit purchased at the institution. The user suspected the file might be corrupt.
We investigated and determined the attachment is in fact in a fancy PDF format which cannot be read by many PDF readers. Those applications which are able to open the document present a cover page inviting the reader to agree to an “intellectual property rights notice”. The terms of the notice forbid “use” of the “material” without “the written permission of the university”. The intent appears to be to ensure only those agreeing to the terms are given access to the document’s content.
The Freedom of Information Act requires authorities to release information whether or not the person making the request wants to enter into an agreement with respect to its use; so one could argue the university are not properly discharging their responsibilities. In any case their stance is silly because anyone who wants to can request their own copy of the information. All they’re doing is creating more work for themselves.
What were they thinking?
An article John Ozimek of The Register has written in response to the correspondence reports the university explaining their actions by saying:
The Freedom of Information Act gives applicants the right to have information held by public authorities communicated to them, not to documents. Applicants are entitled to use that information so long as they do not breach the intellectual property rights of the public authority: taking the information and using it in a document of their own is acceptable, making use of a document which contains not only the information requested but copyright material is not.
One problem with this was pointedly highlighted by a comment on the Register article saying: Ask yourself, “how do I prove my facts if I can’t publish the evidence?”
WhatDoTheyKnow routinely publishes information released under FOI despite a wide variety of copyright and other legal notices and disclaimers suggesting we shouldn’t. (Read the policy on copyright.) This has been the first relatively elaborate technological attempt to circumvent WhatDoTheyKnow’s efforts to make the responses to FOI requests easily accessible to everyone.
Actions
WhatDoTheyKnow’s staff, volunteers, and users have responded to the university’s actions in a number of ways:
They’ve got form
This is not the first time the freedom of information team at Southampton University have been inventive in their use of PDFs; back in May they responded to a request via a password protected PDF version of a compliments slip which they attached to an email.
A few days ago mySociety asked the known possible candidates for Speaker to endorse 3 principles relating to making Parliament more transparent on the Internet.
We’ve now had endorsements which you can read on the individual pages of Sir George Young, Sir Menzies Campbell, Frank Field, Tony Wright and Sir Alan Beith , which until Parmijit Dhanda declared this morning, represented endorsement by 50% of the possible field. We also just recieved a typically frank and interrogative phonecall from Ann Widdecombe, who will be writing a formal response soon.
So, come on, John Bercow, Alan Haselhurst, Patrick Cormack, Sylvia Heal, and Chris Mullin. What’s holding up your replies? The days counter on your pages is telling the world how quick you are to respond…
Update: 11 June - John Bercow has now endorsed, and we’ve written to Margaret Beckett and Parmjit Dhanda, who’ve just declared their candidacies.
Update 2: Chris Mullin has told us he is ‘not a candidate’.
Update 3: Sir Alan Haselhurst has also endorsed.
Update 4 – Speaker Election Day: And Sir Michael Lord endorses too.
The latest “Giving Carnival” question is being organized this month by Peter Deitz, the brains behind Social Actions.
Peter’s question: “Is person-to-person fundraising dead, or just getting started?�?”
I’ll give my brief answer. Person-to-person fundraising is either dead or slowly dying, but that could be good news. Let me explain.
We’ve all probably had some experience with person-to-person fundraising — me, I sold Girl Scout Cookies. Now, I wasn’t selling Girl Scout cookies because I deeply cared about raising money for the organization or because I was interested in sharing the story of the Girl Scouts with those whose doorbells I rang. I was in it for the badges and stuffed unicorns that I could win by selling the most boxes of cookies. Person-to-person fundraising has reached a point, I think, at which there is such commodification of an organization’s story that there is little meaning left in the actual transaction…just stuffed unicorns.
Person-to-person action, however, is a different story — and I think that’s where there is tremendous potential. Take, for instance, what’s happening on PledgeBank. Individuals create a pledge to do “something,” but must make asks of their social network (friends, family, co-workers, etc.) in order to make that “something” happen. Rather than operating via a “transactional” outlook, it’s actually necessary to tell the story of why this “something” is important. There are few drive-by fundraisers on the site — you have to interact with folks and let them know why they should sign up to this social contract with you in order to meet your pledge’s target number of signers.
Similarly, I recently chatted with David Stoker, who’s working with Ashoka’s Citizen Base Initiative. They’re addressing the problem of superficial interactions by encouraging organizations to rally a fan base of individuals who support them…much like a sports team. The analogy is lost when you start imagining people painting their faces on behalf of the Red Cross, but the idea of organizations looking to their network for more than just a donation is certainly compelling.
Don’t get me wrong — I still buy and love Girl Scout cookies. But if I had to place bets on where the future of online organizing lies, it’s on peer-to-peer action that requires interaction, that illuminates an individual’s values, and that is part of a meaningful narrative arc. Thin Mint, anyone?
No lolling about in the sun for us, as we follow an endless chain of projects through the hot months. Inured to hasslebot, we’ve not been posting to this blog much. Instead, busy working on, or soon about to work on:
And that’s without mentioning Neighbourhood Fix-it and the call for proposals. Later in the year. Have I missed anything?
Have a good weekend!
This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.
During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.
We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.
Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.
Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.
Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:
[sv] type=WMS url=path/to/mapserv.fcgi?map=os.map& layers=streetview tms_type=google spherical_mercator=true
MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked :) ).
As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.
Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:
if ( fixmystreet.area ) {
var area = new OpenLayers.Layer.Vector("KML", {
strategies: [ new OpenLayers.Strategy.Fixed() ],
protocol: new OpenLayers.Protocol.HTTP({
url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
format: new OpenLayers.Format.KML()
})
});
fixmystreet.map.addLayer(area);
area.events.register('loadend', null, function(a,b,c) {
var bounds = area.getDataExtent();
if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
});
}
Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.
It seems Transport for London (TfL) really didn’t want anyone to be able to get hold of their internal guidelines describing how they deal with congestion charge appeals. Now though, thanks to mySociety’s freedom of information site WhatDoTheyKnow anyone wanting to find out in what circumstances TfL will cancel a congestion charge penalty can read the document detailing TfL’s criteria themselves.
I believe there are basic principles involved here; we all ought be able to find out what the law is, both “in statue” and “in practice”; it is right that we can find out the detailed rules which are being applied to the application of the congestion charge in London. I think that this request, enabling that, has been an excellent use of the Freedom of Information Act; it also shows how using the access to information laws can redress the balance of power between the citizen and the state.
The Road User Charging (Enforcement and Adjudication) (London) Regulations 2001 (the law) describes six circumstances in which someone served with a Penalty Charge Notice relating to the London Congestion Charge can make representations against it. However in practice TfL are in-fact allowing representations on a much wider basis. Now the internal document is available it can be seen that even the detailed: “Helping you with your Congestion Charging Penalty Charge Notice” information sheet offered by TfL does not contain the full details of what is and is not accepted as a valid reason for not paying the congestion charge.
Members of the public can now find out that TfL is happy to waive the congestion charge for those who are travelling “to register a death or travelling to hospital due to death of relative”, as long as proof of the death is provided, but not for someone who is “travelling to attend a funeral”. The document also suggests TfL won’t waive the charge if your vehicle has a breakdown and is recovered while the charge is operational; but the charge will be waived if, as a result of being clamped by a local authority, you have to collect your car from within the zone. Reassurance that TfL won’t pursue you for your congestion charge after you’re dead (assuming someone proves you’re really dead to TfL) is also provided.
There is a wealth of detail in the document, including revelations that foreign military vehicles are subject to the charge, but UK ones are not; along with details of circumstances where people will be given a second chance ie. their first appeal will be allowed but second and subsequent ones will not.
The public availability of this document may well make TfL’s life easier; some people may no longer bother making appeals in circumstances where they know they’ll be rejected and others will be able to phrase their appeal letters in such a way that it makes it easy for TfL staff to assess them against their criteria and accept them.
Request Details
Transport for London (TfL) is responsible for the London Congestion Charge. A document entitled: “Criteria for dealing with Representations and Appeals” describes the procedure TfL staff use to determine if someone’s appeal against a congestion charge penalty will be be accepted or rejected. This document was the subject of a Freedom of Information request made January 2008 which was initially refused. The argument TfL made against disclosure was that releasing the document would prejudice the exercise of TFL’s functions; Freedom of Information Act exemptions under S.30 (Investigations and proceedings conducted by public authorities) and S.31 (Law enforcement) were claimed.
On appeal the information commissioner’s office issued a decision notice saying it agreed with TfL that “the public interest in maintaining the exemption outweighs the public interest in disclosure”. The individual who had requested the document was persistent, and took his case to the information tribunal; there the information commissioner’s decision was overturned and TfL were ordered to release the document. The tribunal ordered the document be released to the original requestor by the 23rd of December 2009. A WhatDoTheyKnow.com user had made a separate request for the same document on the 8th of December 2009, and received it on the 7th of January 2010.
TfL still have not placed the released document on their disclosure log, which is perhaps an indication they’re still not too keen on the fact they’ve been compelled to release it.
mySociety has never run a campaign before today. And we’re not sure anyone’s ever run a campaign featuring a charismatic duck-billed platypus escaping from under the closing jaws of a Parliamentary portcullis.
Update 15.34 25/03/2008
Conservative Party leader David Cameron has just endorsed the campaign in this video.
Update 17.14 25/03/2008
Now kind words from techy Lib Dem MP Lynne Featherstone.
Update 11.38 1/04/2008
We’ve just recieved this fantastic endorsement from Nick Clegg, leader of the Liberal Democrats:
“Parliament belongs to the people. It’s time to open it up so people can find out what’s going on. mySociety has done a brilliant job in recent years in doing that – and it’s time to take this project to the next level and get information about the laws Parliament passes into the public domain.
“It takes a new MP months to figure out how the tortuous bills procedures work – so how we expect the voters to know what’s going on, I have no idea. The changes MySociety are calling for are vital so that every MP is fully accountable for the decisions they take on behalf of their constituents.
“I fully support the Free our Bills campaign, and will do all I can in Parliament to win this battle.”
Thanks Nick!
Well, I doubt that this will end the mashup trend going on out in the interwebs, but I thought folks might be interested in seeing this…
NetSquared (based in the U.S.) has launched their newest summer contest, the N2Y3 (that’s NetSquared Year Three) Mashup Challenge. You can see the 100+ projects that have been submitted here. One of mySociety’s projects, PledgeBank, is featured in one of the submissions: Social Actions. OK, the name isn’t super-sexy. But the idea is. Peter Deitz is developing a way to lead any given user (an individual or an organization) through the process of selecting a social action platform. Do you want to raise money? Do you need to integrate with a specific CRM? Do you need an online donation processing tool? Do you need a widget for your site? This mashup with combine 29+ (the list keeps growing) “action” tools (including PledgeBank) in that wizard, helping the average Joe or Jane figure out which tool would work best for them.
Of course, in order to move forward in the competition for mentoring and money, Peter needs your vote. To vote for this mashup (and at least four more — NetSquared is smarter than to just let everyone vote for one), just create a free account on the site and add at least five projects to your ballot. There are some really cool ones out there, so browse around a bit. The polls opened on Monday at 8am PST, and they will be closing on Friday at 5pm PST. The 20 mashup proposals with the most votes will attend the annual NetSquared Conference in San Jose, May 27 & 28, 2008. During the conference, the mashup creators will have a chance to pitch their projects to funders, foundations, and fellow nonprofit tech professionals.
As they say in Chicago, vote early and vote often!
A WhatDoTheyKnow user Tom Taylor has posted a cool query to the Royal Mail – he wants a list of where all the postboxes in the UK are (presumably so he can build a ‘find your nearest post box’ web site).
After some delay Colin Young of the Royal Mail responded with a list in a PDF file. However, whilst the list is pretty long, it only contains the postcode location of each postbox, not an actual coordinate that can be plotted on a map. So neither he, nor anyone else, can build a postbox finder service.
Just think about that for a second. The Post Office doesn’t know where its Post Boxes are. Whoda thunk? Good use of WhatDoTheyKnow.com, Tom!
Hansard's Parliamentary Debates by hugovk (cc)
The House of Commons debate coverage on TheyWorkForYou has recently extended back from the 2001 general election to the 1935 general election, and our knowledge of MPs now extends back to the start of the 19th century. This means TheyWorkForYou now includes things such as Anthony Eden on the Suez Canal in 1956, saying “there was not foreknowledge that Israel would attack Egypt“; the debate the day after Bloody Sunday in 1972; Geoffrey Howe’s resignation statement in 1990; Neville Chamberlain on the eve and start of the second World War in 1939; and Winston Churchill‘s speeches to the House, such as We shall fight on the beaches and This was their finest hour in 1940. This and much, much more are available and searchable using our new improved advanced search, which allows you to filter by e.g. date range or person. We hope people enjoy researching this huge wealth of information (I certainly do), and add useful annotations to the text to help other people.
This would not have been possible without the original project by Parliament to digitise historical copies of Hansard and make them available, nor the internal Parliamentary project to clean up the data, match up speaker names, and so on. The project was kindly funded by the Ministry of Justice’s Innovation Fund, which also supported the creation of FixMyStreet and GroupsNearYou.
The following is a message that we’d like to see emailed around within political parties of all stripes. If you work for a party, or know anyone who does, please send it along:
———-
Hi there,
TheyWorkForYou.com has sent online surveys to nearly 3000 candidates
across the UK, including most of your party’s candidates. If you don’t know it, TheyWorkForYou is probably the largest politician transparency website in the UK, with about 3m visitors last year.
The survey we’ve sent is a rigorously neutral attempt to clarify candidates positions on many of the biggest issues at the election. It is also a long-term document – the data that comes from candidate responses will be viewed millions of times between now and the general election after this one. It also contains both local and national questions.
There are 6000+ volunteers now nagging non-responsive candidate. You can help your party improve its responsiveness rating, here, but passing on the word that TheyWorkForYou’s survey is not push-polling, not single issue, not short-termist:
http://www.democracyclub.org.uk/twfy/chart/
Please help us by passing on the message that TheyWorkForYou will be one of the main ways that new MPs from all parties (and none) will be scrutinised and neither we nor new MPs want to start our relationship with a “refused to go on the record” badge on their pages.
If you are a candidate, and you want to do the survey, check your email for TheyWorkForYou (no spaces). If you don’t have it, drop a mail to developers@democracyclub.org.uk and it’ll be sent along shortly.
many thanks,
The staff and volunteers at TheyWorkForYou and Democracy Club
Update: The Telegraph posted a retraction yesterday.
You may have seen coverage on various websites saying that a civil servant was sacked after posting a comment on TheyWorkForYou.
We’ve no idea what this story is about, but we’re pretty certain it has nothing whatsoever to do with TheyWorkForYou. No journalist bothered to contact us before running the story.
So frankly, we’ve no idea what’s going on.
What we do know is that the implication that mySociety would merrily hand over sensitive personal data that ends up in getting someone sacked, without fighting tooth and nail for their privacy every inch of the way, is a complete misinterpretation of the way we work and the things we hold most dear. No-one has ever contacted us to ask us to hand over such data, nor have we ever done so.
We think what might have happened is a simple mis-remembering of the website that contained the problematic comment. We’re hoping to get in touch with Lisa Greenwood so we can get full details before asking the various media companies that have run with this for a correction.
Richard Pope has been redesigning mySociety’s biggest site TheyWorkForYou.com for a couple of months.
He’s done a heroic job, as has Matthew with his epic import of Hansard data from 1935 onwards. TheyWorkForYou is a much better site for their combined work recently. We’ll be writing more on the historic stuff soon.
There are a few things I’d like from you as a member of the mySociety community:
1. Please say a big thanks to Richard. This was not an easy or relaxing task at all, and he’s done it brilliantly. Just check a Lords debate to see the attention to detail. We are a very lucky organisation to have him, as he’s always in demand.
2. Please give some constructive criticism on how it could be even better (please note, focussing on design here, we already have a load of feature priorities to deliver).
3. Anyone who could help supply a redesigned logo, or some nicely processed parliamentary-themed artwork to sit in the background grey-boxes on the homepage would be doing a very Good Deed for mySociety.
And lastly, please do pledge to become a TheyWorkForYou Patron, so we can keep doing things like this in the future!
Our video is streamed via progressive HTTP, using lighttpd and mod_flv_streaming. This works by having keyframe metadata at the start of the FLV (Flash video) file (we add ours using yamdi as that doesn’t load the whole file into memory first), which maps times within the video to byte positions within the file. When someone drags the position slider, or presses a skip button, the player actually changes the source of the video to something like file.flv?start=<byte position> which starts a new download from that point in the video. This means you can seek to parts of the video not yet downloaded, which is definitely a required feature.
The video is split up into programme chunks, according to BBC Parliament’s schedule, so each Oral Questions will (approximately) be its own video chunk, and the main debates will be a couple of chunks. By default, the video player will show a screengrab from the start of the video, as that’s all that’s available when it first loads (you have to load the start of the FLV file to fetch the keyframe metadata in order to move anywhere else :) ). I wanted the player to show a relevant screengrab before you hit Play, so came up with the slightly messy workaround of setting the volume to 0, seeking and playing the video for under a second in order to start it from the new point and show the video, then stopping it and resetting the volume. It works most of the time :-)
Some of our video chunks have jumps in them, due to problems in downloading the original WMV stream. The timestamping interface has a link for people to let us know of such problems, so that we can mark the relevant speeches as missing video and not have them be offered to future timestampers. One valiant volunteer, Tim, let us know about two such videos, but with the added oddity that if you let them play, they would happily carry on past their “end” point, but this made timestamping those speeches quite difficult.
I started investigating, firstly noting that both videos should have been 6 hours long, but were both listed as 1:20:24, which I thought was a bit of an odd coincidence. After reading the FLV file specification, it turned out that 32-bit millisecond timestamps in FLV are split into two – first the low 24 bits, then the high 8 bits. 2^24 = 16,777,216, which in milliseconds is 4 hours, 39 minutes, 37 seconds, which is pretty much exactly what the two videos’ durations were short by! All the timestamps in our FLV files were not setting the high byte, so after 4:39:37, they were wrapping round to 0 (and thus 6 hours became 1:20:24ish).
Our video processing consists of four major steps – the downloading script uses ffmpeg to convert each 75 minute chunk from WMV to MPEG; then nightly processing uses ffmpeg again to convert the right bits of these MPEG files to FLV, mencoder to join the relevant FLV files into one FLV chunk, then yamdi to add the metadata. My first try at a solution was to alter yamdi to increment the high byte itself, which fixed the duration display and let you seek to high times, but when you tried to go to e.g. 5 hours, the video started playing from the right point but the video thought it was playing from 20 minutes in. This would obviously confuse timestamping!
As the FLV files produced by ffmpeg were all under 75 minutes long, they couldn’t have the problem. It turned out we were running an old version of mencoder, and updating that and converting all our long video files fixed the problem. Phew :-)
Join us later today for my third short technical talk on TheyWorkForYou video, where I’ll explain how our Flash application talks to the HTML and vice-versa to enable the “Watch this” and highlighting of speeches.
TheyWorkForYou video timestamping has been launched, over 40% of available speeches have already been timestamped, and (hopefully) all major bugs have been fixed, so I can now take a short breather and write this short series of more technical posts, looking at how the front end bits I wrote work and hang together.
Let’s start with the most obvious feature of video timestamping – the video player itself. :) mySociety is an open-source shop, so it was great to discover that (nearly all of) Adobe Flex is available under the Mozilla Public Licence. This meant I could simply download the compiler and libraries, write some code and compile it into a working SWF Flash file without any worries (and you can do the same!).
Writing a Flex program is split into three main areas – MXML that lays out your application, defines any web services you’re using and so on; CSS to define the style of the various components; and ActionScript to deal with things like events, or talking to the JavaScript in the parent HTML. My code is probably quite shoddy in a number of places – it’s my first application in Flex :-) – but it’s all available to view if you want to take a peek, and it’s obviously running on the live TheyWorkForYou site.
To put a video component in the player is no harder than including an <mx:VideoDisplay> element – set the source of that, and you have yourself a video player, no worrying about stream type, bandwidth detection, or anything else. :) You can then use a very useful feature called data binding to make lots of things trivial – for example, I simply set the value of a horizontal slider to be the current playing time of the video, and the slider is then automatically in the right place at all times. On the downside, VideoDisplay does appear to have a number of minor bugs (the most obvious one being where seeking can cause the video to become unresponsive and you have to refresh the page; it’s more than possible it’s a bug in my code, of course, but there are a couple of related bugs in Adobe’s bug tracker).
As well as the buttons, sliders and the video itself, the current MXML contains two fades (one to fade in the hover controls, one to fade them out), one time formatter (to format the display of the running time and duration), and three web services (to submit a timestamp result, delete a mistaken timestamp, and fetch an array of all existing timestamps for the current debate). These are all called from various places within the ActionScript when certain events happen (e.g. the Now button or the Oops button is clicked).
Compiling is a simple matter of running mxmlc on the mxml file, and out pops a SWF file. It’s all straightforward, although a bit awkward at first working again with a strongly-typed, compiled language after a long time with less strict ones :-) The documentation is good, but it can be hard to find – googling for [flex3 VideoDisplay] and the like has been quite common over the past few weeks.
Tomorrow I will talk about moving around within the videos and some bugs thrown up there, and then how the front end communicates with the video in order to highlight the currently playing speech – for example, have a look at last week’s Prime Minister’s Questions.
In January last year, at our yearly staff and volunteers retreat, we decided that TheyWorkForYou should do something special for the general election. We decided that we wanted to gather information on where every candidate in every seat stood on what most people would think were the biggest issues, not just nationally but locally too.
Our reasons for setting this ambitious goal were two fold. First, we thought that pinning people down to a survey that didn’t reward rhetorical flourishes would help the electorate cut through the spin that accompanies all elections. But even more important was to increase our ability to hold new MPs to account: we want users of TheyWorkForYou in the future to be able to see how Parliamentary voting records align with campaign statements.
This meant doing quite a lot of quite difficult things:
The Volunteer Army
This has turned out to be a massive operation, requiring the creation of the independent Democracy Club set up by the amazing new volunteers Seb Bacon and Tim Green, and an entire candidate database site YourNextMP, built by another new volunteer Edmund von der Burg. Eventually we managed to get at least one local issue in over 80% of constituencies, aided by nearly 6000 new volunteers spread from Lands End to John O’Groats. There’s at least one volunteer in every constituency in Great Britain, and in all but three in Northern Ireland. Volunteers have done more than just submit issues, they’ve played our duck house game (you can still win!) to help gather thousands of email addresses, phone numbers, and postal addresses.
The Survey
What we ended up with is a candidate survey that is different for every constituency – 650 different surveys, in short. The survey always contains the same 15 national issues (chosen by a politically balanced panel held at the Institute for Government) and then anything between zero and ten local issues. We’ve seen everything from cockle protection to subsidies for ferries raised – over 3000 local issues were submitted, before being painstakingly moderated, twice, by uber-volunteers checking for for spelling, grammar, obvious bias and straightforward interestingness (it isn’t really worth asking candidates if they are in favour of Good Things and against Bad Things).
In the last couple of days we’ve started to send out the first surveys – we’ve just passed 1000 emails, and there are at least 2000 still to be sent.
The Output
We’re aiming to release the data we are gathering on candidates positions on 30th April. We’ll build a nice interface to explore it, but we also hope that others will do something with what we are expecting to be quite a valuable dataset.
The Pressure
Candidates are busy people, so how do we get their attention? Happily, some candidates are choosing to answer the survey just because TheyWorkForYou has a well know brand in the political world, but this has limits.
The answer is that we are going to ask Democracy Club, and it’s army of volunteers to help. We’ll shortly roll out a tool that will tell volunteers which of their candidates haven’t taken the opportunity to go on the record , and provide a range of ways for them to push for their candidates to fill it in.
It would be a lie to say we’re confident we’ll get every last candidate. But we are confident we can make sure that no candidate can claim they didn’t see, or didn’t know it was important to their constituents. And every extra voice we have makes that more likely.
Since the petition system went out properly on Wednesday, we’ve been absolutely buried in an avalanche of changes, fixes, feature additions and massive massive amounts of email. I thought that you might be interested as to what sort of stuff has happened in the first two days:
I’ve probably missed some – I’m sure Matthew, Chris, Francis and Ben will let me know!
Some of the work we do at mySociety these days is policy related, and happens behind the scenes. I’m conscious that we haven’t been blogging here like we did in the early days, and that is partly because advice and consultancy often have to be confidential.
Two speeches, both of which mention TheyWorkForYou, were recently given by senior members of the UK’s two main opposing parties. They’re both worth reading, and will set you thinking about how much further mySociety’s work can be taken.
First a recent speech by Tom Watson MP, a Cabinet Office Minister on “Transformational Government”. He talks about the massive change we are living through, in terms of how IT can and will improve Government.
Less than a decade ago, people were just recipients of information, they got what they were given when they were given it. Today, the most successful websites are those that bring together content created by the people who use them (Tom Watson)
Second a speech by David Cameron, Leader of the Opposition, which talks a lot about open knowledge. Can local government be transformed by better information? TheyWorkForYou for local councils?
We will require local authorities to publish this information – about the services they provide, council meetings and how councillors vote – online and in a standardised format. (David Cameron)
On Monday the 12th of April 2010 Parliament was formally dissolved in advance of the forthcoming general election. This has prompted some interesting, and rather bizarre, responses to the Freedom of Information requests which have been made to the House of Commons and House of Lords since dissolution. Each such request made via mySociety’s Freedom of Information website WhatDoTheyKnow.com has received a standard reply stating:
When Parliament has been dissolved there is no ‘House of Commons/Lords’ for the purposes of Part 1 of Schedule 1 to the 2000 Act, and there is therefore no ‘public authority’ to which the 20 day deadline under section 10 of the 2000 Act is capable of applying. The time limits do not, therefore, apply during the period of Dissolution.
The effect of the 2000 Act, including its time limits, resumes when the new House of Commons/Lords first meets.
Requests which were due to receive a response during the period parliament is dissolved have had their “clocks” stopped, with notices saying:
As your request was received before the House dissolved, the 20 working day time limit of your request will be split, ceasing on 12 April 2010 and resuming on 18 May 2010 when the new Parliament first meets.
As mySociety’s Freedom of Information website WhatDoTheyKnow automatically publishes requests, correspondence and responses online it’s not just the requestors who can see those responses, anyone can.
During the 2005 election according to the UK FOI blog Parliament placed a notice on its website saying it had consulted with the Information Commissioner and agreed the procedure for extending the time limit for a response.
Clearly the Houses of Parliament still have staff employed and people are still acknowledging the FOI requests. While both houses have stopped meeting, the institutions behind them must surely still be operating, and to claim they have ceased to exist is bordering on the utterly ludicrous.
Are the staff who would otherwise be in a position to respond to requests for things like viewer statistics for the Parliament Live TV stream, content of the Commons’ Intranet or cost of the Parliament Education Service not at their desks at the moment? If they are who’s employing them? Who’s paying them? While it is presumably a busy time for those staff preparing for a new intake of MPs; you might think that without MPs and Lords around it may be a quite time for many staff who might want to use the opportunity to catch up with correspondence like FOI requests. Perhaps in the midst of all this rather British oddness we should be happy that at least the parliament website hasn’t been turned off at this time of peak interest in the nation about parliament and our democratic system.
WhatDoTheyKnow already has to be aware of public holidays and follows some rather complex rules when it comes to calculating deadlines for responses however we have decided against updating the system to deal with this new and unexpected situation. We don’t think it is right that the institution of Parliament should consider itsself not to exist during an election period. We still be marking FOI requests as “late” when the twenty working day statutory time limit has expired regardless of the odd stance being taken by Parliamentary officials.
Partly for our own internal documentation, and partly because it might be of interest to (some) readers, some notes on how the Number 10 petitions site works. On the face of it you’d imagine this would be very simple, but as usual it’s complicated by performance requirements (our design was motivated by the possibility that a large NGO might mail a very large mailing list inviting each member to sign a particular petition, so that we might have to sustain a very high signup rate for an extended period of time). Here’s a picture of the overall architecture:
(This style of illustration is unaccountably popular in the IT industry but unlike most examples of the genre, I’ve tried to arrange that this one actually contains some useful information. In particular I’ve tried to mark the direction of flow of information, and separate out the various protocols; as usual there are too many of the latter. The diagram is actually a slight lie because it misses out yet another layer of IPC—between the web server, apache, and the front-end FastCGI scripts.)
Viewing petition pages is pretty conventional. Incoming HTTP requests reach a front-end cache (an instance of squid, one per web server, cacheing in memory only); squid passes them to the petition browsing scripts (written in perl running under FastCGI) to display petition information. Those scripts consult the database for the current state of the relevant petition and pass it back to the proxy, and thence to the web client. This aspect of the site is not very challenging.
Signing petitions is harder. The necessary steps are:
The conventional design for this would be to have the web script, when it processes the HTTP request for a new signature, submit a message for sending by a local mail server and write a row into the database and commit it, forcing the data out to disk. The mail server would then write the message into its spool directory, and fsync it, forcing it out to disk. The mail server will then pick the mail out of its queue and send it to a remote server, at which point it will be erased from the queue. Later on the mail will arrive in the
user’s inbox, at which point they will (presumably) click the link, resulting in another HTTP request which causes the web script to update the corresponding database row and commit the result. While this is admirably modular it requires far more disk writes than necessary to actually complete the task, which limits its potential speed. (In particular, there’s no reason to have a separate MTA spool directory and for the MTA to make its own writes to that directory.)
At times of high load, it is also extremely inefficient to do one commit per signature. It takes about as long to commit ten new or changed rows to the database as it is to commit one (because the time spent is determined by the disk seek time). Therefore to achieve high performance it is necessary to batch signatures. Unfortunately this is a real pain to implement because all the common web programming models use one process per concurrent request, and it is inconvenient to share database handles between different processes. The correct answer to this problem would of course be to write the signup web script as a single-process multiplexing implementation, but that’s a bit painful (we’d have had to implement our own FastCGI wire protocol library, or alternatively an HTTP server) and the deadlines were fairly tight. So instead we have a single-process server, petsignupd, which accepts signup and confirmation requests from the front-end web scripts over a simple UDP protocol, and passes them to the database in batches every quarter of a second. In theory, therefore, users should see a maximum latency of a bit over 0.25s, but we should achieve close to the theoretical best throughput of incoming requests. (Benchmarking more-or-less bears this out.)
Sending the corresponding email is also a bit problematic. General-purpose MTAs are not optimised for this sort of situation, and (for instance) exim can’t keep up with the sustained signup rate we were aiming for even if you put all of its spool directories on a RAM disk and accept that you have to repopulate its outgoing queue in the event of a crash. The solution was to write petemaild, a small multiplexed SMTP sending server; unlike a general-purpose MTA this manages its queue in memory and communicates updates directly to the database (when a confirmation email is delivered or delivery fails permanently).
It’s unfortunate that such a complex system is required to fulfil such a simple requirement. If we’d been prepared to write the whole thing ourselves, from processing HTTP requests down to writing signatures out to files on disk, the picture above would look much simpler (and there would be fewer IPC boundaries at which things could go wrong). On the other hand the code itself would be a lot more complex, and there’d be a lot more of it. I don’t think I’d describe this design as a “reasonable” compromise, but it’s at least an adequate one.
TheyWorkForYou has, until now, only covered things that have already happened, be that Commons main chamber debates since 1935, Public Bill committees back to 2000, or all debates in the modern Northern Ireland Assembly.
Upcoming business
From today, we are taking the UK Parliament’s upcoming business calendar and feeding it into our database and search engine, which means some notable new features. Firstly, and most simply, you can browse what’s on today (or the next day Parliament is sitting), or 16th May. Secondly, you can easily search this data, to e.g. see if there will be something happening regarding Twickenham. And best of all, if you’re signed up for an email alert – see below for instructions – you’ll get an email about any matching future business along with the matching new Hansard data we already send. We currently send about 25,000 alerts a day, with over 65,000 email addresses signed up to over 111,000 alerts.
Mark originally wrote some code to scrape Parliament’s business papers, but this sadly proved too fragile, so we settled on Parliament’s calendar which covers most of the same information and more importantly has (mostly) machine-readable data. Duncan and I worked on this intermittently amidst our other activities, with Duncan concentrating on the importer and updating our search indexer (thanks as ever to Xapian) whilst I got on with adding and integrating the new data into the site.
New TheyWorkForYou home page
I’ve also taken the opportunity to rejig the home page (and fix the long-standing bug with popular searches that meant it was nearly always Linda Gilroy MP!) to remove the confusingly dense amount of recent links, bring it more in line with the recently refreshed Scottish Parliament and Northern Ireland Assembly home pages, and provide more information to users who might not have any idea what the site covers.
Signing up for an email alert: If you want to receive an email alert on a particular person (MP, Lord, MLA or MSP), visit their page on TheyWorkForYou and follow the “Email me updates” link. If you would like alerts for a particular word or phrase, or anything else, simply do a search for what you’re after, then follow the email alert or RSS links to the right of the results page.
This pledge helps kids in Ooty, India to broaden their horizons by providing books containing knowledge they would otherwise not have access to. Sirukathai, the organisation behind the pledge, believes that reading material outside the child’s usual textbooks helps opening up new worlds, develop thinking processes, aid communication, foster self confidence and create a better, more balanced and compassionate citizen.
With two days to go before deadline, the getachild2read pledge has 205 people signed up to donate books. That’s five over target and there’s still time join if you wish to make a contribution!
Kalyani, the man behind this worthy project, and his “aids” are already in the process of collecting and cataloguing items coming in. To some it seems that this is a most interesting task and not at all that laboursome :-)). A more thorough follow-up report is on the way so stay tuned.
We’re very excited to announce that our Parliamentary website TheyWorkForYou.com now includes video of debates in the House of Commons – but we need your help to match up each speech with the video footage.
It’s really easy to help out. We’ve built a really simple, rather addictive system that lets anyone with a few spare minutes match up a randomly-selected speech from Hansard against the correct snippet of video. You just listen out for a certain speech, and when you hear it you hit the big red ‘now’ button. Your clip will then immediately go live on TheyWorkForYou next to the relevent speech, improving the site for everyone. Yay!
You can start matching up speeches with video snippets right away, but if you take 30 seconds to register a username then we’ll log every speech that you match up and recognise your contribution on our “top timestampers” league table. We’ll send out mySociety hoodies to the top timestampers – they’re reserved exclusively for our volunteers as a badge of honour.
We think that this really easy approach to crowd-sourcing data about online video could come in useful in many different situations – not just for politics – and we hope that it gets used all over the place. It might even be a world first, we’re not sure. If you’d like us to create something similar for your local legislature, sports team, Am Dram group or anything else that can be audio or video recorded then please get in touch. We’d also really appreciate your feedback on the current beta system – please send your email to hello@mysociety.org.
Note to MPs, researchers, office staff, campaigners and bloggers – we know that you want to concentrate on matching up the speeches of a particular MP, or of a particular debate. If this sounds like you, please send an email to hello@mysociety.org with what you want, and we’ll help you do it.
Background
This project was initially commissioned and funded by the BBC, who asked mySociety to create a searchable, online video archive of debates based on footage from BBC Parliament. We were thrilled to help out, because we think that it will enhance the public understanding of – and respect for – the work of Parliament. The initial goal of this project was to use the BBC’s captions to help chop up the video into different speeches. Tom Loosemore arranged for access to the BBC’s internal captions data, Etienne Pollard was commissioned to build an open source recording/transcoding/web-serving system (and then donated some of his wages back to pay for enough hard drive space for the video!), Stef Magdalinski donated a network storage array to hold the disks. However, after lots of hard work trying to get our computers to automatically slice up the video into chunks according to the BBC’s captions we concluded that this on its own wasn’t sufficiently accurate to reliably match up every speech in Hansard with the appropriate snippet in our video footage.
Adversity, however, is a great source of innovation. Matthew Somerville, working on a spec first sketched out by Tom Steinberg customised the flash interface substantially so that users could watch video and help add correct timestamps. Now that’s built, what remains is for you to do your part! What’s more, once we get a significant number of speeches timestamped we can start providing web feeds and APIs for MPs to embed video footage directly on their own websites, and video of your MP’s most recent speeches on their MP page on TheyWorkForYou.
There are some conflicting views about whether this all online video of Parliament is a good idea – for instance, this video snippet (created using the new system) shows that the Deputy Leader isn’t so keen on the idea of Parliamentary footage appearing on sites like YouTube. Or perhaps she’s just been misunderstood – now you can judge for yourself what she was saying, based on her appearance and intonation. On the other hand, the BBC seem to understand the benefit of putting video content online (and they’re a fully paid up member of ParBol, the Parliamentary Broadcasting group), and Parliament themselves have an alternative set of online video streams. Unfortunately the official Parliamentary video service can’t be integrated with Hansard, is only available in Windows Media format, only has enough storage to keep the most recent 28 days of footage in archive, and doesn’t even attempt to break up the video into individual speeches apparently you can search for speeches after all, although this capability isn’t actively advertised. It perhaps goes without saying that mySociety considers it an important public service for citizens to be able to find footage of their MPs doing their work, and we will resist attempts to deny this service to citizens.
One final thing – we’re currently trying to persuade the clerks in Parliament to tweak their internal processes a bit, and make it easier for people to see how laws are made. It’s called the Free Our Bills campaign, and we need as many people as possible to join the campaign, so that we can bring law-making into the 21st century. Please sign up now!
Update 1.40PM
There are already over 1000 timestamps, and we’ve not even gone for any media coverage yet. Well done all!
Update 11.00AM on Thursday 5 June 2008
6769 speeches have now been timestamped, which is exactly 20% of the current total of 33838 speeches. Thanks for all your efforts, and keep up the good work!
7,000 members on the Facebook group, over 93% of MPs contacted. Lots of news coverage: BBC, Daily Mail, Guardian, Telegraph, Times. John Mann MP makes a good point in a letter to the Guardian: “Few of my constituents care about the detail of how I spend their money as long as I do a good job, but nearly all of them care that they have the right to find out if they really feel the need to.”
Tom has updated the main blog post with a quote from President Obama’s speech that I thought was worth repeating, on why this is a much bigger issue than some bits of paper and some minor embarrassment: “And those of us who manage the public’s dollars will be held to account – to spend wisely, reform bad habits, and do our business in the light of day – because only then can we restore the vital trust between a people and their government.”
If you haven’t already, do write to your MP, and pick up the phone and call your local radio and TV news stations to let them know about this.
We just had our irregular weekly meeting, which we do most Mondays using a conference call. I thought I’d just write up what we’re all up to this week.
Please ask questions in the comments – for example, if you’d like us to post about particular things on this blog.
Whew. It’s exciting times.
As you know, we’ve been looking for a new developer for a little while, and I’m pleased to say we’ve found one. We’re very picky, as we have lots of really convulted, diverse software amazing award winning websites to keep going.
Just as important as finding someone technically skilled, it’s important that they are motivated and excited about what mySociety is doing. If you’re hiring any programmer you should be looking for that, especially so for a small, nimble charity like us. We had lots of good applicants, and were sorry we could only afford to choose one.
Please welcome Angie Ahl! She lives in Cumbria, so keeping with our policy of having staff scattered to the four winds. I can see mountain climbing in my future. Angie runs a web design company working mainly in the film and music industries. I’ll try and persuade her to post here about what she gets up to with us.
Other things – there’s another one of our Disruptive Technology talks in London next Thursday. It’s by Jason Kitcat, who not only is head of technology at netmums.com, but also co-ordinated the Open Rights Group’s electronic voting trials observations earlier in the year. Read the fascinating report – there’s a bit where overall control of the Scottish Parliament literally hung off the edge of the page of an Excel spreadsheet. Sign up now to come and see Jason speak next week.
Speaking of the Open Rights Group, it is astonishingly two years since a PledgeBank pledge got them started with 1000 supporters. Danny O’Brien has written a fun summary of what they’ve been up to. If they can do this, then what could you do with PledgeBank?
An MP page scorecard from the re-launched Mzalendo
When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.
These days dozens of such sites exist around the world. But today sees the launch of a rather-special new transparency site: Mzalendo, covering the Parliament of Kenya.
Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:
The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.
Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.
I’ve been doing lots of research around “cloud computing” recently, so we can change how Mapumental works and take it out of private beta.
One thing that’s struck me is that there doesn’t seem to be a proper, industry standard name to distinguish what to me are two fundamentally different sorts of “cloud computing”. I’m focusing here entirely on cloud services for programmers (let’s leave what it means to end users or businesses for another day).
Here are my own names and descriptions of them:
1) Cloud hardware server provision (Cloud HSP)
Low level APIs for making and destroying (virtual) servers, and loading machine images onto them. e.g. Amazon Elastic Compute Cloud, Rackspace Cloud Servers, Eucalyptus’s EC2 bits. Basically, what Eucalyptus v 1.5 can do and what libcloud should do. (By analogy, this is the assembly language of cloud computing)
2) Cloud developer service provision (Cloud DSP) A service that a developer accesses with one name and a simple API, and behind the scenes it scales for him, automatically. e.g. Amazon Queue Service, Rackspace Cloud Files. (By analogy, this layer is the C programming language of cloud computing)
[as an aside, Google AppEngine is an interesting one. It is definitely in the Cloud DSP category, but I think it is larger than that - it is a whole set of APIs all in that category. Something like Google DataStore is a single Cloud DSP, albeit one apparently only accessible within AppEngine apps]
It’s possible to use a Cloud HSP (assembly language), along with a bunch of your own software or open source software, to build new Cloud DSPs (C code). Right now this is pretty hard – even quite well known open source distributed datasbases like CouchDB still need scripting to even make them replicate. The code that makes and destroys servers and gives the service one name, needs manually stringing with quite new bits of wire (things like scalr and Wackamole).
For this reason, I’m reluctant for mySociety to get into the “making our own Cloud DSP out of Cloud HSP” game. It feels to me like a suck of time, and like we wouldn’t be able to guarantee without lots of careful and expensive testing that it would scale. I’m more tempted to use the commercial Cloud DSP services where possible, even though they are proprietary. But use them via our own abstraction layer, so we can change as we need to. Of course, we have some C++ code (the public transport route finder), so will have to use the Cloud HSP API to get that going, perhaps with Amazon’s Auto Scaling. But it can jolly well use AQS and S3 to talk to other services.
So, what do you think about the names Cloud HSP/DSP? Are there already existing names for the distinction that I’m making? Is it a useful distinction for you? Can you think of better names?
The volunteer team behind our Freedom of Information (FOI) site
WhatDoTheyKnow.com, has used statistics released by the Ministry of Justice to discover the proportion of all FOI requests being made via the site.
They found that in the first quarter of 2009, 8.5% of all requests made to central government departments were made using WhatDoTheyKnow. In absolute terms that was 514 of 6019 requests.
The breakdown by department is given in the below table. Notably, one in five FOI requests to the Home Office (122 of 643) were made via WhatDoTheyKnow.
WDTK = WhatDoTheyKnow; Source for total FOI request statistics : Statistics for Q1 2009 (released on the 25th of June 2009); Extended table covering all monitored bodies available.
The Ministry of Justice only monitors, and provides statistics on, 44 bodies’ compliance with the Freedom of Information Act; WhatDoTheyKnow currently lists 2910. We cover a wide range of local bodies including Primary Care Trusts, Local Councils and the Police. There is no national monitoring of how many FOI requests are made to such bodies, or how well they perform when responding to requests.
If you want to see such performance statistics, please help categorise more of the responses made via the site. It can be quite addictive!
Thanks to Richard Taylor for doing this research – see his blog post for some more details, including some information about Scotland.
Back in November 2006 we launched Number 10’s petitions website. We were pretty proud of the usability-centred site we built – we can still lay a pretty good claim to it being one of the biggest democracy sites (measured in terms of people transacting) that the world’s ever seen.
Over 12 million signatures had been added to petitions by the time the site was switched off after the 2010 general election. We were particularly proud of developing a system that was highly load-tolerant: we once survived over 20,000 people signing within a single hour, all whilst running on a pair of cheap little servers. That performance on so little hardware was down to the raw brilliance represented by a coding team made up of Francis Irving, Matthew Somerville, and the late, great Chris Lightfoot.
We’re also pleased that the popularity of the site led to the irresistible rise of the belief that the public should be able to petition the government via the internet. So even though our site was mothballed, Parliament and DirectGov have taken over the idea, and the commitment has been upped a notch, from ‘we’ll send a reply’ to ‘we’ll talk about it’. To be clear, we are not, nor have ever been a community interested in replacing representative democracy with direct democracy, but anything that can squeeze any drop of change from Parliament is worth a small celebration.
What’s most pleasing, though, is that we’ve been able to take the open source code built for Number 10, improve and expand upon it to develop a hosted petitions service for local councils around the country, or the rest of the world. And this is where we found the most important lesson for us: local petitions can be awesome, and despite the much smaller numbers of signatories involved, we’ve been more widely and frequently impressed by local petitions and responses than at the more glamorous national level. We’re particular fans of Hounslow Borough Council who have given positive and detailed feedback on all sorts of genuine local issues, as well as working hard to let local residents know that the service exists.
Just recently we launched a site to make it really easy to find local council petition websites, because there are hundreds hidden away (we built some; most are supplied by other vendors). If we could see anything result from today’s huge explosion of interest in online petitions, it would be that people might start to look local, and explore what petitions in their community could mean.
Much of my August seems to have been absorbed with maintenance tasks.
For example, Chris and I spent a few days tightening up WriteToThem’s privacy. I made sure the privacy statement correctly describes what happens with backup files, and failed messages. I reduced the timeouts on how long we keep the body of failed messages. I made sure we delete old backup files of the WriteToThem database. I wrote scripts to run periodically to check that no bugs in our queueing demon can accidentally mean we keep the body of messages for longer than we say. I added a cron job to delete Apache log files older than a month for all our sites. As AOL know to their cost, the only really private data is deleted data.
Earlier in the month, I handled some WriteToThem support email for the first time in ages. We get a couple of hundred messages a week, which Matthew mainly slogs through. It’s good for morale to do it, as we get quite a lot of praise mail. It is also hard work, as you realise how complicated even our simple site and the Internet are, and it leads to fixing bugs and improving text on the site. I made a few improvements to our administration tools, and things like the auto-responder if people reply to the questionnaire, to try and reduce the amount of support email, and make it easier to handle.
I did some more work on the geographically cascading pledges (like this prototype one), but I’m still not happy with them. In the end, I realised that it is the structure of wording of the pledge that is the key problem. Our format of “If will A but only if N others will B” just isn’t easily adapted to get across that the pledge applies separately in different geographically areas. Working out how to fix that is one of the things we’ll brainstorm about in the Lake District (see below).
The last couple of days I’ve been configuring one of our new servers who is called Balti, and getting the PledgeBank test harness working on it. Until now, it has only been run on my laptop. This is partly heading towards making a proper test harness for the ePetitions site, running on a server so we properly test nothing can be broken before deploying a new version.
Matthew has wrapped up the TheyWorkForYou API now, and is working on Neighbourhood Fixit next. Chris has been doing lots more performance work for the ePetitions site. And he’s been making some funky monitoring thing to detect PostgreSQL database lock conflicts, which we get occasionally and are hard to debug
Tom’s in Berlin at the moment, he gave a talk last night, and I think has been to see some people from Politik Digital. As we’ve been discussing on the developers email list, there’s an EU grant we’re likely to apply for in collaboration with them.
On Friday, we’re all going to the Lake district for a week, with some of the trustees and volunteers intermitently. We very conveniently and cheaply all work from home, so it’s good and necessary to meet up for a more sustained period of time at least once a year. Last year we were in Wales.
Many MPs and Lords use the Freedom of Information Act to obtain information from public authorities despite the fact they are able to table parliamentary questions. Occasionally they make their requests via mySociety’s freedom of information website WhatDoTheyKnow.com which ensures both the request, and its response, are freely available online. Surprisingly the freedom of information route can result in the release of more, and better quality, information than a written Parliamentary Question.
For example on the 12th of November 2009 Eleanor Laing the Conservative Shadow Minister for Justice submitted the following written Parliamentary Question:
To ask the Secretary of State for Health how many staff in his Department were employed on the management of freedom of information requests submitted to his Department in each year since 2005; and how much his Department spent on the management of such requests in each such year.
The response contained the number of staff per year as requested but with respect to the spending the parliamentary response stated: “The information requested on expenditure could be provided only at disproportionate cost.”
A very similar request for information had been made many months previously, in July, by WhatDoTheyKnow.com user and FOI campaigner Heather Brooke. The response to the FOI request contained more information, and more precise information, than Eleanor Laing had obtained via her parliamentary question. When the request was made via WhatDoTheyKnow how much staff substantially involved in answering requests were paid was disclosed, in detail.
While the costs of complying with a particular request are capped by regulations under the Freedom of Information Act, data on total costs of FOI compliance such as that released by this request allows the average costs of dealing with a request to be calculated.
Do let us know in the comments if you’ve spotted any more!
WhatDoTheyKnow keeps growing and growing, sucking people in from Google as its archive of maybe 8.5% of Freedom of Information requests gets more and more detailed.
(Graph of number of FOI requests made using WhatDoTheyKnow over time; click for larger version)
There’s round about 8Gb of unfettered Government data in the core database, plus a whole bunch more for indexing and caching. For comparison, TheyWorkForYou (which now goes back to 1935) has 12Gb. And it’s catching up on traffic also – WhatDoTheyKnow has about half the number of visitors as TheyWorkForYou.
Unfortunately, this new found traffic has led to performance problems. You might have seen errors when using WhatDoTheyKnow in the last week or two. This post is firstly an apology for that. Thank you for your patience. Hopefully it is fixed now – do let us know if you get problems still. And secondly it is some techy stuff about debugging such problems in Ruby on Rails…
When WhatDoTheyKnow started failing, we did the obvious things to start with – moving the database to a separate server, and moving some other services off the same server, to give WDTK more room to breathe. It still kept breaking.
None of my server monitoring tools shed any very clear light as to the problem. I upgraded to the latest version of Passenger, the best Rails deployment tool I’ve seen yet. It’s pretty good, but still not mature enough for my liking. I was still getting the same problems with it, but reporting tools like passenger-memory-stats were really helpful.
Eventually I worked out that it was to do with memory use of the Rails processes. Individual ones would leap up to 1Gb, and never drop back down. If several did, the server (with 4Gb of RAM) would start swapping and grind to a halt. The world of Ruby and Rails memory monitoring software is patchwork at best, and in the end I found the simplest tools the most useful. Here’s some:
check system localhost
if memory > 3500 MB then exec "/usr/sbin/apache2ctl graceful"
I’ve more work to do, I think there are quite a few other quick wins, all of which are making the site faster too. I’m quite happy that WhatDoTheyKnow also has a bunch more test code as a result of all this.
On the other hand, what a disappointing disaster for open source languages beginning with P/R (as opposed to J). Yes, the help and tools were just about there to work it out, but would seem primitive if you’d used say Java’s Memory Analyzer. Indeed somebody over on StackOverflow suggested running your site in JRuby and using exactly that tool…
The Government is currently proposing to reform the UK’s defamation laws. The WhatDoTheyKnow.com team has responded to the consultation on a Draft Defamation Bill currently being run by the Ministry of Justice.
The bill proposes extending and clarifying the list of types of material subject to “privilege” ie. which can be published without fear of being sued for defamation. “Matter published by or on the authority of a government or legislature anywhere in the world” is already covered but we have been advised that might not extend to all Freedom of Information responses; if it does or not is something which is yet to be tested in court.
We are asking for the law to be clarified and for “privilege” to be extended to a fair and accurate report or summary of, copy of or extract from material released by public bodies. The proposed new provision would enable the republication of Freedom of Information responses from public bodies without fear of libel action. Such a provision would clearly be of value to services such as WhatDoTheyKnow.com. It would also allow campaigners, journalists and others working with such material more freedom from legal threats and uncertainty; as such this addition would appear to be in-line with the coalition’s stated aims of their amendments to libel law.
We would rather see Parliament explicitly clarify the law rather than see a journalist, campaigner or website operator be subjected to an expensive and time consuming legal action.
We have also suggested:
The full consultation response can be read online at FOIWiki
The Ministry of Justice recently released the latest statistics on freedom of information implementation in central government for the first quarter of 2010. We can use this data to roughly calculate the share of FOI requests made via mySocety’s Freedom of Information website WhatDoTheyKnow.com.
According to the Ministry of Justice 6,857 requests were made to “Departments of State” between January and March 2010, of which 707 were made via WhatDoTheyKnow – a 10% share. Defra and DCSF had the highest percentage of WhatDoTheyKnow requests in the first quarter, at 23% and 24% respectively. We have made the departmental breakdown of the statistics for the past two years available as a Google Spreadsheet.
Precise calculations of the percentage share are not possible, due to the way each department defines a valid request. For example, some FOI requests to the UK Border Agency (included in Home Office data) or the FCO made via WhatDoTheyKnow would not have been counted as FOI requests, but instead as “routine requests for information”, falling outside the scope of data collection for the statistics produced by the Ministry of Justice. In addition, requests for data readily available (i.e. exempt under Section 21 of the Freedom of Information Act) are not counted either by the MoJ, but will be included in the WhatDoTheyKnow statistics. Overall, these form a minority of requests made by WhatDoTheyKnow, so the percentage share we quote is fairly accurate.
WDTK share of central departments’ FOI requests
The graph shows that the total numbers and percentage of requests have been fairly consistent over the past year, around 10-13%. Share of requests increased significantly during 2008 as the site grew in popularity following its launch.
The large increase in total FOI requests between Q4 2008 and Q1 2009 may partly be due to the increased popularity of WhatDoTheyKnow and also increased visibility of Freedom of Information to the general public through more mentions of FOI law in the media. Key stories in the press included Jack Straw’s veto against disclosure of the Iraq War Cabinet Minutes and Parliament’s aborted attempt to hide details of MPs’ Expenses. Requests made via WhatDoTheyKnow accounted for around 20% of the total increase in FOI requests.
It’s worth bearing in mind that the above graph only covers the 22 central Departments. WhatDoTheyKnow.com lists over 3,750 authorities including local government councils, police forces, NHS Trusts, non-ministerial departments and many more, and also covers Scotland’s public authorities. There is no national monitoring of how many FOI requests are made to such bodies, or how well they perform when responding to requests.
In total, 6,565 requests were made via WDTK in the first quarter, 87% of which were to non-central Departments. It’s unlikely that the site will ever get full coverage as the majority of FOI requests originate from companies carrying out market research, journalists, political parties and charities, most of whom prefer to make requests in private in order to maintain exclusivity over the released data. WhatDoTheyKnow would like to include these companies in future, possibly by offering a managed FOI service on a fee basis, including an embargo period prior to the released information being made fully available on the internet – more information is available from the WhatDoTheyKnow team.
The Register has an article today Parliament’s take on Freedom of Information which describes an FOI request I made using WhatDoTheyKnow, and the House of Commons’ refusal to respond to it because the response would be automatically republished.
Hopefully the House will choose to waive copyright on the document, and send it soon – I still haven’t seen a good reason why they could or should not.
(Also, I haven’t changed my name to Francis Stirling, hopefully The Register will correct it soon!)
Image by Adam Tinworth
Some time in the middle of last night, our Freedom of Information site WhatDoTheyKnow.com was used to send its 100,000th FOI request. It was a simple one, made to the Queen Victoria Hospital NHS Foundation Trust.
WhatDoTheyKnow was launched in February 2008, with these aims: to make it easy to file a FOI request, and to keep a public archive of the requests and (more importantly) the responses received from public bodies. The Freedom of Information Act had been in force since 2005, but we wanted to make it fully accessible to people who were not journalists, lobbyists or professional operatives – it is a law that gives us all a right, not just those experts.
At base, mySociety is about giving people power to people who don’t believe that they have any way of affecting the world around them. Giving practical access to the right enshrined in this Act was and is a meaningful way of advancing that goal.
Then, thanks to a flash of inspiration from our late colleague Chris, we saw a great opportunity to increase the value created by the existence of the Act: we built a system that published the entire exchange of messages between users and public bodies online.
We believe that because of this decision to publish all exchanges with public bodies, WhatDoTheyKnow represents a very unusual phenomenon: a third-party web site that takes an existing piece of legislation and makes it better value for money for the taxpayer. Public money was already being spent answering FOI, but by running WhatDoTheyKnow we could magnify the value generated by each request by making it public, without requiring anyone who worked in a public sector to retrain, buy a new computer system or spend any new money.
And this theory turned out to be right. For every request made on the site, around twenty people come to read materials contained on WhatDoTheyKnow. The multiplier is remarkable, and one of the things that we think is most worth celebrating about this site.
WhatDoTheyKnow’s success is only possible because of a team of fantastically dedicated volunteers. These loyal enthusiasts have helped countless users, and do a simply amazing amount of maintenance work to keep the site friendly, helpful and effective. They are astonishingly talented, principled and knowledgeable, and mySociety owes them a debt of gratitude it will never really be able to pay back.
However, to give them a bit of the credit they deserve, and to highlight some of the countless uses of WhatDoTheyknow, we asked them to pick out some notable requests from the last four years.
Helen “The use of the site by campaign groups like the Campaign for Better Transport to find out about bus subsidy cuts as part of their save our buses campaign.”
John “There was the accidental release of how tax is applied to the Royal Family – which resulted in a Daily Mail front page story.”
Alex “This request about the Warmfront boiler installation scheme has a significant number of annotations. What makes it different is that the user patiently persisted with her original FOI requests, and then has carried on by continuing to help loads more people with details of how to complain and lobby for help and general warm encouragement.”
WhatDoTheyKnow is one of mySociety’s most visited sites, with one and a half million unique visitors in 2011. Like our other projects, it was built as an open source project. Thanks to the Open Society Foundation, we are in the process of making it much easier to re-deploy around the world, under the brand name ‘Alaveteli’. As we speak, there are sites based on our code in places as far apart as New Zealand, Kosovo, Brazil, and the EU, and we’re looking forward to helping people from around the world create more grandchild sites in the years ahead.
Our 100,000 request milestone comes at an interesting time for the Freedom of Information Act. It’s currently under scrutiny by the Justice Select Committee, who are investigating whether it works effectively and in the way that it was intended.
As you might expect, at mySociety, we’re passionate about the right to information. We’ll be submitting evidence to the Justice Select Committee to show just how vital FOI is to good government and a good society. If FOI has touched your life, you might want to do the same.
The Ministry of Justice have just published their latest quarterly statistics on the handling of Freedom of Information requests by central government bodies. We’ve crunched the numbers to compare them to the requests made using WhatDoTheyKnow.com
The graph shows our share of FOI requests sent to central Departments of State jumped to 14.6% in the 1st quarter of 2011.
This time round, the top 3 departments were: