1. Fixing public transport one email address at a time

    Ever got a problem fixed by reporting it on FixMyStreet? Written to your representative via WriteToThem? Here’s an opportunity to pay the favour forward to someone stranded on a wet Wednesday by the non-arrival of the number seven bus.

    We’ve reached the point in FixMyTransport development where we can start asking for your help. We need to fill in the information we’ll use to report people’s transport problems to the companies that run bus and train routes. If you have five minutes to spare, please spend them adding a contact email address or two for your local bus companies to this spreadsheet:


    …then you can bask in the glory of a karmic balance restored*.

    * Will also work if you accidentally ran over a kitten on your way to work this morning.

  2. ScenicOrNot raw data now available for re-use

    Scenic Or Not

    Matthew’s just updated ScenicOrNot, the little game that we built to provide a ‘Scenicness’ dataset for Mapumental, to include a data dump of the raw data. The dump will update automatically on a weekly basis, but currently it contains averaged scores for 181,188 1*1km grid squares, representing 83% of the Geograph dataset we were using, or 74% of all the grid squares in Great Britain. It is, in other words, really pretty good, and, I think, unprecedented in coverage as a piece of crowd sourced geodata about a whole country.

    It’s available under the Creative Commons Attribution Noncommercial 3 Licence, and we greatly look forward to seeing what people do with it.

  3. mySociety launches ScenicOrNot

    Score - 9.5: Across Loch Ericht to Sron a Chlaonaidh (by Geoff White on Geograph)

    Harry Metcalfe and co have been working for us recently to build a new mini-site, ScenicOrNot.

    The goal of ScenicOrNot is to be a gentle-ambling sort of quasi game that’s just compelling enough to keep clicking on, just in case the next picture is the dream valley in which you wish to be buried.

    mySociety’s obviously not in the business of building games for their own sake, though. This is another crowdsourcing experiment to solve a specific problem – we need a scenicness map of the UK for a major upcoming mySociety project, and there ain’t one to be had any other way, for love or money.

    So if you like mySociety, or just want to ogle the best and worst of this Island, please have a play.

  4. Check the FOI addresses that we have

    We sometimes have incorrect or out of date addresses for sending Freedom of Information requests to. Now anyone can check our addresses. Click “view FOI email address” on the page for any authority, and enter two of those squiggly words to prove you are not a robot.

    If you are using WhatDoTheyKnow, and suspect problems with a request, please do check the address we are using is correct. If you are from an authority, or work closely or know a particular authority, please also check the address.

  5. Amazing Volunteers do Entire Year of TheyWorkForYou Video Clip Timestamping in weeks

    The epic task of manually matching each of the 42,019 video clips of MPs was started way, way back, ooh, about 12 whole weeks ago. Two days ago the Number 1 rated volunteer timestamper in our league table, Abi Broom, completed the last clip in our database, bringing her personal tally to 8,543 clips.

    Abi Broom, No1 timestamping league table champ

    Last night we went out and met with Abi and Robert Whittakker, one of the other super-timestampers who had done over 2,000 himself.

    As a result of their efforts, and those of hundreds of other volunteers, we have put all the video that we have of the House of Commons sitting over the last year online, next to the text of the debates. The many thousands of people per day who visit TheyWorkForYou can, as a direct consequence of this work, now see video of most of the debates for the last year. When people embed clips on their own sites, that’ll also be thanks in part to our volunteers.

    We went out for ice-cream at the end of the evening.

    Robert Whittaker, down in 5th place with a measly 2047 clips to his name

    When Parliament starts again in the Autumn there’ll be another 300-400 clips a day to do, but we have a feeling the only problem doing them will be who gets to them first.

    In the meantime, we’ll soon be working on another game-like toy to help create more data. Hint – it might have something to do with GroupsNearYou.

  6. Mozilla and Ubuntu have an opportunity (and a duty) to unlock the cognitive surplus

    There’s been a lot written recently about the cognitive surplus, a phrase coined by Clay Shirky to describe the amount of human energy that can be deployed to create things if only barriers are lowered and incentives sharpened.

    mySociety has recently been fortunate enough to see a little of this phenomenon through the explosion of volunteering activity which grew up around our TheyWorkForYou video timestamping ‘game’. For those of you not familiar, we needed video clips of politician’s speaking matched with the text of their speeches, and in just a couple of months a gang of volunteers new and old have done almost all of the video in the archive. Other, much larger examples include reCAPTCHA and the ESP game.

    Reflecting on this, my friend Tom Lynn suggested that there was a gap in the market for a service that would draw together different crowdsourcing games, ensure that their usability standards and social benefit were high, and which then syndicate them out in little widgets, recaptcha style, to anyone who wanted to include one on a web page.

    This is where Mozilla and Ubuntu come in. Anyone who uses Firefox knows what the home page is like, essentially the Google homepage with some Firefox branding. Ubuntu’s default browser homepage, post patch upgrade especially, is similarly minimalist and focused on telling you what’s changed.

    Therein lies the opportunity – using pieces of these default home pages (maintained by organisations that claim to have a social purpose, remember) for more good than simply repeatedly reminding users about the the brand of the product. Traditionally that would mean asking people to donate or become volunteers, but the new universe of ultra-easy crowdsourcing games are challenging that assumption.

    Here’s a scenario. One time in ten when I load Firefox, the homepage contains a widget right under the search box that contains an almost entirely self explanatory task that contributed to the public good in some way. This could be spotting an object on a fragment of satellite photo after a disaster, typing in a word that’s difficult to OCR, timestamping a video clip, or adding tags to an image or a paragraph of text. The widgets would be syndicated from the central repository of Cognitive Surplus Foundation ‘games’, and would help groups like Mozilla and Ubuntu to show themselves to millions of tech-disinterested users to be the true 21st century social enterprises that they want to be.

  7. Internal links, and search engine crawlers

    TheyWorkForYou now finds whenever an old version of Hansard is referenced (which they do by date and column number, e.g. Official Report, 29 February 2008, column 1425) and turns the citation into a link to a search for the speeches in that column on that date. This only really became feasible when we moved server, upgraded Xapian, and added date and column number metadata (among others), allowing much more advanced and focussed searching – the advanced search form gives some ideas. Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech, much like our video matching (nearly 80% of our archive done!). 🙂

    Kudos to Google and Yahoo! for spotting this change within a couple of days, as they’re now so busy crawling everything for changes that they’re slowing the whole website down… 😉

  8. TheyWorkForYou video – the Flash player

    TheyWorkForYou video timestamping has been launched, over 40% of available speeches have already been timestamped, and (hopefully) all major bugs have been fixed, so I can now take a short breather and write this short series of more technical posts, looking at how the front end bits I wrote work and hang together.

    Let’s start with the most obvious feature of video timestamping – the video player itself. 🙂 mySociety is an open-source shop, so it was great to discover that (nearly all of) Adobe Flex is available under the Mozilla Public Licence. This meant I could simply download the compiler and libraries, write some code and compile it into a working SWF Flash file without any worries (and you can do the same!).

    Writing a Flex program is split into three main areas – MXML that lays out your application, defines any web services you’re using and so on; CSS to define the style of the various components; and ActionScript to deal with things like events, or talking to the JavaScript in the parent HTML. My code is probably quite shoddy in a number of places – it’s my first application in Flex 🙂 – but it’s all available to view if you want to take a peek, and it’s obviously running on the live TheyWorkForYou site.

    To put a video component in the player is no harder than including an <mx:VideoDisplay> element – set the source of that, and you have yourself a video player, no worrying about stream type, bandwidth detection, or anything else. 🙂 You can then use a very useful feature called data binding to make lots of things trivial – for example, I simply set the value of a horizontal slider to be the current playing time of the video, and the slider is then automatically in the right place at all times. On the downside, VideoDisplay does appear to have a number of minor bugs (the most obvious one being where seeking can cause the video to become unresponsive and you have to refresh the page; it’s more than possible it’s a bug in my code, of course, but there are a couple of related bugs in Adobe’s bug tracker).

    As well as the buttons, sliders and the video itself, the current MXML contains two fades (one to fade in the hover controls, one to fade them out), one time formatter (to format the display of the running time and duration), and three web services (to submit a timestamp result, delete a mistaken timestamp, and fetch an array of all existing timestamps for the current debate). These are all called from various places within the ActionScript when certain events happen (e.g. the Now button or the Oops button is clicked).

    Compiling is a simple matter of running mxmlc on the mxml file, and out pops a SWF file. It’s all straightforward, although a bit awkward at first working again with a strongly-typed, compiled language after a long time with less strict ones 🙂 The documentation is good, but it can be hard to find – googling for [flex3 VideoDisplay] and the like has been quite common over the past few weeks.

    Tomorrow I will talk about moving around within the videos and some bugs thrown up there, and then how the front end communicates with the video in order to highlight the currently playing speech – for example, have a look at last week’s Prime Minister’s Questions.

    1. The Flash player
    2. Seeking
    3. Highlighting the current speech
  9. Awesome progress on video timestamping

    Wow! The video timestamping site on TheyWorkForYou has been live for just under ten days, but you’ve already managed to timestamp almost 14,000 clips – that’s almost 40% of our entire video archive!

    Thank you for all the good work – and especially to our top timestampers. David Jones, Alex Hazell and Lee Maguire are currently the top three in the overall rankings, but there are five more people who have timestamped more than 500 clips, another seventeen people who have done 100 clips or more, and more than 100 people who’ve done anything from 1 to 100 clips. And of course, there’s also a fair few anonymous people who haven’t yet registered, so their individual contributions to the “anonymous” total of 3349 clips are not recorded on the league tables. Remember, we’ll be handing out prizes to the top timestampers, so get registered before you timestamp your next video!

    We’re starting to collect a list of notable clips that we can use to compile a “best of parliament” video gallery – if you would like to nominate a particular speech, please leave a comment below or send an email to team@theyworkforyou.com – just tell us the name of the MP speaking, and the URL of the page where this speech appears on theyworkforyou.com. We’ll put the best of them together and publish a list later this summer.

  10. Video recordings of the House of Commons on TheyWorkForYou.com

    We’re very excited to announce that our Parliamentary website TheyWorkForYou.com now includes video of debates in the House of Commons – but we need your help to match up each speech with the video footage.

    It’s really easy to help out. We’ve built a really simple, rather addictive system that lets anyone with a few spare minutes match up a randomly-selected speech from Hansard against the correct snippet of video. You just listen out for a certain speech, and when you hear it you hit the big red ‘now’ button. Your clip will then immediately go live on TheyWorkForYou next to the relevent speech, improving the site for everyone. Yay!

    You can start matching up speeches with video snippets right away, but if you take 30 seconds to register a username then we’ll log every speech that you match up and recognise your contribution on our “top timestampers” league table. We’ll send out mySociety hoodies to the top timestampers – they’re reserved exclusively for our volunteers as a badge of honour.

    We think that this really easy approach to crowd-sourcing data about online video could come in useful in many different situations – not just for politics – and we hope that it gets used all over the place. It might even be a world first, we’re not sure. If you’d like us to create something similar for your local legislature, sports team, Am Dram group or anything else that can be audio or video recorded then please get in touch. We’d also really appreciate your feedback on the current beta system – please send your email to hello@mysociety.org.

    Note to MPs, researchers, office staff, campaigners and bloggers – we know that you want to concentrate on matching up the speeches of a particular MP, or of a particular debate. If this sounds like you, please send an email to hello@mysociety.org with what you want, and we’ll help you do it.


    This project was initially commissioned and funded by the BBC, who asked mySociety to create a searchable, online video archive of debates based on footage from BBC Parliament. We were thrilled to help out, because we think that it will enhance the public understanding of – and respect for – the work of Parliament. The initial goal of this project was to use the BBC’s captions to help chop up the video into different speeches. Tom Loosemore arranged for access to the BBC’s internal captions data, Etienne Pollard was commissioned to build an open source recording/transcoding/web-serving system (and then donated some of his wages back to pay for enough hard drive space for the video!), Stef Magdalinski donated a network storage array to hold the disks. However, after lots of hard work trying to get our computers to automatically slice up the video into chunks according to the BBC’s captions we concluded that this on its own wasn’t sufficiently accurate to reliably match up every speech in Hansard with the appropriate snippet in our video footage.

    Adversity, however, is a great source of innovation. Matthew Somerville, working on a spec first sketched out by Tom Steinberg customised the flash interface substantially so that users could watch video and help add correct timestamps. Now that’s built, what remains is for you to do your part! What’s more, once we get a significant number of speeches timestamped we can start providing web feeds and APIs for MPs to embed video footage directly on their own websites, and video of your MP’s most recent speeches on their MP page on TheyWorkForYou.

    There are some conflicting views about whether this all online video of Parliament is a good idea – for instance, this video snippet (created using the new system) shows that the Deputy Leader isn’t so keen on the idea of Parliamentary footage appearing on sites like YouTube. Or perhaps she’s just been misunderstood – now you can judge for yourself what she was saying, based on her appearance and intonation. On the other hand, the BBC seem to understand the benefit of putting video content online (and they’re a fully paid up member of ParBol, the Parliamentary Broadcasting group), and Parliament themselves have an alternative set of online video streams. Unfortunately the official Parliamentary video service can’t be integrated with Hansard, is only available in Windows Media format, only has enough storage to keep the most recent 28 days of footage in archive, and doesn’t even attempt to break up the video into individual speeches apparently you can search for speeches after all, although this capability isn’t actively advertised. It perhaps goes without saying that mySociety considers it an important public service for citizens to be able to find footage of their MPs doing their work, and we will resist attempts to deny this service to citizens.

    One final thing – we’re currently trying to persuade the clerks in Parliament to tweak their internal processes a bit, and make it easier for people to see how laws are made. It’s called the Free Our Bills campaign, and we need as many people as possible to join the campaign, so that we can bring law-making into the 21st century. Please sign up now!

    Update 1.40PM
    There are already over 1000 timestamps, and we’ve not even gone for any media coverage yet. Well done all!

    Update 11.00AM on Thursday 5 June 2008
    6769 speeches have now been timestamped, which is exactly 20% of the current total of 33838 speeches. Thanks for all your efforts, and keep up the good work!