1. TheyWorkForYou back to 1935

    Hansards Parliamentary Debates by hugovk (cc)

    Hansard's Parliamentary Debates by hugovk (cc)

    The House of Commons debate coverage on TheyWorkForYou has recently extended back from the 2001 general election to the 1935 general election, and our knowledge of MPs now extends back to the start of the 19th century. This means TheyWorkForYou now includes things such as Anthony Eden on the Suez Canal in 1956, saying “there was not foreknowledge that Israel would attack Egypt“; the debate the day after Bloody Sunday in 1972; Geoffrey Howe’s resignation statement in 1990; Neville Chamberlain on the eve and start of the second World War in 1939; and Winston Churchill‘s speeches to the House, such as We shall fight on the beaches and This was their finest hour in 1940. This and much, much more are available and searchable using our new improved advanced search, which allows you to filter by e.g. date range or person. We hope people enjoy researching this huge wealth of information (I certainly do), and add useful annotations to the text to help other people.

    This would not have been possible without the original project by Parliament to digitise historical copies of Hansard and make them available, nor the internal Parliamentary project to clean up the data, match up speaker names, and so on. The project was kindly funded by the Ministry of Justice’s Innovation Fund, which also supported the creation of FixMyStreet and GroupsNearYou.

  2. TheyWorkForYou Redesign

    Richard Pope has been redesigning mySociety’s biggest site TheyWorkForYou.com for a couple of months.

    He’s done a heroic job, as has Matthew with his epic import of Hansard data from 1935 onwards.  TheyWorkForYou is a much better site for their combined work recently. We’ll be writing more on the historic stuff soon.

    There are a few things I’d like from you as a member of the mySociety community:

    1. Please say a big thanks to Richard. This was not an easy or relaxing task at all, and he’s done it brilliantly. Just check a Lords debate to see the attention to detail. We are a very lucky organisation to have him, as he’s always in demand.

    2. Please give some constructive criticism on how it could be even better (please note, focussing on design here, we already have a load of feature priorities to deliver).

    3. Anyone who could help supply a redesigned logo, or some nicely processed parliamentary-themed artwork to sit in the background grey-boxes on the homepage would be doing a very Good Deed for mySociety.

    And lastly, please do pledge to become a TheyWorkForYou Patron, so we can keep doing things like this in the future!

  3. Internal links, and search engine crawlers

    TheyWorkForYou now finds whenever an old version of Hansard is referenced (which they do by date and column number, e.g. Official Report, 29 February 2008, column 1425) and turns the citation into a link to a search for the speeches in that column on that date. This only really became feasible when we moved server, upgraded Xapian, and added date and column number metadata (among others), allowing much more advanced and focussed searching – the advanced search form gives some ideas. Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech, much like our video matching (nearly 80% of our archive done!). 🙂

    Kudos to Google and Yahoo! for spotting this change within a couple of days, as they’re now so busy crawling everything for changes that they’re slowing the whole website down… 😉

  4. Video recordings of the House of Commons on TheyWorkForYou.com

    We’re very excited to announce that our Parliamentary website TheyWorkForYou.com now includes video of debates in the House of Commons – but we need your help to match up each speech with the video footage.

    It’s really easy to help out. We’ve built a really simple, rather addictive system that lets anyone with a few spare minutes match up a randomly-selected speech from Hansard against the correct snippet of video. You just listen out for a certain speech, and when you hear it you hit the big red ‘now’ button. Your clip will then immediately go live on TheyWorkForYou next to the relevent speech, improving the site for everyone. Yay!

    You can start matching up speeches with video snippets right away, but if you take 30 seconds to register a username then we’ll log every speech that you match up and recognise your contribution on our “top timestampers” league table. We’ll send out mySociety hoodies to the top timestampers – they’re reserved exclusively for our volunteers as a badge of honour.

    We think that this really easy approach to crowd-sourcing data about online video could come in useful in many different situations – not just for politics – and we hope that it gets used all over the place. It might even be a world first, we’re not sure. If you’d like us to create something similar for your local legislature, sports team, Am Dram group or anything else that can be audio or video recorded then please get in touch. We’d also really appreciate your feedback on the current beta system – please send your email to hello@mysociety.org.

    Note to MPs, researchers, office staff, campaigners and bloggers – we know that you want to concentrate on matching up the speeches of a particular MP, or of a particular debate. If this sounds like you, please send an email to hello@mysociety.org with what you want, and we’ll help you do it.


    This project was initially commissioned and funded by the BBC, who asked mySociety to create a searchable, online video archive of debates based on footage from BBC Parliament. We were thrilled to help out, because we think that it will enhance the public understanding of – and respect for – the work of Parliament. The initial goal of this project was to use the BBC’s captions to help chop up the video into different speeches. Tom Loosemore arranged for access to the BBC’s internal captions data, Etienne Pollard was commissioned to build an open source recording/transcoding/web-serving system (and then donated some of his wages back to pay for enough hard drive space for the video!), Stef Magdalinski donated a network storage array to hold the disks. However, after lots of hard work trying to get our computers to automatically slice up the video into chunks according to the BBC’s captions we concluded that this on its own wasn’t sufficiently accurate to reliably match up every speech in Hansard with the appropriate snippet in our video footage.

    Adversity, however, is a great source of innovation. Matthew Somerville, working on a spec first sketched out by Tom Steinberg customised the flash interface substantially so that users could watch video and help add correct timestamps. Now that’s built, what remains is for you to do your part! What’s more, once we get a significant number of speeches timestamped we can start providing web feeds and APIs for MPs to embed video footage directly on their own websites, and video of your MP’s most recent speeches on their MP page on TheyWorkForYou.

    There are some conflicting views about whether this all online video of Parliament is a good idea – for instance, this video snippet (created using the new system) shows that the Deputy Leader isn’t so keen on the idea of Parliamentary footage appearing on sites like YouTube. Or perhaps she’s just been misunderstood – now you can judge for yourself what she was saying, based on her appearance and intonation. On the other hand, the BBC seem to understand the benefit of putting video content online (and they’re a fully paid up member of ParBol, the Parliamentary Broadcasting group), and Parliament themselves have an alternative set of online video streams. Unfortunately the official Parliamentary video service can’t be integrated with Hansard, is only available in Windows Media format, only has enough storage to keep the most recent 28 days of footage in archive, and doesn’t even attempt to break up the video into individual speeches apparently you can search for speeches after all, although this capability isn’t actively advertised. It perhaps goes without saying that mySociety considers it an important public service for citizens to be able to find footage of their MPs doing their work, and we will resist attempts to deny this service to citizens.

    One final thing – we’re currently trying to persuade the clerks in Parliament to tweak their internal processes a bit, and make it easier for people to see how laws are made. It’s called the Free Our Bills campaign, and we need as many people as possible to join the campaign, so that we can bring law-making into the 21st century. Please sign up now!

    Update 1.40PM
    There are already over 1000 timestamps, and we’ve not even gone for any media coverage yet. Well done all!

    Update 11.00AM on Thursday 5 June 2008
    6769 speeches have now been timestamped, which is exactly 20% of the current total of 33838 speeches. Thanks for all your efforts, and keep up the good work!

  5. My second entry of 2006

    I haven’t posted on here for six months; quality over quantity is my motto. 🙂 In that time I have answered a lot of user support email (we’re up to Ticket 5741 in RT, the subject of my last post, and quite a lot of that isn’t spam or out of office replies…) and made sure that Hansard has parsed successfully into TheyWorkForYou’s database every morning. I get all the fun jobs. 😉 Oh, I also quit my day job to work for mySociety basically full time, and wrote or helped write WriteToThem Lords, TheyWorkForYou Lords, various features and improvements to our other sites, and much else I’ve probably forgotten. It’s been fun.

    Today, I gave a talk (my first!) at LUGRadio Live 2006 on mySociety in general, with various open source related ramblings in particular. I wrote the presentation in Slidy, Dave Raggett’s HTML slide generation software, which means anyone should be able to see it online at https://www.mysociety.org/2006/lugradio-live/ – I’ll try and add some speaking notes to explain some of the slides in more detail. The talks were all recorded, though the audio of the talk before mine didn’t work – I’m not sure if mine did, or if so, where they’re going to be uploaded, but I’ll post here if I find out.

    Lastly, as a minor rant, my laptop’s power supply just stopped working yesterday – annoying when you hoped to use it for a talk the next day. I had bought it because my old one had become very frayed and liable to cut out randomly; I’ve had to switch back to that one (cutting out sometimes is better than nothing), and it’s proving very annoying. Dell using differently shaped plugs for all their models is rather annoying too. Grr.

  6. Peer Pressure: The House of Lords Opens Today

    Today mySociety launches a pair of complementary services sprinkling some of our democratic pixie dust on the House of Lords:

    Between them, these new services let you:

    • Identify a Lord who is interested in an issue of interest to you.
    • Write to any Lord you want via WriteToThem.com
    • View individual Peer profile pages including attendance at votes, most recent speeches, rebelliousness and more.
    • Get custom email alerts every time a certain Lord speaks, or when a word or phrase is spoken by anyone in the Lords. Over 5000 people will be mailed with unique updates today alone.
    • Produce league tables of which Lord or which MP has spoken words or phrases the most (handy for identifying which members show a public interest in which issues issues).
    • Search, read and annotate Lords Hansard back to 1999

    Collectively we think that these tools should help pour light into the activities and workings of the House of Lords, and help members of the public develop productive relationships with the peers who vote on issues of importance to all of us.