Skip navigation

  Help us to make more
useful things.
Donate to mySociety

mySociety blog » TheyWorkForYou

Parliamentary boundary changes

Monday, September 21st, 2009 by Matthew Somerville
Current Birmingham parliamentary boundaries

Current Birmingham parliamentary boundaries

Birmingham parliamentary boundaries at the next election

Birmingham parliamentary boundaries at the next election

Parliamentary boundary changes appear to be a source of confusion to many people and organisations. The facts are quite simple – parliamentary boundary changes, proposed by the various Boundary Commissions, do not take effect until the next general election. Until then, your MP remains whoever they have been, no matter what literature you may get through your letter box, or what anyone may tell you.

As one example, take Birmingham City Council. Their page on constituencies and wards correctly states that Birmingham is divided into eleven parliamentary constituencies, but then goes on to list only ten – they are listing the new constituencies which do not yet exist, as Birmingham is losing one constituency at the next election. It appears that they have organised themselves along the new boundaries in advance – which is fine, but this doesn’t affect current Parliamentary representation, and so they should explain this clearly, as otherwise members of the public get confused (and blame us for giving them the “wrong” MP, when we haven’t done so). As you can see from the maps above (which highlight Birmingham, Hall Green), the constituencies will be changing their boundaries quite a bit, and we have had reports of people receiving letters from candidates in the next election who are MPs of different neighbouring constituencies, simply referring to themselves as an MP, which is a great source of confusion.

St Josephs Avenue, and the Royal Orthopaedic Hospital are currently in Selly Oak, but will be in Northfield

St Josephs Avenue is just below the Royal Orthopaedic Hospital

An inhabitant of St Josephs Avenue, Birmingham (behind the Royal Orthopaedic Hospital), which is currently within the Selly Oak parliamentary constituency (red), and the Northfield ward of Birmingham City Council (green), would, on looking at Birmingham City Council’s website, assume they’re in a parliamentary constituency called Northfield. Northfield is currently the constituency to the west of Selly Oak; at the next election, its boundary with Selly Oak will change to the blue line, at which point St Josephs Avenue will be in the Northfield constituency. But not until then.

Map of Streatham constituency at next election

Map of Streatham constituency at next election

Current Streatham parliamentary boundary

Current Streatham parliamentary boundary

As another example (chosen purely as it has come up in user support), the Labour candidate for Streatham has a page about the constituency – obviously you would expect a candidate to be talking about the future constituency, but would it hurt to add some explanation that Streatham is currently a slightly different shape?

Boundaries of different things are all independent – if a ward boundary moves due to some local issue, the corresponding Parliamentary boundary does not necessarily change with it (probably not, in fact). So when Birmingham changed its ward boundaries back in 2003, they became out of sync with the Parliamentary constituencies. From the next election, things will be more in sync as the new Parliamentary boundaries are based on more recent ward boundaries, but this will again separate over time. All we can do is always clearly explain the current situation, and ask that others do the same.

Nine is the number: The different flavours of transparency website in 2009

Monday, September 14th, 2009 by Tom Steinberg

Image from jaygoldman

Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites

I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.

Identifying and categorising these types of site turns out to be surprisingly useful.  First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.

Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.

1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.

2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou.co.nz, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch,  Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.

3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.

4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.

5. Government spending – where the big money goes. In the US the dominant site is FedSpending.org, and in the UK we have ukpublicspending.co.uk.

6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and  Statute Law Database. Much of the legal database field, however, remains essentially private.

7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet’s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.

8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places,  the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others.  mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.

9. Political document archives - This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.

10. Bulk data - Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.

——

Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.

TheyWorkForYou back to 1935

Tuesday, July 14th, 2009 by Matthew Somerville
Hansards Parliamentary Debates by hugovk (cc)

Hansard's Parliamentary Debates by hugovk (cc)

The House of Commons debate coverage on TheyWorkForYou has recently extended back from the 2001 general election to the 1935 general election, and our knowledge of MPs now extends back to the start of the 19th century. This means TheyWorkForYou now includes things such as Anthony Eden on the Suez Canal in 1956, saying “there was not foreknowledge that Israel would attack Egypt“; the debate the day after Bloody Sunday in 1972; Geoffrey Howe’s resignation statement in 1990; Neville Chamberlain on the eve and start of the second World War in 1939; and Winston Churchill’s speeches to the House, such as We shall fight on the beaches and This was their finest hour in 1940. This and much, much more are available and searchable using our new improved advanced search, which allows you to filter by e.g. date range or person. We hope people enjoy researching this huge wealth of information (I certainly do), and add useful annotations to the text to help other people.

This would not have been possible without the original project by Parliament to digitise historical copies of Hansard and make them available, nor the internal Parliamentary project to clean up the data, match up speaker names, and so on. The project was kindly funded by the Ministry of Justice’s Innovation Fund, which also supported the creation of FixMyStreet and GroupsNearYou.

Register of Members’ Financial Interests

Friday, July 10th, 2009 by Matthew Somerville

As a new edition has just been released, and I’ve had to tweak the parser to cope with the new highlighting, it’s a good time to write a brief article on TheyWorkForYou’s handling of the House of Commons Register of Members’ Financial Interests (Register of Members’ Interests as was before the current edition). Way back in the day, a scraper/parser was written (by either Julian or Francis) that monitors the Register pages on www.parliament.uk for new editions, and downloads and broadly parses the HTML into machine-readable data. The XML produced can be found at http://ukparse.kforge.net/parldata/scrapedxml/regmem/ – TheyWorkForYou then pulls in this XML into its database, and makes the latest data available on every MP’s page.

However, as it’s been scraping/parsing the Register since 2000, we can do more than that. Each MP’s page contains a link to a page giving the history of their entry in the Register – when things were added, removed, or changed. You can also view the differences between one edition of the Register and the next, or view a particular edition in a prettier form than the official site. There’s a central page containing everything Register-related at http://www.theyworkforyou.com/regmem/

TheyWorkForYou nothing to do with this sacked civil servant story

Monday, July 6th, 2009 by Matthew Somerville

Update: The Telegraph posted a retraction yesterday.

You may have seen coverage on various websites saying that a civil servant was sacked after posting a comment on TheyWorkForYou.

We’ve no idea what this story is about, but we’re pretty certain it has nothing whatsoever to do with TheyWorkForYou. No journalist bothered to contact us before running the story.

  • There is no comment on TheyWorkForYou containing the text quoted in that article, nor anything like it, nor has there ever been. Nor in fact (as we’ve checked), on HearFromYourMP, WriteToThem, or WhatDoTheyKnow.
  • Only one comment has been left on any contribution by Hazel Blears in 2009, and it’s definitely not related to this.
  • 27 comments were left on 13th May, the date the comment was apparently posted; we’ve read them all and they’re all nothing to do with this.

So frankly, we’ve no idea what’s going on.

What we do know is that the implication that mySociety would merrily hand over sensitive personal data that ends up in getting someone sacked, without fighting tooth and nail for their privacy every inch of the way, is a complete misinterpretation of the way we work and the things we hold most dear. No-one has ever contacted us to ask us to hand over such data, nor have we ever done so.

We think what might have happened is a simple mis-remembering of the website that contained the problematic comment. We’re hoping to get in touch with Lisa Greenwood so we can get full details before asking the various media companies that have run with this for a correction.

TheyWorkForYou Redesign

Friday, July 3rd, 2009 by Tom Steinberg

Richard Pope has been redesigning mySociety’s biggest site TheyWorkForYou.com for a couple of months.

He’s done a heroic job, as has Matthew with his epic import of Hansard data from 1935 onwards.  TheyWorkForYou is a much better site for their combined work recently. We’ll be writing more on the historic stuff soon.

There are a few things I’d like from you as a member of the mySociety community:

1. Please say a big thanks to Richard. This was not an easy or relaxing task at all, and he’s done it brilliantly. Just check a Lords debate to see the attention to detail. We are a very lucky organisation to have him, as he’s always in demand.

2. Please give some constructive criticism on how it could be even better (please note, focussing on design here, we already have a load of feature priorities to deliver).

3. Anyone who could help supply a redesigned logo, or some nicely processed parliamentary-themed artwork to sit in the background grey-boxes on the homepage would be doing a very Good Deed for mySociety.

And lastly, please do pledge to become a TheyWorkForYou Patron, so we can keep doing things like this in the future!

Speaker candidates: Half the field now endorses mySociety’s principles

Wednesday, June 10th, 2009 by Tom Steinberg
Thumbs up by Carf (cc)
‘Thumbs up’ by Carf (cc)

A few days ago mySociety asked the known possible candidates for Speaker to endorse 3 principles relating to making Parliament more transparent on the Internet.

We’ve now had endorsements which you can read on the individual pages of Sir George Young, Sir Menzies Campbell, Frank Field, Tony Wright and Sir Alan Beith , which until Parmijit Dhanda declared this morning, represented endorsement by 50% of the possible field. We also just recieved a typically frank and interrogative phonecall from Ann Widdecombe, who will be writing a formal response soon.

So, come on, John Bercow, Alan Haselhurst, Patrick Cormack, Sylvia Heal, and Chris Mullin. What’s holding up your replies? The days counter on your pages is telling the world how quick you are to respond…

Update: 11 June - John Bercow has now endorsed, and we’ve written to Margaret Beckett and Parmjit Dhanda, who’ve just declared their candidacies.

Update 2: Chris Mullin has told us he is ‘not a candidate’.

Update 3: Sir Alan Haselhurst has also endorsed.

Update 4 – Speaker Election Day: And Sir Michael Lord endorses too.

3 Principles: First endorsement, from Sir George Young

Friday, May 22nd, 2009 by Tom Steinberg

In less than 24 hours we’ve seen the first reply to our emails asking possible Speaker candidates to endorse our three principles. It is from Sir George Young – we’re looking forward to seeing the responses from the others that we wrote to.

Sir George is broadly supportive, which is great, and we’ve printed his reply in full on his own TheyWorkForYou MP page.

In the mean time, please do write to your MP and ask them to ensure that whoever they vote for, it is a candidate who has endorsed mySociety’s three simple principles, You really can have an impact on this issue: MPs are desperate to be seen to be acting for their constituents right now.

NB mySociety is strictly non-partisan and non-party aligned. We want all candidates from all parties to endorse these principles, and we have ensured that none of the wording of the principles leans towards any particular party or set of beliefs not connected to transparency in the modern age.

3 Principles we are asking Speaker candidates to endorse: You can help right now

Thursday, May 21st, 2009 by Tom Steinberg
Speakers Chair (Parliamentary Copyright)
Speaker’s Chair (Parliamentary Copyright)

mySociety has today emailed (and in one case, posted) a set of 3 Principles which we believe it is important that all candidates for Speaker endorse, before the election of a new Speaker by MPs.

1. Voters have the right to know in detail about the money that is spent to support MPs and run Parliament, and in similar detail how the decisions to spend that money are settled upon.

2. Bills being considered must be published online in a much better way than they are now, as the Free Our Bills campaign has been suggesting for some time.

3. The Internet is not a threat to a renewal in our democracy, it is one of its best hopes. Parliament should appoint a senior officer with direct working experience of the power of the Internet who reports directly to the Speaker, and who will help Parliament adapt to a new era of transparency and effectiveness.

We will be posting the status of requests on the likely candidates web pages where we expect large numbers of people to see them before the vote in late June. We have also taken the unusual step of allowing possible candidates to leave a statement of up to 150 words on the principles.

(NB no candidates have actually declared at this stage, so we are starting with the BBC’s list of possibles)

Rationale

mySociety helped lead the campaign back in January to prevent the last ditch attempts to conceal MPs’ expenses. We did so not because, like the newspapers, we wanted to revel in embarrassment and scandal, but because we believe that in the Internet age, the only way for our democracy and government to thrive is if they are open and connected to the net as the rest of us expect them to be. The dramatic events seen in Parliament in recent days vindicate the view that secrecy breeds poor policies and seeds untrustworthy behaviour in the weaker willed.

Furthermore, more than a simple attitude of openness is required of the new Speaker: the public needs a genuine will to push for technological reform using the power of the Internet that will take both open-mindedness and a willing to tread on toes, especially in some parts of the unelected establishment.

Case in point: Over the last two years we have been trying to persuade Parliament to acknowledge that the way it publishes its Bills online is hopelessly inadequate for the Internet age. The campaign has faltered, despite multi partly endorsement from 140 MPs and a campaign membership of thousands. To see why, just take a look at this colourful and error-crammed internal email that we uncovered using the Freedom of Information Act, published for the first time today.

The new Speaker will have a tough job on their hands to overcome resistance of this kind. The best thing we can do is help the new Speaker, whoever they are, assume their new job with a clear mandate from the public, as well as from members.

Act!

That is why, as a final part of this call, we are asking you, our community, to write to your MP today to let them know that you expect them to vote for a candidate that has endorsed the principles above. Your voices to your own constituency MPs can resonate in a way that no blog post or newspaper article ever can. Go to it.

Blimey. It looks like the Internets won

Wednesday, January 21st, 2009 by Tom Steinberg

UPDATE: If you approve of what we did this week, and what to help make sure we can still do it in the future, please pledge to support us: http://www.pledgebank.com/supportmysociety

The vote on concealing MPs’ expenses has been cancelled by the government!

In other words – we won!

This is a huge victory not just for transparency, it’s a bellwether for a change in the way politics works. There’s no such thing as a good day to bury bad news any more, the Internet has seen to that.

Over 7000 people joined a Facebook group, they sent thousands of emails to over 90% of all MPs. Hundreds of thousands of people found out about the story by visiting TheyWorkForYou to find something they wanted to know, reading an email alert, or simply discovered what was going on whilst checking their Facebook or Twitter pages. Almost all of this happened, from nowhere, within 48 hours, putting enough pressure on Parliament to force change.

Make no mistake. This is new, and it reflects the fact that the Internet generation expects information to be made available, and they expect to be able to make up their own minds, not be spoon fed the views of others. This campaign was always about more than receipts, it was about changing the direction of travel, away from secrecy and towards openness.

Today we stopped moving in the wrong direction. Tomorrow we start moving the right way. Sign up to our news mailing list (box on the right) to get updates on what mySociety gets up to.

Wednesday morning update

Wednesday, January 21st, 2009 by Matthew Somerville

7,000 members on the Facebook group, over 93% of MPs contacted. Lots of news coverage: BBC, Daily Mail, Guardian, Telegraph, Times. John Mann MP makes a good point in a letter to the Guardian: “Few of my constituents care about the detail of how I spend their money as long as I do a good job, but nearly all of them care that they have the right to find out if they really feel the need to.”

Tom has updated the main blog post with a quote from President Obama’s speech that I thought was worth repeating, on why this is a much bigger issue than some bits of paper and some minor embarrassment: “And those of us who manage the public’s dollars will be held to account – to spend wisely, reform bad habits, and do our business in the light of day – because only then can we restore the vital trust between a people and their government.”

If you haven’t already, do write to your MP, and pick up the phone and call your local radio and TV news stations to let them know about this.

MP addresses – no relevance

Tuesday, January 20th, 2009 by Matthew Somerville

We’ve been shown or seen a few responses from MPs, after people wrote to them, saying that they are worried about their addresses being made public. If this is their main barrier to voting No on Thursday, they have nothing to worry about: they voted to exclude their residential addresses (and expenses on security, and future/regular travel) from the Freedom of Information Act in July 2008.

In fact, one of the reasons it is costing so much money to collate and edit these expenses is staff have been going through and making sure precisely that such information would not be released.

It is sad that MPs don’t seem to know what the law is, and I hope someone will stand up in the debate on Thursday and make this point.

MPs expenses – pulling out the stops

Monday, January 19th, 2009 by Tom Steinberg

A few moments ago the team rolled out changes to our biggest and best known site, TheyWorkForYou.com meaning that every visitor to any page of the site will be greeted with a call to arms on the issue of some MPs voting this Thursday to conceal their expenses. And after the vote, we’ll be prominently publishing who voted which way – there should be a couple of million visitors at least before the next election.

Our explicit goal is to have a lot of constituents from around the country let their MPs know they won’t be impressed with a ‘yes’ vote or an abstention (the same thing in this case), and to build our Facebook group to the point where the mainstream media starts to take notice of this Net driven discontent.

Please do everything you can to get as many people as possible writing to their MPs and joining that Facebook group. We’re doing our bit – please do yours. Together we can stop the encouraging trend of more openness in our Parliament scrunching into reverse.

Avoid exhausting train journeys!

Monday, November 3rd, 2008 by Francis Irving

Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.

We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.

I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.

  • TheyWorkForYou opens up access to conventional, representational democracy, between and during elections. In 10 years time, I asked for Parliament to publish all information about its work in a structured way, as hinted at in our Free Our Bills campaign. So it is much easier for everyone to help make new laws better.
  • FixMyStreet is local control of the things people care about, a very practical democracy. In 10 years time I’d like to see all councils running their internal systems (planning, tree preservation orders… everything that isn’t about individuals) in public, so everyone can see and be reassured about what is being done, why and where.
  • WhatDoTheyKnow shows the deep interest that there is by the public in the functioning of all areas of government. In 10 years time, I’d like to see document management systems in wide use by public authorities that publish all documents by default. Only if overridden for national security or data protection reasons would they be hidden.

Charles Armstrong, from the Circus Foundation, has written up the workshop.

Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort – Charles is going to work on it for another time.

Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).

PSHE lessons

Thursday, October 2nd, 2008 by Francis Irving

My house mate just said that his friend, who is at sixth form college, just had a PSHE (personal, social and health education) lesson in which they studied the website TheyWorkForYou.com.

Apparently it is good and I should go to it.

Amazing Volunteers do Entire Year of TheyWorkForYou Video Clip Timestamping in weeks

Thursday, August 28th, 2008 by Tom Steinberg

The epic task of manually matching each of the 42,019 video clips of MPs was started way, way back, ooh, about 12 whole weeks ago. Two days ago the Number 1 rated volunteer timestamper in our league table, Abi Broom, completed the last clip in our database, bringing her personal tally to 8,543 clips.

Abi Broom, No1 timestamping league table champ

Last night we went out and met with Abi and Robert Whittakker, one of the other super-timestampers who had done over 2,000 himself.

As a result of their efforts, and those of hundreds of other volunteers, we have put all the video that we have of the House of Commons sitting over the last year online, next to the text of the debates. The many thousands of people per day who visit TheyWorkForYou can, as a direct consequence of this work, now see video of most of the debates for the last year. When people embed clips on their own sites, that’ll also be thanks in part to our volunteers.

We went out for ice-cream at the end of the evening.

Robert Whittaker, down in 5th place with a measly 2047 clips to his name

When Parliament starts again in the Autumn there’ll be another 300-400 clips a day to do, but we have a feeling the only problem doing them will be who gets to them first.

In the meantime, we’ll soon be working on another game-like toy to help create more data. Hint – it might have something to do with GroupsNearYou.

acts_as_xapian

Thursday, July 17th, 2008 by Francis Irving

One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.

So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.

I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline – something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.

The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.

If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.

Internal links, and search engine crawlers

Thursday, July 17th, 2008 by Matthew Somerville

TheyWorkForYou now finds whenever an old version of Hansard is referenced (which they do by date and column number, e.g. Official Report, 29 February 2008, column 1425) and turns the citation into a link to a search for the speeches in that column on that date. This only really became feasible when we moved server, upgraded Xapian, and added date and column number metadata (among others), allowing much more advanced and focussed searching – the advanced search form gives some ideas. Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech, much like our video matching (nearly 80% of our archive done!). :)

Kudos to Google and Yahoo! for spotting this change within a couple of days, as they’re now so busy crawling everything for changes that they’re slowing the whole website down… ;-)

Postcodes on TheyWorkForYou

Tuesday, July 8th, 2008 by Matthew Somerville

If you enter your postcode on TheyWorkForYou and it’s Scottish or Northern Irish, you’re now presented with your MSPs and MLAs as well as your MP, which makes sense given the site covers their Parliament and Assembly respectively. :-) You also get an extra tab in the navigation linking through to Your MSPs or MLAs. In order to do this, I needed a quick way of determining if a postcode was Northern Irish or Scottish. Northern Ireland was easy, as all postcodes there begin with BT. I assumed Scotland was also easy, which turned out to be true apart from the TD postcode area that straddled the border like a mail-sorting Niagara Falls. After some very dull investigation, I eventually worked out that e.g. most of TD15 is in England, but (amongst others) TD15 1X* is in Scotland, except for TD15 1XX which is apparently back in England. The final result was the postcode_is_scottish() function in postcode.inc, which (hopefully) correctly determines if a given postcode is Scottish or not – perhaps someone else will find it useful.

Highlighting the current speech

Friday, June 13th, 2008 by Matthew Somerville

Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?

I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. :-)

Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.

The highlighting works the other way – as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! :) ).

Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.

Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse :-)

Previous articles

  1. The Flash player
  2. Seeking
  3. Highlighting the current speech

News & information:
Projects:
Contact & information:
Technical:

mySociety is a project of UK Citizens Online Democracy (UKCOD). UKCOD is a registered charity in England and Wales, no. 1076346.