TheyWorkForYou has, until now, only covered things that have already happened, be that Commons main chamber debates since 1935, Public Bill committees back to 2000, or all debates in the modern Northern Ireland Assembly.

From today, we are taking the UK Parliament’s upcoming business calendar and feeding it into our database and search engine, which means some notable new features. Firstly, and most simply, you can browse what’s on today (or the next day Parliament is sitting), or 16th May. Secondly, you can easily search this data, to e.g. see if there will be something happening regarding Twickenham. And best of all, if you’re signed up for an email alert – see below for instructions – you’ll get an email about any matching future business along with the matching new Hansard data we already send. We currently send about 25,000 alerts a day, with over 65,000 email addresses signed up to over 111,000 alerts.

Mark originally wrote some code to scrape Parliament’s business papers, but this sadly proved too fragile, so we settled on Parliament’s calendar which covers most of the same information and more importantly has (mostly) machine-readable data. Duncan and I worked on this intermittently amidst our other activities, with Duncan concentrating on the importer and updating our search indexer (thanks as ever to Xapian) whilst I got on with adding and integrating the new data into the site.

I’ve also taken the opportunity to rejig the home page (and fix the long-standing bug with popular searches that meant it was nearly always Linda Gilroy MP!) to remove the confusingly dense amount of recent links, bring it more in line with the recently refreshed Scottish Parliament and Northern Ireland Assembly home pages, and provide more information to users who might not have any idea what the site covers.

Signing up for an email alert: If you want to receive an email alert on a particular person (MP, Lord, MLA or MSP), visit their page on TheyWorkForYou and follow the “Email me updates” link. If you would like alerts for a particular word or phrase, or anything else, simply do a search for what you’re after, then follow the email alert or RSS links to the right of the results page.

As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.

To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).

Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.

Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.

There don’t appear to be any central official lists of candidates in these elections. Amnesty.org.uk has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations 🙂

Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15” to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.

We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

• Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
• As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
• From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
• Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
• As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
• Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

Someone was asking about getting the boundaries of UK counties (and by that, I believe they meant the ceremonial counties for displaying a summary map). However, this is more confusing than at first glance – not everything you would call a county has a county council, which was quickly noticed, and even some of those county councils will have gaps due to bits of them having become unitary authorities.

So here’s how I would do it for Great Britain (we don’t hold any boundaries for Northern Ireland, sadly).

England

Scotland

Depending on how detailed you want to be, either use the 32 unitary authorities (UTA) in Scotland directly, or group them into the 12 old regions as follows:

Wales

Again, depending on the detail needed, you could either use the 22 unitary authorities (UTA) in Wales directly, or group them into the eight regions as follows:

I’m very pleased to announce that mySociety’s upgraded point and postcode lookup service, MaPit, is public and available to all. It can tell you about administrative areas, such as councils, Welsh Assembly constituencies, or civil parishes, by various different lookups including name, point, or postcode. It has a number of features not available elsewhere as far as I know, including:

• Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! 🙂
• Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
• Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
• History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

On Twitter about 15 minutes ago, @greenerleith asked: “Has anyone worked out how to display the most recent #fixmystreet reports on a local map widget that can be embedded? #hyperlocal”

Like this? 🙂

It’s very simple to do:

1. Go to FixMyStreet, and locate any RSS feed of the latest reports you want (for the above map, I used Edinburgh Waverley’s postcode of EH1 1BB; you could have used reports to a particular council, or ward, using the Local alerts section). Copy the URL of the RSS feed.
2. Go to Google Maps, paste the RSS feed URL into its search box, and click Search Maps.
3. Click the “Link” link to the top right of the map, and copy the “Paste HTML to embed in website” code.
4. Paste that code into your blog post, sidebar, or wherever (you can alter the code to change its size etc.).
5. Done. 🙂

The latest reports from FixMyStreet, superimposed on a Google Map, embedded in your blog. Hope that’s helpful.

mySociety is lucky enough to have a number of small donors who give us monthly donations, normally ranging from £5-£20 (if you like our work and want to support us, please do join them!). Today we’re announcing a change to TheyWorkForYou which is supported by these donations.

One of the most popular features on TheyWorkForYou is the vote analysis – the bit that tells you that your MP “voted strongly against introducing a smoking ban” and so on. These voting analyses cut through a massive wall of parliamentary opacity whilst still allowing visitors to examine the details first hand. Despite each analysis resulting in just a single line on TheyWorkForYou, each one is rather time-consuming to construct, and TheyWorkForYou has not updated them as much as our users deserve.

Thanks to our small donors we’ve now been able to commission two part time researchers, Marcus Fergusson and Stephen Young, to help add new vote analyses more regularly. We’re pleased to say that we’ve just rolled out the first new policies, covering issues relating to schools, inquests and the House of Lords. We aim to add a couple of new vote analyses a month for the foreseeable future.

We take the business of authoring analyses that are scrupulously fair and neutrally worded extremely seriously. To this end we have replaced our previously ad-hoc approach with a newly instituted process designed to ensure the maximum rigour and balance, and to ensure we focus on issues which MPs thought were important even if they were not so well covered by the media.

TheyWorkForYou’s analysis of MP’s voting positions relies on The Public Whip, a project run by Julian Todd which tracks which way MPs vote.

The new process for analysing MP’s positions works like this:

1. A list of votes in the current Parliament, ordered with the highest turnout at the top, is taken as a starting point. The turnout figure used is corrected to account for party abstentions.
2. mySociety’s researchers work down that list writing explanations in easy to read terms describing what the vote was about; they also identify other related votes on the same issue and research those too.
3. A “policy position on the issue” is then chosen against which MP’s votes are compared to determine to what extent they agree or disagree with it. Policy positions are written to be intelligible and interesting to a wide range of users and in such a way that votes to change the status quo are ultimately described on TheyWorkForYou MP pages as votes “for” that change.

We hope you find these analyses useful. Thanks to Richard Taylor for his divisions list and help with this post.

The Hansard Society have just published a report entitled MPs Online: Connecting with Constituents. I’m only going to talk about one part, the part that mentions the mySociety project WriteToThem in a section on MPs’ use of email.

We’re surprised and disappointed to see our methodology for collecting data on how well MPs respond to constituency mail being called “unreliable”, especially from a paper that makes a number of simple mistakes of its own in just a few lines on one page.

• On page 5, they state that WriteToThem has been “tracking responsiveness to emails via their website for three years”. Most importantly for the theme of the report, we don’t just send emails – we send faxes to a number of MPs (and other representatives) who do not accept or want messages via email.
• The figures given for survey responses in the table are backwards; 2007 and 2005’s figures should be interchanged – how could we get more survey responses than messages (again, not necessarily emails) sent? 🙂
• They claim there is “no quantification of the response categories provided” – the raw data used to automatically generate these categories is given in the adjacent column (“very high” simply means a response percentage of 80% or more, for example; our code is all open source).
• We exclude MPs with very small sample sizes, and take a range of steps to make sure the data is not abused.
• We have four years of statistics now, not three; our stats for 2008 were published nearly six months ago.

The Hansard Society, to the best of my knowledge, never got in touch with us to request any clarification or ask about our data or methodology, which we would have been more than happy to supply.

In my last blog post, I explained the new service TheyWorkForYou offers to show you what constituency you will be in at the next general election. Now I’m going to show you why you shouldn’t use anything else.

The defintions of the boundaries for the forthcoming constituencies in England were originally published in The Parliamentary Constituencies (England) Order 2007 (SI 2007/1681), based on ward boundaries as they were on 12th April 2005. However, due to some local government changes since that date, The Parliamentary Constituencies (England) (Amendment) Order 2009 (SI 2009/698) was published changing the boundaries for four constituencies – Daventry, South Northamptonshire, Wells, and Somerton & Frome – to be based on the new council wards as they were on 3rd May 2007.

The forthcoming constituencies in Northern Ireland were defined in The Parliamentary Constituencies (Northern Ireland) Order 2008 (SI 2008/1486). In this, Derryaghy ward was split between two constituencies – Belfast West is given “that part of Derryaghy ward lying to the north of the Derryaghy and Lagmore townland boundary.”

All of which means that other sites that try to tell you what constituency you will be in at the election invariably get it wrong.

Both Labour and the Conservatives say that BA6 8NJ is in Wells at the next election, when it will be in Somerton & Frome. Both say that NN12 8NF will be in Daventry, when it will be in South Northamptonshire. I assume that both sites are using boundary data predating the Amendment Order from March 2009. The Conservatives also say that BT17 0XD will be in Lagan Valley when it will be in Belfast West; Labour simply say “Northern Ireland” for any Northern Irish postcode you provide.

The Liberal Democrats site currently returns no results for any postcode, which I assume is a bug 🙂

The /current/ boundary between Belfast West and Lagan Valley. (Image produced from the Ordnance Survey electionmap service. Image reproduced with permission of Ordnance Survey and Land and Property Services)

The official election-maps.co.uk service (from where TheyWorkForYou gets its boundary maps) returns the correct results for BA6 8NJ and NN12 8NF, but doesn’t have future boundaries for Northern Ireland. It’s not clear that it doesn’t, as searching for Lagan Valley with “future boundaries” selected returns a result, but that result is the current boundary. This can be seen from the picture on the right – as is clear from the quote I gave above, everything within Derryaghy ward north of the Lagmore/Derryaghy boundary will be in Belfast West at the next election. Plus the site doesn’t work without JavaScript.

TheyWorkForYou’s “constituency at the next election” service gives BA6 8NJ in Somerton & Frome, NN12 8NF in South Northamptonshire, and BT17 0XD in Belfast West. There is enough confusion with the changes to boundaries for everywhere except Scotland, that it is somewhat frustrating to have it compounded by sites giving incorrect information. The lack of any official service also doesn’t help.

Boundaries by MarkyBon

Constituency boundaries are changing at the next general election in England, Wales, and Northern Ireland. After some amount of fiddling (I’ll go into technical details in another post, but it wasn’t as easy as just importing some shapefiles), as a slightly early Christmas present, TheyWorkForYou now has a section where you can enter your postcode to find out what constituency you are currently in, and what constituency you will be voting in at the election, along with maps of before and after:

http://www.theyworkforyou.com/boundaries/

This service is also available through the TheyWorkForYou API. This is a facility we have been asked for frequently, more so as we approach the forthcoming election; the large amount of boundary changes have led to confusion from our users and elsewhere, so this will hopefully prove useful.

One site that will need the boundaries before the election is DemocracyClub – join to help make this coming election the most transparent ever!

Side effects of the above process include updated council boundaries, so those councils on WriteToThem that we’ve had switched off since May due to lack of boundary data are now back; a more up-to-date postcode dataset; and the beginnings of parish council support (as in they’re now in the database, but the front-end doesn’t know what to do with them yet).

I hope you all have a happy Christmas and New Year.