An MP page scorecard from the re-launched Mzalendo

When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.

These days dozens of such sites exist around the world. But today sees the launch of a rather-special new transparency site: Mzalendo, covering the Parliament of Kenya.

Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:

• It is entirely responsively designed, so that it works on the simplest of mobile web browsers from day one.
• All the lessons we learned from storing political data wrongly have been baked into this site (i.e we can easily cope with people changing names, parties and jobs)
• Every organisation, position and place in the system is now a proper object in the database. So if you want to see all the politicians who went to Nairobi University, you can.
• There is lots of clear information on how parliament functions, what MPs and committees do, and so on.
• It synthesizes some very complex National Taxpayer’s Association data on missing or wasted money into a really clear ‘scorecard‘, turning large sums of money into numbers of teachers.

The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.

Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.

Stroud on FixMyStreet

Birmingham on FixMyStreet

Firstly, we’ve thrown away our old maps and replaced them with new, shiny, zoomable maps. This should make it easier for people to find and report problems, especially in sparser locations. We’re using the OS StreetView layer (hosted internally) when zoomed in, reverting to Bing Maps’ Ordnance Survey layer when zoomed out, as we felt this provided the best combination for reporting problems. In urban areas, you can still see individual houses, whilst in more rural areas the map with footpaths and other such features is probably of more use. FixMyStreet tries to guess initially which map would be most appropriate based upon population density, meaning a search for Stroud looks a bit different from that for Birmingham.

FixMyStreet with OpenStreetMap

OpenStreetMap fans, don’t worry – as part of our mapping technology upgrade, you can now use osm.fixmystreet.com to access your favourite mapping instead.

Secondly, we now have user accounts. We’ve rolled these out alongside our current system of email confirmation, and it’s up to you which you use when reporting a problem or leaving an update. This means that those who come to the site one time only to report a pothole can continue to do so quickly, but have the option of an account if they want. Having an account means you no longer have to confirm reports and updates by email, and you have access to a page listing all the reports you’ve made through FixMyStreet, and showing these reports on a (obviously new and shiny) summary map.

Adur District Council reports

Other improvements include a much nicer All Reports section, so you can see all reports to Adur District Council on a map, paginated and with the boundary of the council marked – and individual wards of councils now each have their own pages too.

I’ll follow up this post with another, more technical, look at the maps and how they work, for anyone who’s interested :)

Our most recent mini-project is an attempt to make it as easy as possible to find your local council’s e-petitioning site, if they have one. PetitionYourCouncil.com (you’ll notice we stuck to our tried and tested format for site names, there) is a way of finding every council e-petitioning website we know about.

Our original motivation for building the site was that we, along with other suppliers, have supplied online e-petitioning sites to numerous councils ourselves – it’s one of the ways in which we fund our charitable activities. Having delivered these sites, we later noticed that many of them are left under-used and in some cases, not used at all: only because people don’t know about them. We hate to think of councils spending money on a splendid resource that could be improving democratic processes for their citizens – and those citizens never knowing that they exist. In particular, we owe Dave Briggs thanks for pushing us into action with this blog post.

And yes, in case you’re wondering, PetitionYourCouncil links to every council petitions site, not just the ones we made.

The site was built by mySociety developer Edmund von der Burg using Django, jQuery, Google maps and Mapit, and like most mySociety projects, it’s open source. There’s a bit more detail on the About page. Please do try it out, and let us know what you think.

Upcoming business

From today, we are taking the UK Parliament’s upcoming business calendar and feeding it into our database and search engine, which means some notable new features. Firstly, and most simply, you can browse what’s on today (or the next day Parliament is sitting), or 16th May. Secondly, you can easily search this data, to e.g. see if there will be something happening regarding Twickenham. And best of all, if you’re signed up for an email alert – see below for instructions – you’ll get an email about any matching future business along with the matching new Hansard data we already send. We currently send about 25,000 alerts a day, with over 65,000 email addresses signed up to over 111,000 alerts.

Mark originally wrote some code to scrape Parliament’s business papers, but this sadly proved too fragile, so we settled on Parliament’s calendar which covers most of the same information and more importantly has (mostly) machine-readable data. Duncan and I worked on this intermittently amidst our other activities, with Duncan concentrating on the importer and updating our search indexer (thanks as ever to Xapian) whilst I got on with adding and integrating the new data into the site.

New TheyWorkForYou home page

I’ve also taken the opportunity to rejig the home page (and fix the long-standing bug with popular searches that meant it was nearly always Linda Gilroy MP!) to remove the confusingly dense amount of recent links, bring it more in line with the recently refreshed Scottish Parliament and Northern Ireland Assembly home pages, and provide more information to users who might not have any idea what the site covers.

Signing up for an email alert: If you want to receive an email alert on a particular person (MP, Lord, MLA or MSP), visit their page on TheyWorkForYou and follow the “Email me updates” link. If you would like alerts for a particular word or phrase, or anything else, simply do a search for what you’re after, then follow the email alert or RSS links to the right of the results page.

At mySociety we’ve spent a lot of time over the last twelve months helping councils to cope with this new duty by offering  them a commercial petitions service that is really good for users and easy to administer for councils. Some of the sites have been live for months, but many of the 35 council e-petitions sites we’re currently contracted to supply launch today.

mySociety’s core developers Matthew Somerville and Dave Whiteland deserve huge credit for all the work they did re-purposing the No10 Petitions codebase and doing dozens of council customisations and rebrands. I’ve just seen one council officer email “Yippeee” at the prospect of launching, so I reckon they’ve done a pretty good job  - well done gents, everyone in mySociety owes you a debt of gratitude for a time consuming job well done.

Here’s the current list of live local petitions sites. We’ll be adding more as they go up. Happy petitioning!

Ashfield http://petitions.ashfield-dc.gov.uk/

Barnet http://petitions.barnet.gov.uk

Barrow http://petitions.barrowbc.gov.uk/

Bassetlaw http://petitions.bassetlaw.gov.uk/

Blackburn with Darwen http://petitions.blackburn.gov.uk/

East Cambridgeshire http://petitions.eastcambs.gov.uk/

East Northants http://petitions.east-northamptonshire.gov.uk

Elmbridge http://petitions.elmbridge.gov.uk

Forest Heath http://petitions.forest-heath.gov.uk

Hounslow http://petitions.hounslow.gov.uk

Ipswich http://petitions.ipswich.gov.uk

Islington http://petitions.islington.gov.uk

Lichfield http://petitions.lichfielddc.gov.uk

Mansfield http://petitions.mansfield.gov.uk/

Melton http://petitions.melton.gov.uk/

New Forest http://petitions.newforest.gov.uk

Nottinghamshire http://petitions.nottinghamshire.gov.uk

Reigate & Banstead http://petitions.reigate-banstead.gov.uk

Runnymede http://petitions.runnymede.gov.uk

Rushcliffe http://petitions.rushcliffe.gov.uk/

South Holland http://petitions.sholland.gov.uk

Spelthorne http://petitions.spelthorne.gov.uk

St Edmundsbury http://petitions.stedmundsbury.gov.uk

Stevenage http://petitions.stevenage.gov.uk

Suffolk Coastal http://petitions.suffolkcoastal.gov.uk/

Surrey County Council http://petitions.surreycc.gov.uk

Surrey Heath http://petitions.surreyheath.gov.uk

Tandridge http://petitions.tandridge.gov.uk

Waveney http://petitions.waveney.gov.uk

Waverley http://petitions.waverley.gov.uk

Wellingborough http://petitions.wellingborough.gov.uk

Westminster http://petitions.westminster.gov.uk

Royal Borough of Windsor and Maidenhead http://petitions.rbwm.gov.uk

Woking http://petitions.woking.gov.uk

We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

• Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
• As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
• From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
• Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
• As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
• Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

• Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! :-)
• Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
• Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
• History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

Making Parliament clearer; mySociety announces a new process for monitoring MP's votes. (Image: Monet)

mySociety is lucky enough to have a number of small donors who give us monthly donations, normally ranging from £5-£20 (if you like our work and want to support us, please do join them!). Today we’re announcing a change to TheyWorkForYou which is supported by these donations.

One of the most popular features on TheyWorkForYou is the vote analysis – the bit that tells you that your MP “voted strongly against introducing a smoking ban” and so on. These voting analyses cut through a massive wall of parliamentary opacity whilst still allowing visitors to examine the details first hand. Despite each analysis resulting in just a single line on TheyWorkForYou, each one is rather time-consuming to construct, and TheyWorkForYou has not updated them as much as our users deserve.

Thanks to our small donors we’ve now been able to commission two part time researchers, Marcus Fergusson and Stephen Young, to help add new vote analyses more regularly. We’re pleased to say that we’ve just rolled out the first new policies, covering issues relating to schools, inquests and the House of Lords. We aim to add a couple of new vote analyses a month for the foreseeable future.

We take the business of authoring analyses that are scrupulously fair and neutrally worded extremely seriously. To this end we have replaced our previously ad-hoc approach with a newly instituted process designed to ensure the maximum rigour and balance, and to ensure we focus on issues which MPs thought were important even if they were not so well covered by the media.

TheyWorkForYou’s analysis of MP’s voting positions relies on The Public Whip, a project run by Julian Todd which tracks which way MPs vote.

The new process for analysing MP’s positions works like this:

1. A list of votes in the current Parliament, ordered with the highest turnout at the top, is taken as a starting point. The turnout figure used is corrected to account for party abstentions.
2. mySociety’s researchers work down that list writing explanations in easy to read terms describing what the vote was about; they also identify other related votes on the same issue and research those too.
3. A “policy position on the issue” is then chosen against which MP’s votes are compared to determine to what extent they agree or disagree with it. Policy positions are written to be intelligible and interesting to a wide range of users and in such a way that votes to change the status quo are ultimately described on TheyWorkForYou MP pages as votes “for” that change.

We hope you find these analyses useful. Thanks to Richard Taylor for his divisions list and help with this post.

derived from http://commons.wikimedia.org/wiki/File:Eastern-Europe-map2.svg

If you live in Central or Eastern Europe, we’re now in a position to help you get effective democracy and transparency websites built. mySociety have teamed up with the Open Society Institute (OSI) and together we are now looking for determined people with great ideas for new digital transparency and accountability services in their countries.

Over the next few months we are running a Call for Proposals, similar to the one we recently ran in the UK. The big difference is that this time we’re not looking for projects that we will build. We’re looking for projects you want to build, but that for lack of funds or lack of the right skills, you can’t get started yourself.

Each month the Open Society Institute and mySociety will work closely together to select a series of projects to fund and mentor. Crucially, the call isn’t solely for existing NGOs: the process is absolutely open to submissions from individuals or groups with no prior direct experience of working in the transparency and accountability sector, but who have a good idea that addresses a problem they see in their country. We will, however, look more favourably on applicants with access to the advanced programming skills required to build sites like this.

The criteria are simple, though demanding:

1. The projects have to generate some kind of meaningful transparency, accountability, or democratic empowerment of another kind.
2. The projects must seize the unique benefits that the Internet brings with it, such as scalability, two way communication, easy data analysis and so on.

More details are available over at our new CEE site, but even if you don’t live in one of the eligible countries please help us spread the word about this exciting new opportunity!

Go straight to the 2009 Call for Proposals

Do you have a ‘mySocietyish’ idea that you’d like to see become reality? Is there something radical about the sites we already run that should change? Do you have any smart ideas about helping more people to benefit from the services we already offer? Or would you just like to read and comment on ideas submitted by other people?

If the answer to any of these questions is yes, then we’d like you to submit your idea to our 2009 Call for Proposals (built for us by Richard Pope). We’ve run these twice before in 2003 and 2006, resulting in the launch and success of sites such as WriteToThem and WhatDoTheyKnow.

Just as on previous occasions we’ll pick a winner and some runners up, but also just as before we can only promise to do our best – we don’t have the resources to solemnly promise to build the winner, whatever it might be.

What We’re Looking For (or, an insight into the mySociety mindset)

The characteristics of the winning and runner up ideas are highly likely to include one or more of the following factors. Don’t try and include all of them, that’d be silly :)

1. They have to involve the Internet. We don’t do clay modelling.
2. They will capitalise on one or more things that the Internet does really well, better than offline or other forms. WhatDoTheyKnow, for example, seizes on the fact that email can be simultaneously published and rerouted – a simple but critical insight.
3. They will either be a whole new website idea, or a smart and impactful modification of something we already do.
4. They will be ideas that have clear social, civic or democratic benefits that are really easy to explain to the least political person you know, even if the technology behind them is fiendishly complicated.
5. They will have some characteristic that will widely spread the word that the service exists, or that other mySociety services exist.
6. They will offer brilliant value for money, even if they’re expensive to build in the first instance.
7. They will be genuinely new ideas
8. They won’t contain the phrase ‘social media’

We might well change these guidelines a bit as the first responses come in. The call will stay open until September 15th, and we’ll hope to announce the winners in early October.

So what are you waiting for – check out the 2009 Call for Proposals