Back in November 2006 we launched Number 10’s petitions website. We were pretty proud of the usability-centred site we built – we can still lay a pretty good claim to it being one of the biggest democracy sites (measured in terms of people transacting) that the world’s ever seen.

Over 12 million signatures had been added to petitions by the time the site was switched off after the 2010 general election. We were particularly proud of developing a system that was highly load-tolerant: we once survived over 20,000 people signing within a single hour, all whilst running on a pair of cheap little servers. That performance on so little hardware was down to the raw brilliance represented by a coding team made up of Francis Irving, Matthew Somerville, and the late, great Chris Lightfoot.

We’re also pleased that the popularity of the site led to the irresistible rise of the belief that the public should be able to petition the government via the internet. So even though our site was mothballed, Parliament and DirectGov have taken over the idea, and the commitment has been upped a notch, from ‘we’ll send a reply’ to ‘we’ll talk about it’. To be clear, we are not, nor have ever been a community interested in replacing representative democracy with direct democracy, but anything that can squeeze any drop of change from Parliament is worth a small celebration.

What’s most pleasing, though, is that we’ve been able to take the open source code built for Number 10, improve and expand upon it to develop a hosted petitions service for local councils around the country, or the rest of the world. And this is where we found the most important lesson for us: local petitions can be awesome, and despite the much smaller numbers of signatories involved, we’ve been more widely and frequently impressed by local petitions and responses than at the more glamorous national level. We’re particular fans of Hounslow Borough Council who have given positive and detailed feedback on all sorts of genuine local issues, as well as working hard to let local residents know that the service exists.

Just recently we launched a site to make it really easy to find local council petition websites, because there are hundreds hidden away (we built some; most are supplied by other vendors). If we could see anything result from today’s huge explosion of interest in online petitions, it would be that people might start to look local, and explore what petitions in their community could mean.

[Note, November 2014: Petition Your Council has now been retired]

Local petitions can be highly effective, and we think that making them easier to create is in the public interest. Many councils have petitions facilities buried deep within their websites,  most often, very deeply. In fact it brings to mind Douglas Adams’ quote about important council documents being “on display on the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard'”.

Our most recent mini-project is an attempt to make it as easy as possible to find your local council’s e-petitioning site, if they have one. PetitionYourCouncil.com (you’ll notice we stuck to our tried and tested format for site names, there) is a way of finding every council e-petitioning website we know about.

Our original motivation for building the site was that we, along with other suppliers, have supplied online e-petitioning sites to numerous councils ourselves – it’s one of the ways in which we fund our charitable activities. Having delivered these sites, we later noticed that many of them are left under-used and in some cases, not used at all: only because people don’t know about them. We hate to think of councils spending money on a splendid resource that could be improving democratic processes for their citizens – and those citizens never knowing that they exist. In particular, we owe Dave Briggs thanks for pushing us into action with this blog post.

And yes, in case you’re wondering, PetitionYourCouncil links to every council petitions site, not just the ones we made.

The site was built by mySociety developer Edmund von der Burg using Django, jQuery, Google maps and Mapit, and like most mySociety projects, it’s open source. There’s a bit more detail on the About page. Please do try it out, and let us know what you think.

Councils all around England have been busy getting ready to comply with the new duty to provide e-Petitions which kicks in today, 15th December. This means that on council sites across England you should now be able to make petitions which will be formally considered by the councils, in accordance with their chosen policies.

At mySociety we’ve spent a lot of time over the last twelve months helping councils to cope with this new duty by offering  them a commercial petitions service that is really good for users and easy to administer for councils. Some of the sites have been live for months, but many of the 35 council e-petitions sites we’re currently contracted to supply launch today.

mySociety’s core developers Matthew Somerville and Dave Whiteland deserve huge credit for all the work they did re-purposing the No10 Petitions codebase and doing dozens of council customisations and rebrands. I’ve just seen one council officer email “Yippeee” at the prospect of launching, so I reckon they’ve done a pretty good job  – well done gents, everyone in mySociety owes you a debt of gratitude for a time consuming job well done.

Here’s the current list of live local petitions sites. We’ll be adding more as they go up. Happy petitioning!

Ashfield http://petitions.ashfield-dc.gov.uk/

Barnet http://petitions.barnet.gov.uk

Barrow http://petitions.barrowbc.gov.uk/

Bassetlaw http://petitions.bassetlaw.gov.uk/

Blackburn with Darwen http://petitions.blackburn.gov.uk/

East Cambridgeshire http://petitions.eastcambs.gov.uk/

East Northants http://petitions.east-northamptonshire.gov.uk

Elmbridge http://petitions.elmbridge.gov.uk

Forest Heath http://petitions.forest-heath.gov.uk

Hounslow http://petitions.hounslow.gov.uk

Ipswich http://petitions.ipswich.gov.uk

Islington http://petitions.islington.gov.uk

Lichfield http://petitions.lichfielddc.gov.uk

Mansfield http://petitions.mansfield.gov.uk/

Melton http://petitions.melton.gov.uk/

New Forest http://petitions.newforest.gov.uk

Nottinghamshire http://petitions.nottinghamshire.gov.uk

Reigate & Banstead http://petitions.reigate-banstead.gov.uk

Runnymede http://petitions.runnymede.gov.uk

Rushcliffe http://petitions.rushcliffe.gov.uk/

South Holland http://petitions.sholland.gov.uk

Spelthorne http://petitions.spelthorne.gov.uk

St Edmundsbury http://petitions.stedmundsbury.gov.uk

Stevenage http://petitions.stevenage.gov.uk

Suffolk Coastal http://petitions.suffolkcoastal.gov.uk/

Surrey County Council http://petitions.surreycc.gov.uk

Surrey Heath http://petitions.surreyheath.gov.uk

Tandridge http://petitions.tandridge.gov.uk

Waveney http://petitions.waveney.gov.uk

Waverley http://petitions.waverley.gov.uk

Wellingborough http://petitions.wellingborough.gov.uk

Westminster http://petitions.westminster.gov.uk

Royal Borough of Windsor and Maidenhead http://petitions.rbwm.gov.uk

Woking http://petitions.woking.gov.uk

Super WhatDoTheyKnow volunteer John Cross has made an interesting petition about Freedom of Information and publicly owned companies

“We the undersigned petition the Prime Minister to support a change to the law to make companies owned two thirds or more by public authorities subject to the Freedom of Information Act 2000.”

The petition goes on to explain (in more details at the bottom right of the petition page) that the situation is quite comical at the moment. If a company is owned by one local authority, then it is subject to FOI, but if it is jointly owned by two then it isn’t. This makes little sense, and it is also very important, as private companies owned by authorities often do important work.

Obviously it’s always great when any paper gives mySociety coverage – it helps get the word about our services out and helps more people get things done that help their lives.

However, today’s look at mySociety’s 5 years in the Guardian makes a few claims I think it’s important to challenge, so instead of writing to the readers editor I thought I’d just seize the power of Citizen Media(TM) to note them here.

First, has the No10 petitions site had “little notable impact” on government policy? Given that that project appears almost single handedly to have bounced Parliament into developing an online petitioning system and devoting debate time to major petitions, I’d say that it certainly has had some impact. But there is indeed a bigger problem of pointing at No10 petitions and going “That one changed policy.” It’s a problem of two halves: scale, and deniability.  Governments almost never acknowledge that they were forced into anything, ever. Policy announcements are almost always framed as if the right course of action was being followed all along. So apart from the fact that I don’t know how one could possibly assess the impacts of so many thousands of petitions without a huge research project, I would expect that even those that do have in impact will still usually be denied by the government, even when shifting policy. I would encourage No10 and the whole of Government to take a look at directly challenging this culture, and employ someone whose job it is to find out which petitions are having an impact, and shout about them in plain English.

Second, the majority of mySociety’s sites are programmed by staff and contractors, not volunteers. The volunteers are super-essential to mySociety running every day, but the sheer size of some of our projects makes it unlikely a volunteer could have built them without giving up their day job for many months. This needs mentioning to explain why it matters if our finances are precarious!

Next – do councils find FixMyStreet an irritation or an asset? Well, last time we did a count a few weeks ago, we had 4 complaining emails from councils, and 62 supportive ones, with several linking directly to us. As for the Customer Relationship Management at councils, we’d be delighted to send reports straight into their databases without going via email first, it’s just that only one council has set up such an interface so far. I hope that FixMyStreet can put pressure on councils and their suppliers to build a small number of standardised interfaces for the good of everyone. And yes, we are building FixMyStreet for iPhone and Android, and I’m happy to talk to anyone who wants to build UIs for any other phones.

There – hope that doesn’t come across as too ungrateful to Michael Cross et al. See you at the next birthday party, I hope!

Update: I also meant to mention that I’ve never been a ‘Downing Street Insider’. I was a junior civil servant in the Prime Minister’s Strategy Unit, which is not in Downing Street and more loosely affiliated than the name might suggest.

Last night was the annual New Statesman New Media Awards, held in Westminster Abbey’s College Gardens. mySociety were finalists in two categories, Modernising Government and Contribution to Civic Society, with both Number 10 petitions and FixMyStreet nominated in both. Also, two other projects we host, PlanningAlerts and The Government Says, were both finalists in the Information & Openness category.

It was a lovely evening, seeing some people I haven’t seen for some time and meeting new people too. We ended up winning in both our categories – the Number 10 petitions site in Modernising Government, and FixMyStreet in Contribution to Civic Society, which is obviously fantastic for everyone involved. The judges were impressed at the open source nature of the petitions site, and the “deceptive simplicity” of FixMyStreet. This is now the third year in a row we’ve won the Civic Society award – TheyWorkForYou won in 2005, and WriteToThem in 2006, so we’re obviously doing something right. 🙂

It’s a shame that Chris could not be with us, but his mother did attend to see the projects he worked on recognised.

Thanks and congratulations to all the other winners and finalists.

A couple of weekends ago when it was still sunny, a group of 20 or so mySociety developers, trustees, and volunteers went away together to a farmhouse in Warwickshire (thanks to everyone especially Tim Morley and Tom Loosemore for their help). This was not only an opportunity for people like me to finally meet all those I’ve been emailing for months if not years, but also to discuss various things about mySociety.

It was an excellent weekend – we learnt lots of new things, like how UKCOD and mySociety have developed over the last 10 years(!), Rob’s excellent NZ TheyWorkForYou, and Richard’s PlanningAlerts.com. We also discussed what mySociety’s core aims and principles should be – here are some thoughts:

Aims

1) Build sites that build civic value, using the internet natively as a medium and that scale elegantly
2) Build sites that are easy to use for those without experience
3) Build sites that are focused on meeting one simple need
4) mySociety should become self-sustaining, financially and staff-wise

Principles for developing mySociety services and products

1) Build things that meet people’s needs, and that they can’t express yet
2) Do one thing really, really, really well (brand on one thing)
3) Treat the entire world as a creative canvas (plug-ins, widgets, etc.)
4) Do not attempt to do everything yourself; use other people’s content
5) Back success, get rid of failure
6) The web is a conversation; join in
7) Any website is only as good as its worst page
8) Make sure your content can be linked to forever
9) Your granny will never use Second Life
10) Maximize roots to content; optimize your site to run high on Google
11) One size does not fit all – users should know they’re on your site
12) Accessibility is not an optional extra
13) Let people paste their content on their own sites
14) Link to discussions on the web, not necessarily host them
15) Personalization should be unobtrusive and coherent

And some more thoughts:

1) Only use html and CSS
2) Ensure accessibility
3) Ensure usability
4) Make it work across the spectrum – screen readers to mobile phones
5) Build things that don’t require key “stick in the muds?? to do anything
6) Don’t ever build anything that might become an empty cupboard, or if you do, make it very easy for people to fill that cupboard.
7) Don’t rely on network effect, but do seek out network effect
8) Engineer serendipity
9) Help users connect with other users
10) Set the bar high for privacy

However, we still have some challenges ahead: we need to think about how to make the most of our existing sites, and had a very good session on how to improve PledgeBank’s outreach; we also need to engage better with both our current and potential volunteers; and, of course, move towards becoming financially self-sustaining to keep up our good work without always relying on grants.

And finally, because we like tangible actions, we launched the UKCOD site on Saturday night too.

So what happens next? Well some of the things have already happened, like Matthew and others transforming FixMyStreet and Francis developing some widgets. We’ll also see what the new PM wants to do with e-petitions (keep it, apparently, which is good), and how the e-democracy landscape is changing. And, soon we hope, we’ll give this site a bit of a facelift.

But we still have much to do, and the weekend wasn’t long enough to get through everything we wanted. So here are a few more things to chew over.

• Have you wanted to volunteer for mySociety but found it difficult, e.g. the tasks were too technical, or didn’t really know where to start?
• Is there something you want to know about mySociety, or our sites, but not been able to find?
• How can we improve our existing sites?
• Do you know any nice millionaires with some spare cash burning a hole in their pockets, and they just don’t know what to do with it?

Let us know why and we’ll try to do something about it.

Lots has happened since I last posted, which was months ago, before Chris died.

Luckily for us, in the last few months Chris had only been working one day a week for us, so it hasn’t been as difficult in practical terms as it could have been. There were various mySociety things running in his flat, such as the WriteToThem fax server, which had to be set up quickly elsewhere.

We miss Chris’s expertise most days (only yesterday I was swearing at gnuplot). Matthew is twice as much for me to handle; he works so quickly, he has hard questions to ask at a ferocious rate that I can’t keep up with. I think before Chris used to handle most of them, so it was much easier for me.

I highly recommend Chris Lightfoot’s obituary by Martin Keegan. Also, Chris’s obituary on Last Word, which was on Radio 4 on 23rd March, is well worth listening to.

We have a few new members of staff.

Keith Garrett is working for us now, mainly tending our servers, but he’s also been working on the E Petitions site. He’s trying to bully us into documenting all our internal processes so it’s easier for new people.

Heather Cronk has started working for us in the US, evangelising PledgeBank. This is funded by the Omidyar Foundation. As well as getting the word out about PledgeBank in the states, this is extra good for us, as she’s forcing us to give PledgeBank the TLC that it deserves.

Deborah Kerr is now doing customer support for us part time. She’s been busy with Neighbourhood Fix-it which has had lots of traffic and attention the last few weeks.

OK, back to the present.

Last week Ben Campbell and I gave a seminar at Technology for a Small Nation in Llandudno. We split into two groups, I got people to add Google maps to an HTML page on their website, and Ben got people to call the TheyWorkForYou API from PHP. It went down well, very satisfying to do practical exercises, and answer all the niggling questions (how to install a testing webserver on Windows, how to use FTP) which are the real things that stop people doing what they want to do.

And Thursday? Just a reminder, unless you’re not apathetic, that there’s an election today.

So, I’ve just had a shower and I’m waiting for Matthew and Tom to turn up. As time goes on, mySociety seems to get more geographically disparate, and I look forward to meeting my coworkers. Then for 1pm we’ll be heading to CB2 for the mySociety developers meeting. Feel free to come along any time afternoon or evening, whatever your skills or interest in mySociety.

I haven’t posted on here for ages, since October. I’ve been away on holiday quite a lot, and when I’ve not been away I’ve been busy, partly with systems administration. We’ve set up lots of servers in the last month for the E Petitions site. When you go from 3 servers to 7 servers, there’s another step change in sorting out systems administration tools. For example, I had to change the monitoring script so every server wouldn’t monitor every other. And I had to work out the quirks and bugs in the system we have for storing config files for different classes of server in CVS. Because we only had one class of server before.

I’ve also had to learn lots about server monitoring and load balancing. Things have slowed down a bit now, to maybe 10 hits per second. But a few weeks ago the road pricing petition was often getting 50 hits per second. I’ve never worked on a site with that level of traffic before. You find all the bugs in your code, all the missing indices in PostgreSQL, all the badly tweaked FastCGI parameters. I’ve been sucking knowledge off Chris like a sponge, so tools like strace and vmstat begin to become instinctive. Seemingly nobody offers a book or a course which teaches this stuff well – every server setup is different, everyone knows different ways to tune and profile. But maybe you can tell me different in the comments.

Louise has been busily working away on lots of things. Amongst that is a major change to WriteToThem, to let you write to all the members in a multi-member constituency in one go. The last day or two, she’s been installing the WriteToThem test code on one of our servers, when it has only run on my laptop before. This will be fantastic – hopefully can get Matthew to be bolder about making changes to WriteToThem, if he has a test script he can easily run (getting Matthew to be bold isn’t normally a problem, but he seems mildly less bold when it comes to the WriteToThem queue daemon).

Tom and I have also been busy on a second travel maps report for the DfT. More on that soon. Lots of running screen scraping jobs on TransportDirect which take days. On the subject of Tom, he seems to have got expert at “stacking meetings” next to each other. In one day last week he had 7 meetings!

Partly for our own internal documentation, and partly because it might be of interest to (some) readers, some notes on how the Number 10 petitions site works. On the face of it you’d imagine this would be very simple, but as usual it’s complicated by performance requirements (our design was motivated by the possibility that a large NGO might mail a very large mailing list inviting each member to sign a particular petition, so that we might have to sustain a very high signup rate for an extended period of time). Here’s a picture of the overall architecture:

(This style of illustration is unaccountably popular in the IT industry but unlike most examples of the genre, I’ve tried to arrange that this one actually contains some useful information. In particular I’ve tried to mark the direction of flow of information, and separate out the various protocols; as usual there are too many of the latter. The diagram is actually a slight lie because it misses out yet another layer of IPC—between the web server, apache, and the front-end FastCGI scripts.)

Viewing petition pages is pretty conventional. Incoming HTTP requests reach a front-end cache (an instance of squid, one per web server, cacheing in memory only); squid passes them to the petition browsing scripts (written in perl running under FastCGI) to display petition information. Those scripts consult the database for the current state of the relevant petition and pass it back to the proxy, and thence to the web client. This aspect of the site is not very challenging.

Signing petitions is harder. The necessary steps are:

• write a database record about the pending signature;
• send the user an email containing a unique link to confirm their signature;
• update the database record when the user clicks the link;
• commit everything to stable storage; and finally
• tell the user that their signature has been entered and confirmed.

The conventional design for this would be to have the web script, when it processes the HTTP request for a new signature, submit a message for sending by a local mail server and write a row into the database and commit it, forcing the data out to disk. The mail server would then write the message into its spool directory, and fsync it, forcing it out to disk. The mail server will then pick the mail out of its queue and send it to a remote server, at which point it will be erased from the queue. Later on the mail will arrive in the
user’s inbox, at which point they will (presumably) click the link, resulting in another HTTP request which causes the web script to update the corresponding database row and commit the result. While this is admirably modular it requires far more disk writes than necessary to actually complete the task, which limits its potential speed. (In particular, there’s no reason to have a separate MTA spool directory and for the MTA to make its own writes to that directory.)

At times of high load, it is also extremely inefficient to do one commit per signature. It takes about as long to commit ten new or changed rows to the database as it is to commit one (because the time spent is determined by the disk seek time). Therefore to achieve high performance it is necessary to batch signatures. Unfortunately this is a real pain to implement because all the common web programming models use one process per concurrent request, and it is inconvenient to share database handles between different processes. The correct answer to this problem would of course be to write the signup web script as a single-process multiplexing implementation, but that’s a bit painful (we’d have had to implement our own FastCGI wire protocol library, or alternatively an HTTP server) and the deadlines were fairly tight. So instead we have a single-process server, petsignupd, which accepts signup and confirmation requests from the front-end web scripts over a simple UDP protocol, and passes them to the database in batches every quarter of a second. In theory, therefore, users should see a maximum latency of a bit over 0.25s, but we should achieve close to the theoretical best throughput of incoming requests. (Benchmarking more-or-less bears this out.)

Sending the corresponding email is also a bit problematic. General-purpose MTAs are not optimised for this sort of situation, and (for instance) exim can’t keep up with the sustained signup rate we were aiming for even if you put all of its spool directories on a RAM disk and accept that you have to repopulate its outgoing queue in the event of a crash. The solution was to write petemaild, a small multiplexed SMTP sending server; unlike a general-purpose MTA this manages its queue in memory and communicates updates directly to the database (when a confirmation email is delivered or delivery fails permanently).

It’s unfortunate that such a complex system is required to fulfil such a simple requirement. If we’d been prepared to write the whole thing ourselves, from processing HTTP requests down to writing signatures out to files on disk, the picture above would look much simpler (and there would be fewer IPC boundaries at which things could go wrong). On the other hand the code itself would be a lot more complex, and there’d be a lot more of it. I don’t think I’d describe this design as a “reasonable” compromise, but it’s at least an adequate one.