1. What we learned from ePetitions

    New Government e-petitions site

    Back in November 2006 we launched Number 10’s petitions website. We were pretty proud of the usability-centred site we built – we can still lay a pretty good claim to it being one of the biggest democracy sites (measured in terms of people transacting) that the world’s ever seen.

    Over 12 million signatures had been added to petitions by the time the site was switched off after the 2010 general election. We were particularly proud of developing a system that was highly load-tolerant: we once survived over 20,000 people signing within a single hour, all whilst running on a pair of cheap little servers. That performance on so little hardware was down to the raw brilliance represented by a coding team made up of Francis Irving, Matthew Somerville, and the late, great Chris Lightfoot.

    We’re also pleased that the popularity of the site led to the irresistible rise of the belief that the public should be able to petition the government via the internet. So even though our site was mothballed, Parliament and DirectGov have taken over the idea, and the commitment has been upped a notch, from ‘we’ll send a reply’ to ‘we’ll talk about it’. To be clear, we are not, nor have ever been a community interested in replacing representative democracy with direct democracy, but anything that can squeeze any drop of change from Parliament is worth a small celebration.

    What’s most pleasing, though, is that we’ve been able to take the open source code built for Number 10, improve and expand upon it to develop a hosted petitions service for local councils around the country, or the rest of the world. And this is where we found the most important lesson for us: local petitions can be awesome, and despite the much smaller numbers of signatories involved, we’ve been more widely and frequently impressed by local petitions and responses than at the more glamorous national level. We’re particular fans of Hounslow Borough Council who have given positive and detailed feedback on all sorts of genuine local issues, as well as working hard to let local residents know that the service exists.

    Just recently we launched a site to make it really easy to find local council petition websites, because there are hundreds hidden away (we built some; most are supplied by other vendors). If we could see anything result from today’s huge explosion of interest in online petitions, it would be that people might start to look local, and explore what petitions in their community could mean.

  2. Local e-Petitions – See if mySociety is providing your local system

    Councils all around England have been busy getting ready to comply with the new duty to provide e-Petitions which kicks in today, 15th December. This means that on council sites across England you should now be able to make petitions which will be formally considered by the councils, in accordance with their chosen policies.

    At mySociety we’ve spent a lot of time over the last twelve months helping councils to cope with this new duty by offering  them a commercial petitions service that is really good for users and easy to administer for councils. Some of the sites have been live for months, but many of the 35 council e-petitions sites we’re currently contracted to supply launch today.

    mySociety’s core developers Matthew Somerville and Dave Whiteland deserve huge credit for all the work they did re-purposing the No10 Petitions codebase and doing dozens of council customisations and rebrands. I’ve just seen one council officer email “Yippeee” at the prospect of launching, so I reckon they’ve done a pretty good job  - well done gents, everyone in mySociety owes you a debt of gratitude for a time consuming job well done.

    Here’s the current list of live local petitions sites. We’ll be adding more as they go up. Happy petitioning!

    Ashfield http://petitions.ashfield-dc.gov.uk/

    Barnet http://petitions.barnet.gov.uk

    Barrow http://petitions.barrowbc.gov.uk/

    Bassetlaw http://petitions.bassetlaw.gov.uk/

    Blackburn with Darwen http://petitions.blackburn.gov.uk/

    East Cambridgeshire http://petitions.eastcambs.gov.uk/

    East Northants http://petitions.east-northamptonshire.gov.uk

    Elmbridge http://petitions.elmbridge.gov.uk

    Forest Heath http://petitions.forest-heath.gov.uk

    Hounslow http://petitions.hounslow.gov.uk

    Ipswich http://petitions.ipswich.gov.uk

    Islington http://petitions.islington.gov.uk

    Lichfield http://petitions.lichfielddc.gov.uk

    Mansfield http://petitions.mansfield.gov.uk/

    Melton http://petitions.melton.gov.uk/

    New Forest http://petitions.newforest.gov.uk

    Nottinghamshire http://petitions.nottinghamshire.gov.uk

    Reigate & Banstead http://petitions.reigate-banstead.gov.uk

    Runnymede http://petitions.runnymede.gov.uk

    Rushcliffe http://petitions.rushcliffe.gov.uk/

    South Holland http://petitions.sholland.gov.uk

    Spelthorne http://petitions.spelthorne.gov.uk

    St Edmundsbury http://petitions.stedmundsbury.gov.uk

    Stevenage http://petitions.stevenage.gov.uk

    Suffolk Coastal http://petitions.suffolkcoastal.gov.uk/

    Surrey County Council http://petitions.surreycc.gov.uk

    Surrey Heath http://petitions.surreyheath.gov.uk

    Tandridge http://petitions.tandridge.gov.uk

    Waveney http://petitions.waveney.gov.uk

    Waverley http://petitions.waverley.gov.uk

    Wellingborough http://petitions.wellingborough.gov.uk

    Westminster http://petitions.westminster.gov.uk

    Royal Borough of Windsor and Maidenhead http://petitions.rbwm.gov.uk

    Woking http://petitions.woking.gov.uk

  3. A few words on the Guardian

    Obviously it’s always great when any paper gives mySociety coverage – it helps get the word about our services out and helps more people get things done that help their lives.

    However, today’s look at mySociety’s 5 years in the Guardian makes a few claims I think it’s important to challenge, so instead of writing to the readers editor I thought I’d just seize the power of Citizen Media(TM) to note them here.

    First, has the No10 petitions site had “little notable impact” on government policy? Given that that project appears almost single handedly to have bounced Parliament into developing an online petitioning system and devoting debate time to major petitions, I’d say that it certainly has had some impact. But there is indeed a bigger problem of pointing at No10 petitions and going “That one changed policy.” It’s a problem of two halves: scale, and deniability.  Governments almost never acknowledge that they were forced into anything, ever. Policy announcements are almost always framed as if the right course of action was being followed all along. So apart from the fact that I don’t know how one could possibly assess the impacts of so many thousands of petitions without a huge research project, I would expect that even those that do have in impact will still usually be denied by the government, even when shifting policy. I would encourage No10 and the whole of Government to take a look at directly challenging this culture, and employ someone whose job it is to find out which petitions are having an impact, and shout about them in plain English.

    Second, the majority of mySociety’s sites are programmed by staff and contractors, not volunteers. The volunteers are super-essential to mySociety running every day, but the sheer size of some of our projects makes it unlikely a volunteer could have built them without giving up their day job for many months. This needs mentioning to explain why it matters if our finances are precarious!

    Next – do councils find FixMyStreet an irritation or an asset? Well, last time we did a count a few weeks ago, we had 4 complaining emails from councils, and 62 supportive ones, with several linking directly to us. As for the Customer Relationship Management at councils, we’d be delighted to send reports straight into their databases without going via email first, it’s just that only one council has set up such an interface so far. I hope that FixMyStreet can put pressure on councils and their suppliers to build a small number of standardised interfaces for the good of everyone. And yes, we are building FixMyStreet for iPhone and Android, and I’m happy to talk to anyone who wants to build UIs for any other phones.

    There – hope that doesn’t come across as too ungrateful to Michael Cross et al. See you at the next birthday party, I hope!

    Update: I also meant to mention that I’ve never been a ‘Downing Street Insider’. I was a junior civil servant in the Prime Minister’s Strategy Unit, which is not in Downing Street and more loosely affiliated than the name might suggest.

  4. New Media Awards 2007

    Last night was the annual New Statesman New Media Awards, held in Westminster Abbey’s College Gardens. mySociety were finalists in two categories, Modernising Government and Contribution to Civic Society, with both Number 10 petitions and FixMyStreet nominated in both. Also, two other projects we host, PlanningAlerts and The Government Says, were both finalists in the Information & Openness category.

    It was a lovely evening, seeing some people I haven’t seen for some time and meeting new people too. We ended up winning in both our categories – the Number 10 petitions site in Modernising Government, and FixMyStreet in Contribution to Civic Society, which is obviously fantastic for everyone involved. The judges were impressed at the open source nature of the petitions site, and the “deceptive simplicity” of FixMyStreet. This is now the third year in a row we’ve won the Civic Society award – TheyWorkForYou won in 2005, and WriteToThem in 2006, so we’re obviously doing something right. :)

    It’s a shame that Chris could not be with us, but his mother did attend to see the projects he worked on recognised.

    Thanks and congratulations to all the other winners and finalists.

  5. mySociety Away Weekend

    A couple of weekends ago when it was still sunny, a group of 20 or so mySociety developers, trustees, and volunteers went away together to a farmhouse in Warwickshire (thanks to everyone especially Tim Morley and Tom Loosemore for their help). This was not only an opportunity for people like me to finally meet all those I’ve been emailing for months if not years, but also to discuss various things about mySociety.

    It was an excellent weekend – we learnt lots of new things, like how UKCOD and mySociety have developed over the last 10 years(!), Rob’s excellent NZ TheyWorkForYou, and Richard’s PlanningAlerts.com. We also discussed what mySociety’s core aims and principles should be – here are some thoughts:

    Aims

    1) Build sites that build civic value, using the internet natively as a medium and that scale elegantly
    2) Build sites that are easy to use for those without experience
    3) Build sites that are focused on meeting one simple need
    4) mySociety should become self-sustaining, financially and staff-wise

    Principles for developing mySociety services and products

    1) Build things that meet people’s needs, and that they can’t express yet
    2) Do one thing really, really, really well (brand on one thing)
    3) Treat the entire world as a creative canvas (plug-ins, widgets, etc.)
    4) Do not attempt to do everything yourself; use other people’s content
    5) Back success, get rid of failure
    6) The web is a conversation; join in
    7) Any website is only as good as its worst page
    8) Make sure your content can be linked to forever
    9) Your granny will never use Second Life
    10) Maximize roots to content; optimize your site to run high on Google
    11) One size does not fit all – users should know they’re on your site
    12) Accessibility is not an optional extra
    13) Let people paste their content on their own sites
    14) Link to discussions on the web, not necessarily host them
    15) Personalization should be unobtrusive and coherent

    And some more thoughts:

    1) Only use html and CSS
    2) Ensure accessibility
    3) Ensure usability
    4) Make it work across the spectrum – screen readers to mobile phones
    5) Build things that don’t require key “stick in the muds?? to do anything
    6) Don’t ever build anything that might become an empty cupboard, or if you do, make it very easy for people to fill that cupboard.
    7) Don’t rely on network effect, but do seek out network effect
    8) Engineer serendipity
    9) Help users connect with other users
    10) Set the bar high for privacy

    However, we still have some challenges ahead: we need to think about how to make the most of our existing sites, and had a very good session on how to improve PledgeBank’s outreach; we also need to engage better with both our current and potential volunteers; and, of course, move towards becoming financially self-sustaining to keep up our good work without always relying on grants.

    And finally, because we like tangible actions, we launched the UKCOD site on Saturday night too.

    So what happens next? Well some of the things have already happened, like Matthew and others transforming FixMyStreet and Francis developing some widgets. We’ll also see what the new PM wants to do with e-petitions (keep it, apparently, which is good), and how the e-democracy landscape is changing. And, soon we hope, we’ll give this site a bit of a facelift.

    But we still have much to do, and the weekend wasn’t long enough to get through everything we wanted. So here are a few more things to chew over.

    • Have you wanted to volunteer for mySociety but found it difficult, e.g. the tasks were too technical, or didn’t really know where to start?
    • Is there something you want to know about mySociety, or our sites, but not been able to find?
    • How can we improve our existing sites?
    • Do you know any nice millionaires with some spare cash burning a hole in their pockets, and they just don’t know what to do with it?

    Let us know why and we’ll try to do something about it.

  6. RIP Chris Lightfoot – 1978 to 2007

    It is with great sadness that I must report the death of Chris Lightfoot, mySociety’s first developer and a good friend to all of us. He was found by friends at his flat on February 11th. The main announcement can be read in this post on his blog.

    Chris was perhaps the pre-eminent example so far of what polymath means in the Internet age. His contributions to the world are more than just a formidable legacy of computer code of the very highest quality, for mySociety and many others. They also include substantial contributions to applied statistics, geographic information systems, economics and a range of public policy issues from identity cards to speed cameras.

    Everything Chris did in these fields combined an incredulity-inducing array of technical and analytical skills with a wickedly funny, savage turn of phrase. To understand what a remarkable intellectual outlier he was, simply sift through his blog and marvel at the quantity of primary research and original coding that went into it. Documenting and exploring his work would provide material for many years of research, and yet all this was accomplished by the age of 28.

    Within mySociety he was involved right from the start through the development of WriteToThem, HearFromYourMP and PledgeBank, building some amazing underpinning geographic and political web services like Gaze, MaPit and DaDem. These components make all our sites work and make a raft other tools and sites possible in the future.

    For the last three or four months he was working at another employer, Media Molecule four days a week, but still helped the full time staff with the petitions work. The last major thing he built for us was the system that serves up the maps for Neighbourhood Fix-It, a site which was only just soft launched before he died, but of which he was apparently fond for its WriteToThem-like habit of getting simple things done that mattered to normal people.

    Building mySociety’s major sites involved mighty team efforts, something which can obscure even huge invididual talent. So perhaps the sort of work for which Chris will be be most remembered is his wonderfully individualistic, virtuoso forays into scholastic areas in which he had no formal training. He wandered into differing disciplines, made a mark, and wandered on again like a giant that had no idea he’d just trodden on a village. The political survey work he did both hugely illuminates our understanding of our own political world, whilst raising the question “how come none of the professional political analysts thought of this?” And his travel-time maps should make everyone in government wonder if they’re sitting on information which could be reused to such amazing, potentially life changing effect.

    Chris’ intellect and appetite for knowledge was surpassed by only one aspect of his character: his integrity. If you’ve ever wondered why WriteToThem goes to such lengths to protect users’ data it is largely because of his rock solid belief in the dignity and social indispensibility of privacy. Chris was an energetic campaigner in this field, notably for No2ID, who have posted a tribute.

    It doesn’t stretch the truth an inch to say that with his death the whole of the UK’s citizenry, not just his family, friends and colleagues, will be worse off. Rest in peace, Chris.

  7. Good morning

    So, I’ve just had a shower and I’m waiting for Matthew and Tom to turn up. As time goes on, mySociety seems to get more geographically disparate, and I look forward to meeting my coworkers. Then for 1pm we’ll be heading to CB2 for the mySociety developers meeting. Feel free to come along any time afternoon or evening, whatever your skills or interest in mySociety.

    I haven’t posted on here for ages, since October. I’ve been away on holiday quite a lot, and when I’ve not been away I’ve been busy, partly with systems administration. We’ve set up lots of servers in the last month for the E Petitions site. When you go from 3 servers to 7 servers, there’s another step change in sorting out systems administration tools. For example, I had to change the monitoring script so every server wouldn’t monitor every other. And I had to work out the quirks and bugs in the system we have for storing config files for different classes of server in CVS. Because we only had one class of server before.

    I’ve also had to learn lots about server monitoring and load balancing. Things have slowed down a bit now, to maybe 10 hits per second. But a few weeks ago the road pricing petition was often getting 50 hits per second. I’ve never worked on a site with that level of traffic before. You find all the bugs in your code, all the missing indices in PostgreSQL, all the badly tweaked FastCGI parameters. I’ve been sucking knowledge off Chris like a sponge, so tools like strace and vmstat begin to become instinctive. Seemingly nobody offers a book or a course which teaches this stuff well – every server setup is different, everyone knows different ways to tune and profile. But maybe you can tell me different in the comments.

    Louise has been busily working away on lots of things. Amongst that is a major change to WriteToThem, to let you write to all the members in a multi-member constituency in one go. The last day or two, she’s been installing the WriteToThem test code on one of our servers, when it has only run on my laptop before. This will be fantastic – hopefully can get Matthew to be bolder about making changes to WriteToThem, if he has a test script he can easily run (getting Matthew to be bold isn’t normally a problem, but he seems mildly less bold when it comes to the WriteToThem queue daemon).

    Tom and I have also been busy on a second travel maps report for the DfT. More on that soon. Lots of running screen scraping jobs on TransportDirect which take days. On the subject of Tom, he seems to have got expert at “stacking meetings” next to each other. In one day last week he had 7 meetings!

  8. Under the bonnet

    Partly for our own internal documentation, and partly because it might be of interest to (some) readers, some notes on how the Number 10 petitions site works. On the face of it you’d imagine this would be very simple, but as usual it’s complicated by performance requirements (our design was motivated by the possibility that a large NGO might mail a very large mailing list inviting each member to sign a particular petition, so that we might have to sustain a very high signup rate for an extended period of time). Here’s a picture of the overall architecture:

    Diagram representing petitions site architecture

    (This style of illustration is unaccountably popular in the IT industry but unlike most examples of the genre, I’ve tried to arrange that this one actually contains some useful information. In particular I’ve tried to mark the direction of flow of information, and separate out the various protocols; as usual there are too many of the latter. The diagram is actually a slight lie because it misses out yet another layer of IPC—between the web server, apache, and the front-end FastCGI scripts.)

    Viewing petition pages is pretty conventional. Incoming HTTP requests reach a front-end cache (an instance of squid, one per web server, cacheing in memory only); squid passes them to the petition browsing scripts (written in perl running under FastCGI) to display petition information. Those scripts consult the database for the current state of the relevant petition and pass it back to the proxy, and thence to the web client. This aspect of the site is not very challenging.

    Signing petitions is harder. The necessary steps are:

    • write a database record about the pending signature;
    • send the user an email containing a unique link to confirm their signature;
    • update the database record when the user clicks the link;
    • commit everything to stable storage; and finally
    • tell the user that their signature has been entered and confirmed.

    The conventional design for this would be to have the web script, when it processes the HTTP request for a new signature, submit a message for sending by a local mail server and write a row into the database and commit it, forcing the data out to disk. The mail server would then write the message into its spool directory, and fsync it, forcing it out to disk. The mail server will then pick the mail out of its queue and send it to a remote server, at which point it will be erased from the queue. Later on the mail will arrive in the
    user’s inbox, at which point they will (presumably) click the link, resulting in another HTTP request which causes the web script to update the corresponding database row and commit the result. While this is admirably modular it requires far more disk writes than necessary to actually complete the task, which limits its potential speed. (In particular, there’s no reason to have a separate MTA spool directory and for the MTA to make its own writes to that directory.)

    At times of high load, it is also extremely inefficient to do one commit per signature. It takes about as long to commit ten new or changed rows to the database as it is to commit one (because the time spent is determined by the disk seek time). Therefore to achieve high performance it is necessary to batch signatures. Unfortunately this is a real pain to implement because all the common web programming models use one process per concurrent request, and it is inconvenient to share database handles between different processes. The correct answer to this problem would of course be to write the signup web script as a single-process multiplexing implementation, but that’s a bit painful (we’d have had to implement our own FastCGI wire protocol library, or alternatively an HTTP server) and the deadlines were fairly tight. So instead we have a single-process server, petsignupd, which accepts signup and confirmation requests from the front-end web scripts over a simple UDP protocol, and passes them to the database in batches every quarter of a second. In theory, therefore, users should see a maximum latency of a bit over 0.25s, but we should achieve close to the theoretical best throughput of incoming requests. (Benchmarking more-or-less bears this out.)

    Sending the corresponding email is also a bit problematic. General-purpose MTAs are not optimised for this sort of situation, and (for instance) exim can’t keep up with the sustained signup rate we were aiming for even if you put all of its spool directories on a RAM disk and accept that you have to repopulate its outgoing queue in the event of a crash. The solution was to write petemaild, a small multiplexed SMTP sending server; unlike a general-purpose MTA this manages its queue in memory and communicates updates directly to the database (when a confirmation email is delivered or delivery fails permanently).

    It’s unfortunate that such a complex system is required to fulfil such a simple requirement. If we’d been prepared to write the whole thing ourselves, from processing HTTP requests down to writing signatures out to files on disk, the picture above would look much simpler (and there would be fewer IPC boundaries at which things could go wrong). On the other hand the code itself would be a lot more complex, and there’d be a lot more of it. I don’t think I’d describe this design as a “reasonable” compromise, but it’s at least an adequate one.

  9. How should we handle categorisation for petitions?

    So, there are now over 600 petitions in the petitions system, and we’re getting a steady stream of appeals from our users to add categories.

    I’m posting to ask how you all think we should handle this. It seems to me that there are a few options:

    • Ask petition creators to pick one very basic top level category of no more than 10 or so, taken from a hierarchical taxonomy like the one the BBC uses.
    • Ask petition creators to pick the top level and the subsequent sub-levels to be more specific.
    • Go all web 2.0 and simply ask people to tag their petitions with some key words

    More than just thinking about the overall philsophy I’d also appreciate thoughts on design. When you come to the homepage, how should the category system be presented to you? Tricky stuff, and I’d really appreciate your thoughts.

  10. This is what Beta means: the first 48 hours of petitions

    Since the petition system went out properly on Wednesday, we’ve been absolutely buried in an avalanche of changes, fixes, feature additions and massive massive amounts of email. I thought that you might be interested as to what sort of stuff has happened in the first two days:

    • Email has taken over our lives. Matthew has responded to over 200 emails since yesterday morning, and I was up at 4am last night just trying to cope with the rate of incoming of mail. Francis, who’s now in Canada, then heroically took up the baton and responded to mail all (UK) night! Many if not most of these mails are giving us suggestions, as well as bug reports, problems with email and bits of praise and the odd conspiracy theory.
    • Changes made to cope with expats and overseas military personnel.
    • Phoned Hotmail to stop their system from eating 95% of the confirmation messages being sent to Hotmail accounts!
    • Redesigned the automated mails no10 get telling them there’s a new petition (they’ve had over 500 of these mails, so they need to be clear and easy to read!)
    • Made the rejected petitions system more granular, so that if a petition has to be rejected, and part of it has to be hidden (say, if it is libellous), then it only hides that bit, not the whole thing. Maximum transparency is the goal, you see.
    • New options added to sort the list of all petitions in different ways, by number of signatures being the most asked for.
    • Limited the length of “more info” fields so people can only write long rants, rather than really really long rants :)
    • Special cased people with AOL accounts, so that their, erm, nonstandard email clients can actually cope with the confirmation links.
    • Made several fixes to the processes involved in sending out confirmation mails.
    • Made RSS changes and improvements.
    • Updated various bits of text, like providing examples of what “party political” means. The BBC initially wrote that this meant no pledges mentioning controversial issues like Iraq, which was grabbing quite the wrong end of the stick about the nature of the rules. Now we have some complaining emails saying we’re being too liberal!
    • Compiled a big list of user suggestions and fixes on the wiki.
    • Made the rejection criteria in the Ts&Cs actually match the ones in the admin interface.
    • Installed a stats packages to watch what’s going on.
    • Added facility to search petitions
    • Improved/fixed logging
    • Added link and text pointing to the open source code.

    I’ve probably missed some – I’m sure Matthew, Chris, Francis and Ben will let me know!