mySociety

So, I’ve just had a shower and I’m waiting for Matthew and Tom to turn up. As time goes on, mySociety seems to get more geographically disparate, and I look forward to meeting my coworkers. Then for 1pm we’ll be heading to CB2 for the mySociety developers meeting. Feel free to come along any time afternoon or evening, whatever your skills or interest in mySociety.

I haven’t posted on here for ages, since October. I’ve been away on holiday quite a lot, and when I’ve not been away I’ve been busy, partly with systems administration. We’ve set up lots of servers in the last month for the E Petitions site. When you go from 3 servers to 7 servers, there’s another step change in sorting out systems administration tools. For example, I had to change the monitoring script so every server wouldn’t monitor every other. And I had to work out the quirks and bugs in the system we have for storing config files for different classes of server in CVS. Because we only had one class of server before.

I’ve also had to learn lots about server monitoring and load balancing. Things have slowed down a bit now, to maybe 10 hits per second. But a few weeks ago the road pricing petition was often getting 50 hits per second. I’ve never worked on a site with that level of traffic before. You find all the bugs in your code, all the missing indices in PostgreSQL, all the badly tweaked FastCGI parameters. I’ve been sucking knowledge off Chris like a sponge, so tools like strace and vmstat begin to become instinctive. Seemingly nobody offers a book or a course which teaches this stuff well – every server setup is different, everyone knows different ways to tune and profile. But maybe you can tell me different in the comments.

Louise has been busily working away on lots of things. Amongst that is a major change to WriteToThem, to let you write to all the members in a multi-member constituency in one go. The last day or two, she’s been installing the WriteToThem test code on one of our servers, when it has only run on my laptop before. This will be fantastic – hopefully can get Matthew to be bolder about making changes to WriteToThem, if he has a test script he can easily run (getting Matthew to be bold isn’t normally a problem, but he seems mildly less bold when it comes to the WriteToThem queue daemon).

Tom and I have also been busy on a second travel maps report for the DfT. More on that soon. Lots of running screen scraping jobs on TransportDirect which take days. On the subject of Tom, he seems to have got expert at “stacking meetings” next to each other. In one day last week he had 7 meetings!

Partly for our own internal documentation, and partly because it might be of interest to (some) readers, some notes on how the Number 10 petitions site works. On the face of it you’d imagine this would be very simple, but as usual it’s complicated by performance requirements (our design was motivated by the possibility that a large NGO might mail a very large mailing list inviting each member to sign a particular petition, so that we might have to sustain a very high signup rate for an extended period of time). Here’s a picture of the overall architecture:

(This style of illustration is unaccountably popular in the IT industry but unlike most examples of the genre, I’ve tried to arrange that this one actually contains some useful information. In particular I’ve tried to mark the direction of flow of information, and separate out the various protocols; as usual there are too many of the latter. The diagram is actually a slight lie because it misses out yet another layer of IPC—between the web server, apache, and the front-end FastCGI scripts.)

Viewing petition pages is pretty conventional. Incoming HTTP requests reach a front-end cache (an instance of squid, one per web server, cacheing in memory only); squid passes them to the petition browsing scripts (written in perl running under FastCGI) to display petition information. Those scripts consult the database for the current state of the relevant petition and pass it back to the proxy, and thence to the web client. This aspect of the site is not very challenging.

Signing petitions is harder. The necessary steps are:

• write a database record about the pending signature;
• send the user an email containing a unique link to confirm their signature;
• update the database record when the user clicks the link;
• commit everything to stable storage; and finally
• tell the user that their signature has been entered and confirmed.

The conventional design for this would be to have the web script, when it processes the HTTP request for a new signature, submit a message for sending by a local mail server and write a row into the database and commit it, forcing the data out to disk. The mail server would then write the message into its spool directory, and fsync it, forcing it out to disk. The mail server will then pick the mail out of its queue and send it to a remote server, at which point it will be erased from the queue. Later on the mail will arrive in the
user’s inbox, at which point they will (presumably) click the link, resulting in another HTTP request which causes the web script to update the corresponding database row and commit the result. While this is admirably modular it requires far more disk writes than necessary to actually complete the task, which limits its potential speed. (In particular, there’s no reason to have a separate MTA spool directory and for the MTA to make its own writes to that directory.)

At times of high load, it is also extremely inefficient to do one commit per signature. It takes about as long to commit ten new or changed rows to the database as it is to commit one (because the time spent is determined by the disk seek time). Therefore to achieve high performance it is necessary to batch signatures. Unfortunately this is a real pain to implement because all the common web programming models use one process per concurrent request, and it is inconvenient to share database handles between different processes. The correct answer to this problem would of course be to write the signup web script as a single-process multiplexing implementation, but that’s a bit painful (we’d have had to implement our own FastCGI wire protocol library, or alternatively an HTTP server) and the deadlines were fairly tight. So instead we have a single-process server, petsignupd, which accepts signup and confirmation requests from the front-end web scripts over a simple UDP protocol, and passes them to the database in batches every quarter of a second. In theory, therefore, users should see a maximum latency of a bit over 0.25s, but we should achieve close to the theoretical best throughput of incoming requests. (Benchmarking more-or-less bears this out.)

Sending the corresponding email is also a bit problematic. General-purpose MTAs are not optimised for this sort of situation, and (for instance) exim can’t keep up with the sustained signup rate we were aiming for even if you put all of its spool directories on a RAM disk and accept that you have to repopulate its outgoing queue in the event of a crash. The solution was to write petemaild, a small multiplexed SMTP sending server; unlike a general-purpose MTA this manages its queue in memory and communicates updates directly to the database (when a confirmation email is delivered or delivery fails permanently).

It’s unfortunate that such a complex system is required to fulfil such a simple requirement. If we’d been prepared to write the whole thing ourselves, from processing HTTP requests down to writing signatures out to files on disk, the picture above would look much simpler (and there would be fewer IPC boundaries at which things could go wrong). On the other hand the code itself would be a lot more complex, and there’d be a lot more of it. I don’t think I’d describe this design as a “reasonable” compromise, but it’s at least an adequate one.

So, there are now over 600 petitions in the petitions system, and we’re getting a steady stream of appeals from our users to add categories.

I’m posting to ask how you all think we should handle this. It seems to me that there are a few options:

• Ask petition creators to pick one very basic top level category of no more than 10 or so, taken from a hierarchical taxonomy like the one the BBC uses.
• Ask petition creators to pick the top level and the subsequent sub-levels to be more specific.
• Go all web 2.0 and simply ask people to tag their petitions with some key words

More than just thinking about the overall philsophy I’d also appreciate thoughts on design. When you come to the homepage, how should the category system be presented to you? Tricky stuff, and I’d really appreciate your thoughts.

Since the petition system went out properly on Wednesday, we’ve been absolutely buried in an avalanche of changes, fixes, feature additions and massive massive amounts of email. I thought that you might be interested as to what sort of stuff has happened in the first two days:

• Email has taken over our lives. Matthew has responded to over 200 emails since yesterday morning, and I was up at 4am last night just trying to cope with the rate of incoming of mail. Francis, who’s now in Canada, then heroically took up the baton and responded to mail all (UK) night! Many if not most of these mails are giving us suggestions, as well as bug reports, problems with email and bits of praise and the odd conspiracy theory.
• Changes made to cope with expats and overseas military personnel.
• Phoned Hotmail to stop their system from eating 95% of the confirmation messages being sent to Hotmail accounts!
• Redesigned the automated mails no10 get telling them there’s a new petition (they’ve had over 500 of these mails, so they need to be clear and easy to read!)
• Made the rejected petitions system more granular, so that if a petition has to be rejected, and part of it has to be hidden (say, if it is libellous), then it only hides that bit, not the whole thing. Maximum transparency is the goal, you see.
• New options added to sort the list of all petitions in different ways, by number of signatures being the most asked for.
• Limited the length of “more info” fields so people can only write long rants, rather than really really long rants
• Special cased people with AOL accounts, so that their, erm, nonstandard email clients can actually cope with the confirmation links.
• Made several fixes to the processes involved in sending out confirmation mails.
• Made RSS changes and improvements.
• Updated various bits of text, like providing examples of what “party political” means. The BBC initially wrote that this meant no pledges mentioning controversial issues like Iraq, which was grabbing quite the wrong end of the stick about the nature of the rules. Now we have some complaining emails saying we’re being too liberal!
• Compiled a big list of user suggestions and fixes on the wiki here.
• Made the rejection criteria in the Ts&Cs actually match the ones in the admin interface.
• Installed a stats packages to watch what’s going on.
• Added facility to search petitions
• Improved/fixed logging
• Added link and text pointing to the open source code.

I’ve probably missed some – I’m sure Matthew, Chris, Francis and Ben will let me know!

We just had our irregular weekly meeting, which we do most Mondays using a conference call. I thought I’d just write up what we’re all up to this week.

• I’m continuing to test the ePetitions site for 10 Downing Street, and developing an interesting branded version of PledgeBank for CAFOD (more when it launches).
• Matthew is going to look at various things that need doing on PledgeBank and WriteTothem. For PledgeBank more chivvying emails, I think something like this ticket but not exactly. For WriteToThem, various bits of code to do with how we handle error cases.
• Chris is making more pretty maps for the Department for Transport.
• Tom is working out in detail how we’re doing to spend the money from DCLG which has finally come through. It’s mentioned in this post, look for “e-Innovations Product and Marketisation strand via Kirklees MBC”. Which means, we’re being paid to do proper marketing and sales of branded version of our services, such as WriteToThem, PledgeBank, and Neighbourhood Fixit. He’s also chasing up some interesting people met at a conference in Eastern Europe (Bratislava, I think?) last week.

Please ask questions in the comments – for example, if you’d like us to post about particular things on this blog.

Much of my August seems to have been absorbed with maintenance tasks.

For example, Chris and I spent a few days tightening up WriteToThem’s privacy. I made sure the privacy statement correctly describes what happens with backup files, and failed messages. I reduced the timeouts on how long we keep the body of failed messages. I made sure we delete old backup files of the WriteToThem database. I wrote scripts to run periodically to check that no bugs in our queueing demon can accidentally mean we keep the body of messages for longer than we say. I added a cron job to delete Apache log files older than a month for all our sites. As AOL know to their cost, the only really private data is deleted data.

Earlier in the month, I handled some WriteToThem support email for the first time in ages. We get a couple of hundred messages a week, which Matthew mainly slogs through. It’s good for morale to do it, as we get quite a lot of praise mail. It is also hard work, as you realise how complicated even our simple site and the Internet are, and it leads to fixing bugs and improving text on the site. I made a few improvements to our administration tools, and things like the auto-responder if people reply to the questionnaire, to try and reduce the amount of support email, and make it easier to handle.

I did some more work on the geographically cascading pledges (like this prototype one), but I’m still not happy with them. In the end, I realised that it is the structure of wording of the pledge that is the key problem. Our format of “If will A but only if N others will B” just isn’t easily adapted to get across that the pledge applies separately in different geographically areas. Working out how to fix that is one of the things we’ll brainstorm about in the Lake District (see below).

The last couple of days I’ve been configuring one of our new servers who is called Balti, and getting the PledgeBank test harness working on it. Until now, it has only been run on my laptop. This is partly heading towards making a proper test harness for the ePetitions site, running on a server so we properly test nothing can be broken before deploying a new version.

Matthew has wrapped up the TheyWorkForYou API now, and is working on Neighbourhood Fixit next. Chris has been doing lots more performance work for the ePetitions site. And he’s been making some funky monitoring thing to detect PostgreSQL database lock conflicts, which we get occasionally and are hard to debug

Tom’s in Berlin at the moment, he gave a talk last night, and I think has been to see some people from Politik Digital. As we’ve been discussing on the developers email list, there’s an EU grant we’re likely to apply for in collaboration with them.

On Friday, we’re all going to the Lake district for a week, with some of the trustees and volunteers intermitently. We very conveniently and cheaply all work from home, so it’s good and necessary to meet up for a more sustained period of time at least once a year. Last year we were in Wales.