1. How (not) to survive a Slashdotting

    So, PledgeBank got Slashdotted a couple of weeks ago when Mike Liveright’s $100 laptop pledge was linked from a post about the laptop. We didn’t cope very well.

    Unfortunately, PledgeBank is a pretty slow site. Generating the individual pledge page (done by mysociety/pb/web/ref-index.php) can take anything up to 150ms. That’s astonishingly slow, given the speed of a modern computer. What takes the time?

    It’s quite hard to benchmark pages on a running web server, but one approach that I’ve found useful in the past is to use an analogue of phase-sensitive detection. Conveniently enough, all the different components of the site — the webserver, the database and the PHP process — run as different users, so you can easily count up the CPU time being used by the different components during an interval. To benchmark a page, then, request it a few times and compute the amount of CPU time used during those requests. Then sleep for the same amount of time, and compute the amount of CPU time used by the various processes while you were sleeping. The difference between the values is an estimate of the amount of CPU time taken servicing your requests; by repeating this, a more accurate estimate can be obtained. Here are the results after a few hundred requests to http://www.pledgebank.com/100laptop, expressed as CPU time per request in ms:

    Subsystem User System
    apache ~0 ~0
    PostgreSQL 55±9 6±4
    PHP 83±8 4±4

    (The code to do the measurements — Linux-specific, I’m afraid — is in mysociety/bin/psdbench.)

    So that’s pretty shocking. Obviously if you spend 150ms of CPU time on generating a page then the maximum rate at which you can serve users is ~1,000 / 150 requests/second/CPU, which is pretty miserable given that Slashdot can relatively easily drive 50 requests/second. But the really astonishing thing about these numbers is the ~83ms spent in the PHP interpreter. What’s it doing?

    The answer, it turns out, is… parsing PHP code! Benchmarking a page which consists only of this:

    <?
    /* ... */
    
    require_once '../conf/general';
    require_once '../../phplib/db.php';
    require_once '../../phplib/conditional.php';
    require_once '../phplib/pb.php';
    require_once '../phplib/fns.php';
    require_once '../phplib/pledge.php';
    require_once '../phplib/comments.php';
    require_once '../../phplib/utility.php';
    
    exit;
    ?>

    reveals that simply parsing the libraries we include in the page takes about 35ms per page view! PHP, of course, doesn’t parse the code once and then run the bytecode in a virtual machine for each page request, because that would be too much like a real programming language (and would also cut into Zend’s market for its “accelerator” product, which is just an implementation of this obvious idea for PHP).

    So this is bad news. The neatest approach to fixing this kind of performance problem is to stick a web cache like squid in front of the main web site; since the pledge page changes only when a user signs the pledge, or a new comment is posted, events which typically don’t occur anywhere near as frequently as the page is viewed, most hits ought to be servable from the cache, which can be done very quickly indeed. But it’s no good to allow the pledge page to just sit in cache for some fixed period of time (because that would be confusing to users who’ve just signed the pledge or written a comment, an effect familiar to readers of the countless “Movable Type” web logs which are adorned with warnings like, “Your comment may take a few seconds to appear — please don’t submit twice”). So to do this properly we have to modify the pledge page to handle a conditional GET (with an If-Modified-Since: or If-None-Match: header) and quickly return a “304 Not Modified” response to the cache if the page hasn’t changed. Unfortunately if PHP is going to take 35ms to process such a request (ignoring any time in the database), that still means only 20 to 30 requests/second, which is better but still not great.

    (For comparison, a mockup of a perl program to process conditional GETs for the pledge page can serve each one in about 3ms, which isn’t much more than the database queries it uses take on their own. Basically that’s because the perl interpreter only has to parse the code once, and then it runs in a loop accepting and processing requests on its own.)

    However, since we unfortunately don’t have time to rewrite the performance-critical bits of PledgeBank in a real language, the best we can do is to try to cut the number of lines of library code that the site has to parse on each page view. That’s reduced the optimal case for the pledge page — where the pledge has not changed — to this:

    <?
    /* ... */
    
    require_once '../conf/general';
    require_once '../../phplib/conditional.php';
    require_once '../../phplib/db.php';
    
    /* Short-circuit the conditional GET as soon as possible -- parsing the rest of
     * the includes is costly. */
    if (array_key_exists('ref', $_GET)
        && ($id = db_getOne('select id from pledges where ref = ?', $_GET['ref']))
        && cond_maybe_respond(intval(db_getOne('select extract(epoch from pledge_last_change_time(?))', $id))))
        exit();
    
    /* ... */
    ?>

    — that, and a rewrite of our database library so that it didn’t use the gigantic and buggy PEAR one, has got us up to somewhere between 60 and 100 reqs/sec, which while not great is enough that we should be able to cope with another similar Slashdotting.

    For other pages where interactivity isn’t so important, life is much easier: we can just emit a “Cache-Control: max-age=…” header, which tells squid that it can re-use that copy of the page for however long we specify. That means squid can serve that page at about 350reqs/sec; unfortunately the front page isn’t all that important (most users come to PledgeBank for a specific pledge).

    There’s a subtlety to using squid in this kind of (“accelerator”) application which I hadn’t really thought about before. What page you get for a particular URL on PledgeBank (as on lots of other sites) vary based on the content of various headers sent by the user, such as cookies, preferred languages, etc.; for instance, if you have a login cookie, you’ll see a “log out” link which isn’t there if you’re an anonymous user. HTTP is set up to handle this kind of situation through the Vary: header, which the server sends to tell clients and proxies on which headers in the request the content of the response depends. So, if you have login cookies, you should say, “Vary: Cookie”, and if you do content-negotiation for different languages, “Vary: Accept-Language” or whatever.

    PledgeBank has another problem. If the user doesn’t have a cookie saying which country they want to see pledges for, the site tries to guess, based on their IP address. Obviously that makes almost all PledgeBank pages potentially uncachable — the Vary: mechanism can’t express this dependency. That’s not a lot of help when your site gets featured on Slashdot!

    The (desperately ugly) solution? Patch squid to invent a header in each client request, X-GeoIP-Country:, which says which country the client’s IP address maps to, and then name that in the Vary: header of the outgoing pledges. It’s horrid, but it seems to work.

  2. Detecting bad contacts

    We decided to be particularly careful about the new WriteToThem statistics, and did lots of checks on the data. In particular, we wanted to make sure we didn’t unfairly impugn MPs whom we had had bad contact details for. It is possible that for a period of time we thought we had their details but we got them wrong, that we were sending messages to an incorrect address, and their constituents were (unknowingly unfairly) reporting them as unresponsive in the questionnaire.

    So, I wrote a script which generates the statistics, and triest to spot such cases. During the 2005 period, it breaks each MPs time up into intervals according to when we changed our contact details for them. We can do this, because every change we make to contact details is recorded in dadem’s database (see the representatives_edited table).

    I’m going to sound a bit like Donald Rumsfield here, but keep with me. For each interval, we either have good or bad contact details, and we either know or we don’t know that they are good or bad. If we know that they’re bad (e.g. we have no details at all), then that interval isn’t a problem. No messages will have been sent, WriteToThem will have apologised to constituents trying to send messages, and no questionnaires will have been sent out. Any questionnaire results we have from good intervals can still be used and will be fine.

    The case when we think we have contact details is harder. The script does some simple checks to work out if they were really valid. For example, if there have been at least a couple of questionnaire responses, and none were affirmative, then it is a bit suspicious. The script a threshold of length of time of suspicious intervals, and outputs as “unknown” MPs which it thinks there may have been a problem with for long enough for it to matter.

    Tom then heroically checked all those MPs. Some we’ve marked as “WriteToThem had possibly bad contact details for this MP” in the league table. For others, we managed to verify that the questionable email or fax that we had (either via the MPs own website, or by ringing up their office) was actually good. The script then spits out, of all things, PHP file, which you might find useful on your own websites. It contains the complete detailed results. Make sure you look at the “category” for each MP. That indicates if we had too little data, or bad contact details, amongst other things.

    Why PHP? And why not update the stats in real time? We’ve decided to make new statistics just once a year. Firstly, this is much easier to describe, we can say for example on TheyWorkForYou (where the responsiveness data also appears) that it is for the ‘year 2005’. Secondly, it lets us do the manual checking, so we are more confident about our data. Thirdly, it’s good for publicity to announce the new statistics as a news story. And finally, it is much easier to manage an unchanging text file (e.g. the PHP file), stored forerver in CVS, than it would be an ephemeral table in a database somewhere.

    After all that, we mailed or faxed all the still sitting MPs who scored 100% responsiveness, to congratulate them on a job well done. Greg Pope, Richard Page, Fraser Kemp, Thomas McAvoy, Bob Laxton, Mark Simmonds, Paul Stinchcombe, Dennis Turner, Nick Ainger, Alan Meale, Adrian Sanders, Tom Cox, Andrew Hunter, Robert Key, Andrew Selous, John Wilkinson, Paul Goodman, Gwyneth Dunwoody, David Evennett, Peter Atkinson, Andrew Bennett, George Young, Terry Lewis, Douglas Hogg, Patrick Cormack, Andrew Robathan, David Stewart, Colin Challen, Harry Barnes MPs and all your staff, congratulations! (That list includes ones who are no longer MPs, for example stood down at the General Election)

  3. mySociety is coming to North America

    Hello everyone,

    This Tuesday I am heading to North America for a couple of weeks to spread the news about PledgeBank and mySociety. I’d love to meet as many people as possible who are interested in mySociety, or who might become so, and I’m travelling all over the continent to do so.

    I’m visiting now because we’ve quietly been doing lots of work to make our most popular site yet,PledgeBank, a truly international site with the ability to create, search and subscribe to pledges in any one of over 2 million cities, towns and villages around the world. We’ve also engineered the site so that volunteer translators can help us translate the whole site into other languages just by editing one easy text file. This is the first time a mySociety project has been truly international, and we’re keen to share our free services beyond the borders of the UK.

    We’ve not done any publicity in the US yet because we know from experience that it is was an essential part of the highly successful UK launch that we had a variety of strong, clear and practical pledges ready to go from day one. Discussing potential US and Canada based pledges and building informal partnerships is what this is all about.

    So, if you are in the US or Canada, or you know someone who is and who would be interested in meeting, please get in touch with tom@mySociety.org

  4. Media Shchmedia

    We thought that the launch of PledgeBank might be a bit more low key than NotApathetic, what with it not having the election as a hook, and being a bit more ‘high concept’ and difficult to explain to people. But what do we know, eh?

  5. Launch day morning

    Last night I should have gone to bed early, but these things being how they are I stayed up late having tea with my housemate and his friend. I wanted to get up early, because I knew a few things needed tidying before we started getting media coverage, so I set my alarm. I haven’t done that for work for years! So I’m a bit sleepy.

    The most important early thing I did was make the front page featured pledges appear in a random order, for more fairness and serendipity. Late last night Chris had added code in to fuzzily find pledges which somebody has typed in. It uses the database to look for the number of common three letter substrings, so if you type in “http://www.pledgebank.com/suirname” it gives a nice error page leading you to go to “http://www.pledgebank.com/Suriname”. It’s pretty good, and all I had to do was tidy up the text a bit, and add it to the search page as well.

    By that time everyone else was up, and the no2id people were publicising their pledge. We were all on IRC, and tailing various logfiles. There were quite a few minor tidy ups for us to make to the launch pledges that were made over the weekend, changing text and signup numbers for the creators a bit.

    Someone spotted that the “all pledges” page had the wrong calculated count for one of the pledges. This was very odd, as it was right for all the others. I downloaded a fresh dump of the database to my local machine, where everything was fine. Meanwhile, Chris noticed the PHP server was crashing. After more investigation, we found a subtle bug was creating a corrupt PHP variable. Calling “gettype” on it caused the PHP process to stop with an error, and calling number_format crashed the whole thing. We’re still not sure quite what PHP bug caused this, and need to investigate it more. But we found a simple workaround which stopped it causing any more trouble.

    You always find all the bugs when your traffic goes up! That’s why staged beta getting larger and larger, of which today is in many ways the next phase, is the way to go.

  6. Welcome Slashdot Readers!

    We hope you enjoy Geoff and Tom’s paper. But while you’re here, why not take a quick look at our current projects:

    • WriteToThem.com — contact your elected representatives, for free
    • NotApathetic.com — giving a voice to those who aren’t voting in the British May 5th General Election
    • PledgeBank.com — “I’ll do this if enough other people promise to help out”
  7. About to have breakfast

    This morning I woke up quite late because I didn’t sleep well (and had been building Public Whip’s How To Vote). Today I’m not really working for mySociety, as I’m going to London this afternoon, but Tom had a couple of important bits for me to do. There was a bug in the admin page, and a missing link on the confirmation page. I fixed these up, improved the test suite a bit, and deployed to the main site.

    PledgeBank is now ready for people to start using it in earnest. We’re still in testing, as we’re sure there’ll be lots of changes needed to it as it is used in the real world. But all the basic features, and fancier ones such as SMS and the auto-generated flyers, work. Tom’s just been on the radio, and he’s making lots of specific example pledges like this one about Shropshire. So, the hunt is now on.

    My next jobs are to tidy up outstanding tickets which we have already fixed, and fix any bugs in there. The next feature we’re adding to the site is comments, so people can discuss the pledge.

  8. Installing statistics packages

    This afternoon I’ve been installing web log analysis software. I chose to use awstats, mainly because I know it already. But also because last time I looked it was the prettiest open source one. Installing it was mostly painless, but hampered by a couple of technical annoyances.

    Firstly, we run everything on the mySociety server as different users, for privilege separation. This involves using suExec and FastCGI, neither of which are as mature as they might be. Now, the problem with suExec is that it is extremely paranoid. For example, it makes sure the owner and group of any files you try and run as CGI are the same as the user you set them to be run by. It also checks that they are under document root – a silly restriction, especially when I’ve just added an alias to httpd.conf point to the awstats.pl file which FreeBSD installed elsewhere.

    The solution to this? Make a short one line bash script which merely calls through to the actual script. That it is so easy to get round them, shows how silly the suExec restrictions are – it just puts everybody off using it at all, and instead they make everything world-readable. And less secure. So by being paranoid about security, they make everyones web server less secure. Nice.

    Secondly, awstats has a strange view of the world. It insists on getting log files in order. That is every row of log fed to it has to arrive, and you can’t go back and feed an older log file in later. I’m not quite sure why this is – perhaps so that its very compact text file summary of the logs can be updated efficiently. So, to run analysis of old files from months back you have to write a quick script to run through them all and call awstats with them.

    Anyway, I’ve now done that for NotApathetic, PledgeBank, WriteToThem and this very website that you’re reading. So Tom will stop hassling me every time a journalist is saying “how many visitors do you get a day?” So far today we have 2807 visits to NotApathetic.com.

  9. Downing Street Says

    Mysociety is proud to announce the launch of DowningStreetSays.com, a site where you can leave and read comments on Number 10’s press briefings.

    This isn’t a full mySociety project, and was thrown together by some amazing volunteers over the last few days. We hope that in a small way it helps to improve the quality of debate at the heart of government, by letting the public at government without first passing through the lens of the mass media.

    Let us know what you think!

    [July 2013 – dead link replaced with Internet Archive link]