We decided to be particularly careful about the new WriteToThem statistics, and did lots of checks on the data. In particular, we wanted to make sure we didn’t unfairly impugn MPs whom we had had bad contact details for. It is possible that for a period of time we thought we had their details but we got them wrong, that we were sending messages to an incorrect address, and their constituents were (unknowingly unfairly) reporting them as unresponsive in the questionnaire.

So, I wrote a script which generates the statistics, and triest to spot such cases. During the 2005 period, it breaks each MPs time up into intervals according to when we changed our contact details for them. We can do this, because every change we make to contact details is recorded in dadem’s database (see the representatives_edited table).

I’m going to sound a bit like Donald Rumsfield here, but keep with me. For each interval, we either have good or bad contact details, and we either know or we don’t know that they are good or bad. If we know that they’re bad (e.g. we have no details at all), then that interval isn’t a problem. No messages will have been sent, WriteToThem will have apologised to constituents trying to send messages, and no questionnaires will have been sent out. Any questionnaire results we have from good intervals can still be used and will be fine.

The case when we think we have contact details is harder. The script does some simple checks to work out if they were really valid. For example, if there have been at least a couple of questionnaire responses, and none were affirmative, then it is a bit suspicious. The script a threshold of length of time of suspicious intervals, and outputs as “unknown” MPs which it thinks there may have been a problem with for long enough for it to matter.

Tom then heroically checked all those MPs. Some we’ve marked as “WriteToThem had possibly bad contact details for this MP” in the league table. For others, we managed to verify that the questionable email or fax that we had (either via the MPs own website, or by ringing up their office) was actually good. The script then spits out, of all things, PHP file, which you might find useful on your own websites. It contains the complete detailed results. Make sure you look at the “category” for each MP. That indicates if we had too little data, or bad contact details, amongst other things.

Why PHP? And why not update the stats in real time? We’ve decided to make new statistics just once a year. Firstly, this is much easier to describe, we can say for example on TheyWorkForYou (where the responsiveness data also appears) that it is for the ‘year 2005’. Secondly, it lets us do the manual checking, so we are more confident about our data. Thirdly, it’s good for publicity to announce the new statistics as a news story. And finally, it is much easier to manage an unchanging text file (e.g. the PHP file), stored forerver in CVS, than it would be an ephemeral table in a database somewhere.

After all that, we mailed or faxed all the still sitting MPs who scored 100% responsiveness, to congratulate them on a job well done. Greg Pope, Richard Page, Fraser Kemp, Thomas McAvoy, Bob Laxton, Mark Simmonds, Paul Stinchcombe, Dennis Turner, Nick Ainger, Alan Meale, Adrian Sanders, Tom Cox, Andrew Hunter, Robert Key, Andrew Selous, John Wilkinson, Paul Goodman, Gwyneth Dunwoody, David Evennett, Peter Atkinson, Andrew Bennett, George Young, Terry Lewis, Douglas Hogg, Patrick Cormack, Andrew Robathan, David Stewart, Colin Challen, Harry Barnes MPs and all your staff, congratulations! (That list includes ones who are no longer MPs, for example stood down at the General Election)

Hello everyone,

This Tuesday I am heading to North America for a couple of weeks to spread the news about PledgeBank and mySociety. I’d love to meet as many people as possible who are interested in mySociety, or who might become so, and I’m travelling all over the continent to do so.

I’m visiting now because we’ve quietly been doing lots of work to make our most popular site yet,PledgeBank, a truly international site with the ability to create, search and subscribe to pledges in any one of over 2 million cities, towns and villages around the world. We’ve also engineered the site so that volunteer translators can help us translate the whole site into other languages just by editing one easy text file. This is the first time a mySociety project has been truly international, and we’re keen to share our free services beyond the borders of the UK.

We’ve not done any publicity in the US yet because we know from experience that it is was an essential part of the highly successful UK launch that we had a variety of strong, clear and practical pledges ready to go from day one. Discussing potential US and Canada based pledges and building informal partnerships is what this is all about.

So, if you are in the US or Canada, or you know someone who is and who would be interested in meeting, please get in touch with tom@mySociety.org

We thought that the launch of PledgeBank might be a bit more low key than NotApathetic, what with it not having the election as a hook, and being a bit more ‘high concept’ and difficult to explain to people. But what do we know, eh?

Last night I should have gone to bed early, but these things being how they are I stayed up late having tea with my housemate and his friend. I wanted to get up early, because I knew a few things needed tidying before we started getting media coverage, so I set my alarm. I haven’t done that for work for years! So I’m a bit sleepy.

The most important early thing I did was make the front page featured pledges appear in a random order, for more fairness and serendipity. Late last night Chris had added code in to fuzzily find pledges which somebody has typed in. It uses the database to look for the number of common three letter substrings, so if you type in “http://www.pledgebank.com/suirname” it gives a nice error page leading you to go to “http://www.pledgebank.com/Suriname”. It’s pretty good, and all I had to do was tidy up the text a bit, and add it to the search page as well.

By that time everyone else was up, and the no2id people were publicising their pledge. We were all on IRC, and tailing various logfiles. There were quite a few minor tidy ups for us to make to the launch pledges that were made over the weekend, changing text and signup numbers for the creators a bit.

Someone spotted that the “all pledges” page had the wrong calculated count for one of the pledges. This was very odd, as it was right for all the others. I downloaded a fresh dump of the database to my local machine, where everything was fine. Meanwhile, Chris noticed the PHP server was crashing. After more investigation, we found a subtle bug was creating a corrupt PHP variable. Calling “gettype” on it caused the PHP process to stop with an error, and calling number_format crashed the whole thing. We’re still not sure quite what PHP bug caused this, and need to investigate it more. But we found a simple workaround which stopped it causing any more trouble.

You always find all the bugs when your traffic goes up! That’s why staged beta getting larger and larger, of which today is in many ways the next phase, is the way to go.

We hope you enjoy Geoff and Tom’s paper. But while you’re here, why not take a quick look at our current projects:

• WriteToThem.com — contact your elected representatives, for free
• NotApathetic.com — giving a voice to those who aren’t voting in the British May 5th General Election
• PledgeBank.com — “I’ll do this if enough other people promise to help out”

This morning I woke up quite late because I didn’t sleep well (and had been building Public Whip’s How To Vote). Today I’m not really working for mySociety, as I’m going to London this afternoon, but Tom had a couple of important bits for me to do. There was a bug in the admin page, and a missing link on the confirmation page. I fixed these up, improved the test suite a bit, and deployed to the main site.

PledgeBank is now ready for people to start using it in earnest. We’re still in testing, as we’re sure there’ll be lots of changes needed to it as it is used in the real world. But all the basic features, and fancier ones such as SMS and the auto-generated flyers, work. Tom’s just been on the radio, and he’s making lots of specific example pledges like this one about Shropshire. So, the hunt is now on.

My next jobs are to tidy up outstanding tickets which we have already fixed, and fix any bugs in there. The next feature we’re adding to the site is comments, so people can discuss the pledge.

This afternoon I’ve been installing web log analysis software. I chose to use awstats, mainly because I know it already. But also because last time I looked it was the prettiest open source one. Installing it was mostly painless, but hampered by a couple of technical annoyances.

Firstly, we run everything on the mySociety server as different users, for privilege separation. This involves using suExec and FastCGI, neither of which are as mature as they might be. Now, the problem with suExec is that it is extremely paranoid. For example, it makes sure the owner and group of any files you try and run as CGI are the same as the user you set them to be run by. It also checks that they are under document root – a silly restriction, especially when I’ve just added an alias to httpd.conf point to the awstats.pl file which FreeBSD installed elsewhere.

The solution to this? Make a short one line bash script which merely calls through to the actual script. That it is so easy to get round them, shows how silly the suExec restrictions are – it just puts everybody off using it at all, and instead they make everything world-readable. And less secure. So by being paranoid about security, they make everyones web server less secure. Nice.

Secondly, awstats has a strange view of the world. It insists on getting log files in order. That is every row of log fed to it has to arrive, and you can’t go back and feed an older log file in later. I’m not quite sure why this is – perhaps so that its very compact text file summary of the logs can be updated efficiently. So, to run analysis of old files from months back you have to write a quick script to run through them all and call awstats with them.

Anyway, I’ve now done that for NotApathetic, PledgeBank, WriteToThem and this very website that you’re reading. So Tom will stop hassling me every time a journalist is saying “how many visitors do you get a day?” So far today we have 2807 visits to NotApathetic.com.

Mysociety is proud to announce the launch of DowningStreetSays.com, a site where you can leave and read comments on Number 10’s press briefings.

This isn’t a full mySociety project, and was thrown together by some amazing volunteers over the last few days. We hope that in a small way it helps to improve the quality of debate at the heart of government, by letting the public at government without first passing through the lens of the mass media.

Let us know what you think!

[July 2013 – dead link replaced with Internet Archive link]