Just a short one for today. Here’s a plot of the rate of take-up of Engineers Without Borders UK’s successful pledge to solicit funding for a development project in Suriname (the first of what we hope will be many projects using PledgeBank to raise funds):

[Plot showing takeup of EWB Suriname fundraising pledge – image gone]

— we’ll be incorporating this (and some other handy information, such as how many people are looking at the pledge pages and which sites link to them) into the site some time soon to help pledge creators see how they’re doing at publicising the pledges (and also because graphs look cool).

Today’s other task was to switch the live site over to using the new login system which Francis mentioned the other day. This was complicated by the fact that there might be some users who had received a signup token, but hadn’t clicked through on the token to actually complete their signup before the site was shut down for the upgrade. The solution to this is a bit nasty, but it seems to have worked OK, which is nice.

The job for this week has been getting WriteToThem back up. We’ve now sorted out the Scottish boundary changes and the names of the new MPs, but we don’t have their contact details yet. Today, among sundry other bits and bobs, including debugging thorny Exim problems and other such uninteresting stuff, I’ve been sending faxes to the Parliamentary fax numbers we have for MPs who’ve been re-elected, asking them whether their details have changed. We’re doing that because we don’t know whether those MPs will be in the same offices in this Parliament as in the last, and obviously it would be a breach of trust to send constituents’ mail to the wrong offices. So, I’ve finally reached rock bottom: Chris Lightfoot, junk fax merchant. Sorry, everyone!

(Previous “rock bottom” moment: being telephoned early in the morning by a Labour MP with a rather cut-glass accent. She was calling to find out she had been receiving phone calls from our fax machine — we had the number wrong — at her office. But she didn’t tell me who she was, and, because over the phone she sounded uncannily like an old friend of mine, I assumed that it was my friend on the line. She must have assumed that she knew who I was too, and we exchanged several rounds of pleasantries before comprehension gradually dawned: “This is Mrs — —, MP; who am I speaking to?” Come to think of it, she may even have asked “to whom am I speaking?”.)

We’re looking for ways to make it easier for volunteers to get involved in mySociety. Like everything in real life this is mostly a question of openness and policy, but there are also a few technical steps we think would make life easier. One of these is to make it easier for us to hand over a test web server to a volunteer or a group of volunteers to develop code on, play with and generally break. At the moment that’s quite hard to do, because we use apache and all our sites are hosted on one machine (yes really — computers are fast and memory is cheap, though in day-to-day life you’d never notice that, because most of the IT industry is involved in developing “technology” — meaning, “programs that don’t work yet” — that are designed to make your computer slow again: Microsoft Windows, Java, modern web browsers, etc. etc.). Apache is monolithic and if one user breaks the configuration of their test site they can bring down all the sites hosted on the machine. Also, apache isn’t very good at crossing security boundaries (arguably that’s a fault of UNIX generally), so unless we’re prepared to give all the volunteers root (not acceptable for policy reasons) they need to hassle us to get things done (not acceptable to them). Indeed, to save time and admin hassle, IVotedForYouBecause was developed and is hosted elsewhere.

So the idea is to strip away all this crap by running lots of instances of apache, and giving one to any group of volunteers who want to play with one of our sites — all running under their own unprivileged UID — and then direct requests from the outside world through to the appropriate internal apache server via a public-facing proxy. The design I’m envisaging looks something like this:

[Basically pointless block diagram – server hosted on has gone]

(Actually it’s not clear to me that that diagram conveys anything you won’t have understood anyway, but there we go.) For the front-end server I’m using Squid, which is balky and overcomplicated, but supports one very handy feature which is invaluable in this setup: external URL rewriting scripts, which can be used to redirect requests that come through the cache to other resources. The classic application of this is to redirect requests for advertising and other pointless content to local resources so that they don’t take up bandwidth or break your web browser; in this case we’ll be using it to rewrite requests for certain publically-visible URLs (“http://fred.pledgebank.com/…” or whatever) into internal URLs which route to individual users’ apache servers (“http://127.0.0.1:8001/…”). One of the nice things about this is that it preserves the Host: header, and (with a further small hack) apache can be persuaded to pretend that requests weren’t proxied at all, so any back-end stuff that needs to know clients’ IP addresses (such as logging, etc.) can be used unmodified. On top of this, squid will cache responses (assuming that we aren’t lazy about the headers we emit on our own content), which may speed things up a bit for certain sites, though I suspect (with little evidence, and none I’m prepared to bore you with now) this won’t be very useful in practice for the types of sites we’re building.

Another attractive feature of this scheme is that it means that we’re not tied to apache: we could use lighttpd or something, if we wanted to. I doubt that a technical reason to do that will arise in practice, but every minute I spend fighting apache configurations is a minute closer to chucking the bloody thing and picking another web server.

So, it’s the usual story: you start off trying to work around the brokenness in one bit of software, and then all sorts of exciting possibilities suddenly open up. At least, that’s one way to look at it.

Well, I’m back from my holiday, suitably sunburned and (relatively) relaxed. As Francis mentions, I was off in the Mediterranean somewhere (Majorca, specifically) suffering from miserable internet withdrawal symptoms. I did manage to get IRC up-and-running over dialup for election night, though this turned out to be surprisingly expensive. For once I was grateful to my iBook, which did actually Just Work when plugged into the wall.

Anyway, today’s job is sorting out the new Scottish constituency boundaries. Scotland’s Parliament was dissolved in 1707 on the passing of the Act of Union, to be reconstituted in 1999. The quid pro quo for the Scots was enhanced representation in the House of Commons; Scottish constituencies had, in 1998, an average of 55,000 electors, compared to 69,000 in England. This anomaly has now been corrected, reducing the number of constituencies in Scotland from 72 to 59; all but three of the latter have different boundaries.

This means updating MaPit, the component we built to map postcodes into electoral geography, to deal with the new boundaries. Ideally the way that we’d do this is to wait for Ordnance Survey to ship us, via our friends in ODPM, the new revision of their Boundary-Line (TM, apparently) product, with the outlines of the new constituencies encoded in attractive machine-readable form, and feed it to our existing import scripts. (As so often in life, it’s not quite that simple, but you get the general idea.) In an ideal world, this would also contain all the changed boundaries of the English counties and their constituent county electoral divisions.

However, this is not an ideal world, and though there is a new revision of Boundary-Line in the works, it hasn’t come out yet, so we have to construct the point-to-constituency mapping in some other way. Happily, at this stage of the boundary revision process, the constituency boundaries are coterminous with ward boundaries, so it’s possible to just lift the definitions of the new constituencies from the relevant Statutory Instrument and fix up the constituencies from the ward boundaries, which haven’t changed. This, sadly, has occasioned a bit of a hack to our code, because we generally don’t assume that electoral geography is hierarchically defined — because it isn’t.

(I don’t feel too bad about committing this hack, actually, because we’re likely to chuck the whole MaPit database and reconstruct it later in the year from OS data. When we built it originally, we did so from data in ESRI shapefile format; unfortunately, OS stuffed up the process of generating this from their own, internal and quite bonkers, NTF format, so the various area ID numbers in the database are not unique and not expected to be stable. We’d rather like stable ID numbers, so that we can cope gracefully with revisions to geography while maintaining continuity of, for instance, statistical data about MPs, so next time round we’re going to work from the NTF instead.)

Sadly this Scottish hack doesn’t get us anywhere with the new county boundaries, and OS have told us that not all of the updated counties will be included in the forthcoming Boundary-Line revision. So it’ll be back to the tedious conversion of statutory instruments into SQL at some point in the near future, except that we’ll probably have to start building things up from parishes, rather than wards. Expect more anguished posts on this in the future.

Meanwhile, Francis and Tom are collecting names and contact details for the new MPs. Tom tells me that this intake looks much more tech-savvy than the last, which could be good news from our (and everyone else’s) point of view. Hopefully WriteToThem will be cranking back into action — as far as MPs go, at least — fairly soon.

We hope you enjoy Geoff and Tom’s paper. But while you’re here, why not take a quick look at our current projects:

• WriteToThem.com — contact your elected representatives, for free
• NotApathetic.com — giving a voice to those who aren’t voting in the British May 5th General Election
• PledgeBank.com — “I’ll do this if enough other people promise to help out”

So, another day, another new version of the PledgeBank source code live on http://www.pledgebank.org/. Actually, the changes are mostly underneath the surface, so you shouldn’t notice any specific differences, unless we’ve broken something, in which case whinge to team@pledgebank.com, as usual. That said, the new posters and SMS signup to pledges are now live, so you should now go out into the world and do Good Virtuous Stuff with them.

This is supposed to be the developers’ ‘blog, so a couple of technical things which have annoyed me today. (presumably you all read my actual web log and therefore expect me to write about things that annoy me):

• You can’t use a variable quantity in a limit clause in a subselect in PostgreSQL. “A what? In a what? What?“, I hear you cry. Well, this did come up in real life. When I was upgrading the PledgeBank code, there were various changes to the database schema which had to be made first. One of them was to change the way that the success of a pledge (i.e., what happens when it reaches its target) is recorded. Previously we had two boolean columns, like this:

  create table pledges (  -- obviously SQL tables should have singular names, but
                          -- in this case nobody asked me...
      -- ...
      success boolean
          not null
          default false,  -- indicates that the pledge has succeeded
      completionnotified boolean
          not null
          default false,  -- indicates that the creator and signers have been
                          -- told that the pledge succeeded
      -- ...
  );

  Now, this is messy and not enough to describe how the site actually works. Specifically, there are some types of messages which should be sent to creators and signers, some which should be sent only to signers who signed before the pledge succeeded (in between success and the deadline, you can still sign the pledge), some which should only be sent to non-SMS recipients, etc. etc. So instead we now have a table of messages with flags indicating where they should go to and so forth. A side-effect of this is that the above structure is replaced with this:

  create table pledges (
      -- ...
      whensucceeded timestamp,    -- indicates when pledge succeeded
      -- ...
  );

  Now, there are Real Pledges on the live site, so unlike the development site we can’t just drop the database in an update; instead, we have to port all the data over to the new data model. So what you’d like to write is, obviously,

  begin work;
  alter table pledges add column whensucceeded timestamp;
  update pledges
  set whensucceeded = (
      select signtime
      from signers
      where pledge_id = pledges.id
      order by signtime
      limit 1 offset pledges.target
  )
  where success;
  alter table pledges drop column success;
  commit work;
      -- I love using a proper database!

  Sadly, you can’t. The two arguments in the limit statement in the subselect have to be constant. (No, I don’t know if/where this is documented. I, uh, asked on IRC.) This sucks. In the end I just set whensucceeded to the current time for currently-successful pledges; it’s not right, but it’ll do.

• Python (2.3, on FreeBSD) either sets O_NONBLOCK on sockets by default, or fails to clear it when creating a socket. Result: program crashes with EAGAIN down in the FastCGI library every so often. Outstanding!

Poll: Should we turn on comments on this ‘blog? (Does anyone read it, anyway?) Mail me at chris@mysociety.org with your answers….

Just as I was getting on with something else, I am called hither to write a blog post. So this’ll be a short one, I’m afraid. And instead of talking about what I was doing today (fixing bugs, modifying database schemas, and other dull interludes in the programming life) I’ll draw your attention to a couple of NotApathetic things.

NotApathetic has (notwithstanding various teething troubles) been going pretty well, and there are some more cool ideas in the pipeline. For those of you who are enjoying the latest web-logger fad, “tags“, Matthew Somerville has implemented this experimental page generated from users’ confessions reasons for not voting. It’s rather fun; since, unlike the “‘blogosphere”, mySociety isn’t about throwing away all the lessons of the last forty years of research in information retrieval, we generate tags automatically rather than expecting users to annotate their posts with the most meaningful set we can manage. (They wouldn’t.) Hopefully this or something like it will become part of the site’s front page soon.

And something which I want to implement — later this week, hopefully — is a voting system to collect two types of data: firstly, which posts people think are particularly interesting, so that we can give them more prominence on the front page; and secondly, given two example posts, how similar (in some general sense) the two reasons are. The point of the second one is that, given a set of similarity data, we should be able to cluster and categorise the country’s apathy and better understand it (and, of course, draw beautiful pictures of it in gnuplot, too). You might not think that sounds like fun, but I do.

So, we’ve launched Not Apathetic. If you aren’t voting, here’s your chance to say why. It’s a grouphug.us for political confessions. Much kudos to Sam, Matthew and Matthew who did all the work on the site.

So, what am I up to right now? As you’ll probably know, the General Election has just been called; as those who aren’t obsessive political geeks will know (full disclosure: I had to look this up) that means that Parliament will be dissolved on Monday 11th April. One peculiarity of our Constitution is that, once Parliament is dissolved, there are no MPs any more (though ex-MPs still get paid, and government ministers retain their posts and their salaries). So my last ten minutes’ work were looking up what FaxYourMP.com said during the 2001 election, and then adding the following note to WriteToThem.com:

Important note about the General Election: there will be a General Election on Thursday 5th May 2005. That means that on Monday 11th April, Parliament will be dissolved. Once Parliament is dissolved, you do not have a Member of Parliament (though your ex-MP still gets paid — nice!). So if you’re going to write to your MP, be quick! Messages sent to MPs which have not been sent by 11th April will be returned to their senders.

We’ll be back as soon after the election as we can manage!

Now that’s done, and I’ve completed today’s service to posterity by writing the above, it’s back to adding the last few features to PledgeBank that we need before we launch. And look out for NotApathetic (coming to a domain name like that, real soon now), too.

So, just a very quick update. Actually Tom was supposed to write this, but he’s off schmoozing at some event for people much cooler than us programmers, so I’ve volunteered. Since I last wrote about SMS, we’ve arranged to do our SMS stuff in partnership with C360, who’ve very kindly offered us a much better deal than the other aggregators we’d talked to. Unfortunately that required a rewrite of the SMS interface, which took up a bit of time. But that’s now done and seems to be working. I’d ask you to test it, but (a) we already think it’s working OK, and (b) because of The Rules on the promotion of “premium rate” (i.e., reverse-billed) SMS services, we can’t advertise the service publically without also advertising a point of contact for users. Because that has to go on the flyers which can be printed out to advertise pledges, we need to arrange a short-and-snappy postal address (PO Box or similar) which won’t take up too much space. Next week, hopefully.

So, that was yesterday and this morning’s work. Today it’s back to the mailing lists which pledge creators will use to send information about pledges out to the signers. But before that, I need to make some more coffee. Toodle-pip.