The meeting day voting application (vote often!) that we’ve been mentioning everywhere all week is a new departure for mySociety. In a frantic bid to catch up with the cool kids, it’s our first deployed Ruby on Rails application. This happened because Louise Crow, who kindly volunteered to make it (thanks Louise!), felt like learning Rails. We used to have a policy of using any language, as long as it was open source and began with the letter P (Python/Perl/PHP…). This has now been extended to the letter R!

You can browse the source code in our CVS repository. One interesting thing about Rails applications is that they are structured things, a deployable directory tree. So are mySociety applications.

For example, take a look at PledgeBank’s directory. It’s a mini, well defined filesystem – the ‘web’ directory is the meat of the stuff, but note also ‘web-admin’ for the administrator tools. Include files are tucked away in ‘perllib’ and ‘phplib’, while script files nestle under ‘bin’. We keep configuration files (analogous to the Windows Registry, or /etc on Unix) under ‘conf’. Database schema files live in ‘db’.

And a rails application is much the same. But much much much more detailed. Some of those are extra directories which we also have, but only when we deploy, not in CVS (for example, log files). All in all they are surprisingly similar structures, which shows we’re either both on the right lines, or both on the wrong false trail.

Like making Frankenstein’s monster, poor Louise and I had to graft these two beasts together just to deploy this small application. For example, we have a standard configuration file format which we read from Perl, Python and PHP. The deploy system does useful things with it like check all entries are present, and generate the file for any sandbox from a template. To get round this, there’s an evil script, possibly the first time PHP has been used to make YAML. (And please don’t look at the thing that makes symlinks.)

We could have extended Rails to be able to read its configuration from our file format, but that would be a lot more work. And we could have discovered how to hack its log file system to write to the mySociety log file directory. But everything is so coupled, it doesn’t ever seem worth it. Any Rails apps we deploy will just have to be an even more confusing mass of directories, application trees inside application trees.

So, a silly post for today: Postcodeine. This is a British version of Ben Fry’s zipdecode, a “tool” for visualising the distribution of zipcodes in the United States. This is, as has been pointed out to me, wholly pointless, but it’s quite fun and writing it was an interesting exercise (it also taught me a little bit about AJAX, the web’s technology trend du jour). If you want the source code, it’s at the foot here; licence is the Affero GPL, as for all the other mySociety code.

How it works: this is pretty obvious, but I might as well spell it out. The web page has four images on it: the big and small base maps, and two overlays. The back-end code is responsible for drawing sets of postcode locations into transparent PNGs, and when you type things in the text field, the src for each of the overlay images is changed. Panning the large map is done by issuing another request from Javascript to grab the mean location of all postcodes matching the given prefix (slightly hobbled, so that this isn’t a generalised postcode-to-coordinates oracle — sorry!); the rightmost pane, with a list of postcodes and their areas, is populated from another HTTP request. It could be done with an iframe but, as Paul Graham puts it, “Javascript works now”, so we might as well use that.

(I should say, by the way, that I wrote this in my copious spare time. It’s copyright mySociety because I don’t have the right to use the postcode database myself.)

And a follow-up to my last post: the population density and customary proximity APIs are now available in Gaze. The additional APIs are:

get_population_density

Parameters:

lat

WGS84 latitude, in decimal degrees

lon

WGS84 longitude, in decimal degrees

Return an estimate of the population density at (lat, lon), in persons per square kilometer, as a decimal number followed by a line feed.

get_radius_containing_population

Parameters:

lat

WGS84 latitude, in decimal degrees

lon

WGS84 longitude, in decimal degrees

number

number of persons

maximum

largest radius returned, in kilometers; optional; default 150

Return an estimate of the smallest radius around (lat, lon) containing at least number persons, or maximum, if that value is smaller, as a decimal number followed by a line feed.

For instance,

• http://gaze.mysociety.org/gaze-rest?f=get_population_density;lat=51.53;lon=-0.1020 gives the population density of Islington in London (about 8,000 persons/km²)
• http://gaze.mysociety.org/gaze-rest?f=get_radius_containing_population;lat=51.53;lon=-0.1020;number=200000 returns the radius of a circle around that point containing 200,000 people (about 2.7km)
• http://gaze.mysociety.org/gaze-rest?f=get_radius_containing_population;lat=43.09;lon=-95.54;number=200000 the same, for a point near Cylinder, Iowa, USA (about 74km)

Enjoy! Questions and comments to chris@mysociety.org, please.

mySociety yesterday launched a pair of Back o’ The Envelope projects based on Google Maps.

Placeopedia.com — Connect Wikipedia articles with the places they represent

YourHistoryHere.com — Share local and geographic history and trivia.

There are a few things to say about both projects:

1 – As is normal with mySociety projects the code for these projects (excepting Google maps) is open source. We hope that by providing a ready-made annotation system, people will find it easier to make their own publicly-authored layers of information.

2 – Both sites syndicate their data under open source licenses, and in a location-queryable fashion. This is really important, as it allows for all types of nice local history to be syndicated to tourism sites, local community discussion boards, blogs and so on.

3 – We’re calling them ‘Back o’ the Envelope’ to contrast them to the big, polished and time consuming projects we run like PledgeBank.com and WriteToThem.com.

For pretty well the last week now I’ve been adding various new messages and alerts to PledgeBank. First up were a set of emails which go to creators. Two of these are “chivvying” ones, which remind the creator to do publicity, and give them some advice.

Quite a few people make pledges and don’t realise that they have to publicise them themselves. Ambient traffic to PledgeBank will get you some sign ups, especially at the start of last week when we had lots of media attention, but pledge creator’s also have to do some of their own work. So we chivvy the creator once after a few days if they don’t have many sign ups.

Secondly, even if you’ve got some publicity and some signups, sometimes you just haven’t got a high enough rate to reach your target on time. I’ve added some code which checks signup rate on average over the last week (for pledges at least a week old). If things were to carry on at that rate until the deadline, and the pledge still wouldn’t succeed, it sends an email with ideas about publicity/marketing of your pledge. Now we just have to hope this is useful rather than annoying…

Another thing we noticed is that lots of pledge creators hadn’t sent a message to their signers after their pledge succeeded. Admittedly this was partly because of a bug in our announce message code (there was no action= in one of the forms, which caused problems on some browsers). So I also added a successful pledge reminder email which is sent a few days after success.

All the stuff above was done using the nice “message” tables which Chris made a while ago. They make it easy to send messages which go out only once, and to creators, signers and late signers as appropriate. However, for the next set of emails, some new code was needed. These are optional “alerts”. For example, to email you when new comments are added to a pledge, and when there is a new local pledge in your area. So, the last day or two I’ve been coding that.

So, it’s my turn to write something here again. Ho-hum. Anyway, lately I’ve been working on adding the geographical features to PledgeBank which everyone thinks are there already: specifically, finding pledges which are nearby. To start with, we’re doing this for the UK only (because we already have the infrastructure to do postcode-to-coordinates lookups through MaPit), but the intention is to do thiis for the whole world as soon as we can, either by having users select their location through a gazetteer, or, where we can get the data, by using a similar postcode/zip-code/whatever-to-coordinates lookup. So that means we have to deal with places which might be anywhere on earth, which means dealing with latitudes and longitudes. And as anyone who’s dealt with this stuff knows, it’s very tedious to get this right. I’m afraid I’ve now spent too long reading about datum ellipsoids and Helmert Transforms to want to spend any time at all talking about them, so in the unlikely event that you’re interested, you’ll just have to read the (relevant parts of) code.

Oh, and if anybody has good suggestions for a hierarchical gazetteer with world-wide coverage, I’d love to hear them. Similarly, does anybody know if this one is any good?

So, another day, another new version of the PledgeBank source code live on http://www.pledgebank.org/. Actually, the changes are mostly underneath the surface, so you shouldn’t notice any specific differences, unless we’ve broken something, in which case whinge to team@pledgebank.com, as usual. That said, the new posters and SMS signup to pledges are now live, so you should now go out into the world and do Good Virtuous Stuff with them.

This is supposed to be the developers’ ‘blog, so a couple of technical things which have annoyed me today. (presumably you all read my actual web log and therefore expect me to write about things that annoy me):

• You can’t use a variable quantity in a limit clause in a subselect in PostgreSQL. “A what? In a what? What?“, I hear you cry. Well, this did come up in real life. When I was upgrading the PledgeBank code, there were various changes to the database schema which had to be made first. One of them was to change the way that the success of a pledge (i.e., what happens when it reaches its target) is recorded. Previously we had two boolean columns, like this:

  create table pledges (  -- obviously SQL tables should have singular names, but
                          -- in this case nobody asked me...
      -- ...
      success boolean
          not null
          default false,  -- indicates that the pledge has succeeded
      completionnotified boolean
          not null
          default false,  -- indicates that the creator and signers have been
                          -- told that the pledge succeeded
      -- ...
  );

  Now, this is messy and not enough to describe how the site actually works. Specifically, there are some types of messages which should be sent to creators and signers, some which should be sent only to signers who signed before the pledge succeeded (in between success and the deadline, you can still sign the pledge), some which should only be sent to non-SMS recipients, etc. etc. So instead we now have a table of messages with flags indicating where they should go to and so forth. A side-effect of this is that the above structure is replaced with this:

  create table pledges (
      -- ...
      whensucceeded timestamp,    -- indicates when pledge succeeded
      -- ...
  );

  Now, there are Real Pledges on the live site, so unlike the development site we can’t just drop the database in an update; instead, we have to port all the data over to the new data model. So what you’d like to write is, obviously,

  begin work;
  alter table pledges add column whensucceeded timestamp;
  update pledges
  set whensucceeded = (
      select signtime
      from signers
      where pledge_id = pledges.id
      order by signtime
      limit 1 offset pledges.target
  )
  where success;
  alter table pledges drop column success;
  commit work;
      -- I love using a proper database!

  Sadly, you can’t. The two arguments in the limit statement in the subselect have to be constant. (No, I don’t know if/where this is documented. I, uh, asked on IRC.) This sucks. In the end I just set whensucceeded to the current time for currently-successful pledges; it’s not right, but it’ll do.

• Python (2.3, on FreeBSD) either sets O_NONBLOCK on sockets by default, or fails to clear it when creating a socket. Result: program crashes with EAGAIN down in the FastCGI library every so often. Outstanding!

Poll: Should we turn on comments on this ‘blog? (Does anyone read it, anyway?) Mail me at chris@mysociety.org with your answers….

Right now I’m working on the email code for Pledgebank.

There are (ignoring SMS) three communications problems in Pledgebank: firstly, how does the pledge setter get to communicate with the pledge signers when the pledge is complete (so that the setter can tell the signers what to do / how to organise themselves, or whatever); secondly, how do the signers communicate amongst themselves before the pledge completes (to discuss tactics for getting more people signed up, or whatever); and thirdly, how can other people get more information or clarification on the pledge.

Ideally, we’d like to be able to do all of the above while keeping individual signers’ email addresses private.

(As an aside, I should say that the proper English words for “pledge creator” and “pledge signer” are “pledgee” and “pledger”. I fought a valiant struggle to have these adopted as the Official Pledgebank Terminology, but inevitably we all got confused as to which was which, and reluctantly abandoned them in favor of “creator” and “signer”, which sound a bit buzzword to me. Ho-hum.)

Anyway, there are lots of different ways that we could allow the creators and signers to communicate. After a long discussion we decided that we’re going to implement,

• an announcements-type mailing list for the pledge creator to use,
• an (optional to join) discussion list for signers and the pledge creator, and
• a web-log-style comments page for each pledge.

which seemed like a sufficient set of features which would avoid having too many confusing configuration options.

(Another way to look at this is to look at this from a permissions perspective:

For the first two functions we need to limit access to what’s said to particular groups of people. That means that either we have to have a login system on Pledgebank, which we’re avoiding, to keep things as simple as possible; or to use email, which is practical because creators and signers have to confirm their email addresses in order to pledge. No access restrictions are needed for comments and clarifications, and it’s useful to show comments publically — so as to avoid ever answering the same question twice — so public web comments are appropriate here.)

So, this week’s task is to get the email stuff all working. Getting email sending from web applications right can be tricky, especially if you care about reliable delivery. As you can imagine, this is a whole heap of fun….