1. 5 Years On: Why Understanding Chris Lightfoot Matters Now More Than Ever

    My friend and mySociety’s first developer Chris Lightfoot died five years ago today. He killed himself in his own flat for reasons that we will never really know, but which are doubtless linked to the depression which he’d been fighting for years. He was just 28, but had already achieved so much that The Times ran an obituary of him. He would have laughed mightily about the fact that this is now behind a paywall.

    To mark this occasion I wanted to write something for mySociety staff and volunteers who never knew Chris, and for a wider audience of people who work in places like GDSCode For America or indeed anyone with an interest in politics and governance. What Chris represented is too important to be lost in the grief at his passing.

    The basic fact to understand about Chris was that he was a very specific kind of polymath – one perfectly suited to the internet age. What I mean by this is that he did much more than simply master varying disciplines:  he saw and drew connections between fields. He wouldn’t just master cartographic principles, engage in politics and, as Francis Irving put it, ‘write Perl like other people write English’: he invariably saw the connections and mixed them up in meaningful and often pioneering ways.

    Moreover, this mixing of disciplines was conducted at a furious, restless pace, and knew absolutely no concept of ‘too hard’ – problems were either fundamentally impossible, or ‘trivially soluble’, to use one of his favourite and most gloriously under-stated phrases. Who else would build the technology to break a captcha, just to investigate what American truck rental costs tell us about internal migration in America, for fun?  The answer is trivial.

    That he was a genius is not what I want you to understand. Telling you that someone you never met was smarter than you is not helpful, and doesn’t fulfil my promise to tell you why understanding Chris matters.

    What is fundamentally valuable about Chris’ legacy (besides piles of code that power services still running today) is that his story signals how we all need to change our conception of what it means to be ‘wise enough to rule’. Let me explain.

    Unlike most of us, Chris had the luxury of being able to pick any field of study that interested him, dig up some books and papers, and teach himself a graduate-level understanding in what felt like a few days. It is hard to express quite how fast he could consume and internalise complex new information, and how relentlessly he went at it. To note that he got six A grades at A-level is too puerile a précis, but it is indicative.*

    Again, I am not telling you this to make you feel stupid: what matters is what he chose to do with this gift.  What he chose to do was built an ever-expanding palette of skills from which he could paint as he pleased. And what he chose to paint was a vision of a better, saner world.

    This painting ranged across a huge expanse of topics and disciplines: nuclear engineering, political ideologies, constitutional law, military history, statistics, psephology, economics, security engineering, behavioural psychology, propaganda, intellectual propery law and more. His favourite brushes were Perl and a blog composed of prose so sharp and funny that George Bernard Shaw would not have been displeased by the comparison. I still wish I could write half as well as him.

    What I want to communicate most is this: the disciplines he chose to study form a combined19th, 20th and 21st century curriculum of skills required by modern leaders, both leaders of political organisations and government bureaucracies.  Chris’s life was the invention of a massively expanded, far more up to date version of the traditional Politics, Philosophy and Economics course that this country still uses to educate its elites.

    Some of these disciplines are timeless, like the understanding of ideologies or economics. Some represent vital new issues that emerged in the 20th century, like nuclear energy and world-scale warfare. But mixed in there are wholly new, alien group of skills that the recent SOPA, Wikileaks and ID cards debacles show that modern leaders haven’t got anywhere near to internalising: they include knowledge about security engineering, intellectual property and how new technologies clash with old laws and ideologies. They are skills that nobody used to think were political, but which are now centre stage in a polity that can’t keep up.

    This doesn’t mean Chris would have made a perfect leader: I used to argue with him a lot about how he weighed up the costs and benefits of different issues. But what he fundamentally had right was the understanding that you could no longer run a country properly if the elites don’t understand technology in the same way they grasp economics or ideology or propaganda. His analysis and predictions about what would happens if elites couldn’t learn were savage and depressingly accurate.

    The canon of Chris’s writings and projects embody the idea that what good governance and the good society look like is now inextricably linked to an understanding of the digital. He truly saw how complex and interesting the world was when you understood power as well as networking principles in a way that few have since.

    There is, of course, much more to say about Chris’s life. His blog, built on software that foresaw Posterous, is wonderful, hilarious and utterly readable, so you can learn more yourself. Martin Keegan’s obituary is touching and a much better portrait of how much fun it was to be friends with Chris. I hope to memorialise what he represents to me, if I can. But for now, I’ll sign off with a quote from a blog commentor:

    “Chris was kind enough to take the time to reply to me, an internet nobody whom he didn’t know from a bar of soap, on a fairly complex statistical question once. He took a lot of time and effort in his response, and he made sure I understood it properly. It’s not often you find knowledgeable people willing to take their own time to educate an unknown person. We need more people like him, not less.”

    * For US readers, this is like having a GPA of 4.0, but achieved across twice as many subjects as you actually need to take.
  2. Technical look at the new FixMyStreet maps

    This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

    Progressive enhancement

    During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

    Zooming

    We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

    OpenStreetMap

    Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

    OS StreetView tile server

    Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

    Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

    [sv]
    type=WMS
    url=path/to/mapserv.fcgi?map=os.map&
    layers=streetview
    tms_type=google
    spherical_mercator=true

    MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration* – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked 🙂 ).

    Other benefits of OpenLayers

    As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

    Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

        if ( fixmystreet.area ) {
            var area = new OpenLayers.Layer.Vector("KML", {
                strategies: [ new OpenLayers.Strategy.Fixed() ],
                protocol: new OpenLayers.Protocol.HTTP({
                    url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                    format: new OpenLayers.Format.KML()
                })
            });
            fixmystreet.map.addLayer(area);
            area.events.register('loadend', null, function(a,b,c) {
                var bounds = area.getDataExtent();
                if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
            });
        }

    Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

     

    *Broken link removed, Nov 2014

  3. RIP Angie Martin 1974-2009

    It is with overwhelming sadness that I write to tell our community that Angie Martin, mySociety’s fourth core developer, has died. She was taken from us by the cancer that she had been fighting since soon after we hired her less than two years ago.

    Possessed of an almost unbelievably upbeat personality, Angie brought not only her formidable Perl skills, but her blazing warmth of character to our team. In remission during our yearly retreat in January this year, she combined laughter with a typically tough line of questioning on ideas she thought insufficiently robust. With typical disgregard for cool, her CV noted that she was “known to enjoy wrangling regular expressions on a Sunday Morning”. She didn’t see any contradiction between being a successful woman and a geek, throwing herself wholeheartedly into the Mac-toting, perlmonger ethos. She even brought her husband Tommy with her, who became a significant volunteer.

    Given her habit of plain speaking, it is pointless to pretend that Angie was able to make the contribution to mySociety’s users or codebase that she wanted to. What she achieved in terms of difficult coding during recovery from chemotherapy was incredible, breathtaking – but she wanted to change the world. It now falls to the rest of us, and our supporters, to live up to the expectations she embodied, to continue to push every day, using skills like those that she had to help people with everyday problems. We now have to ask ‘What would Angie do?’, as well as ‘What would Chris do?’. It is a lot to live up to.

    She was a mySociety core developer: I hope that meant as much to her as it meant for me to have her as one of my coders.  Remember and Respect.

    Updated: Angie changed her surname upon getting married, a couple of months ago. I have just read she wanted to be remembered as Angie Martin, and so I have made that change.

    Updated 21 7 2009: Tommy has just told me that those wishing to may memorial donations should send them to Hospice at Home.

  4. Ada Lovelace Day: Angie Ahl

    One of the most striking uses of PledgeBank in recent times was the pledge made by 1700 people to commemorate Ada Lovelace’s birthday by blogging about an inspiring woman in the technology world.

    I have had several possible choices, but I’ve decided on Angie Ahl, mySociety’s 4th and most recent full time developer.

    Angie is mySociety through and through. A born perl hacker, never happier than knee deep in some grungy regular expressions, she’s also gifted with an inate understanding of the possibilities of technology for democratic reform. At interview I asked her what change she’d like to see happen from the government side of our sector, and she replied that she thought the biggest possible win was to publish Bills in parliament in a proper format. You might have heard all this before, thanks to Free Our Bills, but Angie was commenting several months before we ever discussed the idea for the campaign with anyone else. She’d just looked at the world and the obvious problem had jumped out, clear as day.

    Sadly, it’s no secret than Angie’s been seriously ill for some time. Despite this she’s managed things that’d be beyond me in the best of health: I’m only typing into wordpress now because she migrated the whole of this site from our previous system.  She came to our retreat back in January and showed an unerring ability to ask tough questions of the right people, even when tired.

    Angie beat a 100% male field to get the job she has now.  She’s unpretentious, straight talking and as glowingly warm to be near as a roaring log fire.  She’s also getting married within the next few days. Congrats, Angie: Tommy couldn’t have done better.

    AdaLovelaceDay09

  5. “Truth lies within a little and certain compass, but error is immense.”

    I’ve been working on PledgeBank quite a bit recently. As well as adding survey emails asking whether signers have done their pledge, and a feature for people to contact a pledge’s creator, I’ve been fixing numerous bugs that have sprung up along the way. For starters, people on the Isle of Man and the Channel Islands now get a much more helpful error if they try and enter their postcode anywhere on the site, rather than the confusing postcode not recognised they were getting previously.

    Other errors I found turned out not to be with our code. The PledgeBank test suite (that we run before deploying the site to check it all still works) was throwing lots of warnings about “Parsing of undecoded UTF-8 will give garbage” when it got to the testing of our other language pages. Our code wasn’t doing anything special, and there were multiple places the warning came from – upgrading our libwww-perl removed one, and I’ve submitted bug reports to CPAN for the rest (having patched our copies locally – hooray for open source).

    The Perl warnings were at least understandable, though. While tracking down why the site was having trouble sending a couple of emails, I discovered that we had a helper function splitting very long words up to help with word-wrapping – which when applied to some Chinese text was cutting a UTF-8 multibyte character in two and invalidating the text. No problem, I think, I simply have to add the “/u” modifier to PHP’s regular expression so that it matches characters and not bytes. This didn’t work, and after much playing had to submit a bug report to PHP – apparently in PHP “non-space character followed by non-space character” isn’t the same as “two non-space characters in a row”…

  6. mySociety on a Mine Cart

    The meeting day voting application (vote often!) that we’ve been mentioning everywhere all week is a new departure for mySociety. In a frantic bid to catch up with the cool kids, it’s our first deployed Ruby on Rails application. This happened because Louise Crow, who kindly volunteered to make it (thanks Louise!), felt like learning Rails. We used to have a policy of using any language, as long as it was open source and began with the letter P (Python/Perl/PHP…). This has now been extended to the letter R!

    You can browse the source code in our CVS repository. One interesting thing about Rails applications is that they are structured things, a deployable directory tree. So are mySociety applications.

    For example, take a look at PledgeBank’s directory. It’s a mini, well defined filesystem – the ‘web’ directory is the meat of the stuff, but note also ‘web-admin’ for the administrator tools. Include files are tucked away in ‘perllib’ and ‘phplib’, while script files nestle under ‘bin’. We keep configuration files (analogous to the Windows Registry, or /etc on Unix) under ‘conf’. Database schema files live in ‘db’.

    And a rails application is much the same. But much much much more detailed. Some of those are extra directories which we also have, but only when we deploy, not in CVS (for example, log files). All in all they are surprisingly similar structures, which shows we’re either both on the right lines, or both on the wrong false trail.

    Like making Frankenstein’s monster, poor Louise and I had to graft these two beasts together just to deploy this small application. For example, we have a standard configuration file format which we read from Perl, Python and PHP. The deploy system does useful things with it like check all entries are present, and generate the file for any sandbox from a template. To get round this, there’s an evil script, possibly the first time PHP has been used to make YAML. (And please don’t look at the thing that makes symlinks.)

    We could have extended Rails to be able to read its configuration from our file format, but that would be a lot more work. And we could have discovered how to hack its log file system to write to the mySociety log file directory. But everything is so coupled, it doesn’t ever seem worth it. Any Rails apps we deploy will just have to be an even more confusing mass of directories, application trees inside application trees.

  7. Debian archive

    (Shh, don’t tell anyone, but this post is really just so the bots find debian.mysociety.org, but I’m going to try and fill it with some other content so you don’t think I’m being too rude)

    Debian’s software “packaging” system provides a big database of all the open source software in the world, and makes another smaller database of all the software installed on your computer. We’re using it on our new servers, which the sites are gradually migrating to now. When you’ve got security updates, multiple machines, and complex software dependencies, you need it.

    Unfortunately, though it seems like the Debian people have packaged nearly all the software in the world, sometimes they miss things. Normally we’d just install them using the old Unix configure/make/make install. This time we’ve decided to do it properly, and make our own Debian packages. You can find them at debian.mysociety.org.

    The advantage of this is that we can find out where any file on the system came from. We can easily upgrade multiple machines, and check that they all have the same software installed. This makes it much less likely that there’ll be bugs when you go to a corner of one of the websites, and get an error because a perl module wasn’t installed.

    So far there are a few perl module .deb files in our repository, which the handy dh-make-perl builds easily from a perl module tarball. There’s also Xapian (a search engine library), which we use for quick lookups in Gaze (our gazeteer). That had already been packaged by the Xapian people, but for some reason I had to recompile it. Finally there’s one Python module, PyRTF, which makes Python modules, which I just packaged (probably badly).

    Anyway, this post is here to make sure anybody searching for python2.3-pyrtf on Google will find something…

  8. Testing SMS code

    Today I’m expanding the test harness to thoroughly test out all the SMS code. Adding in SMS creates all sorts of edge cases to check out, and indeed quite a few common cases. For example, it needs to make a pledge with both SMS and email signers, and test SMS conversion to email. And do both failure and success for all those cases. It gets even more complicated when you allow for SMS delivery reports.

    To do this, I’ve got small bits of script which pretend to be C360, our SMS aggregator. This is much cheaper than using real SMS messages every time I run the test script 🙂 The pretend scripts talk C360’s API, and fake telling our scripts as if incoming SMS messages arrive. They also store up outgoing SMS from our scripts, and check all the correct messages have been sent out to users.

    At the moment Chris is making some crazy Perl stuff to override the time() function in the SMS daemon, so when the test harness fakes the advance of time, it notices.

    If I get bored of all this, I’ll tidy up and test the new comments code Chris started the other day.

  9. Welsh time travel

    It’s nice being self employed so you can work where you like, but then it does open you up a bit to distraction… Today I went on a walk up past Llyn-Crafnant, which is near Betws-y-coed in North Wales. I managed to do a spot of work this morning, and I’m doing some more now, but the bank holiday Friday atmosphere has sucked me in a bit against my will. I’ve spent the week at a friend’s house in the country. Ben also works from home, and it has been a pleasant change having tea and excellent food cooked for me, and having walks with hills. There aren’t enough hills in Cambridge.

    At the moment I’m improving the PledgeBank test suite. It’s a perl script which makes sure the PledgeBank code is working. An artificial browser clicks through links on my local development copy of the website, fills in forms to make new pledges, and to sign them. It also gets email, so it can follow confirmation links, and make sure the right messages arrive in the right circumstances. This is not just a good way of finding bugs, but extremely reassuring. Especially as I know we’ll have to change the site a lot as people start using it seriously.

    Now, the problem with testing PledgeBank is that it depends on the passage of time. At the moment the test script gets the “pledge success” and “pledge failure” messages script to lie about which date it is, which tests that part. But that isn’t subtle enough, really the whole application including the website needs to bend in time. On my walk I finally worked out that I need to be able to override “today”, and that the best way to do this is with a function in the database which gets called whenever something wants the date.

    So, that’s what I’m implementing right now.