1. Mapumental’s Secret Sauce: A Map Overlay Rendering Technology You Might Find Interesting

    I am Duncan Parkes,  a developer for mySociety, a non-profit full of web geeks. One of the things we try to do well here is to take complicated data and turn it into really usable tools – tools which are attractive to people who aren’t web (or data) geeks.

    For some considerable time I’ve been working on Mapumental – a project that is about turning public transport timetable data into pretty, interactive maps featuring isochrones, shapes that show people where they can live if they want to have a commute of a particular time. You can play with the new version we just launched here. That particular map shows the commuting options to where the Queen lives. Slide the slider for full effect.

    There are a couple of hard problems that need solving if you want to build a service with an interactive journey times overlay like this. You need to be able to calculate a *huge* number of journeys extremely quickly, and you need to be able to make custom map layers so that it all looks nice. But what I think might be most interesting for you is the way in which the contours get rendered on top of the maps.

    It all started about three years ago, when the first version of the app – co-developed with the geniuses at Stamen – used Flash/Flex to draw contours on the maps, and to let people play with them. You can still play with a couple of versions of that technology from way back in 2007, that is, unless you’re using an iPad or iPhone, which of course don’t do Flash.

    Colour Cycling

    What was going on inside this Flash app was as follows. We needed to show the user any one of hundreds of different combinations of journey times (5 minutes, 12 minutes, 56 minutes, etc) depending on where they set the slider. Sending each one from the server as a tiled map overlay would be dead slow. Even Google – who have chosen to send new tiles each time – end up with a service which is surprisingly slow (try choosing a different time on this map).

    With some help from Stamen, we decided that the way of making it possible to show many different contours very quickly was send the client just one set of tiles, where each tile contained all the data for a variety of journey times. What does that mean? Simple: each colour in the tile represented a different number of minutes travelling on the map. So a batch of pixels that are colour X, all show places that are 15 minutes from the centre of the map.

    So, in this old Flash system, when you slide the slider along, the Flash app makes some of the coloured pixels opaque, and the others transparent. It was, in short, a form of colour cycling, familiar to lovers of 8 and 16 bit computer games.

    However, from about 2010 onwards, the march of iOS spelt the end of Flash. And that meant that we couldn’t launch a shiny new site based on this technology, as lovely as it was. We had to work out some approach that would use modern web standards instead.

    The Death of Flash Makes Life Difficult – for a while

    How do we replicate the experience of dragging a slider and seeing the map change like in the original Mapumental demo, but without Flash? One of the things that made the original Mapumental nice to use was how smooth the image changes were when you dragged the slider. Speed really matters to create that sort of organic effect that makes the demo so mesmerising.

    So as we started to tackle the question “How do we make this work in a post-Flash world?”. And the first thought was “Let’s do away with those map tiles, filled with all that journey time data!”. After all – why send any tiles to a modern browser, if it can just render nice shapes on the fly?

    So we had a go. Several goes. At first we tried rendering SVG circles around each public transport stop – but that was too slow, particularly when zoomed out. Then we tried rendering circles in Canvas, and whilst that was OK in sparsely populated places it sucked in the cities, where people would actually want to use it.

    Eventually, we decided that as wonderful and powerful as modern web rendering techniques are, if you exclude WebGL (on the grounds that so few people have it still), all the current techniques result in pages that just hammer your browser, whilst producing an experience which isn’t up to our previous Flash-based standards. To see this in action, see the wonderful Mapnificent site, developed by the super talented Stefan Wehrmeyer. He’s a great guy and a friend of mySociety, but the Javascript circle rendering just grinds, and that’s with far, far fewer data points than we have in our system. (Sorry for potentially crashing your browser there. This is in the interest of Science.)

    Back to Colour Cycling – Using Web Standards

    So, we thought, why not look again at colour cycling the pixels within pre-rendered map tiles? After all, there are some examples out there, like this waterfall, all in Javascript.

    So, I had a bit of a look at the waterfall. It seems to work by holding in memory a structure which has all the pixels which change and all the colours they should change to and when. This works beautifully for the waterfall picture, but only a limited number of the pixels in that image actually change colour, and the image is quite small. For a full screen web browser with a big map in, this didn’t seem promising, although I’d love to see someone try.

    Then I thought: browsers have always been very good at displaying images quickly – that’s sort of vital. Perhaps we could get our tile generating server to output PNG images where, as before, the colours represent travel times, but using a palette. Then by putting this in a canvas layer in the JavaScript mapping framework Leaflet, and by changing the palette of the images on the canvas as the slider is dragged, we could get our animation.

    Unfortunately, there is no way to change the palette of an image that you’ve put on the canvas. In fact, there’s no way to change the palette of an HTML img element: all you can do is assign it a new src attribute.

    But this gets back to the original problem – we don’t want to download new mapping for every different position on the time slider. We definitely can’t afford to have the client downloading a new image source for every tile whenever the slider is moved, so we had to find a way to make that src at the client end and get that into the src attribute.

    The Breakthrough – Data URIs and Base64 encoding

    So we started trying data URIs. For those of you not familiar, these allow you to put a whole object into your HTML or CSS, encoded in Base64. They’re commonly used to prevent pages having to make extra downloads for things like tiny icons.

    So I thought, “Here’s a way we can set an image src in JavaScript to something we’ve calculated, rather than something we’ve downloaded.” And this turned out to be the key insight that allows for the relatively smooth, attractive overlays you see in Mapumental today.

    My new plan was that the client, having downloaded each palette-based image, would make a Base64 encoded version of it, which it could then use to build a version with the right palette and assign this as a data URI of the tile.

    However Base64 encoding all these images in the JavaScript seemed like unnecessary processing to do there, so the final evolution of this technique was to do the Base64 encoding at the tile server end, and while we’re at it, not to bother sending over the parts of the image that we always replace at the client end.

    So in summary, what we built does this:

    1. The server calculates the journey times and renders them to palette-based tiles.
    2. It sends these to the client, encoded in Base64, and with the initial bits up to the palette and transparency chunks removed.
    3. At the client end, we have a pre-prepared array of 255 ‘starts’ of PNGs that we combine with the later parts of the ’tiles’ from the tile server to make data URIs.
    4. When you drag the slider it combines the appropriate ‘start’ of a PNG with the bulk of the tile that has been downloaded from the server, and assigns that to the src attribute of the tile.

    And that’s how the nice overlays on Mapumental work. But as so often in coding, the really interesting devil is in the detail – read on if you’re interested.

    Diving into Base64 and the PNG file format – The Gnarly Bits

    So – why are there 255 of these ‘starts’ of these PNGs, and what do I mean by a ‘start’ anyway?

    PNG files are divided up into an 8 byte signature (the same for every PNG file) and a number of chunks, where each chunk consists of 4 bytes to tell you its length, 4 bytes of its name, some data, and 4 bytes of cyclic redundancy check. In this case, what I call a ‘start’ of a PNG is the 8 byte signature, the 25 byte of the IHDR chunk, and the PLTE (palette) and tRNS (transparency) chunks. The PLTE chunk has 12 bytes of overhead and 3 bytes per colour, and the tRNS chunk has 12 bytes of overhead and 1 byte per colour.

    Base64 encoding is a way of representing binary data in text so that it can be used in places where you would normally expect text – like URIs. Without going into too much detail, it turns groups of 3 bytes of binary gumpf into 4 bytes of normal ASCII text without control characters in it, which can then be put into a URI.

    Why do we have 255 colours, rather than the maximum 256 which are available in a palette? Because we need the break between the end of the tRNS chunk and the start of the IDAT chunk in the PNG file to align with a break between groups of three bytes in the Base64 encoded image. We need the length of these starts to be a multiple of 3 bytes in the original PNG format, which translates into a multiple of 4 bytes in the Base64 encoded version, so we can cut and shut the images without corruption.

    Which just goes to show that even though web GIS technologies may feel like they are approaching a zenith of high level abstraction, there’s still some really gnarly work to be done to get the best out of current browsers.

  2. mySociety Launches FixMyStreet Platform Version 1.0 – Testers and Translators Sought

    FixMyStreet logo (square)Today sees the official launch of FixMyStreet’s open source codebase as a proper tool that we hope people will want to deploy in cities and countries around the world. It is based on FixMyStreet.com which we believe is the most usable, most mature street problem reporting tool in the world, but which is only available to British users.

    We’re shouting about this launch a bit because we need your help to make the service ever better. First, we need feedback from programmers about whether we’ve got the install process right – whether it’s as easy and clear as we want it to be. And for non-coders who want to get involved, we want to ask for help with the process of translating the site’s text into different languages.

    Over the years there have been many copies of FixMyStreet set up in many countries, often using the site’s original name, but always written by developers from scratch. We’re delighted to have inspired people, but all too often the people trying to build copies have stumbled as they realise just how hard it is to build a tool like this with the polish that users expect. We think that people everywhere would be better off if they could have a local FixMyStreet that was really usable, and really connected to the right people.

    So we’re very happy to be able to open up a codebase that has been extensively modified in the last year, to help users around the world manage easy, successful deployments. Steps we have taken include:

    • Putting the translation text into Transifex, so that non-technical translators can get started whenever they feel like it
    • Developing Amazon Machine Images so people who want to tinker can get started in the minimum possible time
    • Rewriting the entire codebase in order to make it a less confusing installation
    • Building a global version of our MapIt political boundaries web service, so you can get going without having to wrestle administrative data out of your government before you get started.

    Plus with the help of the wonderful OpenStreetMap, you can get maps without licensing hassles too.

    Calling it version 1.0 is our way of saying two things. First, that the tool still has a lot of evolution left to do, and a long way to go before it is as good as we want it to become. But more ambitiously, calling it 1.0 is also our way of saying that it’s no longer just a codebase dumped into Github. It’s a real open source project, which we plan to support, and which we hope will make a real difference in the lives of ordinary people. Check it out.

  3. Migrating to Bytemark (who rock)

    Labyrinthine circuit board lines by Karl-Ludwig G. Poggemann

    We want to give a big shout out to Bytemark, the York-based hosting company who are generously sponsoring a large portion of our server space requirements. Thanks guys: as a result, mySociety sites will be more robust and more responsive than ever before.

    Why is this gesture such a huge help to us? Well, to start with, our sites keep getting bigger! Yes, our websites are transactional, but most are also archives – so for example you can summon up FixMyStreet reports dating back to the site’s launch in 2007.

    Of course, FixMyStreet grows each day, as more reports are submitted. So does WhatDoTheyKnow, and FixMyTransport. As for TheyWorkForYou, well, MPs keep on talking, and we keep on archiving their words.

    Second, it’s incredibly important for us to have hosting from a provider who will quickly and competently respond to our requests. Bytemark have been brilliant on this front during the migration phase.

    Our sites should be fast – whether you’re looking at archived content or submitting a new report. You, as a user, should never have to think about the capacity of our servers, or their load tolerance. The pages you want to view should simply be there.

    The move over to higher capacity servers should help with all of these aims. Bytemark has two separate data centres, and our sites are now split across them.

    If you’ve been enjoying a smooth ride on our sites just recently, you might want to thank Bytemark – although of course, the strongest sign of good hosting is that you’ll never notice a thing.

    Thank you to everyone at Bytemark – your generosity is really important to us.

     

     

     Image (CC): Karl-Ludwig G. Poggeman

     

  4. Installing FixMyStreet and MapIt

    A photo of some graffiti saying "SIMPLE"

    Photo credit: duncan on Flickr

    One of the projects we’ve been working on at mySociety recently is that of making it easier for people to set up new versions of our sites in other countries.  Something we’ve heard again and again is that for many people, setting up new web applications is a frustrating process, and that they would appreciate anything that would make it easier.

    To address that, we’re pleased to announce that for both FixMyStreet and MapIt, we have created AMIs (Amazon Machine Images) with a default installation of each site:

    You can use these AMIs to create a running version of one of these sites on an Amazon EC2 instance with just a couple of clicks. If you haven’t used Amazon Web Services before, then you can get a Micro instance free for a year to try this out.  (We have previously published an AMI for Alaveteli, which helped many people to get started with setting up their own Freedom of Information sites.)

    Each AMI is created from an install script designed to be used on a clean installation of Debian squeeze or Ubuntu precise, so if you have a new server hosted elsewhere, you can use that script to similarly create a default installation of the site without being dependent on Amazon:

    In addition, we’ve launched new sites with documentation for FixMyStreet and MapIt, which will tell you how to customize those sites and import data if you’ve created a running instance using one of the above methods.

    These documentation sites also have improved instructions for completely manual installations of either site, for people who are comfortable with setting up PostgreSQL, Apache / nginx, PostGIS, etc.

    Another notable change is that we’re now supporting running FixMyStreet and MapIt on nginx, as an alternative to Apache, using FastCGI and gunicorn respectively.

    We hope that these changes make it easier than ever before to reuse our code, and set up new sites that help people fix things that matter to them.

    Photo credit: duncan

  5. Google Summer of Code Ideas List

    Note: mySociety will not be taking part in this year’s Google Summer of Code.

    We feel strongly that, if we were to participate, we would want to offer a truly useful and productive experience to our students, and this year, we simply can’t spare enough developer time to ensure that. We are a small organisation, and our time is fully scheduled.

    Our experience of participation in 2012 was really positive, and we do hope to take part again some time in the future when we have the mentoring capacity available.

    Here’s our project text from 2012:

    If you’re looking at this page, you probably already know all about Google’s fantastic Summer of Code program. In 2012, mySociety will be applying to be a mentoring organisation for students.

    We’re a distributed working team, with members of staff all across the UK (and in Kenya and Estonia as well). We haven’t got an office and most interaction is remote. IRC, email and Skype feature heavily in our working practices. If you want to work with us you can do so from the comfort of your own home.

    We’ve been thinking about the sort of things that would make good projects, and here are some ideas (we’re going to continue to add to this list).

    TheyWorkForYou

    TheyWorkForYou.com is our parliamentary monitoring site for the United Kingdom. It’s based in part on the UKParse project, which aims to produce structured versions of publicly available data from the UK parliament(s).

    We are always looking to improve the site by adding new data, so that people can find even more information about what their elected representatives are saying and doing in Parliament.  However, we don’t always have the time to do so!  There are several things that you could work on:

    • Welsh Assembly. TWFY covers the UK Parliament, Scottish Parliament and Northern Ireland Assembly, but it doesn’t cover the debates of the devolved Welsh Assembly.
    • House of Lords Grand Committee. Where some bills are considered at committee stage in the Lords.
    • Northern Ireland Assembly written answers. We cover written answers for the other parliaments, but not for Northern Ireland, so it would be great to complete the set.

    Each of these projects would involve identifying a suitable source for the data, writing a scraper (in a language of your choice) to get the data automatically each time it is updated, and parsing that data into structured XML so that it can be imported into the TheyWorkForYou database.

    Existing scrapers are in Python.  Some knowledge of PHP and MySQL might be helpful when working on TWFY, but not essential.  Here’s the github repo if you’d like to have a look around, and an issues list.

    SayIt

    Lately we’ve been thinking about the components that underpin many of our sites. We’ve got a webservice for geographical data, called MapIt; we’re building a webservice for storing and sharing the names and career histories of public figures like politicians and celebrities, called PopIt.  SayIt would be a platform for capturing speeches and public statements.

    MapIt and PopIt are Django apps, so we think it would be a good idea if SayIt was too.

    VoteIt

    Lately we’ve been thinking about the components that underpin many of our sites. We’ve got a webservice for geographical data, called MapIt; we’re building a webservice for storing and sharing the names and career histories of public figures like politicians and celebrities, called PopIt.  VoteIt would be a platform for capturing individual votes and the outcomes of votes, interacting with the existing services.

    MapIt and PopIt are Django apps, so we think it would be a good idea if VoteIt was too.

    Mobile apps

    More and more people are accessing the internet through app-enabled smartphones, and we’d like to offer app versions (for all platforms) of some of our popular websites.

    FixMyTransport is our top candidate. It’s a website that helps people to report problems with public transport. Using an app, they could do this while sitting on a broken-down train, or waiting for a bus that never seems to arrive.

    Some of our other sites could also make use of the capabilities of smartphones, such as geolocation.

    If you’re interested in learning how to build apps for iPhone or Android – even if you’re a complete beginner – this could be for you.

    WriteToThem

    WriteToThem.com helps thousands of people every month to write to politicians. Written in Perl (back end) and PHP (front end), it has been ticking over nicely for years, but we think it could do with some love.  Things you could do to help out include:

    Add parish councils – source the details of parish councils across the UK, and the contact details of their councillors, and add this data and functionality to the site.

    A full design refresh - making the site look slicker and more modern, mobile-friendly and tied in to the look and feel of our other sites.  Suitable for someone with an interest in CSS, usability and responsive design.

    Create a white label version that charities, campaign groups and so on can easily integrate into their own website, customising it to their own branding and wording.

    Here’s the github repo if you’d like to have a look around, and an issues list.

  6. Own a piece of mySociety: servers to give away

    After several years of hosting, mySociety has migrated out of Easynet’s Brick Lane Data Center, and we now live on virtual machines hosted at a top secret location in the north of England. Most of our old hardware is no longer in use. It’s redundant. We don’t need it any more. But we don’t want to throw it all in a skip, that would be terrible.

    Sooo… we’d like to find a lovely new home for it all. And this unique opportunity to own a piece of mySociety may be of interest to YOU.

    Here are the technical details of the various machines we are getting rid of – check it out for what’s currently available.

    Update: all gone!

    The Rules:

    1. The servers don’t come with disks: as part of our privacy procedures, all disks have been removed and destroyed to protect our and our users’ data.

    2. You’ll need to be able to pick them up ASAP from a central London location (WC2).

    3. You’ll need to be available to do so during office hours or very shortly afterwards (evenings or weekends aren’t possible, sorry).

    4. Note that they are large and heavy so you may well need a car.

    For more information, or even better, if you’re thinking “Sold! I’ll take the lot!”, then get in touch with Abi, our friendly office manager, on hello@mysociety.org.

    If you feel guilty about relieving a registered charity of its assets, you can even make us a donation (voluntary, but we’d be appropriately grateful).

  7. Mobile operators altering (and breaking) web content

    We had a complaint that FixMyStreet maps weren’t displaying on someone’s computer. We hadn’t had any other complaints, and we quickly narrowed it down to the fact that the person was on the internet using a tethered T-Mobile phone.

    T-Mobile (and Orange, and quite possibly others) are injecting JavaScript and altering content served over their networks. Their reason for doing this, according to their websites (T-Mobile, Orange), is to compress images and video sent to your browser, so as to speed up your browsing. Seeing it in action, they also inline some CSS and JavaScript, though not all, and remove comments from external files.

    However, their implementation breaks things. In this particular instance, the T-Mobile JavaScript comment stripper appears to be searching for “/*” and “*/” and removing everything inbetween. This might work in most cases; however in the jQuery library, we find a string containing “*/*”, and later down the file, another string containing “*/*”. T-Mobile remove everything between the things it thinks are comment markers, even though they’re actually contained within strings, causing the jQuery library to be invalid JavaScript and stopping anything using jQuery from running.

    Their decision to inline lots of the CSS also seems a bit odd – sure, on a mobile this might be quicker, but even ignoring tethering nowadays plenty of mobiles have caches too and having the CSS download once and be cached would seem better than adding weight to every page download. But I’m sure they’ve studied their decision there, and it doesn’t make any difference to the actual browsing, as opposed to the comment removal.

    To turn off this feature on your mobile phone or broadband, visit accelerator.t-mobile.co.uk or accelerator.orange.co.uk on your connection and pick the relevant option – if anyone knows of similar on other networks, do leave updates in the comments.

    From a FixMyStreet point of view – whilst FixMyStreet functions just fine without JavaScript, I had made the (perhaps incorrect) decision to put the map inside a <noscript> element, to prevent a flash of map-oddity as the JavaScript map overlaid the non-JS one. However, this meant in this circumstance the map did not work, as JavaScript was enabled, but jQuery was unable to be loaded. I haven’t decided whether to change this behaviour yet; obviously it would help people in this situation as the map would still display and function as it does for all those without JavaScript, but for those with JavaScript it does look a bit jarring as the page loads. Any suggestions on a better approach welcome :)

  8. Technical look at the new FixMyStreet maps

    This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

    Progressive enhancement

    During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

    Zooming

    We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

    OpenStreetMap

    Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

    OS StreetView tile server

    Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

    Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

    [sv]
    type=WMS
    url=path/to/mapserv.fcgi?map=os.map&
    layers=streetview
    tms_type=google
    spherical_mercator=true

    MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked :) ).

    Other benefits of OpenLayers

    As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

    Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

        if ( fixmystreet.area ) {
            var area = new OpenLayers.Layer.Vector("KML", {
                strategies: [ new OpenLayers.Strategy.Fixed() ],
                protocol: new OpenLayers.Protocol.HTTP({
                    url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                    format: new OpenLayers.Format.KML()
                })
            });
            fixmystreet.map.addLayer(area);
            area.events.register('loadend', null, function(a,b,c) {
                var bounds = area.getDataExtent();
                if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
            });
        }

    Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

  9. Helping voters in the devolved elections

    As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.

    To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).

    Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.

    Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.

    There don’t appear to be any central official lists of candidates in these elections. http://www.amnesty.org.uk/content.asp?CategoryID=11993 has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations :)

    Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15″ to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.

  10. Why I’d like mySociety to run a Masters in Public Technology

    WWII students looking at an Engine

    It is a cliché for any manager to say that they are proud of their team, and mildly nausea-inducing to listen to anyone who goes on about it too long. However, the purpose of this post is to argue that the world would benefit from a new kind of post-graduate Masters programme – something that is hard to do without  describing the virtues of the type of people who should come out of it. So please bear with me, and keep a sick bag to one hand.

    mySociety’s core development team is very, very good. But they’re not just good at turning out code. Louise Crow, for example, has a keen eye for things that will and won’t make a difference in the offline world, as well as the skills to build virtually whatever she can think of. And the exact same thing is true of the whole coding team:  Duncan, Matthew, Edmund and Dave in the current team, plus Francis, Chris and Angie before them.

    mySociety didn’t give these people their raw talent, nor the passion to be involved with projects that make a difference.   What it has given them, though, is the chance to spend a lot of time talking to each other, learning from their triumphs and their mistakes, and listening to users. This space and peer-contact made them into some of the world’s few genuine experts in the business of conceptualising and then delivering digital projects that deliver new kinds of civic and democratic benefits.

    So, why am I sitting here unashamedly blowing my colleagues trumpets like this? (I don’t have these skills, after all!) Well, in order to point out that there are quite simply far too few people like this out there.

    Too few experts

    “Too few for what?” you may well ask. Too few for any country that wants to be a really great place to live in the 21st century, is my answer.

    There is barely a not-for-profit, social enterprise or government body I can think of that wouldn’t benefit from a Duncan Parkes or a Matthew Somerville on the payroll, so long as they had the intelligence and self-discipline not to park them in the server room. Why? Because just one person with the skills, motivation and time spent learning can materially increase the amount of time that technology makes a positive contribution to almost any public or not-for-profit organisation.

    What they can do for an organistion

    Such people can tell the management which waves of technology are hype, and which bring real value, because they care more about results than this week’s craze, or a flashy presentation. They can build small or medium sized solutions to an organisation’s problems with their bare hands, because they’re software engineers. They can contract for larger IT solutions without getting ripped off or sold snake oil. And they can tell the top management of organisations how those organisations look to a digital native population, because they come from that world themselves.

    And why they don’t

    Except such experts can’t do any of these things for not-for-profit or public institutions: they can’t help because they’re not currently being employed by such bodies. There are two reasons why not, reasons which just may remind you of a chicken and an egg.

    First, such institutions don’t hire this kind of expert because they don’t know what they are missing – they’re completely outside of the known frame of reference. Before you get too snarky about dumb, insular institutions, can you honestly say you would try to phone a plumber if you had never heard that they existed? Or would you just treat the water pouring through the ceiling as normal?

    Second, these institutions don’t hire such experts because there just aren’t enough on the market: mySociety is basically the main fostering ground in UK for new ones, and we greedily keep hold of as many of our people as possible. Hands off my Dave!

    Which leads me to the proposal, a proposal to create more such experts for public and non-profit institutions, and to make me feel less guilty about mySociety hoarding the talent that does exist.

    Describing the Masters in Public Technology

    The proposal is this: there should be a new Masters level course at at least one university which would take people with the raw skill and the motivation and puts them on a path to becoming experts in the impactful use of digital technologies for social purposes. Here’s how I think it might work.

    In the first instance, the course would only be for people who could already code well (if all went well, we could develop a sister course for non-coders later on). Over the course of a single year it would teach its students a widely varied curriculum, covering the structure and activities of government, campaigns, NGOs and companies. It would involve dissecting more and less impactful digital services and campaigns, like biology students dissect frogs, looking for strengths and weaknesses. It would involve teaching the basics of social science methodologies, such as how to look for statistical significance, and good practice in privacy management. It would encourage good practice in User Experience design, and challenge people to think about how serious problems could be solved playfully. It would involve an entire module on explaining the dos and don’t of digital technology to less-literate decision makers. And most important, it would end with a ‘thesis’ that would entail  the construction of some meaningful tool, either alone or in collaboration with other students and external organisations.

    I would hope we could get great guest lecturers on a wide range of topics. My fantasy starter for 10 would include names as varied in their disciplines as Phil Gyford, David Halpern, Martha Lane Fox, Ben Goldacre, Roz Lemieux, William Perrin, Jane McGonigal, Denise Wilton, Ethan Zuckerman, as well as lots of people from in and around mySociety itself.

    What would it take?

    I don’t know the first thing about how universities go about creating new courses, so having someone who knew about that step up as a volunteer would be a brilliant start!

    Next, it would presumably take some money to make it worth the university’s time. I would like to think that there might be some big IT company that would see the good will to be gleaned from educating a new generation of socially minded, organisation-reforming technologists.

    Third, we’d actually need a university with a strong community of programmers attached, willing and ready to do something different. It wouldn’t have to be in the UK, either, necessarily.

    Then it would need a curriculum, and teaching, which I would hope mySociety could lead on, but which would doubtless best be created and taught in conjunction with real academics. We’d need some money to cover our time doing this, too.

    And finally it would need some students. But my hunch is that if we do this right, the problem will probably be fending people off with sticks.

    What next?

    I’m genuinely not sure – I hope this post sparks some debate, and I hope it provokes some people to go “Yeah, me too”. Maybe you could tell me what I should do next?