This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

Progressive enhancement

During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

Zooming

We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

OpenStreetMap

Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

OS StreetView tile server

Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

[sv]
type=WMS
url=path/to/mapserv.fcgi?map=os.map&
layers=streetview
tms_type=google
spherical_mercator=true

MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration* – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked 🙂 ).

Other benefits of OpenLayers

As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

    if ( fixmystreet.area ) {
        var area = new OpenLayers.Layer.Vector("KML", {
            strategies: [ new OpenLayers.Strategy.Fixed() ],
            protocol: new OpenLayers.Protocol.HTTP({
                url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                format: new OpenLayers.Format.KML()
            })
        });
        fixmystreet.map.addLayer(area);
        area.events.register('loadend', null, function(a,b,c) {
            var bounds = area.getDataExtent();
            if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
        });
    }

Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

*Broken link removed, Nov 2014

As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.

To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).

Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.

Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.

There don’t appear to be any central official lists of candidates in these elections. Amnesty.org.uk has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations 🙂

Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15” to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.

It is a cliché for any manager to say that they are proud of their team, and mildly nausea-inducing to listen to anyone who goes on about it too long. However, the purpose of this post is to argue that the world would benefit from a new kind of post-graduate Masters programme – something that is hard to do without  describing the virtues of the type of people who should come out of it. So please bear with me, and keep a sick bag to one hand.

mySociety’s core development team is very, very good. But they’re not just good at turning out code. Louise Crow, for example, has a keen eye for things that will and won’t make a difference in the offline world, as well as the skills to build virtually whatever she can think of. And the exact same thing is true of the whole coding team:  Duncan, Matthew, Edmund and Dave in the current team, plus Francis, Chris and Angie before them.

mySociety didn’t give these people their raw talent, nor the passion to be involved with projects that make a difference.   What it has given them, though, is the chance to spend a lot of time talking to each other, learning from their triumphs and their mistakes, and listening to users. This space and peer-contact made them into some of the world’s few genuine experts in the business of conceptualising and then delivering digital projects that deliver new kinds of civic and democratic benefits.

So, why am I sitting here unashamedly blowing my colleagues trumpets like this? (I don’t have these skills, after all!) Well, in order to point out that there are quite simply far too few people like this out there.

Too few experts

“Too few for what?” you may well ask. Too few for any country that wants to be a really great place to live in the 21st century, is my answer.

There is barely a not-for-profit, social enterprise or government body I can think of that wouldn’t benefit from a Duncan Parkes or a Matthew Somerville on the payroll, so long as they had the intelligence and self-discipline not to park them in the server room. Why? Because just one person with the skills, motivation and time spent learning can materially increase the amount of time that technology makes a positive contribution to almost any public or not-for-profit organisation.

What they can do for an organistion

Such people can tell the management which waves of technology are hype, and which bring real value, because they care more about results than this week’s craze, or a flashy presentation. They can build small or medium sized solutions to an organisation’s problems with their bare hands, because they’re software engineers. They can contract for larger IT solutions without getting ripped off or sold snake oil. And they can tell the top management of organisations how those organisations look to a digital native population, because they come from that world themselves.

And why they don’t

Except such experts can’t do any of these things for not-for-profit or public institutions: they can’t help because they’re not currently being employed by such bodies. There are two reasons why not, reasons which just may remind you of a chicken and an egg.

First, such institutions don’t hire this kind of expert because they don’t know what they are missing – they’re completely outside of the known frame of reference. Before you get too snarky about dumb, insular institutions, can you honestly say you would try to phone a plumber if you had never heard that they existed? Or would you just treat the water pouring through the ceiling as normal?

Second, these institutions don’t hire such experts because there just aren’t enough on the market: mySociety is basically the main fostering ground in UK for new ones, and we greedily keep hold of as many of our people as possible. Hands off my Dave!

Which leads me to the proposal, a proposal to create more such experts for public and non-profit institutions, and to make me feel less guilty about mySociety hoarding the talent that does exist.

Describing the Masters in Public Technology

The proposal is this: there should be a new Masters level course at at least one university which would take people with the raw skill and the motivation and puts them on a path to becoming experts in the impactful use of digital technologies for social purposes. Here’s how I think it might work.

In the first instance, the course would only be for people who could already code well (if all went well, we could develop a sister course for non-coders later on). Over the course of a single year it would teach its students a widely varied curriculum, covering the structure and activities of government, campaigns, NGOs and companies. It would involve dissecting more and less impactful digital services and campaigns, like biology students dissect frogs, looking for strengths and weaknesses. It would involve teaching the basics of social science methodologies, such as how to look for statistical significance, and good practice in privacy management. It would encourage good practice in User Experience design, and challenge people to think about how serious problems could be solved playfully. It would involve an entire module on explaining the dos and don’t of digital technology to less-literate decision makers. And most important, it would end with a ‘thesis’ that would entail  the construction of some meaningful tool, either alone or in collaboration with other students and external organisations.

I would hope we could get great guest lecturers on a wide range of topics. My fantasy starter for 10 would include names as varied in their disciplines as Phil Gyford, David Halpern, Martha Lane Fox, Ben Goldacre, Roz Lemieux, William Perrin, Jane McGonigal, Denise Wilton, Ethan Zuckerman, as well as lots of people from in and around mySociety itself.

What would it take?

I don’t know the first thing about how universities go about creating new courses, so having someone who knew about that step up as a volunteer would be a brilliant start!

Next, it would presumably take some money to make it worth the university’s time. I would like to think that there might be some big IT company that would see the good will to be gleaned from educating a new generation of socially minded, organisation-reforming technologists.

Third, we’d actually need a university with a strong community of programmers attached, willing and ready to do something different. It wouldn’t have to be in the UK, either, necessarily.

Then it would need a curriculum, and teaching, which I would hope mySociety could lead on, but which would doubtless best be created and taught in conjunction with real academics. We’d need some money to cover our time doing this, too.

And finally it would need some students. But my hunch is that if we do this right, the problem will probably be fending people off with sticks.

What next?

I’m genuinely not sure – I hope this post sparks some debate, and I hope it provokes some people to go “Yeah, me too”. Maybe you could tell me what I should do next?

As you might know, we’ve currently got an open-call for new developers, we’re hiring quite a bit in the next six months.

Thus far our list of people interested in the job contains no women’s names at all – zip, zero, zilch – despite us having taken soundings on how to get a more diverse sample of applicants.

I’m really, really not OK with this. I understand the gender imbalance in tech as well as anyone, but I interpret this as ‘mySociety hasn’t reached out well enough’, not ‘blame the women for not applying’.

So my question to you, the world at large, is this: what can we do right now, or this week anyway, to get some women’s names on this list before we start to vet the CVs?

Applications are still very definitely open, so anyone – male, female or other – who’d like to apply should see the original blog post for how to go about it.

This vacancy is now filled.

How would you like to be a coder in an organisation that is as determined to make a difference in the world as it is to be a truly high quality, engineer-led software team?

mySociety is that organisation. We’re a project of a registered charity, currently running award-winning civic and democratic websites like TheyWorkForYou.com and FixMyStreet.com, and we’re looking to grow our already-celebrated development team by several new members over the next six months.

We’re looking for people with at least two years experience (professional or keen amateur) in at least one of Python, Ruby, Perl, PHP, C++, Javascript or Adobe Flex, and who have ambitions to learn more languages in the future.

We’re looking for developers willing to commit to full or mostly-full time positions (no freelancers, sorry) and who are up for a career change that will see them stay with us for a little while. You’ll get to work with volunteers, mix commercial and charitable projects, and travel far and wide. Plus, you can work from wherever you live (in the UK), and we pay salaries from £28k to £50k depending on skills.

Most of all, we’re looking for coders who look at the services we have built so far and think “I wish I’d been on that project”. Projects you’ll likely be working on over the next few months include (but are not limited to):

• A/B testing and conversion tracking of our charitable sites
• Commercial spinoffs from FixMyStreet
• Mapumental
• Enhancements to TheyWorkForYou and WhatDoTheyKnow
• Commercial development for clients

And if you’ve any questions, please post them in the comments below so we can share the answers.

We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

• Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
• As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
• From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
• Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
• As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
• Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

On Twitter about 15 minutes ago, @greenerleith asked: “Has anyone worked out how to display the most recent #fixmystreet reports on a local map widget that can be embedded? #hyperlocal”

Like this? 🙂

It’s very simple to do:

1. Go to FixMyStreet, and locate any RSS feed of the latest reports you want (for the above map, I used Edinburgh Waverley’s postcode of EH1 1BB; you could have used reports to a particular council, or ward, using the Local alerts section). Copy the URL of the RSS feed.
2. Go to Google Maps, paste the RSS feed URL into its search box, and click Search Maps.
3. Click the “Link” link to the top right of the map, and copy the “Paste HTML to embed in website” code.
4. Paste that code into your blog post, sidebar, or wherever (you can alter the code to change its size etc.).
5. Done. 🙂

The latest reports from FixMyStreet, superimposed on a Google Map, embedded in your blog. Hope that’s helpful.

When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.

Peter Collingbourne got fed up with this, and luckily for us, he can code too. He forked our source code repository, and made a nice patch in his own copy of it.

He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.

Now if you go to an Outlook attachment on WhatDoTheyKnow,such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).

Thanks Peter!

If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.

We are very happy to announce that Duncan Parkes has joined mySociety, bringing our team of full time core developers up to four.

Duncan is the incredibly prolific author of screen scrapers for the lovely PlanningAlerts.com which he runs with Richard Pope.

He also has a PhD in Mathematics, which I expect you’ll want to read all of here, and is an editor of Open Source programming books with APress. During the vetting process he listed one of the passions of his life as being ‘Unit Testing‘, which, combined with his love of postbox crowdsourcing, made picking him more or less a no brainer.

In the short run we’ve let him loose, under the tutelage of Francis Irving, on the scaling challenges presented by Mapumental – I can’t wait to see what comes out of it.

I’ve been doing lots of research around “cloud computing” recently, so we can change how Mapumental works and take it out of private beta.

One thing that’s struck me is that there doesn’t seem to be a proper, industry standard name to distinguish what to me are two fundamentally different sorts of “cloud computing”. I’m focusing here entirely on cloud services for programmers (let’s leave what it means to end users or businesses for another day).

Here are my own names and descriptions of them:

1) Cloud hardware server provision (Cloud HSP)
Low level APIs for making and destroying (virtual) servers, and loading machine images onto them. e.g. Amazon Elastic Compute Cloud, Rackspace Cloud Servers, Eucalyptus’s EC2 bits. Basically, what Eucalyptus v 1.5 can do and what libcloud should do. (By analogy, this is the assembly language of cloud computing)

2) Cloud developer service provision (Cloud DSP) A service that a developer accesses with one name and a simple API, and behind the scenes it scales for him, automatically. e.g. Amazon Queue Service, Rackspace Cloud Files. (By analogy, this layer is the C programming language of cloud computing)

[as an aside, Google AppEngine is an interesting one. It is definitely in the Cloud DSP category, but I think it is larger than that – it is a whole set of APIs all in that category. Something like Google DataStore is a single Cloud DSP, albeit one apparently only accessible within AppEngine apps]

It’s possible to use a Cloud HSP (assembly language), along with a bunch of your own software or open source software, to build new Cloud DSPs (C code). Right now this is pretty hard – even quite well known open source distributed datasbases like CouchDB still need scripting to even make them replicate. The code that makes and destroys servers and gives the service one name, needs manually stringing with quite new bits of wire (things like scalr and Wackamole).

For this reason, I’m reluctant for mySociety to get into the “making our own Cloud DSP out of Cloud HSP” game. It feels to me like a suck of time, and like we wouldn’t be able to guarantee without lots of careful and expensive testing that it would scale. I’m more tempted to use the commercial Cloud DSP services where possible, even though they are proprietary. But use them via our own abstraction layer, so we can change as we need to. Of course, we have some C++ code (the public transport route finder), so will have to use the Cloud HSP API to get that going, perhaps with Amazon’s Auto Scaling. But it can jolly well use AQS and S3 to talk to other services.

So, what do you think about the names Cloud HSP/DSP? Are there already existing names for the distinction that I’m making? Is it a useful distinction for you? Can you think of better names?