1. New vacancies at mySociety

    Now Is A Good Time To Explore by Minivan Ninja

    What’s on your Christmas wishlist? If ‘a meaningful job’ is high on the list, then we’ve got important news for you. We’re looking for talented, passionate and diversely skilled people to join our team.

    In 2013, we’ll be pushing out internationally, improving our core UK sites and doing more commercial business. And we need some more lovely, dilligent people to help us.

    All the details are on our jobs page. There’s plenty of time to get your application in, so why not give it some thought over the mince pies?

    Not quite for you? Then please tell your nicest friends!

    Photo by Minivan Ninja (CC)

  2. Looking back: our experience of the Google Summer of Code

    Summer may seem like a long time ago, but despite the cold outside, we’ve been looking back over our participation in Google’s Summer of Code project. It’s almost enough to warm us up!

    This post is an attempt to record the process from our point of view. We hope it will be useful for other organisations considering participating next year, and for students who want to know more about how the scheme works.

    What is Google Summer of Code?

    It’s a programme sponsored by Google’s philanthropic arm, giving students the chance to experience real-life coding on open source software.

    The scheme is open to students all over the world, who are then paired up with open source organisations like us. The students gain paid work experience and mentoring; the organisations gain willing workers and some fresh new perspectives; the world gains some more open source code to use or develop further.

    Everyone’s a winner, basically.

    The beginnings

    2012 was our first year on the programme: once we had been accepted on the scheme, we were given two student slots – the maximum allowed for a first-time organisation.

    Given mySociety’s wide suite of codebases, there were several projects that could have benefited. We listed all our ideas, and let people apply for the ones they found appealing.

    Goodness, there were a lot of applicants! It was very heartening to discover that there is such an enthusiastic community of young coders all around the world – even if it did take us a long time to sift through them all and make our choices.

    You might remember our post back in May, when we announced that we’d made our choices. We were delighted to get working with Dominik from Germany and Chetan from India.

    The project

    As things turned out, our students ended up working on a project that wasn’t even on our original list: PopIt, our super-easy ‘people and positions’ software.

    That’s because once we spoke to our chosen students, we realised they had the skills that could really help us forge ahead with this project – and once we discussed it with them, they were keen. So PopIt it was.

    Logistics

    Germany and India are a bit of a commute away, but fortunately development work can be managed remotely. We know this particularly well at mySociety: our core team work from home and are scattered across the UK.

    The only difference here was the 6+ hour time difference between us and India: it was important to be rigorous about checking in at times when Chetan would be awake!

    We communicated via IRC (instant chat), email, and occasionally Skype, and it all worked well.

    Edmund, the team member chosen to be mentor, broke the required tasks down into big pieces so that the students would have realistic work units of several days each.

    What was achieved

    PopIt is primarily a tool for helping people create and run parliamentary monitoring websites (like TheyWorkForYou) with minimal coding knowledge/effort, though we anticipate that it will have many other uses too.

    Our students spent the first half of the summer learning and improving the PopIt codebase. Once they were confident in it, they created their own sites using PopIt as a datasource to test the API, and, hopefully, create a valuable reference resource for the community.

    Dominik added a migration tool to PopIt, which lets you upload data as a CSV. This means that you can start a site with a database of names, positions and dates at its heart – within seconds.

    His test site was a professors’ database (the code is here). Dom also wrote some helpful posts on the dev blog like this one.

    Chetan created an image proxy that lets us serve images in a smart way that makes sense for APIs. His test site was for Indian representatives (here’s the code).

    Neither site is being maintained now, which just confirms that it is harder to run a site than to start it. This is not a failing, though. The creation of these sites, along with Chetan and Dom’s feedback, helped us understand where improvements needed to be made. In the course of one summer, PopIt became much more mature.

    Looking back on the Summer of Code

    Edmund attended a follow-up ‘mentors’ summit’ at the Googleplex in California – he found it very helpful to compare notes with other organisations and find out what had worked best for them all, and he made some good contacts too.

    Assuming we get the chance again, would we participate in 2013? Our experience was very positive, but as yet we are undecided, purely because of the fluid nature of our workflow: we don’t yet know whether time and resources will permit.

    Obviously, we have enjoyed great benefits from the scheme, but that has depended on quite a bit of input from our side, and we need to be sure that we can ensure that happens again.

    Edmund has compiled a list of advice, from the practical (ask students to treat the placement like a full-time job; test coding skills before acceptance) to the desirable (a weekly blog post from participants; make sure you over-estimate the time you’ll spend mentoring). If you’re thinking of participating next year, he’d be happy to pass on his tips for ensuring that you, and your assigned students, get the best out of the Google Summer of Code. Just drop him a line.

  3. mySociety Christmas Pub Meet

    Inside the Prince Edward, Bayswater, London W2 by Kake Pugh

    We hope friends, supporters and indeed anyone who fancies it will join us for a festive drink at the Prince Edward pub in London’s Notting Hill.

    When? 7.30pm onwards, Tuesday 11th December.

    Where? 73, Prince’s Square, W2 4NY. Google Map

    Who? Everyone’s welcome.

    Why? Come and chat about any of our projects, becoming a volunteer, new ideas you have – or just enjoy a drink.

    You can add your name, and see who else is planning on coming, on our Lanyrd page.

     

    Image (CC): Kake Pugh

  4. Extracting Boundaries from OpenStreetMap – Part 2

     

    Hadrian's Wall by Joe Dunckley

    This is the second part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML files for every useful administrative and political boundary in OpenStreetMap.

    The previous post on this subject described how to take the ID for a particular relation or way that represents a boundary in OpenStreetMap, and generate a KML file for it. That’s much of the work that we needed to create MapIt Global, but there are a few more significant steps that were required:

    Efficiently extracting boundaries en masse

    The code I previously described for extracting a boundary from OpenStreetMap used a public Overpass API server.  This is fine for occasional boundaries, but, given that there are hundreds of thousands of boundaries in OSM, ideally we don’t want to be hitting a public server that many times – it puts a large load on that server, and is extremely slow. As an alternative, I tried parsing the OSM planet file with a SAX-based parser, but this also turned out to be very slow – multiple passes of the file were required to pick out the required nodes, ways and elements, and keeping the memory requirements down to something reasonable was tricky. (Using the PBF format would have helped a bit, but presented the same algorithmic problems.) Eventually, I decided that a better approach was simply to set up a local Overpass API server, and to query that.  This is a great improvement – it allows very fast lookups of the data we need, and the query language is flexible enough to be able to retrieve huge sets of relations and ways that match the tags we’re interested in.  It also means we would no longer have problems if connectivity to the remote server went down.

    Another question that arose when scaling up the boundary extraction was, “Which set of tags we should consider as boundaries of interest?” On the first import, we considered any relation or way with the tag boundary=”adminstrative” and where the admin_level tag was one of “2″, “3″, “4″, … “11″.  At the time, there were about 225,000 such elements that represented closed boundaries. Afterwards, it was pointed out to me that we should also include elements with the tag boundary=”political”, which includes parliamentary constituencies in the UK, for example.  For later import purposes, I gave each of these boundary types a 3-letter code in MapIt, which are as follows:

    • O02 (boundary="administrative", admin_level="2")
    • O03 (boundary="administrative", admin_level="3")
    • [...]
    • O11 (boundary="administrative", admin_level="11")
    • OLC (boundary="political", political_division="linguistic_community")
    • OIC (boundary="political", political_division="insular_council")
    • OEC (boundary="political", political_division="euro_const")
    • OCA (boundary="political", political_division="canton")
    • OCL (boundary="political", political_division="circonscription_législative")
    • OPC (boundary="political", political_division="parl_const")
    • OCD (boundary="political", political_division="county_division")
    • OWA (boundary="political", political_division="ward")

     

    Importing Boundaries into MapIt

    The next step in building our service was to take the 236,000 KML files generated by the previous step and import them into MapIt.

    The code that creates the KML file for an element includes all its OpenStreetMap tags in the <ExtendedData/> section.  On importing the KML into MapIt, there are only a few of those tags that we’re interested in – chiefly those that describe the alternative names of the area.  We have to pick a canonical name for the boundary, which is currently done by taking the first of name, name:en and place_name that exists. If none of those exist, the area is given a default name of the form “Unknown name for (way|relation) with ID (element ID)”. There are also tags for the name of a country in different languages, which we also import into the database so that localized versions of the name of the boundaries will be available through MapIt with their ISO 639 language code.

    Another tricky consideration when importing the data is how to deal with boundaries that have changed or disappeared since the previous import. MapIt has a concept of generations, so we could perfectly preserve the boundaries from the previous import as an earlier generation. This would certainly be desirable in one respect, since if someone is depending on the service they should be able to pick a generation that they have tested their application against, and then not have to worry about a boundary disappearing on the next import. However, with quarterly imports the size of our database would grow quite dramatically: I found that approximately 50% of the boundaries in MapIt Global had changed over the 5 months since the initial import. Our proposed compromise solution is that we will only keep the polygons associated with areas in the two most recent generations, and notify any known users of the service when a new generation is available for them to test and subsequently migrate to.

    For reference, you can see the script that extracts boundaries and generates the KML files and the Django admin command for importing these files into MapIt.

    The end result: MapIt Global

    The aim of all this work was to create our now-launched web service, MapIt Global. As far as we know, this is the only API that allows you to take the latitude and longitude of any point on the globe, and look up  the administrative and political boundaries in OpenStreetMap that surround that point. We hope it’s of use to anyone trying to build services that need to look up administrative boundaries – please let us know!

    Photo credit: Hadrian’s Wall by Joe Dunckley,

  5. Most viewed requests – 19-25 November 2012

    One WhatDoTheyKnow statistic that we often quote, is that for each request written on the site, around 30 requests are read.  I’ve been recently taking a look at our web statistics, and thought you mind find it useful or interesting to see which are our most read requests.

    So, here are the top 10 for last week.

    There’s a definite theme, most readers are looking for information on the new Universal Jobmatch service which went live last week, without much mainstream media coverage, so people have a lot of questions which are being discussed on a number of DWP-watching blogs, Facebook & Twitter.

    1. Medicine A100 Admissions Statistics (Imperial College London)   – 629 unique views. Find out detailed statistics on who applied for Medicine at ICL – seemingly the most popular university out of these similar requests.
    2. Universal Jobmatch (DWP) – 702 views. This request asked for “leaflets or training info or guides given to jobcentre staff or customers to explain Universal Jobmatch”. A summary of training guides was provided, but no detail – one for future follow-up FOI requests?
    3. Universal Jobmatch and the Government Gateway (DWP) – 508 views. Detailed information was asked on the procedures and guidance issued to Jobcentre Plus staff relating to Jobmatch. The request is still in progress.
    4. Universal Jobmatch is Mandatory (DWP) – 446 views. The requester asked whether Universal Jobmatch will be mandatory for anyone on Jobseekers Allowance. The DWP refused the request, claiming Section 35 (formulation of government policy) and Section 42 (legal professional privilege). They said that the policy was still being developed, and (or?) that information was legally privileged. An internal review has been requested.
    5. Medicine A100 Statistics (Imperial College London) – 427 views. This asks for more recent 2011 & 2012 data, following on from #1. There’s a thestudentroom.co.uk forum poston this which is tracking all the medical schools’ statistics.
    6. Is signing up to Universal Jobmatch compulsory to claim JSA? (DWP) – 421 views. We don’t know as there’s no answer just yet.
    7. List of research used by the Secretary of State supporting the Academisation of state schools (Department for Education) – 406 views. Most traffic for this request came from this tweet from @alanmills405. This request is in still progress, so click its “Follow” button to find out the DfE’s response.
    8. Location of every post box that the Royal Mail Group operates (Royal Mail) – 324 views. This is one of our all-time top 3 most viewed requests, with around 22,000 visitors each year looking at it. Data from this request has been incorporated into a number of different apps and websites – follow the various links added to the request. I wonder why Royal Mail hasn’t done more to open up this data…
    9. Universal Jobmatch and Monster Worldwide emails (DWP) – 292 views. Apparently if you’re in a “junior” position, and not public facing, then your work emails are “personal data”, and exempt under FOI. Doesn’t sound right to me.
    10. New appointments opening time (UK Border Agency) – 284 views. Apparently, sometime after midnight seems to be the best time to try to bag your appointment (although the UKBA didn’t provide this information). Disappointment likely though – we get a lot of correspondence from people frustrated with the UK Border Agency’s slow pace at casework, poor communication and lack of detailed guidance on their website.

    If you’re interested in keeping up with any of these requests, especially if there’s no response yet, then click the “Follow” button at the top of each request page to be alerted when an update is received, either by email or RSS.

  6. mySociety is inviting people to become trustees of its parent charity

    Summary

    mySociety is looking to recruit new trustees to help us, as we transition from being a small digital non-profit into a mature international social enterprise.

    If you are interested in helping to guide one of the earliest ‘digitally native’ charities through to its next stage of growth, this may be an opportunity of interest to you.

    Our mission is to discover how technology can (or cannot) help make people more powerful. As a team and a community we are driven by a desire to build tools that help people exert a little control over the world around them – especially people who have never tried to do so, and who don’t think they would succeed if they tried.

    If that is a goal that motivates you in the way it motivates our staff and volunteers, we should have a conversation.

    What is a trustee?

    Trustees oversee charities to ensure that they are well-run, solvent, operating within the law, and making the right strategic decisions. These are unpaid roles with an ultimate legal responsibility for the charity. To understand more about what being a trustee means legally, please see this introduction from the charities commission.

    mySociety is the public brand of the registered charity UK Citizens Online Democracy: the positions we are advertising for today are trustees of that charity.

    As a UKCOD trustee, you will advise on the organisation’s priorities, help with the approval of budgets and staffing, and assess legal matters. In concrete terms, that means attending meetings in London every three months, and dealing with the associated emails and documents – a commitment of about six hours per month.

    History

    Between 2003 and the present day, mySociety has built and grown a series of British democratic and civic websites and apps, including FixMyStreet.com, WhatDoTheyKnow.com, WriteToThem.com and TheyWorkForYou.com.

    In the last two years our organisation has experienced a great deal of growth, with our staff tripling in number (to nearly 20) and our objectives becoming ever more international. This is largely due to major investments by groups like the Omidyar Network and the Open Society Foundation, as well as an increase in our commercial software and consultancy services, which generate about half our revenues.

    Future Challenges

    We have numerous challenges to face as we approach our 10th birthday, in late 2013.

    • How do we balance the need to maintain and improve the quality of UK services, whilst working increasingly in other countries?
    • How do we ensure that the people trying to build copies of the services we run in other countries succeed in adapting them to very different environments?
    • How do we become a successful, substantial social enterprise that can drive quality improvement and higher ethical standards across the entire government IT sector?
    • How do we do all this whilst ensuring that the high standards of talent – and niceness – of the people within our organisation do not slip?

    What we’re looking for in trustees

    mySociety is still small enough that it often needs very practical support from the trustees, such as opinions on legal matters. This means we need trustees equally comfortable with big questions and small ones.

    We are interested in acquiring trustees from a range of different occupational backgrounds. If you have skills in any of the disciplines listed below, you could really help us.

    • Marketing
    • Campaigning and community organising
    • International development
    • Human resources
    • Legal
    • Governance structures
    • Digital product development
    • Finance
    • Local Government
    • Advertising
    • Quality assurance

    We also welcome applications from mySociety volunteers, whether past or present.

    Timelines

    We will be happy to meet people and arrange phonecalls for no-commitment discussions up to 21st December 2012. Please contact abi@mysociety.org if you would like to book in a conversation with someone who could tell you a bit more about the role.

    If you would actually like to apply, please send a CV and covering letter, explaining why becoming a trustee is of interest, to abi@mysociety.org by 5th January 2012 latest.

    We will conduct interviews on Monday and Tuesday 21st and 22nd January – we can arrange them in the evenings if that is necessary.

    We will notify applicants of our appointment decisions on Monday 28th January.

  7. Dear Ninja Campaigner Geek: Why I work on non-partisan tech, and why I encourage you to take a look

    The last few weeks since the US election have seen an explosion in articles and blog posts about how Obama’s tech team pulled out the stops in their race against the Republicans. It’s been an exciting time to learn about the new techniques dreamed up, and the old ones put to the test.

    For those of us who develop non-partisan services to help people report broken street lights or make Freedom of Information Requests, such stories certainly seem unimaginably glamorous: I don’t think any of my colleagues will ever get hugged by Barack Obama!

    But it has also been an interesting time to reflect on the difference between choosing to use tech skills to win a particular fight, versus  trying to improve the workings of the democratic system, or helping people to self-organise and take some control of their own lives.

    At one level there’s no competition at all: the partisan tech community is big and economically healthy. It raises vast amounts of cold hard cash through credit card payments (taking a cut to pay its own bills) and produces squillons of donors, signers, visitors, tweeters, video watchers and so on. The non-partisan tech community is much smaller – has fewer sustainable organisations, and with the exception of some big online petitions, doesn’t get the same sort of traffic spikes. By these metrics there’s absolutely no doubt which use of tech is the most important: the partisan kind where technology is used to beat your opponent, whether they are a political candidate, a policy, company, or an idea.

    But I am still filled with an excitement about the prospects for non-partisan technologies that I can’t muster for even the coolest uses of randomized control trial-driven political messaging. The reason why all comes down to the fact that major partisan digital campaigns change the world, but they don’t do it in the way that services like  eBay, TripAdvisor and Match.com do.

    What all these sites have in common – helping people sell stuff they own, find a hotel, or a life partner – is that they represent a positive change in the lives of millions of people that is not directly opposed by a counter-shift.  These sites have improved the experience of selling stuff, finding hotels and finding life partners in ways that don’t attract equal and opposite forces, driven by similar technologies.

    This is different from the case of campaigning tech: here a huge mailing list is pitted against another even huger mailing list. Epic fund-raising tools are pitched against even more epic fund-raising tools. Orca vs Narwhal. Right now, in US politics, the Democrats have a clear edge over the technology lined up against them, and I totally understand why that must feel amazing to be part of. But everything you build in this field always attracts people trying to undo your work by directly opposing it. There is something inate to the nature of partisanship which means that one camp using a technology will ultimately attract counter-usage by an opposing camp.

    This automatic-counterweighting doesn’t happen with services that shift whole sectors – like TripAdvisor did. In the hotel-finding world, the customer has been made stronger, the hotel sector weaker, and the net simply doesn’t provide tools to the hotel industry to counter what TripAdvisor does.

    It is this model – the model of scaleable, popular technology platforms that help people to live their lives better – that I aspire to bring to the civic, democratic and community spheres in my work. Neither I nor mySociety has yet come up with anything even remotely on the scale of a TripAdvisor, but there remains the tantalising possibility that someone might manage it – a huge, scaleable app of meaningful positive impact on democratic, civic or governance systems*. Our sites are probably about as big as it gets so far, and that’s not big enough by far.

    If someone does manage to find and deliver the dream – some sort of hugely scalable, impactful non-partisan civic or democratic app & website, it is unlikely that the net will instantly throw up an equal and opposite counterweight. There is a real possibility that the whole experience of being a citizen, the whole task of trying to govern a country well will be given a shot in the arm that won’t go away as soon as someone figures out how to oppose it. I’m not talking Utopia, I’m just talking better. But what motivates me is that it could be better for good, not just until the Other Team matches your skills.

    And that – in rather more words than I meant to use – is why I am still excited by non-partisan tech, and why I really hope that some of the awesome technologists who worked in the political campaigns of 2012 get involved in our scene.

    I’m on Twitter if anyone wants to talk about this more.

    *  You can certainly make an argument that Twitter and Facebook sort-of represent non-partisan democracy platforms that have scaled.  But some people disagree vehemently, and I don’t want to get into that here.

     

     

     

  8. Mapumental Property – extra insight for househunters

    If you’re searching for a new home, give Mapumental Property a try. lt narrows property results down, only showing you houses that fall within a decent commute time from the places you visit regularly – like work, school, or the shops. Here, have a go – it’s fun.

    Mapumental Property screenshot

    Irritation is the mother of invention

    Several years ago,  some of our colleagues were looking for a house to rent.

    They weren’t set on a particular town. There were two important factors: that it was within a reasonable commute from central London, where they frequently attended meetings; and that the rent was affordable.

    Faced with these requirements, most of us would sift through property sites and cross-reference the listings manually with public transport information. It’s rather time-consuming, and slightly irritating, but hey-ho, it has to be done.

    But mySociety is in the business of building useful web tools, so when something irritates us like this, we look to see if we can solve the problem through the magic of code. In a stroke of good timing, it was at just around this time that the Department for Transport approached us to ask us to work with their public transport data – and Mapumental was conceived.

    Early days

    The key was to combine Ordnance Survey postcodes with the DfT’s data about journey times, NPTDR (National Public Transport Data Repository). This data set takes a  ‘snapshot’  of every public transport journey in Great Britain for a selected week in October each year.

    Sounds simple? The process was not without its challenges. Prime among them was the problem of displaying map tiles, plus the vast quantities of transport data, within a reasonable amount of time, no matter which postcode or zoom level the user chose. As we know, a  ‘reasonable amount of time’  for a page to load is a metric which is forever shrinking.

    By 2006, we had created Mapumental’s first iteration. Users could input a postcode and see all areas of the country that could be reached by public transport, divided into coloured travel-time bands. In 2009, Francis Irving, the mySociety coder behind Mapumental’s early endeavours, explained the technology he’d used. It was Flash-dependent, and a few years later, developer Duncan wrote about some of the technical hurdles he overcame replacing the Flash elements, in view of the rise of the iPhone, which famously doesn’t  ‘do’  Flash.

    Hoorah! Now our colleagues could type in a central London postcode and see everywhere that fell within a 40-minute journey from there. It wasn’t long before we added median house price data, too.

    Beauty is in the eye of the crowd

    We even added a ‘scenicness’ rating: if the beauty of your surroundings was important to you, you could rule out anywhere below a certain level of attractiveness.

    How did we assess how scenic every area in the UK is? By crowdsourcing the information – our ScenicOrNot website displays a random photograph from every square mile of the British isles, inviting people to rate them. It is surprisingly compulsive.

    ScenicOrNot from mySociety

    A showcase tool

    Mapumental may have been born from our own needs, but we knew from the beginning that it would have wider applications. It has always been the sort of project that got people excited, once they saw it in action.

    We wanted to show how elegantly Mapumental can handle all kinds of data, starting with houses for sale and rent – so we developed Mapumental Property. It’s not intended as a serious competitor to the giant property websites out there. Rather, it’s an all-singing, all-dancing demonstration of Mapumental’s strengths.

    In this case, the data is from the property website Zoopla, and you can narrow it down to show rental or sales property within your chosen price bands and commute distances. You can even add multiple destination points, so that households of two or more people can find their optimum location.

    Versatility rules

    But Mapumental is not just about property: swap out that Zoopla layer, and you could put in anything else you can imagine – hospital locations, supermarkets, schools, job vacancies… you name it.

    The beauty of Mapumental is that now we’ve done the really hard part, incorporating new data layers is relatively simple. Recent work for the Fire Protection Association and the Welsh Government, among others, has shown its versatility.

    Now how about you?

    We believe that Mapumental’s possibilities are pretty much endless. Have you got an unloved, difficult-to-navigate dataset that Mapumental could breathe new life into? Or would your stakeholders benefit from being able to see your data displayed on a map? Let us know.

  9. Mapumental’s Secret Sauce: A Map Overlay Rendering Technology You Might Find Interesting

    I am Duncan Parkes,  a developer for mySociety, a non-profit full of web geeks. One of the things we try to do well here is to take complicated data and turn it into really usable tools – tools which are attractive to people who aren’t web (or data) geeks.

    For some considerable time I’ve been working on Mapumental – a project that is about turning public transport timetable data into pretty, interactive maps featuring isochrones, shapes that show people where they can live if they want to have a commute of a particular time. You can play with the new version we just launched here. That particular map shows the commuting options to where the Queen lives. Slide the slider for full effect.

    There are a couple of hard problems that need solving if you want to build a service with an interactive journey times overlay like this. You need to be able to calculate a *huge* number of journeys extremely quickly, and you need to be able to make custom map layers so that it all looks nice. But what I think might be most interesting for you is the way in which the contours get rendered on top of the maps.

    It all started about three years ago, when the first version of the app – co-developed with the geniuses at Stamen – used Flash/Flex to draw contours on the maps, and to let people play with them. You can still play with a couple of versions of that technology from way back in 2007, that is, unless you’re using an iPad or iPhone, which of course don’t do Flash.

    Colour Cycling

    What was going on inside this Flash app was as follows. We needed to show the user any one of hundreds of different combinations of journey times (5 minutes, 12 minutes, 56 minutes, etc) depending on where they set the slider. Sending each one from the server as a tiled map overlay would be dead slow. Even Google – who have chosen to send new tiles each time – end up with a service which is surprisingly slow (try choosing a different time on this map).

    With some help from Stamen, we decided that the way of making it possible to show many different contours very quickly was send the client just one set of tiles, where each tile contained all the data for a variety of journey times. What does that mean? Simple: each colour in the tile represented a different number of minutes travelling on the map. So a batch of pixels that are colour X, all show places that are 15 minutes from the centre of the map.

    So, in this old Flash system, when you slide the slider along, the Flash app makes some of the coloured pixels opaque, and the others transparent. It was, in short, a form of colour cycling, familiar to lovers of 8 and 16 bit computer games.

    However, from about 2010 onwards, the march of iOS spelt the end of Flash. And that meant that we couldn’t launch a shiny new site based on this technology, as lovely as it was. We had to work out some approach that would use modern web standards instead.

    The Death of Flash Makes Life Difficult – for a while

    How do we replicate the experience of dragging a slider and seeing the map change like in the original Mapumental demo, but without Flash? One of the things that made the original Mapumental nice to use was how smooth the image changes were when you dragged the slider. Speed really matters to create that sort of organic effect that makes the demo so mesmerising.

    So as we started to tackle the question “How do we make this work in a post-Flash world?”. And the first thought was “Let’s do away with those map tiles, filled with all that journey time data!”. After all – why send any tiles to a modern browser, if it can just render nice shapes on the fly?

    So we had a go. Several goes. At first we tried rendering SVG circles around each public transport stop – but that was too slow, particularly when zoomed out. Then we tried rendering circles in Canvas, and whilst that was OK in sparsely populated places it sucked in the cities, where people would actually want to use it.

    Eventually, we decided that as wonderful and powerful as modern web rendering techniques are, if you exclude WebGL (on the grounds that so few people have it still), all the current techniques result in pages that just hammer your browser, whilst producing an experience which isn’t up to our previous Flash-based standards. To see this in action, see the wonderful Mapnificent site, developed by the super talented Stefan Wehrmeyer. He’s a great guy and a friend of mySociety, but the Javascript circle rendering just grinds, and that’s with far, far fewer data points than we have in our system. (Sorry for potentially crashing your browser there. This is in the interest of Science.)

    Back to Colour Cycling – Using Web Standards

    So, we thought, why not look again at colour cycling the pixels within pre-rendered map tiles? After all, there are some examples out there, like this waterfall, all in Javascript.

    So, I had a bit of a look at the waterfall. It seems to work by holding in memory a structure which has all the pixels which change and all the colours they should change to and when. This works beautifully for the waterfall picture, but only a limited number of the pixels in that image actually change colour, and the image is quite small. For a full screen web browser with a big map in, this didn’t seem promising, although I’d love to see someone try.

    Then I thought: browsers have always been very good at displaying images quickly – that’s sort of vital. Perhaps we could get our tile generating server to output PNG images where, as before, the colours represent travel times, but using a palette. Then by putting this in a canvas layer in the JavaScript mapping framework Leaflet, and by changing the palette of the images on the canvas as the slider is dragged, we could get our animation.

    Unfortunately, there is no way to change the palette of an image that you’ve put on the canvas. In fact, there’s no way to change the palette of an HTML img element: all you can do is assign it a new src attribute.

    But this gets back to the original problem – we don’t want to download new mapping for every different position on the time slider. We definitely can’t afford to have the client downloading a new image source for every tile whenever the slider is moved, so we had to find a way to make that src at the client end and get that into the src attribute.

    The Breakthrough – Data URIs and Base64 encoding

    So we started trying data URIs. For those of you not familiar, these allow you to put a whole object into your HTML or CSS, encoded in Base64. They’re commonly used to prevent pages having to make extra downloads for things like tiny icons.

    So I thought, “Here’s a way we can set an image src in JavaScript to something we’ve calculated, rather than something we’ve downloaded.” And this turned out to be the key insight that allows for the relatively smooth, attractive overlays you see in Mapumental today.

    My new plan was that the client, having downloaded each palette-based image, would make a Base64 encoded version of it, which it could then use to build a version with the right palette and assign this as a data URI of the tile.

    However Base64 encoding all these images in the JavaScript seemed like unnecessary processing to do there, so the final evolution of this technique was to do the Base64 encoding at the tile server end, and while we’re at it, not to bother sending over the parts of the image that we always replace at the client end.

    So in summary, what we built does this:

    1. The server calculates the journey times and renders them to palette-based tiles.
    2. It sends these to the client, encoded in Base64, and with the initial bits up to the palette and transparency chunks removed.
    3. At the client end, we have a pre-prepared array of 255 ‘starts’ of PNGs that we combine with the later parts of the ’tiles’ from the tile server to make data URIs.
    4. When you drag the slider it combines the appropriate ‘start’ of a PNG with the bulk of the tile that has been downloaded from the server, and assigns that to the src attribute of the tile.

    And that’s how the nice overlays on Mapumental work. But as so often in coding, the really interesting devil is in the detail – read on if you’re interested.

    Diving into Base64 and the PNG file format – The Gnarly Bits

    So – why are there 255 of these ‘starts’ of these PNGs, and what do I mean by a ‘start’ anyway?

    PNG files are divided up into an 8 byte signature (the same for every PNG file) and a number of chunks, where each chunk consists of 4 bytes to tell you its length, 4 bytes of its name, some data, and 4 bytes of cyclic redundancy check. In this case, what I call a ‘start’ of a PNG is the 8 byte signature, the 25 byte of the IHDR chunk, and the PLTE (palette) and tRNS (transparency) chunks. The PLTE chunk has 12 bytes of overhead and 3 bytes per colour, and the tRNS chunk has 12 bytes of overhead and 1 byte per colour.

    Base64 encoding is a way of representing binary data in text so that it can be used in places where you would normally expect text – like URIs. Without going into too much detail, it turns groups of 3 bytes of binary gumpf into 4 bytes of normal ASCII text without control characters in it, which can then be put into a URI.

    Why do we have 255 colours, rather than the maximum 256 which are available in a palette? Because we need the break between the end of the tRNS chunk and the start of the IDAT chunk in the PNG file to align with a break between groups of three bytes in the Base64 encoded image. We need the length of these starts to be a multiple of 3 bytes in the original PNG format, which translates into a multiple of 4 bytes in the Base64 encoded version, so we can cut and shut the images without corruption.

    Which just goes to show that even though web GIS technologies may feel like they are approaching a zenith of high level abstraction, there’s still some really gnarly work to be done to get the best out of current browsers.

  10. Mapumental Property Launches

    If you’ve been following mySociety for a while, you’ll know that we have been interested in making maps that show commuting times for several years.

    However, we’ve never made public a simple, free, useful version of our slidy-swooshy Mapumental journey times technology. Until today.

    Today we pull the wraps off Mapumental Property , a house-hunting service covering England, Scotland and Wales, designed to help you work out where you might live if you want a public transport commute of a particular maximum duration. Have a go, and we guarantee you’ll find it an oddly compelling experience.

    We think it’s a genuinely useful tool – especially since unlike some of the other players in this space, we’ve got all the different kinds of public transport, right across the whole of Great Britain. We hope that some of you will find it helpful when deciding where to live.

    However, this launch doesn’t mean mySociety is bent on taking over the property websites sector. Mapumental Property isn’t a challenger to the likes of Rightmove, it’s a calling card – an advertisement for our skills – which we hope will help mySociety to attract people and organisations who want beautiful, useful web tools built for them.

    In particular we’d like people interested in Mapumental to note that:

    • We like to build attractive, usable web tools for clients of all kinds.
    • We know how to use complex data to make simple, lovely things.
    • We can do some mapping technology that others haven’t worked out yet.

    If any of that is of interest, please get in touch, or read about our software development and consulting services.

    I’d like to thank quite a few people for helping with this launch. Duncan Parkes was the lead developer, Matthew Somerville ably assisted. Jedidiah Broadbent did the design. The idea originally came from the late Chris Lightfoot, and me, Tom Steinberg. Francis Irving built the first version, and Stamen came up with the awesome idea of using sliders in the first place (and built some early tech). Kristina Glushkova worked on business development, and Zoopla’s API provides the property data. I’m also grateful to Ed Parsons of Google for very kindly giving us a hat tip when they built some technology that was inspired by Mapumental.  Thanks to everyone – this has been a long time coming.

    We’ll follow up soon with a post about the technology – and in particular how we got away from using Flash. It has been an interesting journey.