1. 12 exciting projects mySociety was hired to deliver last year

    Image by Craig Sunter

    Not many people realise that we fund a proportion of our charitable work by carrying our commercial development and consultancy work for a wide range of clients.

    Last year, we scoped, developed and delivered a real variety of digital tools and projects. Some of the projects were surprising. Some of them made us gnash our teeth, a bit, as we grappled with new problems. But all of them (and call us geeks if you like) got us very excited.

    Here are just twelve of our personal high points from last year. If you have a project that you think we might be able to help you with in 2015, we’d love to hear from you!

    1. We Changed the Way in Which Parliament Does Digital

    Palace of Westminster by Greg DunlapThis time last year, a small team from mySociety was poring over analytics, interview content and assorted evidence from Parliament projects dating back last 2-3 years, to help us put together a simple set of recommendations to conclude our review.

    11 months later, Parliament have announced their first Head of Digital, fulfilling one of our key recommendations.

    2. We helped the MAS and the FCA protect financial consumers

    Bubble Car by Allen WatkinTwo of our projects helped people financially.

    We built the Money Advice Service’s (MAS) first responsive web application, the Car Cost Calculator.

    This tool takes one simple thing you know (the car you wish to buy) and tells you roughly how much it’ll cost to run that car against any others you might be interested in. It has been one of MAS’ most successful online tools in terms of traffic and conversion.

    We also built the Financial Conduct Authority’s Scam Smart tool, aiming to prevent financial scams.

    This tool helps users considering a financial investment to check a potential investment. Users enter information about the type of investment, how they heard about it and the details of the company offering it to them and get back tailored guidance and suggested next steps to help them ensure the investment is bona fide.

    3. We Gave Power to the People of Panama (soon)

    Alaveteli homepageWorking with the The National Authority for Transparency & Access to Information (ANTAI) and the Foreign & Commonwealth Office (FCO), we set up our first government-backed instance of our Freedom of Information platform, Alaveteli, in Panama.

    This project will ensure that Panama’s FOI legislation is promoted and used, but it will also shine a light on ANTAI, who are responsible for ensuring ministries and organisations publish their information, and handling case appeals.

    4. We Mapped All the Public Services in Wales

    Bws Ysgol - Image by Aqwis via Wikimedia, CCAfter we extended the Mapumental API to produce data output suitable for GIS (geographical information systems), the Welsh Government were able to map public services in Wales for their Index of Multiple Deprivation calculations.

    Over the course of the year they have calculated travel times for over seventy thousand points of interest.

    5. We Launched a New Organisation in Four Weeks

    Simply SecureSimply Secure approached us in dire need of a brand, an identity and a website to accompany the launch of their new organisation to help the world build user-friendly security tools and technologies.

    Cue four weeks of very intense work for mySociety’s designer, supported by members of the commercial team. And we did it.

    6. We Printed Stuff BIG (and found people jobs)

    Public transport travel times to Birmingham meet-up, from Mapumental by mySocietymySociety developer Dave Arter figured out how to generate A1 sized maps from Mapumental for every job centre in the UK – all 716 of them.

    Xerox will be using these with the DWP to help job seekers find work that is within reach by public transport. As a byproduct, Mapumental now handles high-fidelity print based outputs: get in touch if that is of interest.

    7. We Opened Up Planning Applications

    open-planning-shotWith Hampshire County Council we had the opportunity to build a new application to help assist members of the public and business better understand what was happening around them. For us, it was also the first application in which we worked closely with a provider of a linked data store, in this case Swirrl.

    When Open Planning goes live, it will look to help improve social engagement and the economy of Hampshire through better understanding and transparency of planning data.

    8. We Proved (Again) That FixMyStreet Isn’t All About Potholes

    CollideoscopeAfter a spate of cyclists’ deaths in London last year, we felt that the moment was right to build something that would support cycle safety in the UK.

    We launched Collideoscope on October the 7th with our first sponsor—Barts Charity, with the aim of generating data both on incidents involving cycles, and near misses.

    9. We Helped Launch a Film

    A map of old Norse place namesWe built a tool for the British Museum, to go alongside the general release of Vikings Live. The Norse Names project brought a sense of context and personalisation to a dataset gathered by the University of Nottingham.

    10. We Made Data More Exciting

    To the Trains by Nic McPheeIn 2013, we built an interface to help people explore the data in the National Rail Passenger Survey (NRPS) data explorer  for Passenger Focus.

    This year, they asked us to build something similar for bus users. We’re entering the final week of development now, and the finished product should be launched in March.

    The main aim of this site? To take data that could be considered pretty dry, and make it a lot more engaging.

    11. We Fixed Yet More Potholes

    Fixed, by Tup WandersThis year Warwickshire, East Sussex, Hart & Harrogate joined the list of councils using FixMyStreet as their main street fault reporting platform.

    That means that residents of those places can now make their reports direct from their council’s website, or via FixMyStreet, and either way they’ll have all the benefits of FixMyStreet’s smooth report-making interface.

    12. We Showed Parliament the Way

    Parliament Square by Duncan HarrisAnd so, we end where we began. While Parliament were busy interviewing candidates for their new ‘Head of Digital’ position, we were commissioned to demonstrate what Hansard might look like were a platform like SayIt used instead of the largely print-based publishing mechanisms used today.

    The result was shared internally. While SayIt may not be the end solution for Parliament, it’s great to have had some input into what that solution might be.

    And in 2015…?

    Got a project that you’d like us to be involved in?

    Get in touch and tell us about it.

    Image credits:

    Eggs: Craig Sunter; Parliament: Greg Dunlap; Bubble car: Allen Watkin; To the Trains: Nic McPhee; Potholes: Tup Wanders; Parliament Duncan Harris. All Creative Commons.

     

  2. FixMyStreet mobile design decisions

    So we wanted to build an app for FixMyStreet. Easy: we just had to make a cut-down version of the website, right?

    Hmm, not quite.

    In our two recent posts, we announced the new FixMyStreet apps, and their developer Struan went into some technical detail about why and how we built them.

    Now he explains a little more about what informed the decisions he made during the apps’ development.

    Moving the map, not the pin

    Linked by Cali4Beach

    When you use the desktop version of FixMyStreet, the first thing it asks for is your location, and there’s a good reason for that. It’s such a good reason that it needed to apply to the app as well.

    On the desktop site we ask you to input your postcode or street name. With a mobile app, it’s much more likely that you’ll be reporting a problem that’s right in front of you, so we can usually skip that step and show you a map of your current location.

    However, while the accuracy of geolocation technology is pretty good, it’s not perfect, so we wanted to let users fine-tune the location.

    On the website you click on the map to drop a pin where the problem is, but we’ve found this isn’t the best solution on a small screen. Fingers are big and the end of a pin is small so it can take several clicks to correctly position the pin.

    We quickly realised that having a central static crosshair, and moving the map to the location was a much easier and more accurate way to set a location.

    Sending reports made offline

    Adapted from an image by Premshree PillaiAs we explained in the previous post, one of the benefits of the apps over the mobile site is that you can make reports even if you have no phone coverage. The app stores all the details until you get back within range and you’re ready to send it off.

    One decision we made, which might seem initially puzzling, is that these offline reports don’t automatically get sent off to the council once you’re back within range of a phone or wifi signal.

    There are two linked reasons for this, and they’re both related to the fact that FixMyStreet lets you report a problem even if you don’t know who to send it to.

    Simply, before we can work out who to send the report to, we need to know exactly where you are – and that you are within FixMyStreet’s area of coverage (ie, within the UK).

    Your location also dictates the categories that we show you. Each council has its own categories, and in areas covered by two tiers of government, each council will deal with different types of report. So for example, your county council might deal with potholes, while your district council handles dog fouling.

    Once you’re back online we can check that the location is one we can accept a report about, and then fetch the list of categories for you to pick from.

    In effect, this delay is also a second chance for you to check your report before you send it off, although that was never the reason for the decision!

    The constant map

    Ordnance Survey "One-Inch" Map by Christopher Bulle
    The final decision we made was to leave the map in the background at every step of the app process.

    We initially designed it with the map only appearing when you needed it, but having the map underlying the reporting process provides a nice bit of continuity with the website, and seemed to make the app cohere better too. So, while there’s no particular reason for it to be there, we made the decision to keep things uniform.

    If anything else about the app has got you wondering, do feel free to leave a comment below!

    Photos by Cali4Beach and Christopher Bulle, and adapted from an image by Premshree Pillai (CC)

  3. Accessibility of Welsh schools by public transport – visualised

    A commission from the Welsh Government has resulted in new functionality for Mapumental, which now has the capability to display multiple points and to produce RAW data compatible with GIS applications. Here’s how it happened.

    Bws Ysgol - Image by Aqwis via Wikimedia, CCHow accessible is your nearest school, post office, or GP’s surgery? In Wales, that’s not always a simple question: the country’s mountainous landscapes, rural populations, and sometimes infrequent bus services can mean that those without cars are rather cut off from public service provision.

    But of course, like any other authority, the Welsh Government has an obligation to quantify just how accessible their services are.

    For many years, they have done so using a number of different methods. Some of these involve literally millions of point to point calculations – so, naturally, when Bill Oates, Head of Geography & Technology, Knowledge Services at the Welsh Government, approached Mapumental, he was keen to discover whether we could simplify things.

    We were keen to try it, too – plotting multiple points would add a whole new slew of possibilities to Mapumental. Previously, Mapumental has been all about travel from a single point, and this functionality would bring new applications across all kinds of industries and users.

    There’s only one way to find out

    The sensible way forward was to pick a single service and see what we could do. One of the government’s open data sets showed positioning of all the secondary schools in the country, and would give us a very good indication of how manageable the task would be across all other provisions.

    So we set ourselves this aim: to display the shortest transit time to get to any secondary school in Wales, from any point in that country.

    This project was not like the map we made for the Fire Protection Association last year, with its postcode input and interactive sliders. It bore more relation to our static maps, but with the additional dimension that the single map would have multiple points plotted on it. Each point would display its own associated journey times, and where travel to one school was quicker than to another, it would have to override the data of the school that was further away.

    And here’s the (very pretty) result

    Welsh secondary schools travel times mapTransit times by public transport to secondary schools in Wales, with an arrival time of 9:00am.

    Time bands are in 15-minute increments, with red areas being those where schools are accessible within a 15-minute journey (the centres of the red dots therefore also represent the positions of the schools).

    Purple areas are those where journey time is between 1.75 and 2 hours, and the colours in between run in the order you see bottom right of the map. White areas (much of which are mountainous and sparsely-populated) are outside the two-hour transit time.

    But there’s more – data for GIS

    Plotting all the schools on a single map required quite a bit of modification to Mapumental, but there was another important part of the project that also had to be worked on, if the output was to meet all the needs of the Welsh Government.

    They needed to be able to export the raw transit time data to their own GIS tools – the tools that they use to feed into official statistics. This allows the transit time data to be  combined with other datasets, such as population density, for in-depth analysis.

    We added a feature which allows Mapumental to produce what is known as a ‘raster grid’ output – basically, an enormous matrix that gives every pixel on the map a travel time value. To do this, we used the open source GRASS format.

    What’s next?

    Bill Oates is keen to see where this project can go:

    “I’m really excited at the prospect of combining the power of Mapumental with our open data, and fully understanding how accessible Welsh public services are by public transport.”

     

    To him, the benefits are clear:

    “Mapumental’s approach is significantly quicker than our current methods, so this work will help save us time as well as providing a more engaging output.

     

    “We hope that future work with mySociety will give us a sustainable approach to calculating the accessibility of local shops, hospitals, post offices and other services on an ongoing basis to help ensure that we’re meeting the needs of our citizens.”

     

    We’re looking to build on our success, and offer this service to others – initially on request but via our API as soon as we can. We’ll keep you posted as to our progress.
    You can see multiple-point mapping in action, on our Mapumental Property project – now the tool allows house-hunters to take more than one person’s commute into consideration when choosing where to live.

    Who might use Mapumental?

    Now that Mapumental can plot transit times from multiple points, and provide RAW data for GIS applications, we have great potential for use by anyone interested in travel and accessibility. That could be in central and local government strategy, town planning, architectural consultancy, transport provision, large enterprises looking to save on parking, or start-ups in the green transport space…to name but a few.

    Could Mapumental help you with your mapping needs? If so, please do drop us a line at hello@mysociety.org.

    Photo of Welsh school bus (bws ysgol) by Aqwis (CC)

  4. Fire, fire! Mapumental and fire engine journey times

    Image by William Murphy Mapumental can turn vast datasets into visual tools that everyone understands. Faced with highly complex, yet crucial data from the Fire Protection Association, we had a chance to really put our technology through its paces.

    Just how quickly could fire engines reach a given postcode in case of a fire? It’s a question that’s pivotal to decisions made by both the emergency services and the insurance industry.

    But previously, it has been a challenge to present the data simply, because it involves so many variables.

    Every region has its own factors, each of which will impact on fire engine response time. The number of vehicles at each station, the hours during which the station is manned, and the response policy of each individual fire authority will all play a part – and that’s before you even consider how geography might affect things.

    Dr. Jim Glockling is Technical Director at the Fire Protection Association and Head of the Risk Insight, Strategy and Control Authority (RISCAuthority), an organisation for the advancement of risk management within the fire and security sectors. Jim approached mySociety with this question: how could we map this crucial, yet complicated data in a way that could be understood by RISCAuthority members at a glance?

    It was clearly a job for Mapumental. Our transit-time mapping software was originally built to visualise public transport journey times, but its beauty is that ‘layers’ of data can be swapped out, allowing it to be used for all kinds of purposes.

    Read more about mySociety’s data visualisation services here.

    Assessing a property or postcode

    How quickly could 4 fire engines get to AL10 0XR in 10 minutes and 10 seconds?

    How quickly could 4 fire engines get to AL10 0XR in 10 minutes and 10 seconds? (Click to enlarge)

    And here’s the result of our pilot project. The maps on the right answer the following questions (click each image to see it at full size):

    How quickly could 4 fire engines get to AL10 0XR ?

    FPA: arrival times one engine

    FPA: arrival times one engine (click to enlarge)

    How does that change if the severity of the fire just requires one fire engine?

     A user inputs a postcode, and can assess exactly how quickly a fire could be tackled in that area. The different levels of severity are measured by how many response vehicles are required, and changes in this number are immediately reflected on the map.

    Assessing the general area

    FPA  - what it looks like when you have no postcode

    Which areas can four fire engines get to within 9 minutes 30 seconds at midday on a Saturday?

    It’s also possible to assess the region’s overall response capability, without inputting a postcode. The user sets severity levels (number of fire engines, or High Volume Pumping or Aerial Appliance (ladder) is needed), the time and day of the week.

    FPA - aerial response arrival within 15 minutes

    Where can an aerial appliance get to within 15 minutes at 2am on a weekday?

    The FPA tool immediately highlights the areas that are accessible within the chosen parameters, drawing on the underlying data of journey times and information such as vehicle numbers and hours of operation for each individual fire station in the region.

    Simplicity itself

    With RISCAuthority, we tested the concept using data from one fire authority –  Hertfordshire. mySociety’s task was to create a usable, elegant web interface that was as simple as possible to use, while still giving insurers the key data they needed.

    The project called on everything we knew about clean design, usability and data structures. A key part of what makes Mapumental’s data visualisation so intuitive are its sliders: this enables the user to quickly explore variables on a map.

    A tool with purpose

    Dr Glockling explains: “Whilst not necessarily used as a component of insurance pricing, this information helps insurers administer risk control and fire protection advice to their customers in the context of what the Fire and Rescue Services will be able to achieve on their behalf.”

    The response time is just one factor that insurance surveyors will take into account when they are assessing a building. “Where response and arrival times are not coherent with protecting the viability of the business in the event of fire, additional forms of in-built protection and control might be recommended, such as the installation of sprinkler systems.”

    “In the longer term it is hoped such information will impact beneficially on the annual cost of fire in the UK.”

    Results

    The pilot tool was well received by the FPA community, and the plan is now to work with RISCAuthority to roll it out to more fire authorities shortly, and then nationwide.

    Dr Glockling explains the pilot study helped them to understand two factors:

    • Would they get buy-in from both insurers and Fire and Rescue services on the viability and usefulness of the project?

    • Was it possible to present such a massive amount of data in a format that was readily palatable to the intended audience?

    He says, “Mapumental’s team displayed an immediate understanding of our requirement. Delivery was to time and the result has perfectly satisfied the de-risking ambition. The working relationships were very good throughout and we intend now to extend the pilot to full UK rollout.”

    During this phase, we will be inputting still more detail to the data, including information on the types of fire engine available to each region, and the plotting of fire stations on the map.

    The tool will be a valuable resource for the FPA and the insurance industry, and we really look forward to the roll-out later this year.

    Mapumental specialises in visualising complex geographic data sets on intuitive, easy to use map tools. If you have a data visualisation project that will benefit from Mapumental, just get in touch. Or read more about mySociety’s data visualisation services here.

    Photo by William Murphy (CC)

  5. Mapumental Property – extra insight for househunters

    If you’re searching for a new home, give Mapumental Property a try. lt narrows property results down, only showing you houses that fall within a decent commute time from the places you visit regularly – like work, school, or the shops. Here, have a go – it’s fun.

    Mapumental Property screenshot

    Irritation is the mother of invention

    Several years ago,  some of our colleagues were looking for a house to rent.

    They weren’t set on a particular town. There were two important factors: that it was within a reasonable commute from central London, where they frequently attended meetings; and that the rent was affordable.

    Faced with these requirements, most of us would sift through property sites and cross-reference the listings manually with public transport information. It’s rather time-consuming, and slightly irritating, but hey-ho, it has to be done.

    But mySociety is in the business of building useful web tools, so when something irritates us like this, we look to see if we can solve the problem through the magic of code. In a stroke of good timing, it was at just around this time that the Department for Transport approached us to ask us to work with their public transport data – and Mapumental was conceived.

    Early days

    The key was to combine Ordnance Survey postcodes with the DfT’s data about journey times, NPTDR (National Public Transport Data Repository). This data set takes a  ‘snapshot’  of every public transport journey in Great Britain for a selected week in October each year.

    Sounds simple? The process was not without its challenges. Prime among them was the problem of displaying map tiles, plus the vast quantities of transport data, within a reasonable amount of time, no matter which postcode or zoom level the user chose. As we know, a  ‘reasonable amount of time’  for a page to load is a metric which is forever shrinking.

    By 2006, we had created Mapumental’s first iteration. Users could input a postcode and see all areas of the country that could be reached by public transport, divided into coloured travel-time bands. In 2009, Francis Irving, the mySociety coder behind Mapumental’s early endeavours, explained the technology he’d used. It was Flash-dependent, and a few years later, developer Duncan wrote about some of the technical hurdles he overcame replacing the Flash elements, in view of the rise of the iPhone, which famously doesn’t  ‘do’  Flash.

    Hoorah! Now our colleagues could type in a central London postcode and see everywhere that fell within a 40-minute journey from there. It wasn’t long before we added median house price data, too.

    Beauty is in the eye of the crowd

    We even added a ‘scenicness’ rating: if the beauty of your surroundings was important to you, you could rule out anywhere below a certain level of attractiveness.

    How did we assess how scenic every area in the UK is? By crowdsourcing the information – our ScenicOrNot website displays a random photograph from every square mile of the British isles, inviting people to rate them. It is surprisingly compulsive.

    ScenicOrNot from mySociety

    A showcase tool

    Mapumental may have been born from our own needs, but we knew from the beginning that it would have wider applications. It has always been the sort of project that got people excited, once they saw it in action.

    We wanted to show how elegantly Mapumental can handle all kinds of data, starting with houses for sale and rent – so we developed Mapumental Property. It’s not intended as a serious competitor to the giant property websites out there. Rather, it’s an all-singing, all-dancing demonstration of Mapumental’s strengths.

    In this case, the data is from the property website Zoopla, and you can narrow it down to show rental or sales property within your chosen price bands and commute distances. You can even add multiple destination points, so that households of two or more people can find their optimum location.

    Versatility rules

    But Mapumental is not just about property: swap out that Zoopla layer, and you could put in anything else you can imagine – hospital locations, supermarkets, schools, job vacancies… you name it.

    The beauty of Mapumental is that now we’ve done the really hard part, incorporating new data layers is relatively simple. Recent work for the Fire Protection Association and the Welsh Government, among others, has shown its versatility.

    Now how about you?

    We believe that Mapumental’s possibilities are pretty much endless. Have you got an unloved, difficult-to-navigate dataset that Mapumental could breathe new life into? Or would your stakeholders benefit from being able to see your data displayed on a map? Let us know.

  6. Mapumental’s Secret Sauce: A Map Overlay Rendering Technology You Might Find Interesting

    I am Duncan Parkes,  a developer for mySociety, a non-profit full of web geeks. One of the things we try to do well here is to take complicated data and turn it into really usable tools – tools which are attractive to people who aren’t web (or data) geeks.

    For some considerable time I’ve been working on Mapumental – a project that is about turning public transport timetable data into pretty, interactive maps featuring isochrones, shapes that show people where they can live if they want to have a commute of a particular time. You can play with the new version we just launched here. That particular map shows the commuting options to where the Queen lives. Slide the slider for full effect.

    There are a couple of hard problems that need solving if you want to build a service with an interactive journey times overlay like this. You need to be able to calculate a *huge* number of journeys extremely quickly, and you need to be able to make custom map layers so that it all looks nice. But what I think might be most interesting for you is the way in which the contours get rendered on top of the maps.

    It all started about three years ago, when the first version of the app – co-developed with the geniuses at Stamen – used Flash/Flex to draw contours on the maps, and to let people play with them. You can still play with a couple of versions of that technology from way back in 2007, that is, unless you’re using an iPad or iPhone, which of course don’t do Flash.

    Colour Cycling

    What was going on inside this Flash app was as follows. We needed to show the user any one of hundreds of different combinations of journey times (5 minutes, 12 minutes, 56 minutes, etc) depending on where they set the slider. Sending each one from the server as a tiled map overlay would be dead slow. Even Google – who have chosen to send new tiles each time – end up with a service which is surprisingly slow (try choosing a different time on this map).

    With some help from Stamen, we decided that the way of making it possible to show many different contours very quickly was send the client just one set of tiles, where each tile contained all the data for a variety of journey times. What does that mean? Simple: each colour in the tile represented a different number of minutes travelling on the map. So a batch of pixels that are colour X, all show places that are 15 minutes from the centre of the map.

    So, in this old Flash system, when you slide the slider along, the Flash app makes some of the coloured pixels opaque, and the others transparent. It was, in short, a form of colour cycling, familiar to lovers of 8 and 16 bit computer games.

    However, from about 2010 onwards, the march of iOS spelt the end of Flash. And that meant that we couldn’t launch a shiny new site based on this technology, as lovely as it was. We had to work out some approach that would use modern web standards instead.

    The Death of Flash Makes Life Difficult – for a while

    How do we replicate the experience of dragging a slider and seeing the map change like in the original Mapumental demo, but without Flash? One of the things that made the original Mapumental nice to use was how smooth the image changes were when you dragged the slider. Speed really matters to create that sort of organic effect that makes the demo so mesmerising.

    So as we started to tackle the question “How do we make this work in a post-Flash world?”. And the first thought was “Let’s do away with those map tiles, filled with all that journey time data!”. After all – why send any tiles to a modern browser, if it can just render nice shapes on the fly?

    So we had a go. Several goes. At first we tried rendering SVG circles around each public transport stop – but that was too slow, particularly when zoomed out. Then we tried rendering circles in Canvas, and whilst that was OK in sparsely populated places it sucked in the cities, where people would actually want to use it.

    Eventually, we decided that as wonderful and powerful as modern web rendering techniques are, if you exclude WebGL (on the grounds that so few people have it still), all the current techniques result in pages that just hammer your browser, whilst producing an experience which isn’t up to our previous Flash-based standards. To see this in action, see the wonderful Mapnificent site, developed by the super talented Stefan Wehrmeyer. He’s a great guy and a friend of mySociety, but the Javascript circle rendering just grinds, and that’s with far, far fewer data points than we have in our system. (Sorry for potentially crashing your browser there. This is in the interest of Science.)

    Back to Colour Cycling – Using Web Standards

    So, we thought, why not look again at colour cycling the pixels within pre-rendered map tiles? After all, there are some examples out there, like this waterfall, all in Javascript.

    So, I had a bit of a look at the waterfall. It seems to work by holding in memory a structure which has all the pixels which change and all the colours they should change to and when. This works beautifully for the waterfall picture, but only a limited number of the pixels in that image actually change colour, and the image is quite small. For a full screen web browser with a big map in, this didn’t seem promising, although I’d love to see someone try.

    Then I thought: browsers have always been very good at displaying images quickly – that’s sort of vital. Perhaps we could get our tile generating server to output PNG images where, as before, the colours represent travel times, but using a palette. Then by putting this in a canvas layer in the JavaScript mapping framework Leaflet, and by changing the palette of the images on the canvas as the slider is dragged, we could get our animation.

    Unfortunately, there is no way to change the palette of an image that you’ve put on the canvas. In fact, there’s no way to change the palette of an HTML img element: all you can do is assign it a new src attribute.

    But this gets back to the original problem – we don’t want to download new mapping for every different position on the time slider. We definitely can’t afford to have the client downloading a new image source for every tile whenever the slider is moved, so we had to find a way to make that src at the client end and get that into the src attribute.

    The Breakthrough – Data URIs and Base64 encoding

    So we started trying data URIs. For those of you not familiar, these allow you to put a whole object into your HTML or CSS, encoded in Base64. They’re commonly used to prevent pages having to make extra downloads for things like tiny icons.

    So I thought, “Here’s a way we can set an image src in JavaScript to something we’ve calculated, rather than something we’ve downloaded.” And this turned out to be the key insight that allows for the relatively smooth, attractive overlays you see in Mapumental today.

    My new plan was that the client, having downloaded each palette-based image, would make a Base64 encoded version of it, which it could then use to build a version with the right palette and assign this as a data URI of the tile.

    However Base64 encoding all these images in the JavaScript seemed like unnecessary processing to do there, so the final evolution of this technique was to do the Base64 encoding at the tile server end, and while we’re at it, not to bother sending over the parts of the image that we always replace at the client end.

    So in summary, what we built does this:

    1. The server calculates the journey times and renders them to palette-based tiles.
    2. It sends these to the client, encoded in Base64, and with the initial bits up to the palette and transparency chunks removed.
    3. At the client end, we have a pre-prepared array of 255 ‘starts’ of PNGs that we combine with the later parts of the ’tiles’ from the tile server to make data URIs.
    4. When you drag the slider it combines the appropriate ‘start’ of a PNG with the bulk of the tile that has been downloaded from the server, and assigns that to the src attribute of the tile.

    And that’s how the nice overlays on Mapumental work. But as so often in coding, the really interesting devil is in the detail – read on if you’re interested.

    Diving into Base64 and the PNG file format – The Gnarly Bits

    So – why are there 255 of these ‘starts’ of these PNGs, and what do I mean by a ‘start’ anyway?

    PNG files are divided up into an 8 byte signature (the same for every PNG file) and a number of chunks, where each chunk consists of 4 bytes to tell you its length, 4 bytes of its name, some data, and 4 bytes of cyclic redundancy check. In this case, what I call a ‘start’ of a PNG is the 8 byte signature, the 25 byte of the IHDR chunk, and the PLTE (palette) and tRNS (transparency) chunks. The PLTE chunk has 12 bytes of overhead and 3 bytes per colour, and the tRNS chunk has 12 bytes of overhead and 1 byte per colour.

    Base64 encoding is a way of representing binary data in text so that it can be used in places where you would normally expect text – like URIs. Without going into too much detail, it turns groups of 3 bytes of binary gumpf into 4 bytes of normal ASCII text without control characters in it, which can then be put into a URI.

    Why do we have 255 colours, rather than the maximum 256 which are available in a palette? Because we need the break between the end of the tRNS chunk and the start of the IDAT chunk in the PNG file to align with a break between groups of three bytes in the Base64 encoded image. We need the length of these starts to be a multiple of 3 bytes in the original PNG format, which translates into a multiple of 4 bytes in the Base64 encoded version, so we can cut and shut the images without corruption.

    Which just goes to show that even though web GIS technologies may feel like they are approaching a zenith of high level abstraction, there’s still some really gnarly work to be done to get the best out of current browsers.

  7. Mapumental Property Launches

    If you’ve been following mySociety for a while, you’ll know that we have been interested in making maps that show commuting times for several years.

    However, we’ve never made public a simple, free, useful version of our slidy-swooshy Mapumental journey times technology. Until today.

    Today we pull the wraps off Mapumental Property , a house-hunting service covering England, Scotland and Wales, designed to help you work out where you might live if you want a public transport commute of a particular maximum duration. Have a go, and we guarantee you’ll find it an oddly compelling experience.

    We think it’s a genuinely useful tool – especially since unlike some of the other players in this space, we’ve got all the different kinds of public transport, right across the whole of Great Britain. We hope that some of you will find it helpful when deciding where to live.

    However, this launch doesn’t mean mySociety is bent on taking over the property websites sector. Mapumental Property isn’t a challenger to the likes of Rightmove, it’s a calling card – an advertisement for our skills – which we hope will help mySociety to attract people and organisations who want beautiful, useful web tools built for them.

    In particular we’d like people interested in Mapumental to note that:

    • We like to build attractive, usable web tools for clients of all kinds.
    • We know how to use complex data to make simple, lovely things.
    • We can do some mapping technology that others haven’t worked out yet.

    If any of that is of interest, please get in touch, or read about our software development and consulting services.

    I’d like to thank quite a few people for helping with this launch. Duncan Parkes was the lead developer, Matthew Somerville ably assisted. Jedidiah Broadbent did the design. The idea originally came from the late Chris Lightfoot, and me, Tom Steinberg. Francis Irving built the first version, and Stamen came up with the awesome idea of using sliders in the first place (and built some early tech). Kristina Glushkova worked on business development, and Zoopla’s API provides the property data. I’m also grateful to Ed Parsons of Google for very kindly giving us a hat tip when they built some technology that was inspired by Mapumental.  Thanks to everyone – this has been a long time coming.

    We’ll follow up soon with a post about the technology – and in particular how we got away from using Flash. It has been an interesting journey.

  8. mySociety design tips: how we choose the best map zoom level

    When you report a problem on FixMyStreet.com, the site displays a map for you to click on to indicate its exact location. Of course, you can zoom in and out of that map, but when it is first displayed, FixMyStreet needs to use an initial ‘default’ zoom level. Ideally, this is a zoom level that reduces the number of clicks required before a user can pinpoint the location of their problem.

    “In the middle of America” by Kathie M Ceballos (CC BY-NC-ND licensed from Flickr)

    And here’s where we encounter a tricky problem. The world is a varied place – some towns are very dense with buildings and streets crammed close together. In these areas you need to default to a zoom that’s quite ‘close in’, otherwise it can be hard to locate your problem.

    But out in the countryside, we have the opposite problem. You can have huge areas where there’s nothing but blank fields or moorlands. If the default map zoom is ‘close in’ here then users are likely to see a big map full of nothingness, or maybe just a single stretch of unidentifiable road.

    So, what is to be done?

    The answer is this – every time you search for a location in FixMyStreet the website does a check to see whether the location you typed is in an area where a lot of people live, or very few people live.

    mySociety has been storing this population density data in a webservice which we call Gaze. If the area you searched for is in a densely populated area we assume that it must be an urban location, and the map starts with a helpfully zoomed-in map. But if you’re in a sparsely populated area then it’s probably rural, so FixMyStreet starts zoomed out, making it easier to get an overview of the whole area.

    Where do we get the data from? Our late colleague Chris discusses this in a blog post from 2005 — the short answer is NASA SEDAC and LandScan. It’s an interesting example of how unexpected things can happen when data is made public — if population density wasn’t available to us, we wouldn’t have been able to put this small but clever detail into FixMyStreet’s interface.

  9. mySociety design tips: why you can bypass the FixMyStreet map

    When you report a problem on FixMyStreet, we ask you to click on a map to pinpoint its location. For example, if you want to tell the council a tree has fallen over, or there is a hole in the road, you can click exactly where it is. This is an easy way to provide the most accurate information in the report that we then send to the council.

    If you’re a programmer then you probably think that it’s obvious to use a map for such problem reporting. We agree: maps are ideal for this, and it’s a shame that so many councils still aren’t doing it this way.

    However, even though it’s very useful to have an accurate location for a problem, with FixMyStreet there are several reasons (below, we list just four) why someone might not be able to simply “click on the map”. In these cases, the map is no longer helpful — it’s a barrier. So we have to ask: is an accurate location so important that we should refuse to accept reports without one?

    It turns out, for FixMyStreet, we don’t really need that accuracy. Sure, it’s best if we have it, and over 95% of reports do contain pinpoint locations, because most people do click on the map. But if we don’t have that location data, then we can still make a fair guess from the postcode or area name that the user has already provided (that’s how we knew which map to display, after all). So we allow the user to submit the report without clicking on the map.

    Consequently, every time someone reports a problem without using the map, it means they’ve not given up, or clicked on the wrong place just to submit the form. In fact, they’ve reported a problem because we removed what would otherwise have been a barrier.

    So, here are some reasons why we didn’t make clicking on the map mandatory in FixMyStreet:

    • Map-reading isn’t a skill everybody is comfortable with
      When you’re building and testing FixMyStreet, you tend to imagine people will be reporting problems in places they know well. It’s easy to find somewhere on a map you are familiar with. But FixMyStreet users could just as easily be reporting a problem they passed on their way to work, on holiday, or at a party. So if they can’t read maps well it might be difficult or frustrating to locate a unfamiliar road junction or building.
    • Navigating a map requires more challenging user skills
      FixMyStreet is easy to use, deliberately — in fact you can report a problem just using your keyboard. Zooming and panning a map element is much harder than any other part of the process. Remember that if you’re building a website like this yourself you are already comfortable with using complex controls that lots of people — FixMyStreet-using people — are not. This is true even before considering other limitations such as small-screen devices or visual disabilities.
    • The map might not be helpful
      We rely on third-party maps. Most of the time, they are excellent. But what if the map is out of date? What if the problem is on a new road which hasn’t been added yet? What if the user remembers that the pothole was outside a distinctive shop or remarkable tree, only to find such landmarks aren’t on the map?
    • The problem might not have a location
      Potholes are easy: they have a fixed position on the road. What about smells or flooding? These problems sometimes simply don’t seem to have an obvious pinpoint location.

    In summary: we think clicking on a map is the best way to ask for a location from FixMyStreet users. But if we forced everybody to do it, some problems would never be reported, and some people would never become FixMyStreet users.

  10. New MapIt Global: An Administrative Boundaries Web Service for the World

    Introducing MapIt Global

    All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.

    Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.

    As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.

    Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.

    We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.

    Thanks OpenStreetMap!

    We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.

    The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.

     

    From Britain to the World

    For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.

    MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.

    If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.

    About the Data

    The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.

    The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global.  We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.

    If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.

    Commercial Licenses and Local Copies

    MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.

     Feedback

    As with any new service, we’re sure there will be problems that need sorting out. Please drop us an email, or tweet us @mySociety.