1. Mapumental’s Secret Sauce: A Map Overlay Rendering Technology You Might Find Interesting

    I am Duncan Parkes,  a developer for mySociety, a non-profit full of web geeks. One of the things we try to do well here is to take complicated data and turn it into really usable tools – tools which are attractive to people who aren’t web (or data) geeks.

    For some considerable time I’ve been working on Mapumental – a project that is about turning public transport timetable data into pretty, interactive maps featuring isochrones, shapes that show people where they can live if they want to have a commute of a particular time. You can play with the new version we just launched here. That particular map shows the commuting options to where the Queen lives. Slide the slider for full effect.

    There are a couple of hard problems that need solving if you want to build a service with an interactive journey times overlay like this. You need to be able to calculate a *huge* number of journeys extremely quickly, and you need to be able to make custom map layers so that it all looks nice. But what I think might be most interesting for you is the way in which the contours get rendered on top of the maps.

    It all started about three years ago, when the first version of the app – co-developed with the geniuses at Stamen – used Flash/Flex to draw contours on the maps, and to let people play with them. You can still play with a couple of versions of that technology from way back in 2007, that is, unless you’re using an iPad or iPhone, which of course don’t do Flash.

    Colour Cycling

    What was going on inside this Flash app was as follows. We needed to show the user any one of hundreds of different combinations of journey times (5 minutes, 12 minutes, 56 minutes, etc) depending on where they set the slider. Sending each one from the server as a tiled map overlay would be dead slow. Even Google – who have chosen to send new tiles each time – end up with a service which is surprisingly slow (try choosing a different time on this map).

    With some help from Stamen, we decided that the way of making it possible to show many different contours very quickly was send the client just one set of tiles, where each tile contained all the data for a variety of journey times. What does that mean? Simple: each colour in the tile represented a different number of minutes travelling on the map. So a batch of pixels that are colour X, all show places that are 15 minutes from the centre of the map.

    So, in this old Flash system, when you slide the slider along, the Flash app makes some of the coloured pixels opaque, and the others transparent. It was, in short, a form of colour cycling, familiar to lovers of 8 and 16 bit computer games.

    However, from about 2010 onwards, the march of iOS spelt the end of Flash. And that meant that we couldn’t launch a shiny new site based on this technology, as lovely as it was. We had to work out some approach that would use modern web standards instead.

    The Death of Flash Makes Life Difficult – for a while

    How do we replicate the experience of dragging a slider and seeing the map change like in the original Mapumental demo, but without Flash? One of the things that made the original Mapumental nice to use was how smooth the image changes were when you dragged the slider. Speed really matters to create that sort of organic effect that makes the demo so mesmerising.

    So as we started to tackle the question “How do we make this work in a post-Flash world?”. And the first thought was “Let’s do away with those map tiles, filled with all that journey time data!”. After all – why send any tiles to a modern browser, if it can just render nice shapes on the fly?

    So we had a go. Several goes. At first we tried rendering SVG circles around each public transport stop – but that was too slow, particularly when zoomed out. Then we tried rendering circles in Canvas, and whilst that was OK in sparsely populated places it sucked in the cities, where people would actually want to use it.

    Eventually, we decided that as wonderful and powerful as modern web rendering techniques are, if you exclude WebGL (on the grounds that so few people have it still), all the current techniques result in pages that just hammer your browser, whilst producing an experience which isn’t up to our previous Flash-based standards. To see this in action, see the wonderful Mapnificent site, developed by the super talented Stefan Wehrmeyer. He’s a great guy and a friend of mySociety, but the Javascript circle rendering just grinds, and that’s with far, far fewer data points than we have in our system. (Sorry for potentially crashing your browser there. This is in the interest of Science.)

    Back to Colour Cycling – Using Web Standards

    So, we thought, why not look again at colour cycling the pixels within pre-rendered map tiles? After all, there are some examples out there, like this waterfall, all in Javascript.

    So, I had a bit of a look at the waterfall. It seems to work by holding in memory a structure which has all the pixels which change and all the colours they should change to and when. This works beautifully for the waterfall picture, but only a limited number of the pixels in that image actually change colour, and the image is quite small. For a full screen web browser with a big map in, this didn’t seem promising, although I’d love to see someone try.

    Then I thought: browsers have always been very good at displaying images quickly – that’s sort of vital. Perhaps we could get our tile generating server to output PNG images where, as before, the colours represent travel times, but using a palette. Then by putting this in a canvas layer in the JavaScript mapping framework Leaflet, and by changing the palette of the images on the canvas as the slider is dragged, we could get our animation.

    Unfortunately, there is no way to change the palette of an image that you’ve put on the canvas. In fact, there’s no way to change the palette of an HTML img element: all you can do is assign it a new src attribute.

    But this gets back to the original problem – we don’t want to download new mapping for every different position on the time slider. We definitely can’t afford to have the client downloading a new image source for every tile whenever the slider is moved, so we had to find a way to make that src at the client end and get that into the src attribute.

    The Breakthrough – Data URIs and Base64 encoding

    So we started trying data URIs. For those of you not familiar, these allow you to put a whole object into your HTML or CSS, encoded in Base64. They’re commonly used to prevent pages having to make extra downloads for things like tiny icons.

    So I thought, “Here’s a way we can set an image src in JavaScript to something we’ve calculated, rather than something we’ve downloaded.” And this turned out to be the key insight that allows for the relatively smooth, attractive overlays you see in Mapumental today.

    My new plan was that the client, having downloaded each palette-based image, would make a Base64 encoded version of it, which it could then use to build a version with the right palette and assign this as a data URI of the tile.

    However Base64 encoding all these images in the JavaScript seemed like unnecessary processing to do there, so the final evolution of this technique was to do the Base64 encoding at the tile server end, and while we’re at it, not to bother sending over the parts of the image that we always replace at the client end.

    So in summary, what we built does this:

    1. The server calculates the journey times and renders them to palette-based tiles.
    2. It sends these to the client, encoded in Base64, and with the initial bits up to the palette and transparency chunks removed.
    3. At the client end, we have a pre-prepared array of 255 ‘starts’ of PNGs that we combine with the later parts of the ’tiles’ from the tile server to make data URIs.
    4. When you drag the slider it combines the appropriate ‘start’ of a PNG with the bulk of the tile that has been downloaded from the server, and assigns that to the src attribute of the tile.

    And that’s how the nice overlays on Mapumental work. But as so often in coding, the really interesting devil is in the detail – read on if you’re interested.

    Diving into Base64 and the PNG file format – The Gnarly Bits

    So – why are there 255 of these ‘starts’ of these PNGs, and what do I mean by a ‘start’ anyway?

    PNG files are divided up into an 8 byte signature (the same for every PNG file) and a number of chunks, where each chunk consists of 4 bytes to tell you its length, 4 bytes of its name, some data, and 4 bytes of cyclic redundancy check. In this case, what I call a ‘start’ of a PNG is the 8 byte signature, the 25 byte of the IHDR chunk, and the PLTE (palette) and tRNS (transparency) chunks. The PLTE chunk has 12 bytes of overhead and 3 bytes per colour, and the tRNS chunk has 12 bytes of overhead and 1 byte per colour.

    Base64 encoding is a way of representing binary data in text so that it can be used in places where you would normally expect text – like URIs. Without going into too much detail, it turns groups of 3 bytes of binary gumpf into 4 bytes of normal ASCII text without control characters in it, which can then be put into a URI.

    Why do we have 255 colours, rather than the maximum 256 which are available in a palette? Because we need the break between the end of the tRNS chunk and the start of the IDAT chunk in the PNG file to align with a break between groups of three bytes in the Base64 encoded image. We need the length of these starts to be a multiple of 3 bytes in the original PNG format, which translates into a multiple of 4 bytes in the Base64 encoded version, so we can cut and shut the images without corruption.

    Which just goes to show that even though web GIS technologies may feel like they are approaching a zenith of high level abstraction, there’s still some really gnarly work to be done to get the best out of current browsers.

  2. Mapumental Property Launches

    If you’ve been following mySociety for a while, you’ll know that we have been interested in making maps that show commuting times for several years.

    However, we’ve never made public a simple, free, useful version of our slidy-swooshy Mapumental journey times technology. Until today.

    Today we pull the wraps off Mapumental Property , a house-hunting service covering England, Scotland and Wales, designed to help you work out where you might live if you want a public transport commute of a particular maximum duration. Have a go, and we guarantee you’ll find it an oddly compelling experience.

    We think it’s a genuinely useful tool – especially since unlike some of the other players in this space, we’ve got all the different kinds of public transport, right across the whole of Great Britain. We hope that some of you will find it helpful when deciding where to live.

    However, this launch doesn’t mean mySociety is bent on taking over the property websites sector. Mapumental Property isn’t a challenger to the likes of Rightmove, it’s a calling card – an advertisement for our skills – which we hope will help mySociety to attract people and organisations who want beautiful, useful web tools built for them.

    In particular we’d like people interested in Mapumental to note that:

    • We like to build attractive, usable web tools for clients of all kinds.
    • We know how to use complex data to make simple, lovely things.
    • We can do some mapping technology that others haven’t worked out yet.

    If any of that is of interest, please get in touch, or read about our software development and consulting services.

    I’d like to thank quite a few people for helping with this launch. Duncan Parkes was the lead developer, Matthew Somerville ably assisted. Jedidiah Broadbent did the design. The idea originally came from the late Chris Lightfoot, and me, Tom Steinberg. Francis Irving built the first version, and Stamen came up with the awesome idea of using sliders in the first place (and built some early tech). Kristina Glushkova worked on business development, and Zoopla’s API provides the property data. I’m also grateful to Ed Parsons of Google for very kindly giving us a hat tip when they built some technology that was inspired by Mapumental.  Thanks to everyone – this has been a long time coming.

    We’ll follow up soon with a post about the technology – and in particular how we got away from using Flash. It has been an interesting journey.

  3. mySociety design tips: how we choose the best map zoom level

    When you report a problem on FixMyStreet.com, the site displays a map for you to click on to indicate its exact location. Of course, you can zoom in and out of that map, but when it is first displayed, FixMyStreet needs to use an initial ‘default’ zoom level. Ideally, this is a zoom level that reduces the number of clicks required before a user can pinpoint the location of their problem.

    map_zoom_by_-kathiemcl_pty

    “In the middle of America” by Kathie M Ceballos (CC BY-NC-ND licensed from Flickr)

    And here’s where we encounter a tricky problem. The world is a varied place – some towns are very dense with buildings and streets crammed close together. In these areas you need to default to a zoom that’s quite ‘close in’, otherwise it can be hard to locate your problem.

    But out in the countryside, we have the opposite problem. You can have huge areas where there’s nothing but blank fields or moorlands. If the default map zoom is ‘close in’ here then users are likely to see a big map full of nothingness, or maybe just a single stretch of unidentifiable road.

    So, what is to be done?

    The answer is this – every time you search for a location in FixMyStreet the website does a check to see whether the location you typed is in an area where a lot of people live, or very few people live.

    mySociety has been storing this population density data in a webservice which we call Gaze. If the area you searched for is in a densely populated area we assume that it must be an urban location, and the map starts with a helpfully zoomed-in map. But if you’re in a sparsely populated area then it’s probably rural, so FixMyStreet starts zoomed out, making it easier to get an overview of the whole area.

    Where do we get the data from? Our late colleague Chris discusses this in a blog post from 2005 — the short answer is NASA SEDAC and LandScan. It’s an interesting example of how unexpected things can happen when data is made public — if population density wasn’t available to us, we wouldn’t have been able to put this small but clever detail into FixMyStreet’s interface.

  4. mySociety design tips: why you can bypass the FixMyStreet map

    pilot_and_observer_map

    When you report a problem on FixMyStreet, we ask you to click on a map to pinpoint its location. For example, if you want to tell the council a tree has fallen over, or there is a hole in the road, you can click exactly where it is. This is an easy way to provide the most accurate information in the report that we then send to the council.

    If you’re a programmer then you probably think that it’s obvious to use a map for such problem reporting. We agree: maps are ideal for this, and it’s a shame that so many councils still aren’t doing it this way.

    However, even though it’s very useful to have an accurate location for a problem, with FixMyStreet there are several reasons (below, we list just four) why someone might not be able to simply “click on the map”. In these cases, the map is no longer helpful — it’s a barrier. So we have to ask: is an accurate location so important that we should refuse to accept reports without one?

    It turns out, for FixMyStreet, we don’t really need that accuracy. Sure, it’s best if we have it, and over 95% of reports do contain pinpoint locations, because most people do click on the map. But if we don’t have that location data, then we can still make a fair guess from the postcode or area name that the user has already provided (that’s how we knew which map to display, after all). So we allow the user to submit the report without clicking on the map.

    Consequently, every time someone reports a problem without using the map, it means they’ve not given up, or clicked on the wrong place just to submit the form. In fact, they’ve reported a problem because we removed what would otherwise have been a barrier.

    So, here are some reasons why we didn’t make clicking on the map mandatory in FixMyStreet:

    • Map-reading isn’t a skill everybody is comfortable with
      When you’re building and testing FixMyStreet, you tend to imagine people will be reporting problems in places they know well. It’s easy to find somewhere on a map you are familiar with. But FixMyStreet users could just as easily be reporting a problem they passed on their way to work, on holiday, or at a party. So if they can’t read maps well it might be difficult or frustrating to locate a unfamiliar road junction or building.
    • Navigating a map requires more challenging user skills
      FixMyStreet is easy to use, deliberately — in fact you can report a problem just using your keyboard. Zooming and panning a map element is much harder than any other part of the process. Remember that if you’re building a website like this yourself you are already comfortable with using complex controls that lots of people — FixMyStreet-using people — are not. This is true even before considering other limitations such as small-screen devices or visual disabilities.
    • The map might not be helpful
      We rely on third-party maps. Most of the time, they are excellent. But what if the map is out of date? What if the problem is on a new road which hasn’t been added yet? What if the user remembers that the pothole was outside a distinctive shop or remarkable tree, only to find such landmarks aren’t on the map?
    • The problem might not have a location
      Potholes are easy: they have a fixed position on the road. What about smells or flooding? These problems sometimes simply don’t seem to have an obvious pinpoint location.

    In summary: we think clicking on a map is the best way to ask for a location from FixMyStreet users. But if we forced everybody to do it, some problems would never be reported, and some people would never become FixMyStreet users.

  5. New MapIt Global: An Administrative Boundaries Web Service for the World

    Introducing MapIt Global

    All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.

    Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.

    As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.

    Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.

    We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.

    Thanks OpenStreetMap!

    We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.

    The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.

     

    From Britain to the World

    For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.

    MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.

    If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.

    About the Data

    The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.

    The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global.  We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.

    If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.

    Commercial Licenses and Local Copies

    MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.

     Feedback

    As with any new service, we’re sure there will be problems that need sorting out. Please drop us an email, or tweet us @mySociety.

  6. FixMyStreet’s been redesigned

    FixMyStreet, our site for reporting things like potholes and broken street lights, has had something of a major redesign, kindly supported in part by Kasabi. With the help of Supercool, we have overhauled the look of the site, bringing it up to date and making the most of some lovely maps. And as with any mySociety project, we’d really appreciate your feedback on how we can make it ever more usable.

    The biggest change to the new FixMyStreet is the use of responsive design, where the web site adapts to fit within the environment in which it’s being viewed. The main difference on FixMyStreet, besides the obvious navigation changes, is that in a small screen environment, the reporting process changes to have a full screen map and confirmation step, which we thought would be preferable on small touchscreens and other mobiles. There are some technical details at the end of this post.

    Along with the design, we’ve made a number of other improvements along the way. For example, something that’s been requested for a long time, we now auto-rotate photos on upload, if we can, and we’re storing whatever is provided rather than only a shrunken version. It’s interesting that most photos include correct orientation information, but some clearly do not (e.g. the Blackberry 9800).

    We have many things we’d still like to do, as a couple of items from our github repository show. Firstly, it would be good if the FixMyStreet alert page could have something similar to what we’ve done on Barnet’s planning alerts service, providing a configurable circle for the potential alert area. We also are going to be adding faceted search to the area pages, allowing you to see only reports in a particular category, or within a certain time period.

    Regarding native phone apps – whilst the new design does hopefully work well on mobile phones, we understand that native apps are still useful for a number of reasons (not least, the fact photo upload is still not possible from a mobile web app on an iPhone). We have not had the time to update our apps, but will be doing so in the near future to bring them more in line with the redesign and hopefully improve them generally as well.

    The redesign is not the only news about FixMyStreet today

    As part of our new DIY mySociety project, we are today publishing an easy-to-read guide for people interested in using the FixMyStreet software to run versions of FixMyStreet outside of Britain. We are calling the newly upgraded, more re-usable open source code the FixMyStreet Platform.

    This is the first milestone in a major effort to upgrade the FixMyStreet Platform code to make it easier and more flexible to run in other countries. This effort started last year, and today we are formally encouraging people to join our new mailing list at the new FixMyStreet Platform homepage.

    Coming soon: a major upgrade to FixMyStreet for Councils

    As part of our redesign work, we’ve spoken to a load of different councils about what they might want or need, too. We’re now taking that knowledge, combining it with this redesign, and preparing to relaunch a substantially upgraded FixMyStreet for Councils product. If you’re interested in that, drop us a line.

    Kasabi: Our Data is now in the Datastore

    Finally, we are also now pushing details of reports entered on FixMyStreet to Kasabi’s data store as open linked data; you can find details of this dataset on their site. Let us know if it’s useful to you, or if we can do anything differently to help you.

    Technical details

    For the web developers amongst you – we have a base stylesheet for everyone, and another stylesheet that is only included if your browser width is 48em or above (an em is a unit of measurement dependent on your font size), or if you’re running Internet Explorer 6-8 (as they don’t handle the modern CSS to do this properly, we assume they’ll want the larger styles) using a conditional comment. This second stylesheet has slight differences up to 61em and above 61em. Whilst everything should continue to work without JavaScript, as FixMyStreet has done with its map-based reporting since 2007, where it is enabled this allows us to provide the full screen map you can see at large screen sizes, and the adjusted process you see at smaller resolutions.

    We originally used Modernizr.mq() in our JavaScript, but found that due to the way this works (adding content to the end of the document), this can cause issues with e.g. data() set on other elements, so we switched to detecting which CSS is being applied at the time.

    On a mobile, you can see that the site navigation is at the end of the document, with a skip to navigation link at the top. On a desktop browser, you’ll note that visually the navigation is now at the top. In both cases, the HTML is the same, with the navigation placed after the main content, so that it hopefully loads and appears first. We are using display: table-caption and caption-side: top in the desktop stylesheet in order to rearrange the content visually (as explained by Jeremy Keith), a simple yet powerful technique.

    From a performance point of view, on the front page of the site, we’re e.g. using yepnope (you can get it separately or as part of Modernizr) so that the map JavaScript is downloading in the background whilst you’re there, meaning the subsequent map page is hopefully quicker to load. I’m also adding a second tile server today – not because our current one isn’t coping, it is, but just in case something should happen to our main one – we already have redundancy in our postcode/area server MapIt and our population density service Gaze.

    If you have any technical questions about the design, please do ask in the comments and I’ll do my best to answer.

  7. Mapumental: mapping a new path

    Mapumental map showing travel times to the Royal Festival Hall

    It’s high time we updated you on Mapumental, our journey-time mapping project. For those who may not remember, Mapumental is based on a simple idea: to visualise transit times, by public transport, from or to any postcode in Great Britain.

    It all began in 2006, when the Department for Transport approached us to see what we might do with public transport data; in 2009 we won an investment loan from Channel 4 and Screen West Midlands which enabled us to build a beta tool – you might have played with it. If not, go on, have a go. It’s fun!

    It’s been quite a long journey to where we are today. Unlike many mySociety projects, funding for Mapumental’s development came from a commercial investment loan, with a condition that we set it up as a business. For that reason, it’s not enough that it’s beautiful and useful – we need to find ways for it to be profitable, too. All revenues are set to come back to fund our not-for-profit activities.

    We could tell from very early on in the project that Mapumental would be a sought-after tool for all sorts of purposes, from business to personal use. For example, you can see commute times at a glance, so it’s great for house-hunters and job-seekers. Consequently, it’s also great for the property and recruitment industries.

    “Your maps look amazing, such a great way of representing what could be really boring data, but isn’t.” – A jobseeker

    We can see loads of other possibilities too – like urban planning. This sort of analysis would have been far more expensive in the past; with Mapumental, planners can see at a glance how accessible a new development would be by public transport. Its potential uses are wide-ranging, answering questions for businesses, organisations, charities, and public facilities – especially those wanting to maximise accessibility or encourage use of greener transport options.

    “The maps are a fantastic, a great tool and should be used for every planning application. I will be using Mapumental for all of our projects!” – Lee Taylor, Veridis Design

    We’ve recently refined a product that’s pared down from the dynamic maps you may remember from that beta tool: static maps. These are simple, non-interactive maps which show transit time in bands. They’re flexible in that they can be generated for any postcode, with any maximum travel time, and depict travel at any given time of day.

    We can provide a one-off map for personal use, or batches of many thousands of maps – as we have done for estate agents Foxtons, who now have a Mapumental map on every property listing.

    As we generate more and more maps for different uses, showing different parts of the country, we’re really enjoying digging out all sorts of surprising facts – like how it’s quicker to travel from Watford to Westminster than it is from some parts of Harringay. Or how Cardiff University students might sensibly live at all points east as far as Newport, but will be stymied for transport in the west if they live anywhere other than Barry or Bridgend.

    In fact, our very favourite use so far has come from an individual who centred his map around his home postcode. He tells us he has printed it off and put it up by the front door, so that on his way out of the house, he can find a new and surprising destination for day-trips.

    Find out more on the Mapumental website – and please do spread the word among friends and colleagues who might benefit from a Mapumental map.

  8. Mobile operators altering (and breaking) web content

    We had a complaint that FixMyStreet maps weren’t displaying on someone’s computer. We hadn’t had any other complaints, and we quickly narrowed it down to the fact that the person was on the internet using a tethered T-Mobile phone.

    T-Mobile (and Orange, and quite possibly others) are injecting JavaScript and altering content served over their networks. Their reason for doing this, according to their websites (T-Mobile, Orange), is to compress images and video sent to your browser, so as to speed up your browsing. Seeing it in action, they also inline some CSS and JavaScript, though not all, and remove comments from external files.

    However, their implementation breaks things. In this particular instance, the T-Mobile JavaScript comment stripper appears to be searching for “/*” and “*/” and removing everything inbetween. This might work in most cases; however in the jQuery library, we find a string containing “*/*”, and later down the file, another string containing “*/*”. T-Mobile remove everything between the things it thinks are comment markers, even though they’re actually contained within strings, causing the jQuery library to be invalid JavaScript and stopping anything using jQuery from running.

    Their decision to inline lots of the CSS also seems a bit odd – sure, on a mobile this might be quicker, but even ignoring tethering nowadays plenty of mobiles have caches too and having the CSS download once and be cached would seem better than adding weight to every page download. But I’m sure they’ve studied their decision there, and it doesn’t make any difference to the actual browsing, as opposed to the comment removal.

    To turn off this feature on your mobile phone or broadband, visit accelerator.t-mobile.co.uk or accelerator.orange.co.uk on your connection and pick the relevant option – if anyone knows of similar on other networks, do leave updates in the comments.

    From a FixMyStreet point of view – whilst FixMyStreet functions just fine without JavaScript, I had made the (perhaps incorrect) decision to put the map inside a <noscript> element, to prevent a flash of map-oddity as the JavaScript map overlaid the non-JS one. However, this meant in this circumstance the map did not work, as JavaScript was enabled, but jQuery was unable to be loaded. I haven’t decided whether to change this behaviour yet; obviously it would help people in this situation as the map would still display and function as it does for all those without JavaScript, but for those with JavaScript it does look a bit jarring as the page loads. Any suggestions on a better approach welcome 🙂

  9. Technical look at the new FixMyStreet maps

    This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

    Progressive enhancement

    During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

    Zooming

    We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

    OpenStreetMap

    Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

    OS StreetView tile server

    Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

    Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

    [sv]
    type=WMS
    url=path/to/mapserv.fcgi?map=os.map&
    layers=streetview
    tms_type=google
    spherical_mercator=true

    MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration* – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked 🙂 ).

    Other benefits of OpenLayers

    As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

    Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

        if ( fixmystreet.area ) {
            var area = new OpenLayers.Layer.Vector("KML", {
                strategies: [ new OpenLayers.Strategy.Fixed() ],
                protocol: new OpenLayers.Protocol.HTTP({
                    url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                    format: new OpenLayers.Format.KML()
                })
            });
            fixmystreet.map.addLayer(area);
            area.events.register('loadend', null, function(a,b,c) {
                var bounds = area.getDataExtent();
                if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
            });
        }

    Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

     

    *Broken link removed, Nov 2014

  10. Zoomable maps and user accounts on FixMyStreet

    Last week, FixMyStreet gained a number of new features that we hope you will find useful.

    Birmingham on FixMyStreetStroud on FixMyStreetFirstly, we’ve thrown away our old maps and replaced them with new, shiny, zoomable maps. This should make it easier for people to find and report problems, especially in sparser locations. We’re using the OS StreetView layer (hosted internally) when zoomed in, reverting to Bing Maps’ Ordnance Survey layer when zoomed out, as we felt this provided the best combination for reporting problems. In urban areas, you can still see individual houses, whilst in more rural areas the map with footpaths and other such features is probably of more use. FixMyStreet tries to guess initially which map would be most appropriate based upon population density, meaning a search for Stroud looks a bit different from that for Birmingham.

    FixMyStreet with OpenStreetMap

    OpenStreetMap fans, don’t worry – as part of our mapping technology upgrade, you can now use osm.fixmystreet.com to access your favourite mapping instead.

    Secondly, we now have user accounts. We’ve rolled these out alongside our current system of email confirmation, and it’s up to you which you use when reporting a problem or leaving an update. This means that those who come to the site one time only to report a pothole can continue to do so quickly, but have the option of an account if they want. Having an account means you no longer have to confirm reports and updates by email, and you have access to a page listing all the reports you’ve made through FixMyStreet, and showing these reports on a (obviously new and shiny) summary map.

    Adur District Council reports

    Other improvements include a much nicer All Reports section, so you can see all reports to Adur District Council on a map, paginated and with the boundary of the council marked – and individual wards of councils now each have their own pages too.

    I’ll follow up this post with another, more technical, look at the maps and how they work, for anyone who’s interested 🙂