1. The whys and hows of the FixMyStreet mobile app

    Fixed, by Tup WandersYesterday, we announced the new FixMyStreet apps (download here for Apple devices and here for Android).

    In this post, we want to explain a bit more about why we spent time and effort on making them, when normally we advocate for mobile websites.

    Plus, for our technical audience, we’ll explain some of the tools we used in the build.

    The why bit

    When we redesigned FixMyStreet last year, one of the goals was to provide a first class experience for people using the website on their mobile phone browsers. We’re pretty happy with the result.

    We’re also believers in only building an app if it offers you something a mobile website can’t. There are lots of reasons for that: for a start, apps have to be installed, adding a hurdle at which people may just give up. They’re time-consuming to build, and you probably can’t cater for every type of phone out there.

    Given that, why did we spend time on making a mobile app? Well, firstly, potholes aren’t always in places with a good mobile signal, so we wanted to provide a way to start reporting problems offline and then complete them later.

    Secondly, when you’re on the move you often get interrupted, so you might well start reporting a problem and then become distracted. When that happens, reports in browsers may get lost, so we wanted an app that would save it as you went along, and allow you to come back to it later.

    And the how bit (with added technical details, for the interested)

    Having decided to build an app the next question is how to build it. The main decision is whether to use the native tools for each phone operating system, or to use one of the various toolkits that allow you to re-use code across multiple operating systems.

    We quickly decided on using Phonegap, a cross platform toolkit, for several reasons: we’d already used Phonegap successfully in the past to build an app for Channel 4’s Great British Property Scandal (that won an award, so clearly something went right) and for Züri Wie Neu in Zurich, so it was an easy decision to use it again.

    It’s a good fit for a small team like ours, because as well as reducing the amount of code we have to write it also makes use of the skills we already have in JavaScript, HTML and CSS. We could even re-use parts of the JavaScript from the main FixMyStreet site. There are compromises in using this sort of toolkit, but for our circumstances we think they’re the right ones to make.

    We decided to focus on apps for Android and iOS, as these are the two most popular operating systems. Even with this limitation, there is a lot of variety in the size and capability of devices that could be running the app – think iPads and tablets – but we decided to focus primarily on providing a good experience for people using phone-sized devices. This decision was partly informed by the resources we have at hand, but the main decider was that we mostly expect the app to be used on phones.

    Phonegap gives you the means to run a JavaScript app on the phone, but it doesn’t provide much else. We used JQuery mobile to handle the UI and Backbone.js for the structure. This worked out reasonably well for us but given the pace of development in JavaScript frameworks at the moment we’ll probably look to see what’s available for any future app projects.

    Caution, by Nicholas A Tonelli

    There was one big challenge: the functionality that allows you to take photos  in-app. We just couldn’t get it to work with older versions of Android  – and it’s still not really adequate. We just hope most people are updating their operating systems! Later versions of Android (and iOS) were considerably less frustrating, and perhaps an earlier decision to focus on these first would have led to a shorter development process.

    So, there you have it. The app does what we wanted it to (on the majority of systems), and you can download it right now (Android users go here; iOS here).

    On balance though? We’d still advocate a mobile-optimised browser site almost every time. But sometimes circumstances dictate – like they did for FixMyStreet – that you really need an app.

    We’d give you the same advice, too, if you asked us. And we’d happily build you an app, or a mobile-friendly site, whichever was more suitable.

    Photos by Tup Wanders and Nicholas A Tonelli (CC)

  2. Mobile operators altering (and breaking) web content

    We had a complaint that FixMyStreet maps weren’t displaying on someone’s computer. We hadn’t had any other complaints, and we quickly narrowed it down to the fact that the person was on the internet using a tethered T-Mobile phone.

    T-Mobile (and Orange, and quite possibly others) are injecting JavaScript and altering content served over their networks. Their reason for doing this, according to their websites (T-Mobile, Orange), is to compress images and video sent to your browser, so as to speed up your browsing. Seeing it in action, they also inline some CSS and JavaScript, though not all, and remove comments from external files.

    However, their implementation breaks things. In this particular instance, the T-Mobile JavaScript comment stripper appears to be searching for “/*” and “*/” and removing everything inbetween. This might work in most cases; however in the jQuery library, we find a string containing “*/*”, and later down the file, another string containing “*/*”. T-Mobile remove everything between the things it thinks are comment markers, even though they’re actually contained within strings, causing the jQuery library to be invalid JavaScript and stopping anything using jQuery from running.

    Their decision to inline lots of the CSS also seems a bit odd – sure, on a mobile this might be quicker, but even ignoring tethering nowadays plenty of mobiles have caches too and having the CSS download once and be cached would seem better than adding weight to every page download. But I’m sure they’ve studied their decision there, and it doesn’t make any difference to the actual browsing, as opposed to the comment removal.

    To turn off this feature on your mobile phone or broadband, visit accelerator.t-mobile.co.uk or accelerator.orange.co.uk on your connection and pick the relevant option – if anyone knows of similar on other networks, do leave updates in the comments.

    From a FixMyStreet point of view – whilst FixMyStreet functions just fine without JavaScript, I had made the (perhaps incorrect) decision to put the map inside a <noscript> element, to prevent a flash of map-oddity as the JavaScript map overlaid the non-JS one. However, this meant in this circumstance the map did not work, as JavaScript was enabled, but jQuery was unable to be loaded. I haven’t decided whether to change this behaviour yet; obviously it would help people in this situation as the map would still display and function as it does for all those without JavaScript, but for those with JavaScript it does look a bit jarring as the page loads. Any suggestions on a better approach welcome 🙂

  3. Technical look at the new FixMyStreet maps

    This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

    Progressive enhancement

    During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

    Zooming

    We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

    OpenStreetMap

    Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

    OS StreetView tile server

    Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

    Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

    [sv]
    type=WMS
    url=path/to/mapserv.fcgi?map=os.map&
    layers=streetview
    tms_type=google
    spherical_mercator=true

    MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration* – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked 🙂 ).

    Other benefits of OpenLayers

    As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

    Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

        if ( fixmystreet.area ) {
            var area = new OpenLayers.Layer.Vector("KML", {
                strategies: [ new OpenLayers.Strategy.Fixed() ],
                protocol: new OpenLayers.Protocol.HTTP({
                    url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                    format: new OpenLayers.Format.KML()
                })
            });
            fixmystreet.map.addLayer(area);
            area.events.register('loadend', null, function(a,b,c) {
                var bounds = area.getDataExtent();
                if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
            });
        }

    Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

     

    *Broken link removed, Nov 2014

  4. FixMyStreet in Norway

    At mySociety we love our site FixMyStreet – it’s the epitome of a web tool that gives simple tangible benefits whilst generating a little accountability at the same time. Reports through the site were up 40% last year, so it’s clear that users quite like it too.

    FixMyStreet has been copied in many different countries, which makes everyone in mySociety very happy, too, even apparently appearing in a slide deck the White House uses to show innovative services. However, it turns out that the cheerfully minimalist, almost wantonly unfashionable user-interface has an unfortunate down side: most people who copy the site look at it, think “That looks easy!” and then cheerfully start coding their own clone.

    Deceptive simplicity

    Alas – the very simplicity that makes the site good hides the fact that making a site like FixMyStreet really work well is actually way harder than it looks. What will you do when a government email inbox fills up? What about when administrative boundaries change, due to an election or restructuring? How do you know you’re not scaring users away with careless wording? All the hard-won lessons from these questions have been baked into the FixMyStreet codebase, and we’re only too keen to talk to people about them.

    We were therefore particularly pleased when the Nowegian Unix User’s Group (NUUG) came to us to ask if we could help improve FixMyStreet to make it easier for them to install. Over the last month mySociety Senior Developer Matthew Somerville has been working hard with Petter Reinholdtsen and Christer Gundersen of NUUG, and here’s what they’ve managed in just a handful of weeks.

    • The launch of a Norwegian FixMyStreet called Fiksgatami, covering nearly every corner of Norway’s 300,000 square kilometers.
    • Problems reported anywhere within Norway will be correctly directed to any of the 400+ responsible municipalities, thanks to Petter and Christer’s amazing data sourcing skills.
    • As a necessary side-effect of developing this, Norway now has a free, public administrative web service gazeteer – http://mapit.nuug.no. If Norway is anything like the UK this will soon become an indispensable service for many other web sites and mobile tools.
    • The standard mapping is now OpenStreetMap, pulled together by the brilliant Norwegian OpenStreetMap community*. We couldn’t take a technological step backward, and so whilst the site uses OpenLayers if you have JavaScript, the map continues to work just fine without as well.
    • The open source FixMyStreet codebase has been upgraded to make it easier to translate into other languages, easier to use different mapping with, and easier to install. These efforts will continue, as we realise this has been one reason why others have made their own versions.
    • All this has been done without forking, so various major upgrades we have planned for the UK version will be exportable later in the year.

    NUUG’s Fiksgatami is the epitome of what makes civic open source at its best so unmatchably good. It was developed incredibly quickly: just a month to create what is effectively a fully fledged, best-of-breed nationwide e-government service – albeit an unofficial one. Thanks to the hard work of the public servants who fix problem reports, it will make small but meaningful improvements to the lives of a lot of people in Norway.  And it has made the free FixMyStreet codebase better and easier for other people to use to help them do the same thing in other countries.

    I know that at mySociety we are all looking forward to working with NUUG again. And I hope that this story inspires others to look at our code, and to work with mySociety to make FixMyStreet a service that can help everyone who would benefit from it.

    * We’ll be rolling out updated mapping (including OSM) and more in the UK, soon.

  5. Highlighting the current speech

    Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?

    I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. 🙂

    Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.

    The highlighting works the other way – as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! 🙂 ).

    Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.

    Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse 🙂

    Previous articles

    1. The Flash player
    2. Seeking
    3. Highlighting the current speech
  6. TheyWorkForYou video – the Flash player

    TheyWorkForYou video timestamping has been launched, over 40% of available speeches have already been timestamped, and (hopefully) all major bugs have been fixed, so I can now take a short breather and write this short series of more technical posts, looking at how the front end bits I wrote work and hang together.

    Let’s start with the most obvious feature of video timestamping – the video player itself. 🙂 mySociety is an open-source shop, so it was great to discover that (nearly all of) Adobe Flex is available under the Mozilla Public Licence. This meant I could simply download the compiler and libraries, write some code and compile it into a working SWF Flash file without any worries (and you can do the same!).

    Writing a Flex program is split into three main areas – MXML that lays out your application, defines any web services you’re using and so on; CSS to define the style of the various components; and ActionScript to deal with things like events, or talking to the JavaScript in the parent HTML. My code is probably quite shoddy in a number of places – it’s my first application in Flex 🙂 – but it’s all available to view if you want to take a peek, and it’s obviously running on the live TheyWorkForYou site.

    To put a video component in the player is no harder than including an <mx:VideoDisplay> element – set the source of that, and you have yourself a video player, no worrying about stream type, bandwidth detection, or anything else. 🙂 You can then use a very useful feature called data binding to make lots of things trivial – for example, I simply set the value of a horizontal slider to be the current playing time of the video, and the slider is then automatically in the right place at all times. On the downside, VideoDisplay does appear to have a number of minor bugs (the most obvious one being where seeking can cause the video to become unresponsive and you have to refresh the page; it’s more than possible it’s a bug in my code, of course, but there are a couple of related bugs in Adobe’s bug tracker).

    As well as the buttons, sliders and the video itself, the current MXML contains two fades (one to fade in the hover controls, one to fade them out), one time formatter (to format the display of the running time and duration), and three web services (to submit a timestamp result, delete a mistaken timestamp, and fetch an array of all existing timestamps for the current debate). These are all called from various places within the ActionScript when certain events happen (e.g. the Now button or the Oops button is clicked).

    Compiling is a simple matter of running mxmlc on the mxml file, and out pops a SWF file. It’s all straightforward, although a bit awkward at first working again with a strongly-typed, compiled language after a long time with less strict ones 🙂 The documentation is good, but it can be hard to find – googling for [flex3 VideoDisplay] and the like has been quite common over the past few weeks.

    Tomorrow I will talk about moving around within the videos and some bugs thrown up there, and then how the front end communicates with the video in order to highlight the currently playing speech – for example, have a look at last week’s Prime Minister’s Questions.

    1. The Flash player
    2. Seeking
    3. Highlighting the current speech
  7. Postcodeine

    So, a silly post for today: Postcodeine. This is a British version of Ben Fry’s zipdecode, a “tool” for visualising the distribution of zipcodes in the United States. This is, as has been pointed out to me, wholly pointless, but it’s quite fun and writing it was an interesting exercise (it also taught me a little bit about AJAX, the web’s technology trend du jour). If you want the source code, it’s at the foot here; licence is the Affero GPL, as for all the other mySociety code.

    How it works: this is pretty obvious, but I might as well spell it out. The web page has four images on it: the big and small base maps, and two overlays. The back-end code is responsible for drawing sets of postcode locations into transparent PNGs, and when you type things in the text field, the src for each of the overlay images is changed. Panning the large map is done by issuing another request from Javascript to grab the mean location of all postcodes matching the given prefix (slightly hobbled, so that this isn’t a generalised postcode-to-coordinates oracle — sorry!); the rightmost pane, with a list of postcodes and their areas, is populated from another HTTP request. It could be done with an iframe but, as Paul Graham puts it, “Javascript works now”, so we might as well use that.

    (I should say, by the way, that I wrote this in my copious spare time. It’s copyright mySociety because I don’t have the right to use the postcode database myself.)

  8. Esperanto

    PledgeBank is now available in Esperanto. Getting this translation up and running was very instructive. During the process, we found numerous strings missed from the translation file, a variety of interesting bugs, and had to make various improvements all over the place (things that are the same in English whether something is singular or plural are not so in other languages, and the fonts that we use for PDF posters don’t contain the glyphs needed, to name but two examples).

    From a technical point of view, we now have a suite of scripts that bring our main .po file up to date with what’s currently on the site (including coping with oddities like email templates, JavaScript text), merge this in with the translations that already exist (so nothing is lost), and update everything on the site, giving stats of how much has been translated in a particular language. Very handy.

    Many thanks go to Tim Morley, the translator, who now finds himself having to respond to any Esperanto contact emails we get (sorry!) 🙂

    Spanish also went live recently (thanks Hugo!), Russian and Ukranian are both near completion, German is on its way, and there are others we’ve heard about too, like Greek and Polish.

  9. Well, I’ve got a bit of time…

    …and I think I’ve ignored the last two or three “hassle”s I’ve received. 🙂

    Over the last few weeks, I have been working on improving PledgeBank’s translations – Welsh has gone live, Russian is coming along very well, and German and Spanish too – and using feedback from the translators to help make the translation .po files complete (making sure they even include strings used in our JavaScript, and adding any strings accidentally missed out). I’ve also been working on HearFromYourMP, which though currently live and with initial MP messages, is having its “launch” on Monday in the Houses of Parliament – quite exciting. So today I’m watching out for anything going wrong (we moved the site to a brand new server yesterday), tweaking the text and emails, adding some more features, and making sure everything is generally ship-shape.