1. An award for the Great British Property Scandal

    Broadcast Digital Awards

    You may remember our recent post announcing that we’d been nominated for an Emmy and a BAFTA award. Always the bridesmaid and never the bride, you might have thought.

    But no! For, last night, The Great British Property Scandal picked up a Broadcast Digital Award for Best Multiplatform Project.

    As you’ll recall, we created the app and the website tools for the Channel 4 programme alongside production companies Tiger Aspect and the Project Factory, who share the accolade.

    So: yays all round – and don’t forget, our award-winning skills are for hire.

  2. mySociety – BAFTA and Emmy nominations

    empty homes spotter

    We won’t insist on being addressed this way, but you can now append ‘BAFTA and Emmy nominated’ to our name. We were very chuffed to be nominated for two television awards in the last month: the BAFTA for Digital Creativity in Television Craft, and the Emmy for best Digital Non-fiction Programme.

    ‘TV?’, you might be thinking, ‘I thought mySociety were all about digital stuff.’ Well, increasingly, of course, the lines are blurred. Television programmes come bundled with their own website, Twitter hashtag, or app. These days, TV is less about being a passive viewer, more about becoming part of an active, engaged conversation online.

    Last year, we worked with Channel 4 and TV production company Tiger Aspect to create the app and the website tools that accompanied their programme about empty houses – The Great British Property Scandal. A repurposing of the software that underlies FixMyStreet, the app enabled viewers to report empty homes; the site petition amassed 119k signatures – so the audience certainly got involved.

    We were, of course, delighted to have been recognised, along with C4 and Tiger Aspect. In the end, we didn’t need the space we’d hastily cleared on the mySociety mantelpiece, but as the BAFTA went to the incomparable Paralympics, we really can’t begrudge it.

    And of course, if you’re a TV company looking for help with your digital tie-ins, we’re happy to help.

  3. Fire, fire! Mapumental and fire engine journey times

    Image by William Murphy Mapumental can turn vast datasets into visual tools that everyone understands. Faced with highly complex, yet crucial data from the Fire Protection Association, we had a chance to really put our technology through its paces.

    Just how quickly could fire engines reach a given postcode in case of a fire? It’s a question that’s pivotal to decisions made by both the emergency services and the insurance industry.

    But previously, it has been a challenge to present the data simply, because it involves so many variables.

    Every region has its own factors, each of which will impact on fire engine response time. The number of vehicles at each station, the hours during which the station is manned, and the response policy of each individual fire authority will all play a part – and that’s before you even consider how geography might affect things.

    Dr. Jim Glockling is Technical Director at the Fire Protection Association and Head of the Risk Insight, Strategy and Control Authority (RISCAuthority), an organisation for the advancement of risk management within the fire and security sectors. Jim approached mySociety with this question: how could we map this crucial, yet complicated data in a way that could be understood by RISCAuthority members at a glance?

    It was clearly a job for Mapumental. Our transit-time mapping software was originally built to visualise public transport journey times, but its beauty is that ‘layers’ of data can be swapped out, allowing it to be used for all kinds of purposes.

    Read more about mySociety’s data visualisation services here.

    Assessing a property or postcode

    How quickly could 4 fire engines get to AL10 0XR in 10 minutes and 10 seconds?

    How quickly could 4 fire engines get to AL10 0XR in 10 minutes and 10 seconds? (Click to enlarge)

    And here’s the result of our pilot project. The maps on the right answer the following questions (click each image to see it at full size):

    How quickly could 4 fire engines get to AL10 0XR ?

    FPA: arrival times one engine

    FPA: arrival times one engine (click to enlarge)

    How does that change if the severity of the fire just requires one fire engine?

     A user inputs a postcode, and can assess exactly how quickly a fire could be tackled in that area. The different levels of severity are measured by how many response vehicles are required, and changes in this number are immediately reflected on the map.

    Assessing the general area

    FPA  - what it looks like when you have no postcode

    Which areas can four fire engines get to within 9 minutes 30 seconds at midday on a Saturday?

    It’s also possible to assess the region’s overall response capability, without inputting a postcode. The user sets severity levels (number of fire engines, or High Volume Pumping or Aerial Appliance (ladder) is needed), the time and day of the week.

    FPA - aerial response arrival within 15 minutes

    Where can an aerial appliance get to within 15 minutes at 2am on a weekday?

    The FPA tool immediately highlights the areas that are accessible within the chosen parameters, drawing on the underlying data of journey times and information such as vehicle numbers and hours of operation for each individual fire station in the region.

    Simplicity itself

    With RISCAuthority, we tested the concept using data from one fire authority –  Hertfordshire. mySociety’s task was to create a usable, elegant web interface that was as simple as possible to use, while still giving insurers the key data they needed.

    The project called on everything we knew about clean design, usability and data structures. A key part of what makes Mapumental’s data visualisation so intuitive are its sliders: this enables the user to quickly explore variables on a map.

    A tool with purpose

    Dr Glockling explains: “Whilst not necessarily used as a component of insurance pricing, this information helps insurers administer risk control and fire protection advice to their customers in the context of what the Fire and Rescue Services will be able to achieve on their behalf.”

    The response time is just one factor that insurance surveyors will take into account when they are assessing a building. “Where response and arrival times are not coherent with protecting the viability of the business in the event of fire, additional forms of in-built protection and control might be recommended, such as the installation of sprinkler systems.”

    “In the longer term it is hoped such information will impact beneficially on the annual cost of fire in the UK.”

    Results

    The pilot tool was well received by the FPA community, and the plan is now to work with RISCAuthority to roll it out to more fire authorities shortly, and then nationwide.

    Dr Glockling explains the pilot study helped them to understand two factors:

    • Would they get buy-in from both insurers and Fire and Rescue services on the viability and usefulness of the project?

    • Was it possible to present such a massive amount of data in a format that was readily palatable to the intended audience?

    He says, “Mapumental’s team displayed an immediate understanding of our requirement. Delivery was to time and the result has perfectly satisfied the de-risking ambition. The working relationships were very good throughout and we intend now to extend the pilot to full UK rollout.”

    During this phase, we will be inputting still more detail to the data, including information on the types of fire engine available to each region, and the plotting of fire stations on the map.

    The tool will be a valuable resource for the FPA and the insurance industry, and we really look forward to the roll-out later this year.

    Mapumental specialises in visualising complex geographic data sets on intuitive, easy to use map tools. If you have a data visualisation project that will benefit from Mapumental, just get in touch. Or read more about mySociety’s data visualisation services here.

    Photo by William Murphy (CC)

  4. Design thinking in Cape Town

    The Open Democracy Advice Centre (ODAC) in South Africa will be using mySociety’s Alaveteli software in their latest project – and, with a bit of match-making from mySociety, the preparation period has been rigorous.

    Alaveteli is our open source Freedom of Information platform. It underlies our own UK-based WhatDoTheyKnow, and right-to-know sites around the world. Alaveteli sites make it easy for citizens to ask questions of those bodies who operate under Freedom of Information law and, significantly, they automatically publish all responses.

    Dassies (rock-dwelling hyrax) by Arthur Chapman

    Before any coding or implementation began, we got ODAC together with the “Governance Collaboratory”, an initiative from the d.school in Stanford University that seeks to apply the “design thinking” approach to projects that intend to make government more open, more effective, and more accountable. We’ve observed quite a few Alaveteli installs, but while we’re always on hand to offer support and answer development queries, we’ve never prepared the ground quite like this.

    Gabriella Razzano of ODAC welcomed Jeremy Weinstein and Jenny Stefanotti, both from the d.school, to Cape Town for an intensive few days of assessing how the design thinking approach could shape the project. Two staff from mySociety also went along — Paul (our Head of International Projects) and Dave (one of our developers) — because we’re keen to understand how the d.school’s approach might improve the way we go about building our new projects.

    Now, at mySociety we already know a thing or two about building civic systems that engage with the public, because we have considerable experience in the field. We are expert at combining user experience and current tech to create simple, usable interfaces. We conduct usability tests, we apply A/B testing, and we think hard about what our analytics tell us. But actually much of this is reactive, iterative design: it’s being applied after the core product has already been built.

    Design thinking challenges this approach by suggesting that the user on which initial designs are often based is purely imaginary. As a result, the site inevitably includes the assumptions and prejudices of its creators. This won’t necessarily lead to a bad design — especially if the creators are benign and experienced — but it must fail, by definition, to account for the unexpected things that may motivate or concern actual users. The design thinking process attempts to change this by approaching the initial problem in a prescribed way and following a process that isolates genuine, existing requirements. This includes, in design thinking terms, processing the initial interviews into empathy maps from which requirements emerge, and which themselves become features that are rapidly prototyped in isolation from other parts of the system.

    This is uncomfortable for those of us used to building loose iterations from the bottom up and refining them later. It means introducing empathy and rapid, offline prototyping much earlier in the process than we’d normally expect. Certainly in the commercial world it’s common for a company to prototype against their target consumers early on. But for civic projects such as mySociety’s, it’s often much harder to identify who the users will be, for the impressive yet overwhelming reason that often we are building our platforms for everybody. This can lead to generalisations which may miss specific issues that could make a huge difference to some users.

    The d.school advocates a “learning by doing” way of teaching, so the days we spent in Cape Town were a busy mix of practice as well as theory. We interviewed people who had a variety of reasons to want to make Freedom of Information requests, including an activist who’s already used South Africa’s Freedom of Information legislation to make requests regarding housing projects, the head of a rape crisis centre, and law students who may well become a nation’s most empowered activists. From these interviews we isolated specific needs, which at this stage were nearly all unconnected to any digital or web requirement. Jeremy and Jenny then led us through the process of rapid, analogue prototyping intended to address those needs.

    Inevitably we could only scratch the surface in the few days we had available, but we hope ODAC will be able to apply the process to the development of their project, just as we aim to use it to benefit the work on ours.

    Image credit: Procavia capensis (Rock-dwelling Hyrax or Dassie) by Arthur Chapman, released under CC BY-NC-SA on Flickr.

    They tell visitors that dassies such as these live atop Table Mountain. We went up there and saw none. Similarly, Freedom of Information requests exist in South Africa under the Promotion of Access to Information Act 2000 (PAIA), but most people have never seen one — fewer than 200 PAIA requests were made nationally in 2012. This tenuous comparison allows us to illustrate the blog post with a cute picture of fuzzy mammals.

  5. Open311: Extended

    This is the third of our recent series of Open311 blog posts: we started by explaining why we think Open311 is a good idea, and then we described in a non-techie way how Open311 works. In this post we’ll introduce our proposed extension to Open311, and show how we use it in FixMyStreet.

    The crux of our suggested improvement is this: normal people want to know what has happened to their problem, and Open311 currently isn’t good enough at telling them whether or not it has been dealt with. To be more specific, our additions are all about reports’ status change, by which we mean something like this:

    That pothole?

    I just totally fixed it.

    That’s robot-311 from the previous post, if you’ve dropped in here without reading the previous posts. Once again we’re blurring the distinction between client and user (the girl you’ll see below) a little, to make things simpler to follow.

    Create→send→fix→update

    Every month in the UK, thousands of problems are reported on www.fixmystreet.com and, moments later, sent on to the councils who will fix them. Here’s what happens with a problem report for something like a pothole or a flickering streetlight:

    1. You create the report on FixMyStreet.
    2. FixMyStreet sends that report to the right department at the right council.
    3. That body puts it into its own back-end system.
    4. Later, when the council fixes the problem, FixMyStreet is updated, and everyone knows it’s fixed.

    On the face of it, you might think we need only care about 1 and 2. But really, FixMyStreet isn’t just about dispatching reports, it’s about helping to get things like potholes actually fixed. And neither citizens nor local governments benefit if work gets done but nobody finds out about it – which is part 4 on the list above.

    What do we mean by “status change”?

    The example at the top of the page shows the robot effectively changing a problem’s status to “fixed”.

    Actually, statuses can be simple, such as either OPEN or CLOSED, or more detailed, such as “under investigation”, “crew has been dispatched”, “fixed”, and so on. But since we’re only concerned here with the status changing, that specific vocabulary deployed doesn’t really matter – it can be anything.

    In situations where FixMyStreet is not integrated with council systems (i.e we just send email problem reports) FixMyStreet problems still frequently get marked as fixed, because anyone can change the status of a report just by visiting the page and clicking the button.  Obviously, though, we prefer to have FixMyStreet directly connected to the local government back-end databases, so that news of a fixed report can be automatically bubbled from the back-office up into FixMyStreet and out onto the net.

    And here’s where the problem lies: Open311 doesn’t quite support this business of getting problem updates from the back office out to the public. So first, we’ll show you how it can be done today, using Open311, and we’ll explain why this isn’t a good option. Then we’ll show our preferred solution, which we’ve proposed as an extension.

    Looking at everything just to spot one change (bad)

    One way to notice if any problems’ statuses have changed is to use Open311 to ask for every single service request, and see if any of them have a different status since the last time you checked.

    Tell me all the service requests you’ve ever received

    OK:

    request 981276 the pothole on the corner by Carpenter Street is now CLOSED (I filled in the pothole)

    request 988765 the pothole by bus stop on Nigut Road is now CLOSED (I filled in the pothole)

    request 998610 gaping hole at the end of Sarlacc Road is now OPEN (the pothole fell through)

    request 765533 where the street was cracked outside Taffey’s Snake Pit is now CLOSED (I filled in the pothole)

    . . .

    continues for thousands of requests

    Um, OK. Now I’ll look at all these and see if any have changed since I last asked *sigh*

    Obviously there are some problems with this. Even though Open311 lets you ask for quite specific service requests, you have to ask for all of them, because by definition you don’t know which ones might have changed. Remember, too, that problems can potentially change status more than once, so just because it’s been marked as CLOSED once doesn’t mean it won’t become OPEN again later. This exchange is very wasteful, very slow and ultimately (with enough reports) may become de facto impossible.

    Asking for just the changes (good)

    So here’s a better way of doing it. We’ve actually been doing this for some months, and now seems the time to share.

    The client asks the server for just the updates on a regular basis, so any requests that have recently changed get updated on FixMyStreet automatically, usually just a few minutes later.

    Have you changed the status of any of service requests today?

    Yes, request 981276 was CLOSED at 3 o’clock (I filled in the pothole)

    Or, more practically for keeping FixMyStreet up to date:

    Have you changed the status of any of service requests in the last 15 minutes?

    Nope.

    This is handled by our extension to Open311, GET Service Request Updates. There’s also an optional equivalent call for putting updates into the server (POST Service Request Update), which would apply if the client changed the status after the service request had been submitted.

    Note that the server identifies the problem with its own reference (that is, 981276 is the council’s reference, not a FixMyStreet ID, for example). This is important because not all these requests necessarily came from this particular client. Remember that all service requests are available through the Open311 GET Service Requests call anyway (as shown above). So the server doesn’t send each service request back in its entirety: just its ID, the new status, when it changed, and a brief description.

    In practice the client wouldn’t usually ask for “today”. In fact, we typically send a request asking for any updates in the last 15 minutes, and then at the end of the day ask for the whole day’s updates, just to check none were missed.

    The technical bit

    From a client’s point of view, this is simply an extra call like others in the Open311 API. So it’s just a request over HTTP(S) for XML (or JSON, if required).

    We deliberately make the client poll the server for updates and pull them in, rather than expecting the server to push updates out. This frees the server from any obligation to track which clients (for there may be more than just one) care about which updates. The requests themselves are sent with unique IDs, allocated by the server, so the client can dismiss duplicates. It’s also robust in the event of connection failures, so if there are timeouts or retry logic, that’s for the clients to worry about, not the server. Basically, this is all to make it as light on the server as possible: the only real issue is that it must be able to provide a list of updates. This usually means adding a trigger to the database, so that when a problem’s status is updated a record of that update is automatically created. It’s the table of those “service request update” records that incoming requests are really querying.

    We have published our own recommendation of this mechanism, which FixMyStreet already implements, alongside the FixMyStreet codebase.

    Is that it?

    Yup, that’s it.

    This extension is in addition to the Open311 specification — it doesn’t break existing implementations in any way. Obviously this means FixMyStreet’s Open311 implementation is compatible with existing Open311 servers. But we hope that others working on Open311 systems will consider our extension so that clients are kept better informed of the status of the problems being fixed.

    Why are statuses so important that it is worth extending the Open311 spec?

    mySociety didn’t originally build FixMyStreet because we wanted to get potholes fixed. We built it because we wanted nervous, politically inexperienced people to know what it felt like to ask the government to do something, and to be successful at that. We wanted to give people the buzz of feeling like they have a bit of power in this world, even if the most tiny amount.

    If the government fixes a problem and the citizen doesn’t find out it’s a double loss. The citizen becomes disillusioned and weakened, and the government doesn’t get the credit it is due. Everyone loses. We think that Open311 is a key mechanism for making large numbers of people feel that the government does respond to their needs. It just needs a bit of an upgrade to do it better. We hope very much that the wider community tests and endorses our extensions, and it can be folded in to the next official version of the Open311 standard.

    Find out more about FixMyStreet for Councils

  6. Open311: Explained

    In the previous blog post we explained why we think Open311 is a good idea. In this post we’ll explain what it actually does.

    Open311 is very simple, but because it’s fundamentally a technical thing it’s usually explained from a technical point of view. So this post describes what Open311 does without the nerdy language (but with some nerdy references for good measure). At the end there’s a round-up of the terms so you can see how it fits in with the actual specification.

    We’re using an unusual example here — a blue cat stuck up a tree — to show how applicable Open311 is to a wide range of problems. Or, to put it another way, this is not just about potholes.

    Cat up a tree and an Open311 robot

    So… someone has a problem they want to report (for this discussion, she’s using a service like FixMyStreet).

    There’s one place where that report needs to be sent (in the UK, that’s your council). That administrative body (the council) almost certainly has a database full of problems which only their staff can access.

    I have a problem :–(

    the “client”

    I fix problems!

    the “server”

    In this example, FixMyStreet is an Open311 client and the council is an Open311 server. The server is available over HTTP(S), so the client can access it, and the server itself connects to the council’s database. In reality it’s a little bit more complicated than that (for now we’ll ignore clients that implement only part of Open311, multiple servers, and decent security around these connections), but that is the gist of it.

    Although it’s not technically correct to confuse the client with the user, or the server with the council, it makes things a lot easier to see it this way, so we’ll use those terms throughout.

    Service discovery

    To start things off, the client can ask the server: what services do you provide?

    Until the client has asked the server what problems it can fix, it can’t sensibly request any of them.

    What services do you offer?

    I can:
    POT: fix potholes
    TELE: clean public teleports
    PET: get pets down from trees
    JET: renew jetpack licenses …

    FixMyStreet can use the response it gets from such a service discovery to offer different categories to people reporting problems. We actually put them into the drop-down menu that appears on the report-a-problem page.

    In the Open311 API, this is handled by GET Service List. Each service has its own service_code which the client must use when requesting it. Note that these services and their codes are decided by the server; they are not defined by the Open311 specification. This means that service discovery can easily fit around whatever services the council already offers. The list of services can (and does) vary widely from one council to the next.

    Service definitions

    Some services require specific information when they are requested. For example, it might be important to know how deep a pothole is, but it’s not relevant for a streetlight repair.

    Tell me more about the PET service!

    I can get pets down from trees, but when you request the service, you *must* tell me what kind of animal the pet is, OK?

    In the Open311 API, this is handled by the GET Service Definition method. It’s not necessary for a simple Open311 implementation. In fact, it only makes sense if the service discovery explicitly told the client to ask about the extra details, which the server does by adding metadata="true" to its response for a given service.

    Requesting a service

    This is where it gets useful. The client can request a service: this really means they can report a problem to the server for the body to deal with. Some submissions can be automatically rejected:

    My hoverboots are broken :–( I need BOOT service!

    404: Bzzzt error! I don’t fix hoverboots (use service discovery to see what I *do* fix)

    Hey! Blueblue is up a tree! I need PET service (for cats)!

    400: error! You forgot to tell me where it is.

    If the report is in good order, it will be accepted into the system. Open311 insists that every problem has a location. In practice this is usually the exact position, coordinates on planet Earth, of the pin that the reporter placed on the map in the client application (in this case FixMyStreet.com).

    I need PET service (for cats)! Blueblue is stuck up the biggest tree in the park :–(

    200: OK, got it… the unique ID for your request is now 981276

    In the Open311 API, this is handled by POST Service Request. You need an API key to do this, which simply means the server needs to know which client this is. Sometimes it makes sense for the server to have additional security such as IP address restriction, and login criteria that’s handled by the machines (not the user).

    Listing known requests

    The server doesn’t keep its reports secret: if asked, it will list them out. The client can ask for a specific report (using the ID that the server gave when the report was submitted, for example) or for a range of dates.

    Did anyone ask you for help yesterday?

    Yes, I got two requests:

    request 981299: TELE dirty teleport at the cantina (I’m waiting for a new brush)

    request 971723: POT pothole at the junction of Kirk and Solo (I filled it in)

    In the Open311 API this is handled by GET Service Request(s). The client can indicate which requests should be listed by specifying the required service request id, service code, start date, end date or status.

    Does Open311 work?

    Oh yes. On the Open311 website, you can see the growing list of places, organisations, and suppliers who are using it.

    The technical bit

    In a nutshell: Open311 responds to HTTP requests with XML data (and JSON, if it’s wanted). There’s no messing around with SOAP and failures are reported as the HTTP status code with details provided in the content body.

    You can see the specification for Open311 (GeoReport v2). It doesn’t feature blue cats, but if you look at the XML examples you’ll be able to recognise the same interaction described here. And remember the specification is an open standard, which means anyone can (and, we think, should) implement it when connecting a client and server in order to request civic services.

    Coming next…

    In the next blog post we’ll look at how FixMyStreet uses Open311 to integrate with local council systems, and explain why we’re proposing, and utilising, some additions to the Open311 specification.

    Find out more about FixMyStreet for Councils

    Illustrated especially for us by René Carbonell.

  7. Looking back: our experience of the Google Summer of Code

    Summer may seem like a long time ago, but despite the cold outside, we’ve been looking back over our participation in Google’s Summer of Code project. It’s almost enough to warm us up!

    This post is an attempt to record the process from our point of view. We hope it will be useful for other organisations considering participating next year, and for students who want to know more about how the scheme works.

    What is Google Summer of Code?

    It’s a programme sponsored by Google’s philanthropic arm, giving students the chance to experience real-life coding on open source software.

    The scheme is open to students all over the world, who are then paired up with open source organisations like us. The students gain paid work experience and mentoring; the organisations gain willing workers and some fresh new perspectives; the world gains some more open source code to use or develop further.

    Everyone’s a winner, basically.

    The beginnings

    2012 was our first year on the programme: once we had been accepted on the scheme, we were given two student slots – the maximum allowed for a first-time organisation.

    Given mySociety’s wide suite of codebases, there were several projects that could have benefited. We listed all our ideas, and let people apply for the ones they found appealing.

    Goodness, there were a lot of applicants! It was very heartening to discover that there is such an enthusiastic community of young coders all around the world – even if it did take us a long time to sift through them all and make our choices.

    You might remember our post back in May, when we announced that we’d made our choices. We were delighted to get working with Dominik from Germany and Chetan from India.

    The project

    As things turned out, our students ended up working on a project that wasn’t even on our original list: PopIt, our super-easy ‘people and positions’ software.

    That’s because once we spoke to our chosen students, we realised they had the skills that could really help us forge ahead with this project – and once we discussed it with them, they were keen. So PopIt it was.

    Logistics

    Germany and India are a bit of a commute away, but fortunately development work can be managed remotely. We know this particularly well at mySociety: our core team work from home and are scattered across the UK.

    The only difference here was the 6+ hour time difference between us and India: it was important to be rigorous about checking in at times when Chetan would be awake!

    We communicated via IRC (instant chat), email, and occasionally Skype, and it all worked well.

    Edmund, the team member chosen to be mentor, broke the required tasks down into big pieces so that the students would have realistic work units of several days each.

    What was achieved

    PopIt is primarily a tool for helping people create and run parliamentary monitoring websites (like TheyWorkForYou) with minimal coding knowledge/effort, though we anticipate that it will have many other uses too.

    Our students spent the first half of the summer learning and improving the PopIt codebase. Once they were confident in it, they created their own sites using PopIt as a datasource to test the API, and, hopefully, create a valuable reference resource for the community.

    Dominik added a migration tool to PopIt, which lets you upload data as a CSV. This means that you can start a site with a database of names, positions and dates at its heart – within seconds.

    His test site was a professors’ database (the code is here). Dom also wrote some helpful posts on the dev blog like this one.

    Chetan created an image proxy that lets us serve images in a smart way that makes sense for APIs. His test site was for Indian representatives (here’s the code).

    Neither site is being maintained now, which just confirms that it is harder to run a site than to start it. This is not a failing, though. The creation of these sites, along with Chetan and Dom’s feedback, helped us understand where improvements needed to be made. In the course of one summer, PopIt became much more mature.

    Looking back on the Summer of Code

    Edmund attended a follow-up ‘mentors’ summit’ at the Googleplex in California – he found it very helpful to compare notes with other organisations and find out what had worked best for them all, and he made some good contacts too.

    Assuming we get the chance again, would we participate in 2013? Our experience was very positive, but as yet we are undecided, purely because of the fluid nature of our workflow: we don’t yet know whether time and resources will permit.

    Obviously, we have enjoyed great benefits from the scheme, but that has depended on quite a bit of input from our side, and we need to be sure that we can ensure that happens again.

    Edmund has compiled a list of advice, from the practical (ask students to treat the placement like a full-time job; test coding skills before acceptance) to the desirable (a weekly blog post from participants; make sure you over-estimate the time you’ll spend mentoring). If you’re thinking of participating next year, he’d be happy to pass on his tips for ensuring that you, and your assigned students, get the best out of the Google Summer of Code. Just drop him a line.

  8. Dear Ninja Campaigner Geek: Why I work on non-partisan tech, and why I encourage you to take a look

    The last few weeks since the US election have seen an explosion in articles and blog posts about how Obama’s tech team pulled out the stops in their race against the Republicans. It’s been an exciting time to learn about the new techniques dreamed up, and the old ones put to the test.

    For those of us who develop non-partisan services to help people report broken street lights or make Freedom of Information Requests, such stories certainly seem unimaginably glamorous: I don’t think any of my colleagues will ever get hugged by Barack Obama!

    But it has also been an interesting time to reflect on the difference between choosing to use tech skills to win a particular fight, versus  trying to improve the workings of the democratic system, or helping people to self-organise and take some control of their own lives.

    At one level there’s no competition at all: the partisan tech community is big and economically healthy. It raises vast amounts of cold hard cash through credit card payments (taking a cut to pay its own bills) and produces squillons of donors, signers, visitors, tweeters, video watchers and so on. The non-partisan tech community is much smaller – has fewer sustainable organisations, and with the exception of some big online petitions, doesn’t get the same sort of traffic spikes. By these metrics there’s absolutely no doubt which use of tech is the most important: the partisan kind where technology is used to beat your opponent, whether they are a political candidate, a policy, company, or an idea.

    But I am still filled with an excitement about the prospects for non-partisan technologies that I can’t muster for even the coolest uses of randomized control trial-driven political messaging. The reason why all comes down to the fact that major partisan digital campaigns change the world, but they don’t do it in the way that services like  eBay, TripAdvisor and Match.com do.

    What all these sites have in common – helping people sell stuff they own, find a hotel, or a life partner – is that they represent a positive change in the lives of millions of people that is not directly opposed by a counter-shift.  These sites have improved the experience of selling stuff, finding hotels and finding life partners in ways that don’t attract equal and opposite forces, driven by similar technologies.

    This is different from the case of campaigning tech: here a huge mailing list is pitted against another even huger mailing list. Epic fund-raising tools are pitched against even more epic fund-raising tools. Orca vs Narwhal. Right now, in US politics, the Democrats have a clear edge over the technology lined up against them, and I totally understand why that must feel amazing to be part of. But everything you build in this field always attracts people trying to undo your work by directly opposing it. There is something inate to the nature of partisanship which means that one camp using a technology will ultimately attract counter-usage by an opposing camp.

    This automatic-counterweighting doesn’t happen with services that shift whole sectors – like TripAdvisor did. In the hotel-finding world, the customer has been made stronger, the hotel sector weaker, and the net simply doesn’t provide tools to the hotel industry to counter what TripAdvisor does.

    It is this model – the model of scaleable, popular technology platforms that help people to live their lives better – that I aspire to bring to the civic, democratic and community spheres in my work. Neither I nor mySociety has yet come up with anything even remotely on the scale of a TripAdvisor, but there remains the tantalising possibility that someone might manage it – a huge, scaleable app of meaningful positive impact on democratic, civic or governance systems*. Our sites are probably about as big as it gets so far, and that’s not big enough by far.

    If someone does manage to find and deliver the dream – some sort of hugely scalable, impactful non-partisan civic or democratic app & website, it is unlikely that the net will instantly throw up an equal and opposite counterweight. There is a real possibility that the whole experience of being a citizen, the whole task of trying to govern a country well will be given a shot in the arm that won’t go away as soon as someone figures out how to oppose it. I’m not talking Utopia, I’m just talking better. But what motivates me is that it could be better for good, not just until the Other Team matches your skills.

    And that – in rather more words than I meant to use – is why I am still excited by non-partisan tech, and why I really hope that some of the awesome technologists who worked in the political campaigns of 2012 get involved in our scene.

    I’m on Twitter if anyone wants to talk about this more.

    *  You can certainly make an argument that Twitter and Facebook sort-of represent non-partisan democracy platforms that have scaled.  But some people disagree vehemently, and I don’t want to get into that here.

     

     

     

  9. Mapumental’s Secret Sauce: A Map Overlay Rendering Technology You Might Find Interesting

    I am Duncan Parkes,  a developer for mySociety, a non-profit full of web geeks. One of the things we try to do well here is to take complicated data and turn it into really usable tools – tools which are attractive to people who aren’t web (or data) geeks.

    For some considerable time I’ve been working on Mapumental – a project that is about turning public transport timetable data into pretty, interactive maps featuring isochrones, shapes that show people where they can live if they want to have a commute of a particular time. You can play with the new version we just launched here. That particular map shows the commuting options to where the Queen lives. Slide the slider for full effect.

    There are a couple of hard problems that need solving if you want to build a service with an interactive journey times overlay like this. You need to be able to calculate a *huge* number of journeys extremely quickly, and you need to be able to make custom map layers so that it all looks nice. But what I think might be most interesting for you is the way in which the contours get rendered on top of the maps.

    It all started about three years ago, when the first version of the app – co-developed with the geniuses at Stamen – used Flash/Flex to draw contours on the maps, and to let people play with them. You can still play with a couple of versions of that technology from way back in 2007, that is, unless you’re using an iPad or iPhone, which of course don’t do Flash.

    Colour Cycling

    What was going on inside this Flash app was as follows. We needed to show the user any one of hundreds of different combinations of journey times (5 minutes, 12 minutes, 56 minutes, etc) depending on where they set the slider. Sending each one from the server as a tiled map overlay would be dead slow. Even Google – who have chosen to send new tiles each time – end up with a service which is surprisingly slow (try choosing a different time on this map).

    With some help from Stamen, we decided that the way of making it possible to show many different contours very quickly was send the client just one set of tiles, where each tile contained all the data for a variety of journey times. What does that mean? Simple: each colour in the tile represented a different number of minutes travelling on the map. So a batch of pixels that are colour X, all show places that are 15 minutes from the centre of the map.

    So, in this old Flash system, when you slide the slider along, the Flash app makes some of the coloured pixels opaque, and the others transparent. It was, in short, a form of colour cycling, familiar to lovers of 8 and 16 bit computer games.

    However, from about 2010 onwards, the march of iOS spelt the end of Flash. And that meant that we couldn’t launch a shiny new site based on this technology, as lovely as it was. We had to work out some approach that would use modern web standards instead.

    The Death of Flash Makes Life Difficult – for a while

    How do we replicate the experience of dragging a slider and seeing the map change like in the original Mapumental demo, but without Flash? One of the things that made the original Mapumental nice to use was how smooth the image changes were when you dragged the slider. Speed really matters to create that sort of organic effect that makes the demo so mesmerising.

    So as we started to tackle the question “How do we make this work in a post-Flash world?”. And the first thought was “Let’s do away with those map tiles, filled with all that journey time data!”. After all – why send any tiles to a modern browser, if it can just render nice shapes on the fly?

    So we had a go. Several goes. At first we tried rendering SVG circles around each public transport stop – but that was too slow, particularly when zoomed out. Then we tried rendering circles in Canvas, and whilst that was OK in sparsely populated places it sucked in the cities, where people would actually want to use it.

    Eventually, we decided that as wonderful and powerful as modern web rendering techniques are, if you exclude WebGL (on the grounds that so few people have it still), all the current techniques result in pages that just hammer your browser, whilst producing an experience which isn’t up to our previous Flash-based standards. To see this in action, see the wonderful Mapnificent site, developed by the super talented Stefan Wehrmeyer. He’s a great guy and a friend of mySociety, but the Javascript circle rendering just grinds, and that’s with far, far fewer data points than we have in our system. (Sorry for potentially crashing your browser there. This is in the interest of Science.)

    Back to Colour Cycling – Using Web Standards

    So, we thought, why not look again at colour cycling the pixels within pre-rendered map tiles? After all, there are some examples out there, like this waterfall, all in Javascript.

    So, I had a bit of a look at the waterfall. It seems to work by holding in memory a structure which has all the pixels which change and all the colours they should change to and when. This works beautifully for the waterfall picture, but only a limited number of the pixels in that image actually change colour, and the image is quite small. For a full screen web browser with a big map in, this didn’t seem promising, although I’d love to see someone try.

    Then I thought: browsers have always been very good at displaying images quickly – that’s sort of vital. Perhaps we could get our tile generating server to output PNG images where, as before, the colours represent travel times, but using a palette. Then by putting this in a canvas layer in the JavaScript mapping framework Leaflet, and by changing the palette of the images on the canvas as the slider is dragged, we could get our animation.

    Unfortunately, there is no way to change the palette of an image that you’ve put on the canvas. In fact, there’s no way to change the palette of an HTML img element: all you can do is assign it a new src attribute.

    But this gets back to the original problem – we don’t want to download new mapping for every different position on the time slider. We definitely can’t afford to have the client downloading a new image source for every tile whenever the slider is moved, so we had to find a way to make that src at the client end and get that into the src attribute.

    The Breakthrough – Data URIs and Base64 encoding

    So we started trying data URIs. For those of you not familiar, these allow you to put a whole object into your HTML or CSS, encoded in Base64. They’re commonly used to prevent pages having to make extra downloads for things like tiny icons.

    So I thought, “Here’s a way we can set an image src in JavaScript to something we’ve calculated, rather than something we’ve downloaded.” And this turned out to be the key insight that allows for the relatively smooth, attractive overlays you see in Mapumental today.

    My new plan was that the client, having downloaded each palette-based image, would make a Base64 encoded version of it, which it could then use to build a version with the right palette and assign this as a data URI of the tile.

    However Base64 encoding all these images in the JavaScript seemed like unnecessary processing to do there, so the final evolution of this technique was to do the Base64 encoding at the tile server end, and while we’re at it, not to bother sending over the parts of the image that we always replace at the client end.

    So in summary, what we built does this:

    1. The server calculates the journey times and renders them to palette-based tiles.
    2. It sends these to the client, encoded in Base64, and with the initial bits up to the palette and transparency chunks removed.
    3. At the client end, we have a pre-prepared array of 255 ‘starts’ of PNGs that we combine with the later parts of the ’tiles’ from the tile server to make data URIs.
    4. When you drag the slider it combines the appropriate ‘start’ of a PNG with the bulk of the tile that has been downloaded from the server, and assigns that to the src attribute of the tile.

    And that’s how the nice overlays on Mapumental work. But as so often in coding, the really interesting devil is in the detail – read on if you’re interested.

    Diving into Base64 and the PNG file format – The Gnarly Bits

    So – why are there 255 of these ‘starts’ of these PNGs, and what do I mean by a ‘start’ anyway?

    PNG files are divided up into an 8 byte signature (the same for every PNG file) and a number of chunks, where each chunk consists of 4 bytes to tell you its length, 4 bytes of its name, some data, and 4 bytes of cyclic redundancy check. In this case, what I call a ‘start’ of a PNG is the 8 byte signature, the 25 byte of the IHDR chunk, and the PLTE (palette) and tRNS (transparency) chunks. The PLTE chunk has 12 bytes of overhead and 3 bytes per colour, and the tRNS chunk has 12 bytes of overhead and 1 byte per colour.

    Base64 encoding is a way of representing binary data in text so that it can be used in places where you would normally expect text – like URIs. Without going into too much detail, it turns groups of 3 bytes of binary gumpf into 4 bytes of normal ASCII text without control characters in it, which can then be put into a URI.

    Why do we have 255 colours, rather than the maximum 256 which are available in a palette? Because we need the break between the end of the tRNS chunk and the start of the IDAT chunk in the PNG file to align with a break between groups of three bytes in the Base64 encoded image. We need the length of these starts to be a multiple of 3 bytes in the original PNG format, which translates into a multiple of 4 bytes in the Base64 encoded version, so we can cut and shut the images without corruption.

    Which just goes to show that even though web GIS technologies may feel like they are approaching a zenith of high level abstraction, there’s still some really gnarly work to be done to get the best out of current browsers.

  10. Mapumental Property Launches

    If you’ve been following mySociety for a while, you’ll know that we have been interested in making maps that show commuting times for several years.

    However, we’ve never made public a simple, free, useful version of our slidy-swooshy Mapumental journey times technology. Until today.

    Today we pull the wraps off Mapumental Property , a house-hunting service covering England, Scotland and Wales, designed to help you work out where you might live if you want a public transport commute of a particular maximum duration. Have a go, and we guarantee you’ll find it an oddly compelling experience.

    We think it’s a genuinely useful tool – especially since unlike some of the other players in this space, we’ve got all the different kinds of public transport, right across the whole of Great Britain. We hope that some of you will find it helpful when deciding where to live.

    However, this launch doesn’t mean mySociety is bent on taking over the property websites sector. Mapumental Property isn’t a challenger to the likes of Rightmove, it’s a calling card – an advertisement for our skills – which we hope will help mySociety to attract people and organisations who want beautiful, useful web tools built for them.

    In particular we’d like people interested in Mapumental to note that:

    • We like to build attractive, usable web tools for clients of all kinds.
    • We know how to use complex data to make simple, lovely things.
    • We can do some mapping technology that others haven’t worked out yet.

    If any of that is of interest, please get in touch, or read about our software development and consulting services.

    I’d like to thank quite a few people for helping with this launch. Duncan Parkes was the lead developer, Matthew Somerville ably assisted. Jedidiah Broadbent did the design. The idea originally came from the late Chris Lightfoot, and me, Tom Steinberg. Francis Irving built the first version, and Stamen came up with the awesome idea of using sliders in the first place (and built some early tech). Kristina Glushkova worked on business development, and Zoopla’s API provides the property data. I’m also grateful to Ed Parsons of Google for very kindly giving us a hat tip when they built some technology that was inspired by Mapumental.  Thanks to everyone – this has been a long time coming.

    We’ll follow up soon with a post about the technology – and in particular how we got away from using Flash. It has been an interesting journey.