1. We’re still offering help!

    11 months ago we put up a post offering free development time for people wanting to reuse our software.

    Since then we’ve had loads of responses and helped a few dozen people take their first steps. But this week, in a short and succinct post, I wanted to remind you that this offer still stands.

    There are some caveats.

    In order to qualify, you must be a group or an individual who can show us that you have a desire to run online civic and democratic projects like FixMyStreet or WhatDoTheyKnow in the long term, and that you have access to some kind of web developer skills. You can be anywhere in the world bar the USA (apologies US people, our funding won’t cover your country :( ).

    What does commitment mean? Nothing impossible, but to make a project successful you will need a few things:

    You need long-lasting enthusiasm. We’ll be looking to make sure that you understand the ongoing time and energy commitments a project like this will involve. The technical set up may be easy, but there is a lot of data that needs to be collected. There’s also awareness raising, user support and general love for the site that you’ll need to keep things going. Things start slowly…You have to give them attention to drive usage!

    You may need access to a web developer – at least sometimes. While these kinds of sites do, to some extent, run themselves, some work will always be necessary to keep them running smoothly. And while our developers will help you get your site off the ground, you will need your own developer too, both at set-up, and as the site continues to run.

    If you don’t have access to a developer, or you’re an NGO that’s doing a wider project of which this could be a small part. We’re also happy to talk to you to see if we can still help! Either way,  just fill in this form to get in touch and give us some information about your project!

    Image credit: Building by bartb_pt from Flickr under the creative commons licence.

  2. New: Get free developer time to set up FixMyStreet or Alaveteli in your country, city or region

    Free Sign by Ken Hawkins

    Would you like to run a site like WhatDoTheyKnow or FixMyStreet in your own country, but have worries that you won’t be able to get them running and correctly adapted to your local situation?

    If so, mySociety has some news of an offer that may interest you.

    As of May 2013 we will be offering free technical time from mySociety’s developers to a limited number of people and organisations who want to get versions of Alaveteli (Freedom of Information requests) and FixMyStreet (street problem reports) working, anywhere in the world.

    All you have to do to be considered is to send us a message expressing an interest in gaining our support, and telling us a bit about you and what you hope to achieve.

    If you’re selected, we’ll help modify the software to make sense in relation to your own region’s laws or local authority’s systems, and we’ll even host the service if that is a problem.

    More importantly, we’ll help to explain how the software works at a technical level, so you or a local developer can really understand how the open source code works, and how to make changes to it.

    This service from mySociety is worth thousands of dollars a time. We are offering it because we think it is important to support people who have the enthusiasm, but perhaps not the means, to run a service like FixMyStreet or WhatDoTheyKnow.

    In order to qualify, you must be a group or an individual who can show us that you have a desire to run online civic and democratic projects like FixMyStreet or WhatDoTheyKnow in the long term, and that you have access to some kind of web developer skills. You can be anywhere in the world.

    What does commitment mean? Nothing impossible, but there are a couple of requirements.

    You need long-lasting enthusiasm. We’ll be looking to make sure that you understand the ongoing time and energy commitments a project like this will involve. To put it frankly, we don’t want to invest in a project that may close down after a few months. So, we’ll want to have a chat to ensure that you really know what you’re getting into.

    You need access to a web developer – at least sometimes. While these kinds of sites do, to some extent, run themselves, some work will always be necessary to keep them running smoothly*. And while our developers will help you get your site off the ground, you will need your own developer too, both at set-up, and as the site continues to run.

    But don’t let that put you off – we also want to hear from you even if you haven’t yet got a group in place. The important thing is that you have the desire and the motivation to drive a project to completion.

    Interested? Drop us a line now and let’s talk. Don’t forget to tell us what country, city or region you’re interested in covering, and what resources you can contribute to making your site into a success.

    * See the following resources to understand what sort of work is involved in running a civic or democratic website:

    Interview with David Cabo, who runs a Right To Know site in Spain on our Alaveteli platform

    Blog post by Richard Taylor, volunteer on WhatDoTheyKnow.

    Photo by Ken Hawkins (CC)

  3. Extracting Boundaries from OpenStreetMap – Part 2


    Hadrian's Wall by Joe Dunckley

    This is the second part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML files for every useful administrative and political boundary in OpenStreetMap.

    The previous post on this subject described how to take the ID for a particular relation or way that represents a boundary in OpenStreetMap, and generate a KML file for it. That’s much of the work that we needed to create MapIt Global, but there are a few more significant steps that were required:

    Efficiently extracting boundaries en masse

    The code I previously described for extracting a boundary from OpenStreetMap used a public Overpass API server.  This is fine for occasional boundaries, but, given that there are hundreds of thousands of boundaries in OSM, ideally we don’t want to be hitting a public server that many times – it puts a large load on that server, and is extremely slow. As an alternative, I tried parsing the OSM planet file with a SAX-based parser, but this also turned out to be very slow – multiple passes of the file were required to pick out the required nodes, ways and elements, and keeping the memory requirements down to something reasonable was tricky. (Using the PBF format would have helped a bit, but presented the same algorithmic problems.) Eventually, I decided that a better approach was simply to set up a local Overpass API server, and to query that.  This is a great improvement – it allows very fast lookups of the data we need, and the query language is flexible enough to be able to retrieve huge sets of relations and ways that match the tags we’re interested in.  It also means we would no longer have problems if connectivity to the remote server went down.

    Another question that arose when scaling up the boundary extraction was, “Which set of tags we should consider as boundaries of interest?” On the first import, we considered any relation or way with the tag boundary=”adminstrative” and where the admin_level tag was one of “2”, “3”, “4”, … “11”.  At the time, there were about 225,000 such elements that represented closed boundaries. Afterwards, it was pointed out to me that we should also include elements with the tag boundary=”political”, which includes parliamentary constituencies in the UK, for example.  For later import purposes, I gave each of these boundary types a 3-letter code in MapIt, which are as follows:

    • O02 (boundary="administrative", admin_level="2")
    • O03 (boundary="administrative", admin_level="3")
    • [...]
    • O11 (boundary="administrative", admin_level="11")
    • OLC (boundary="political", political_division="linguistic_community")
    • OIC (boundary="political", political_division="insular_council")
    • OEC (boundary="political", political_division="euro_const")
    • OCA (boundary="political", political_division="canton")
    • OCL (boundary="political", political_division="circonscription_législative")
    • OPC (boundary="political", political_division="parl_const")
    • OCD (boundary="political", political_division="county_division")
    • OWA (boundary="political", political_division="ward")


    Importing Boundaries into MapIt

    The next step in building our service was to take the 236,000 KML files generated by the previous step and import them into MapIt.

    The code that creates the KML file for an element includes all its OpenStreetMap tags in the <ExtendedData/> section.  On importing the KML into MapIt, there are only a few of those tags that we’re interested in – chiefly those that describe the alternative names of the area.  We have to pick a canonical name for the boundary, which is currently done by taking the first of name, name:en and place_name that exists. If none of those exist, the area is given a default name of the form “Unknown name for (way|relation) with ID (element ID)”. There are also tags for the name of a country in different languages, which we also import into the database so that localized versions of the name of the boundaries will be available through MapIt with their ISO 639 language code.

    Another tricky consideration when importing the data is how to deal with boundaries that have changed or disappeared since the previous import. MapIt has a concept of generations, so we could perfectly preserve the boundaries from the previous import as an earlier generation. This would certainly be desirable in one respect, since if someone is depending on the service they should be able to pick a generation that they have tested their application against, and then not have to worry about a boundary disappearing on the next import. However, with quarterly imports the size of our database would grow quite dramatically: I found that approximately 50% of the boundaries in MapIt Global had changed over the 5 months since the initial import. Our proposed compromise solution is that we will only keep the polygons associated with areas in the two most recent generations, and notify any known users of the service when a new generation is available for them to test and subsequently migrate to.

    For reference, you can see the script that extracts boundaries and generates the KML files and the Django admin command for importing these files into MapIt.

    The end result: MapIt Global

    The aim of all this work was to create our now-launched web service, MapIt Global. As far as we know, this is the only API that allows you to take the latitude and longitude of any point on the globe, and look up  the administrative and political boundaries in OpenStreetMap that surround that point. We hope it’s of use to anyone trying to build services that need to look up administrative boundaries – please let us know!

    Photo credit: Hadrian’s Wall by Joe Dunckley,

  4. Installing FixMyStreet and MapIt

    A photo of some graffiti saying "SIMPLE"

    Photo credit: duncan on Flickr

    One of the projects we’ve been working on at mySociety recently is that of making it easier for people to set up new versions of our sites in other countries. Something we’ve heard again and again is that for many people, setting up new web applications is a frustrating process, and that they would appreciate anything that would make it easier.

    To address that, we’re pleased to announce that for both FixMyStreet and MapIt, we have created AMIs (Amazon Machine Images) with a default installation of each site:

    You can use these AMIs to create a running version of one of these sites on an Amazon EC2 instance with just a couple of clicks. If you haven’t used Amazon Web Services before, then you can get a Micro instance free for a year to try this out. (We have previously published an AMI for Alaveteli, which helped many people to get started with setting up their own Freedom of Information sites.)

    Each AMI is created from an install script designed to be used on a clean installation of Debian squeeze or Ubuntu precise, so if you have a new server hosted elsewhere, you can use that script to similarly create a default installation of the site without being dependent on Amazon:

    In addition, we’ve launched new sites with documentation for FixMyStreet and MapIt, which will tell you how to customize those sites and import data if you’ve created a running instance using one of the above methods.

    These documentation sites also have improved instructions for completely manual installations of either site, for people who are comfortable with setting up PostgreSQL, Apache / nginx, PostGIS, etc.

    Another notable change is that we’re now supporting running FixMyStreet and MapIt on nginx, as an alternative to Apache, using FastCGI and gunicorn respectively.

    We hope that these changes make it easier than ever before to reuse our code, and set up new sites that help people fix things that matter to them.

    Photo credit: duncan

  5. mySociety design tip: search box landing pages

    We use Google analytics to understand how people use, and find, our sites. This blog post is a summary of one deceptively simple way we use this information to get more people to use TheyWorkForYou, our UK parliamentary tracking website.

    mallard landing

    "Mallard landing" (at Slimbridge) by LHG Creative Photography (CC BY-NC-ND licensed from Flickr)

    One technical term: landing pages are the pages on a website that a user first arrives on from a particular link or search result. In this case, we created explicit landing pages. That is, we made these pages solely to be the target of search results. They serve no other purpose in the structure of the site; they are like super-focussed homepages. If you’ve read the blog post about minimising clicks on the homepage you’ll see that this is a clear example of that principal in action.

    We added two specific landing pages to TheyWorkForYou — here’s one of them: http://www.theyworkforyou.com/parliament/

    This page exists for a simple reason: we know that someone who uses the word “parliament” when searching on Google is probably looking for something that TheyWorkForYou can help with. By creating the “parliament” landing page, we are able to offer a page that is more likely to serve them. Incidentally, we try to make sure that searches lead to the landing page by using Google Adwords, and encouraging the Google bots to associate the page with the keywords, and other things too — but those are not the details this tip is about. Instead, look what we’ve put on that page. You’ll see it includes a set of useful explanations about parliament (it’s very common for people not to know how their parliament works), as well as a search box.

    That search box is the same one that appears on every other page on the website, in the top right-hand corner.

    But on this landing page, we’ve changed the prominence of the search box by putting it right in front of your eyes. And we added the caption: “Search for any word or phrase – see if it’s been mentioned in Parliament”.

    Yes, there’s no technical magic here. No special code. It’s our regular search box. The difference is that on this page we know the user is thinking about parliament, and that search box does indeed search extensive records of what’s been happening in parliament. If we didn’t do this, we could leave the user to figure it out for themselves. They might of course… or they might give up. As you’ll have noticed if you’ve read the previous design tips, we don’t want users to give up, ever.

    We did it again for “Hansard”. What’s Hansard? It’s the name of the transcript of all UK parliamentary debates, published by Parliament itself. Someone who knows about Hansard might be searching for it (on Google, say) and not realise that TheyWorkForYou has the entire transcript available for search. So we added another landing page. Again, it’s the same search that’s on all the other TheyWorkForYou pages, but now — for a user who’s already identified as someone who knows that Hansard exists — it’s clear that the search will do what they expect.

  6. mySociety design tips: avoiding duplicate messages


    One of the common elements you will find across mySociety’s sites is that they have features designed to reduce duplicate messages or reports being sent to politicians, governments or companies. We often do this in quite a subtle way, so it is worth spelling out here how we do this across several sites:

    • If you start to report a broken street light or pothole on FixMyStreet, you’ll see local problems before you start to type in your own details. This means if the problem is already there, you can see before you waste any effort.
    • If you use WhatDoTheyKnow to send a Freedom of Information request to a public body, we provide a facility which encourages users to search through other people’s requests before they type their new request in.
    • If the 08:10 train you take to work is always late, when you go to report it on FixMyTransport, we show you all the other problems already reported on that route. If someone else has already set up a page, you can press the big green ‘join’ button, and show your support.
    • If more than a handful of people try to use WriteToThem to send an exact duplicate of the same message to a politician, it will prevent it. This is because we know that politicians listen much, much more to individual messages from constituents than bulk mails.

    This pattern – trying to intervene before people write identical messages or reports – is a design decision that makes a big difference to the way these sites operate. As usual with mySociety sites, this little feature seems like the sort of thing that would be quite tempting to skip when building a copy. But it really matters to the long term success of the sites. There are three reasons why.

    First, there is a simple public benefit that comes from saving time. There’s no point us wasting your time if a report or request has already been sent, especially around minor issues. Saving your users time makes them happier and more likely to enjoy their experience.

    Second, if you can spot that someone is about to send a duplicate message, we may be able to encourage that user to support the existing report instead of making a new one. For example, on FixMyStreet you can add an update to an existing pothole report (“it’s getting worse!”).

    This feature is most visible, and most mature, on FixMyTransport, where users are clearly encouraged to ‘support’ pre-existing reports, rather than making new copies.  By discouraging duplicate reports, we let people with a shared problem work together, even if this only means adding themselves as a “supporter” and doing nothing else. We know that many people search for, and find, problem reports which have turned into these little campaigns, which they then join and help. So even if they are only reading them (not joining them) that exposure can have some value to the people affected. This would be diluted if we created lots of similar reports about the same problem.

    Third, we discourage duplicates for the benefit of the governments and companies receiving messages. We don’t think FixMyStreet is effective because it lets people moan: we think it’s effective because it helps local government to be effective by giving them good quality reports about local problems, in formats that area easy to handle. This good quality reporting increases the chance that the government will understand the problem and act on it, which leads to our main goal – citizen empowerment. Recipients are unlikely to help users if many of the messages they get are confused, inaccurate or duplicates, so we work on all these fronts.

    So if you haven’t thought about this before, notice how the “work flow” through our sites makes you see similar problems before you’ve finished reporting your own. This is the implicit way to prevent duplication. We don’t have “Stop! Warning! Check this is a new problem!” messages, because we never want to discourage genuine users. But the careful design of the interface gently discourages, successfully, duplicate reports, and encourages supporting of other items.

    It’s never possible to entirely prevent duplication. But we try hard, because it’s always better to join people together around common causes, than it is to let them struggle alone.

  7. mySociety design tips: data collection is not just filling in forms

    When a user on FixMyTransport tells us about a journey they’ve had, we ask them if it was on a bus, train, underground, or ferry. This is a simple question, but of course our lead developer Louise thought carefully about how to ask it.

    As programmers, we sometimes approach collecting data from the user just like filling in a form — a paper form. So, we could ask the bus/train/ferry question by using a list to select from (in HTML, that’s a select tag) — perhaps spread out, or as a drop-down list — or maybe as a set of radio buttons. But in this case we have deliberately broken away from idea of the form.

    Instead, FixMyTransport uses four big buttons. They look like this:

    Effectively, this means we have a whole webpage devoted to one single question. That’s perhaps not what you’d expect if you’re building an online form. Often it seems easier not to break a task across several pages. But here we have a single page with a single question on it.

    There are some important things to note about this page:

    • the four buttons are big and colourful and beautiful (we can thank Supercool for this, who did the design work on the initial FixMyTransport site)
    • it’s a very easy question, even though the answer is critical (because it affects the kind of data we need to ask next)
    • it’s a reasonable question — the user isn’t surprised or confused to be asked it

    This page has been very successful. We know this because we study our web analytics (that is, how people use our sites) as well as running usability testing. It’s true to say nobody gets stuck on this page, nobody drops out, and in fact most people don’t even think about it (that’s the “reasonable” thing, above).

    After the latest session of FixMyTransport usability testing, run by our developer Mark, we had a discussion with other team members. We agreed this was one of our favourite FixMyTransport pages. It’s pretty, it does the job, it moves the user forward (see the earlier blog post about “why the FixMyStreet homepage asks one easy question”)… and best of all it shows that some of us were wrong to be cautious about introducing another step into the problem-reporting process.

    Incidentally, Mark recommends the book Rocket Surgery Made Easy if you’re interested in running your own usability tests.

    In summary, sometimes a super-simple choice is strong enough to be presented as a single webpage all by itself. Don’t fall into the trap of thinking web forms should be like paper forms.

  8. mySociety design tips: how we choose the best map zoom level

    When you report a problem on FixMyStreet.com, the site displays a map for you to click on to indicate its exact location. Of course, you can zoom in and out of that map, but when it is first displayed, FixMyStreet needs to use an initial ‘default’ zoom level. Ideally, this is a zoom level that reduces the number of clicks required before a user can pinpoint the location of their problem.

    “In the middle of America” by Kathie M Ceballos (CC BY-NC-ND licensed from Flickr)

    And here’s where we encounter a tricky problem. The world is a varied place – some towns are very dense with buildings and streets crammed close together. In these areas you need to default to a zoom that’s quite ‘close in’, otherwise it can be hard to locate your problem.

    But out in the countryside, we have the opposite problem. You can have huge areas where there’s nothing but blank fields or moorlands. If the default map zoom is ‘close in’ here then users are likely to see a big map full of nothingness, or maybe just a single stretch of unidentifiable road.

    So, what is to be done?

    The answer is this – every time you search for a location in FixMyStreet the website does a check to see whether the location you typed is in an area where a lot of people live, or very few people live.

    mySociety has been storing this population density data in a webservice which we call Gaze. If the area you searched for is in a densely populated area we assume that it must be an urban location, and the map starts with a helpfully zoomed-in map. But if you’re in a sparsely populated area then it’s probably rural, so FixMyStreet starts zoomed out, making it easier to get an overview of the whole area.

    Where do we get the data from? Our late colleague Chris discusses this in a blog post from 2005 — the short answer is NASA SEDAC and LandScan. It’s an interesting example of how unexpected things can happen when data is made public — if population density wasn’t available to us, we wouldn’t have been able to put this small but clever detail into FixMyStreet’s interface.

  9. mySociety design tips: why you can bypass the FixMyStreet map

    When you report a problem on FixMyStreet, we ask you to click on a map to pinpoint its location. For example, if you want to tell the council a tree has fallen over, or there is a hole in the road, you can click exactly where it is. This is an easy way to provide the most accurate information in the report that we then send to the council.

    If you’re a programmer then you probably think that it’s obvious to use a map for such problem reporting. We agree: maps are ideal for this, and it’s a shame that so many councils still aren’t doing it this way.

    However, even though it’s very useful to have an accurate location for a problem, with FixMyStreet there are several reasons (below, we list just four) why someone might not be able to simply “click on the map”. In these cases, the map is no longer helpful — it’s a barrier. So we have to ask: is an accurate location so important that we should refuse to accept reports without one?

    It turns out, for FixMyStreet, we don’t really need that accuracy. Sure, it’s best if we have it, and over 95% of reports do contain pinpoint locations, because most people do click on the map. But if we don’t have that location data, then we can still make a fair guess from the postcode or area name that the user has already provided (that’s how we knew which map to display, after all). So we allow the user to submit the report without clicking on the map.

    Consequently, every time someone reports a problem without using the map, it means they’ve not given up, or clicked on the wrong place just to submit the form. In fact, they’ve reported a problem because we removed what would otherwise have been a barrier.

    So, here are some reasons why we didn’t make clicking on the map mandatory in FixMyStreet:

    • Map-reading isn’t a skill everybody is comfortable with
      When you’re building and testing FixMyStreet, you tend to imagine people will be reporting problems in places they know well. It’s easy to find somewhere on a map you are familiar with. But FixMyStreet users could just as easily be reporting a problem they passed on their way to work, on holiday, or at a party. So if they can’t read maps well it might be difficult or frustrating to locate a unfamiliar road junction or building.
    • Navigating a map requires more challenging user skills
      FixMyStreet is easy to use, deliberately — in fact you can report a problem just using your keyboard. Zooming and panning a map element is much harder than any other part of the process. Remember that if you’re building a website like this yourself you are already comfortable with using complex controls that lots of people — FixMyStreet-using people — are not. This is true even before considering other limitations such as small-screen devices or visual disabilities.
    • The map might not be helpful
      We rely on third-party maps. Most of the time, they are excellent. But what if the map is out of date? What if the problem is on a new road which hasn’t been added yet? What if the user remembers that the pothole was outside a distinctive shop or remarkable tree, only to find such landmarks aren’t on the map?
    • The problem might not have a location
      Potholes are easy: they have a fixed position on the road. What about smells or flooding? These problems sometimes simply don’t seem to have an obvious pinpoint location.

    In summary: we think clicking on a map is the best way to ask for a location from FixMyStreet users. But if we forced everybody to do it, some problems would never be reported, and some people would never become FixMyStreet users.

  10. mySociety design tips: using focus to avoid a click

    Remember how we put one simple question on the FixMyStreet homepage? This is possible because we know what most people coming to the page will want to do. So, because we’re confident about their intention, there’s something else we do too. We set the focus to that input. That is, we use JavaScript to put the cursor into that input box.

    This means that you can go to www.fixmystreet.com and start typing your postcode straight away. There’s no extra click to put your cursor in the input box.

    castanets (click click)

    “Folk Andal. Castañuelas” by calafellvalo (CC BY-NC-ND licensed from Flickr)

    Does it matter? On any single visit, if we didn’t do this, then probably this happens: you simply click in the box, and start typing, and do not even think about it. No problem! But sometimes you don’t notice that the cursor is not in the text box, and you start typing, and those letters do not appear. You look up, and you are a tiny bit annoyed; you click in the box and type it in again. That’s all.

    But remember we’ve had well over one hundred thousand different users access FixMyStreet since it went live. On a site that gets that busy, even if we annoyed only 1% of those users on the homepage, it adds up to a lot of frowning. A user who is annoyed on the homepage is more likely to give up — and remember that the homepage often is not the first page they came to, so some users will already be getting tired of unnecessary clicking. This is why we do believe it is worth getting things like this right, and that it does matter.

    In fact, in this case there are no other input boxes on the homepage: all the other interaction is by clicking on a link. This means that everything that page offers is just one click away (and yes, the way browsers handle forms means you can press Enter instead of clicking “GO” when you’ve finished typing, but we count that as one click).

    This is a very small detail, and almost nobody would notice it if we did not point it out. But actually, that is one of the reasons we’re posting these design tips: good design, especially usability design, usually means you don’t notice when an interface is doing the right thing.