1. Alaveteli: it’s easier than ever to run your own Right To Know website

    Alaveteli homepage Alaveteli is our platform for anyone who wants to run their own Freedom of Information website. It’s what underpins our own site, WhatDoTheyKnow, and it’s open source software available for anyone, in any part of the world, to use.

    Over the past few weeks, we’ve been making improvements to Alaveteli – the aim is to make it as easy as possible to install and use.

    You’ll now find that – as well as a snazzy new look for the Alaveteli site – there’s much better documentation and installation instructions. We’ve also consolidated documentation into one place, and we’re working through it all, making sure that everything in there is completely up to date.

    In short, if you’ve been thinking about running your own Right To Know site, there’s no better time to get started! The Alaveteli community are standing by to help you – and the documentation’s never been clearer.

  2. A Federation Is Born

    Poplus conference

     

    The right conference, held at the right time and attended by people with common problems, can sometimes give birth to whole new organisations. I was at OpenTech when the Open Rights Group was born, and on a grander scale the Red Cross and the UN both featured conferences at catalytic moments in their early history.

    Last week in Santiago, Chile, a conference took place that felt like exactly such a moment – PoplusCon. People from 27 countries spent two days talking about their shared goals and desires, and from it the skeleton of a new federation – the Poplus federation – started to take shape.

    Not everyone at the conference worked on identical projects, or had identical skills. Some people were specialists in tracking suspicious relationships (‘This guy’s brother-in-law gets all the contracts’), others were big into training journalists how to use FOI, others specialised in making important datasets more accessible to members of the public, others still were journalists, skilled at constructing stories. But one theme emerged pretty quickly – people wanted better, easier, more reliable ways of sharing knowledge and sharing technology, so that they could all save time, effort and money.

    What could a new federation do for you?

    And so that is how the conversation turned to the idea of founding a new federation – an organisation that could serve the needs of many different groups without being run or owned by any one of them. In a brainstorm session about what people wanted from a new federation, the following ideas were raised:

    • Running events to facilitate more sharing of ideas and tech
    • Publishing stories about successful and unsuccessful projects, especially where those stories need to cross language barriers to spread
    • Vetting and endorsing data standards
    • Access to a community of peers (for sharing experience, encouragement, tips and tricks etc)
    • Resources for projects that are running short
    • Help and advice on making projects sustainable
    • Certification of what counts as a Poplus Component
    • Where groups face common challenges, perhaps coordinate advocacy
    • Organisation of mentorship, exchanges and placements

    This wish list is clearly far more than a nascent organisation could arrange in the near future, but there was some informal voting and the top priorities fairly quickly emerged. People really wanted access to their peers, and to the stories that they tell. And there was a strong wish to see Poplus Components become more official, and better explained.

    Getting Real – Getting Involved

    But a list is just a list without people willing to make it real. And so without doubt the most awesome thing that took place at PoplusCon was that eight people immediately volunteered to form a committee that would bring Poplus into being, representing half a dozen countries in different parts of the world.

    This committee, which is completely open for anyone to join, will be meeting a couple of times in the next few weeks to agree on a plan for the first 12 months of the Poplus federation. It will work out how the new-born federation should govern itself, and what the first things that this entirely volunteer-run group should be doing. It’s an exciting, fragile moment and I’ve not seen anything like it in my ten-odd years working in this field. There’s no boss, no leader, just some people trying to build something of shared value.

    Right now there are no rules, no barriers to entry, no bureaucracy. In fact there’s nothing but some hope, enthusiasm and some shared dreams of a stronger community of individuals and organisations.

    I hope that if you read this and think that Poplus sounds cool, that you’ll consider joining the committee too. All you have to do is join the mailing list and ask where and when to show up. If you come to online committee meetings a couple of times, you’re de facto one of the people who runs Poplus. What happens next is – quite literally – down to you.

    Image: Fundación Ciudadano Inteligente

  3. We’re coming back to Cardiff

    Welsh daffodils Cardiff CastleInterested in civic coding, open data and eDemocracy?

    Then come and join mySociety for a chat, beer and pizza at the Founders Hub in Cardiff on Wednesday 2nd April!

    We’ll be joined by guest speakers:

    Sam Knight: Founder and developer of Your Senedd and Labour Councillor for Cathays.

    Your Senedd was launched in 2011 and aims to make the Welsh Assembly more accessible to the public by providing information on all Assembly Members and publishing all Assembly debates. Sam will be chatting about his motivations behind setting up Your Senedd and the development process.

    Daniele Procida: Daniele manages Cardiff University’s School of Medicine’s website, and the applications that publish it. These include Arkestra and other open-source Python/Django applications.

    Come along to hear Daniele’s talk “The bodiless head of the programmer”.

    He’ll also be around to chat about his work on django CMS and encouraging others to contribute to open source software.

    When: Wednesday 2nd April, drop in any time between 6pm and 9pm
    Where: The Founders Hub, 119 St Mary Street, Cardiff, CF10 1DY
    How: Add your name to the Lanyrd page: http://lanyrd.com/2014/mysocial-2-april/, so we know you’re coming.
    Who: Anyone who fancies it.

    NB: Look out for the mySociety hoodie (they look like this, only usually with a person inside). Watch our Twitter stream on @mySociety to check for last minute advice about where we are sitting or if we have moved venues for unforseen reasons.

    Photo by [Duncan] (CC)

  4. Introducing SayIt: a Poplus Component to bring transcript publication into the modern world

    Dictaphone Operator, via Wikimedia CommonsTranscripts – the written records of who says what in a conversation – aren’t sexy.

    However, they can be very important, or even historic. They can reveal big plans that will affect lots of people, and they are a basic requirement of political accountability.

    But the way in which transcripts are made available online today doesn’t reflect this importance. They tend to be published as hundreds of PDFs, and look more or less like they were made in the 1950s.

    We think that the people who are affected by the decisions and plans announced in transcribed meetings deserve better.

    What is SayIt?

    SayIt screenshot featuring Sienna Miller

    SayIt is an open source tool for publishing speeches, discussions and dialogues, simply and clearly, online. Search functionality is built in, you can link to any part of a transcript, and the whole thing works nicely on mobile devices.

    SayIt can be used either as a hosted service, or it can be built directly into your own website, as a Django app. Here are some examples of what it looks like in its hosted, standalone form:

    However, SayIt’s main purpose is to be built into other sites and apps.  We don’t have a live demo of this today, but one of our international partners will soon be launching a new Parliamentary Monitoring site which uses SayIt to publish years of parliamentary transcripts.

    SayIt is also 100% open data compatible, and we use a cut-down version of the Akoma Ntoso open standard for data import.

    What isn’t SayIt?

    SayIt is:

    • Not a site full of data curated and uploaded by mySociety – it’s a tool for redeployment all over the net. We’ll host deployments where that’s helpful to people, though.

    • Not primarily about Britain – whilst we’re a social enterprise based in the UK, SayIt has been built with an international perspective. We hope it will serve the needs of people watching politicians in places like Kenya and South Africa.

    • Not solely a mySociety project – it’s actually an international collaboration, via the Poplus network (see more below).

    • Not (yet) a tool to replace Microsoft Word as the way you write down transcripts in the first place. This is coming as we move from Alpha to Beta, though.

    Why are we building SayIt?

    SayIt is one of the Poplus Components. Poplus is a global collaboration of groups that believe it is currently too difficult and expensive to build effective new digital tools to help citizens exert power over institutions.

    Poplus Components are loosely joined tools, mostly structured as web services, that can be used to radically decrease the development time of empowerment sites and apps.

    SayIt is the newest component, and aims to reduce the difficulty and cost of launching services that contain transcripts – in particular websites that allow people to track the activities of politicians. Using SayIt or other Poplus components you can build your site in whatever language and framework suits your wishes, but save time by using the components to solve time-consuming problems for you.

    The founders of Poplus are FCI in Chile, and mySociety in the UK – and we are hoping that the launch of SayIt will help grow the network. The project has been made possible by a grant from Google.org, while early iterations were aided by the Technology Strategy Board.

    Interested in publishing transcripts via SayIt? Here’s what to do…

    Having taken a look at the demos, we hope at least some of you are thinking ‘I know of some transcripts that would be better if published like this’.

    If you are interested, then there are two approaches we’d recommend:

    • If you’re a coder, or if you have access to technical skills, read about how to convert your data into the open standard we use. Then talk to us about how to get this data online.

    • If you don’t have access to technical skills,  get in touch about what you’re interested in publishing, and we’ll explore the options with you.

    Note to coders – We’ve not yet spent a lot of time making SayIt easy to deploy locally, so we know it may be a challenge. We’re here to help.

    Where might SayIt help?

    SayIt comes from a desire to publish the speeches of politicians. But we know that there are many other possible uses, which is why we built the Shakespeare demo.

    We think SayIt could be useful for publishing and storing transcripts of:

    • Local council meetings

    • Court hearings

    • Election hustings

    • Academic research interviews and focus groups

    • Academic seminars, lectures, etc

    • Plays

    • Market research focus groups

    • Historic archives of events such as a coronation or key debate

    These are just a few of our ideas, but we bet you have others – please do tell us in the comments below.

    What’s coming next

    At the moment, SayIt only covers publishing transcripts, not creating them. Needless to say, this lack of an authoring interface is a pretty big gap, but we are launching early (as an Alpha) because we want to know how you’ll use it, what features you want us to build, and what doesn’t work as well as we anticipated. We also want to see if we can attract other people to co-develop the code with us, which is the real spirit of the Poplus network.

    We’ll also be adding the ability to subscribe to alerts so that you’ll get an email every time a keyword occurs (just as you can on our other websites, such as FixMyStreet, TheyWorkForYou and WhatDoTheyKnow). This feature will come into its own for ongoing series of transcripts such as council meetings.

    Image by Columbia Phonograph Co. [Public domain], via Wikimedia Commons

  5. Open Source: trying to make it more open

    Hopeful Helping Hands by Matt Katzenberger

    This month we released a new version of FixMyStreet. Amongst the new features, fixes, and thingamajigs were some small improvements added by two volunteers, Andrew and Andy.

    Even though these are not core pieces of functionality — in fact, precisely because they are not — we want to draw your attention to why they were included, and why this is a Very Good Thing. And perhaps, if you’re a coder who wants to put something into an Open Source project but hasn’t quite found a way in, Andrew and Andy’s work will nudge you into becoming a contributor too.

    One of the axioms of Open Source is that, because anyone can read the source code, in theory anyone can contribute to it. In practice, though, it’s not really as simple as that. Both ends of the “anyone can contribute” idea require effort:

    • Before contributing to a project of any complexity (as we hope you want to do), there’s often a lot to learn, or figure out, before any work can even begin;
    • Before accepting contributions to such a project (keen as we are to do so), there’s an overhead of testing, checking, and managing the incoming code.

    The ugly real world

    The basic issue here is that software is complex — no matter how well-written, tested and documented program code is, if the problem it’s solving is in the real world, it’s not going to be simple.

    This is especially true of anything used by the public, because often you can only make things seem simple at the front (such as a clean web interface or “user journey”) by working hard behind the scenes with data structures and processes that handle the underlying complexity. It’s inevitably true of any projects which have been developed over time — programmers like to use the term “legacy code” to describe anything that wasn’t written then way they’d choose to write it now.

    Often the problems that software is solving are not quite as obvious as they first appear. At mySociety we’ve got a wealth of experience and actual usage data that ultimately changes the way we build, and develop, our platforms. We understand the fields we work in well (technically, the nerds like to call these the “problem domain”), whether it’s governmental practice or civic user behaviour, and that’s often knowledge that’s not encapsulated anywhere in the program code.

    Furthermore, any established platform must protect against the risk that new changes break old behaviour — something that regression testing is designed to catch. This is especially important on platforms like FixMyStreet or Alaveteli where the software is already running in multiple installations.

    This is why we have a team of full-time, experienced, and (thanks to our rigorous recruitment process) talented programmers who can invest the time and effort to be familiar with all these things when they set to work coding.

    But this builds up to an impediment: sensitivity to any of these issues is enough to make anyone think twice about simply forking our code and starting to hack on it for us.

    How it sometimes works

    In practice, then, how does anything get contributed? How come it doesn’t all get written by our own coders?

    The answer is, of course, we do work with major contributors outside our own team (have a look at the activity on our github repos to see them) — but it always requires a period of support and on-line discussion both before and during the process. There’s also the business of testing, and sometimes politely pushing back on, pull requests (which is how code contributions are submitted). But the fact of the matter is that this is only possible for people who are willing to spend time familiarising themselves with the specific code, technologies, and practices that we’re using on that project. These tend to be hard-code devs, and — here’s the point — they’re always experienced Open-Sourcers: this will never be the first time they’ve worked on such a project.

    Which is where the little features come in.

    The joy of small

    We noticed this problem — that contributing code to our projects is simply not easy for us or for contributors. Importantly, it’s not just us: it’s Open Source everywhere. But we can’t simply dismiss the opportunity for contribution. We want to encourage coders to do this, because we believe that Open Source is intrinsically a good thing.

    We do two things to make it easier to contribute:

    • We identify small features that a coder can approach without a full understanding of the code and the problem domain;
    • We help people (like you!) get started by opening up a laptop at our weekly meetups.

    The first of these seems obvious now: when we add issues (an idea for a new feature, or maybe a bugfix) to our github repos, if we think they’re candidates for manageable, isolated work, we tag them with the label: Suitable for volunteers (like this).

    Often these turn out to be “nice-to-haves” that one of our full-time devs can’t be pulled off more pressing problems to add just now. (Case in point: Andrew added a date-picker to the FixMyStreet admin stats page, and three of our own staff had stumbled upon and applauded the difference it had made within a week of it going live).

    It means it’s much easier for you to get involved, because often it’s a little, isolated piece of code. And it’s much more manageable for us, because the change you’ll be submitting is also isolated.

    So if you’re looking for something to tackle, pick one of those issues, and let us know (just to check nobody else has baggsied it already). Fork the repo, cut the code, write the tests, submit a pull request!

    But wait — if that last paragraph made you gulp, here’s the second thing we do: meetups. Of course, this is less helpful if you can’t make it to London on Wednesdays, but the concept is sound. Put simply, if there is a barrier to entry to diving in, and if one-on-one time with a dev, and some pizza, is what it takes to overcome that, it’s time well spent for you to come and see us.

    Not 100% confident with git? Not sure when db/schema.sql gets used or how we like to handle migrations? No problem: we’re happy to guide you.

    You?

    If this has struck a chord with you — you’d love to be an Open Source contributor one day, and you think mySociety projects make the world a better place — perhaps you should take a poke in our repos, or come along to a meetup. Start small, but do start.

    Oh, and Andrew and Andy — thanks guys ;-)

    More about volunteering for mySociety

    Photo by Matt Katzenberger (CC)

  6. Where to meet mySociety

    For an organisation whose members normally work from home, we’ve been pretty sociable recently, with meet-ups, conferences, and our annual retreat. We’re glad to discover that we haven’t actually lost the ability to communicate face to face…

    If you’d like to come and sample our sharply honed social skills for yourselves, there are a number of opportunities still to come.

    Every Wednesday: London meet-ups

    Jed and Firefox[Above : mySociety designer Jed, chilling in front of the Mozilla Firefox]

    If you’re in London, do feel free to drop by and say hello, any Wednesday from 6:00 pm – 9:00 pm. We meet at the Mozilla London space – and there are often other interesting things going on too.

    Meet-ups are not just for coders – they’re for anyone who would like to talk more about mySociety projects or the wider eDemocracy field. On October 30th, we’re tying in with the Open Government Partnership event; you’re welcome to attend then or any other week.

    Fancy coming along? Add your name to our Lanyrd pages here.

    mySociety and Mozilla meet-up

    [Above: Our meet-ups are not always this busy! On this night, we happened to coincide with a Mozfest planning event.. speaking of which, see below]

    25th – 27th October: Mozfest

    Mozfest in Ravensbourne, London, is Mozilla’s annual innovative open web event for ‘technologists and creators’.

    The event kicks off on the night of Friday 25th with a Science Fair. We’ll be there, showing our wares – in this case, we’ll be hoping to meet many of the internatonal attendeees and let them know about our open source software. But if you’re not an international attendee, you should totally swing by and say hello too.

    30th October: Edinburgh

    The next non-London mySociety meet-up will be in Edinburgh – watch this blog, our Twitter stream and Facebook page for details of precisely where (it’ll be a nice, central pub that serves food… suggestions are welcome).

    That’s in advance of our attendance at the Channel Shift conference – but you don’t have to be a council employee to drop in. Come and share a pint and have a chat, whichever aspect of our work interests you.

    20th November: Online Information conference

    mySociety’s Director Tom will be giving the keynote presentation at the Online Information conference, the theme of which is “adapting to disruptive technologies and creating value with people, platforms and information”. Feel free to grab Tom afterwards for a chat!

    4th December: Manchester

    As with Edinburgh, we’re pitching up in Manchester for a Channel Shift conference, and will be taking the opportunity to mingle with lovely locals the night before. Again, pub suggestions are more than welcome.

    We hope to see you soon at one of these events. And, if you’re wondering what we look like, well, you’re in luck. At our recent retreat we took a photo of the entire team (plus a few guests). Here we are in all our glory – click to see a larger version, if you dare.

    mySociety team October 2013
  7. Coding, chat, and… hexagons

    Graphene by Michael StathamThanks to everyone who dropped into last night’s meet-up: we hope you had a good time. From our point of view, it was a fine mix of technical and social, with just as much chatting as coding.

    mySociety’s developers Mark and Dave, project manager Ben, and director Tom were on hand, and there were plenty of new folk, who’d come along simply to find out more about mySociety and the work we do.

    Dave reports:

    Quite often these discussions spill over into the whole Open Source world and how that works. After all, we’re at the Mozilla London space because we’re part of that sector, and Mozilla are a great example of people who generously share their (splendid) resources to make wonderful things happen in the tech community.

    Meanwhile, Andrew continued to chip away at issues on FixMyStreet, and we also enjoyed Viktoria’s impromptu explanation of how graphene works (she works with it) by ingeniously using the Moz space’s hexagonal-patterned flooring.

    These meet-ups are open to everyone. So whether you’re an open source veteran, or just a curious newbie who is bemused to discover that there is such a thing as free software – if you’re interested in anything you see on mysociety.org, please come along.

    Every Wednesday, 6:00 -9:00 pm. Sign up here.

    Photo by Michael Statham (CC)

  8. A Thousand Questions

    We love to hear that someone has taken our code and is launching a new website. But it is even better to see that site become fully established.

    How can you tell when a site has reached that milestone? Hard to say – but we reckon that processing over 1,000 Freedom of Information requests is a pretty good sign that an Alaveteli install has succeeded.

    Ki Mit Tud in Hungary passed that mark back in April; in Spain, Tu Derecho A Saber has also dealt with just over 1,000 requests. And in New Zealand, FYI.org.nz is also celebrating the big thou, under the care of Rowan Crawford.

    https://fyi.org.nz/

    We made Alaveteli to be used in any jurisdiction, no matter what Freedom of Information laws are in place, and no matter what the prevailing culture – and we are always fascinated to know how the resulting sites are impacting their respective communities. When it comes to New Zealand, some aspects are familiar; some are distinct to the region.

    Rowan was kind enough to tell us a little about how it’s been, watching those first thousand FOI requests roll through FYI.org.nz.

    How is it running the site? Do you have a volunteer team helping you, like we do with WhatDoTheyKnow in the UK?

      “It’s just me at the moment — it hasn’t been a high burden at all. Generally things “just work”.
      “As the Admin, I do get unusual questions directed at me sometimes, that strain my knowledge of the law. Fortunately I have several lawyer contacts on Twitter I can run to for help!”

    Is there anything we should know about the background to FOI in NZ? 

    “The Official Information Act came in to force almost exactly thirty years ago, on 1 July 1983. It is used extensively, though just how much is hard to say.

    “The only body which reports OIA statistics is the Ombudsman, which only reports the numbers of complaints – about 1200 in 2012. I know journalists use it routinely.”

    How has NZ reacted to having its own Right To Know site?

    “The response from people using it has been very, very positive. The press haven’t covered it that much, but bloggers have. One measure of success is that it has its own section in the Ombudsman’s FAQ.

    Can you recall any particularly noteable requests that have gone through the site?

    “I have an affection for a couple of successful ones from the police: they replied to one request with their interrogation manual, and to another with information on how much they paid informants.”

    How are you finding Alaveteli?

    “Installing it has improved hugely since the first time I tried. The current method of theming works pretty well, too.

    “There are several things on my road map: integrating documentcloud instead of the current HTML conversion, using S3 for resources, and a mobile version.”

    What kind of user numbers are you seeing?

    “There are just under 500 users registered in the system, and we’re getting 5,000 unique visitors a month. It’s growing month by month by about 15% at the moment — who knows how long that will keep up!”

    Run a Freedom of Information website

    Rowan makes it look pretty easy!

    We won’t pretend that you can run a Freedom of Information site with no effort at all, but we have made it as simple as we possibly can. We have the code, we have the documentation, and most important of all, we have a community waiting to give you support and advice.

    If you’re interested in finding out more, drop us a line at international@mysociety.org.

  9. Alaveteli gets an upgrade

    Brass Band Serenade by .sashi

    Today, we are using the phrase “Alaveteli upgrade” rather a lot – and not just because it’s such a great tongue-twister. It’s also a notable milestone for our open-source community.

    Alaveteli is the software that underlies WhatDoTheyKnow, our Freedom of Information website. The code can also be deployed by people in other countries who wish to set up a similar site. If you’re a ‘front-end user’, someone who just uses WhatDoTheyKnow to file or read FOI requests, this upgrade will go unnoticed… assuming all goes well at our end, that is. But if you’re a developer who’d like to use the platform in your own country, it makes several things easier for you.

    Alaveteli will now be using the Rails 3 series – the series we were previously relying on, 2, has become obsolete. One benefit is that we’re fully supported by the core Rails team for security patches. But, more significant to our aim of sharing our software with organisations around the world, it makes Alaveteli easier to use and easier to contribute to. It’s more straightforward to install, dependencies are up-to-date, code is clearer, and there’s good test coverage – all things that will really help developers get their sites up and running without a problem.

    Rails cognoscenti will be aware that series 4.0 is imminent – and that we’ve only upgraded to 3.1 when 3.2 is available. We will be upgrading further in due course – it seemed sensible to progress in smaller steps. But meanwhile, we’re happy with this upgrade! The bulk of the work was done by Henare Degan and Matthew Landauer of the Open Australia Foundation, as volunteers – and we are immensely grateful to them. Thanks, guys.

    Find the Alaveteli code here – or read our guide to getting started.

     

    Image credit: Sashi Manek (cc)

  10. Open311: Extended

    This is the third of our recent series of Open311 blog posts: we started by explaining why we think Open311 is a good idea, and then we described in a non-techie way how Open311 works. In this post we’ll introduce our proposed extension to Open311, and show how we use it in FixMyStreet.

    The crux of our suggested improvement is this: normal people want to know what has happened to their problem, and Open311 currently isn’t good enough at telling them whether or not it has been dealt with. To be more specific, our additions are all about reports’ status change, by which we mean something like this:

    That pothole?

    I just totally fixed it.

    That’s robot-311 from the previous post, if you’ve dropped in here without reading the previous posts. Once again we’re blurring the distinction between client and user (the girl you’ll see below) a little, to make things simpler to follow.

    Create→send→fix→update

    Every month in the UK, thousands of problems are reported on www.fixmystreet.com and, moments later, sent on to the councils who will fix them. Here’s what happens with a problem report for something like a pothole or a flickering streetlight:

    1. You create the report on FixMyStreet.
    2. FixMyStreet sends that report to the right department at the right council.
    3. That body puts it into its own back-end system.
    4. Later, when the council fixes the problem, FixMyStreet is updated, and everyone knows it’s fixed.

    On the face of it, you might think we need only care about 1 and 2. But really, FixMyStreet isn’t just about dispatching reports, it’s about helping to get things like potholes actually fixed. And neither citizens nor local governments benefit if work gets done but nobody finds out about it – which is part 4 on the list above.

    What do we mean by “status change”?

    The example at the top of the page shows the robot effectively changing a problem’s status to “fixed”.

    Actually, statuses can be simple, such as either OPEN or CLOSED, or more detailed, such as “under investigation”, “crew has been dispatched”, “fixed”, and so on. But since we’re only concerned here with the status changing, that specific vocabulary deployed doesn’t really matter – it can be anything.

    In situations where FixMyStreet is not integrated with council systems (i.e we just send email problem reports) FixMyStreet problems still frequently get marked as fixed, because anyone can change the status of a report just by visiting the page and clicking the button.  Obviously, though, we prefer to have FixMyStreet directly connected to the local government back-end databases, so that news of a fixed report can be automatically bubbled from the back-office up into FixMyStreet and out onto the net.

    And here’s where the problem lies: Open311 doesn’t quite support this business of getting problem updates from the back office out to the public. So first, we’ll show you how it can be done today, using Open311, and we’ll explain why this isn’t a good option. Then we’ll show our preferred solution, which we’ve proposed as an extension.

    Looking at everything just to spot one change (bad)

    One way to notice if any problems’ statuses have changed is to use Open311 to ask for every single service request, and see if any of them have a different status since the last time you checked.

    Tell me all the service requests you’ve ever received

    OK:

    request 981276 the pothole on the corner by Carpenter Street is now CLOSED (I filled in the pothole)

    request 988765 the pothole by bus stop on Nigut Road is now CLOSED (I filled in the pothole)

    request 998610 gaping hole at the end of Sarlacc Road is now OPEN (the pothole fell through)

    request 765533 where the street was cracked outside Taffey’s Snake Pit is now CLOSED (I filled in the pothole)

    . . .

    continues for thousands of requests

    Um, OK. Now I’ll look at all these and see if any have changed since I last asked *sigh*

    Obviously there are some problems with this. Even though Open311 lets you ask for quite specific service requests, you have to ask for all of them, because by definition you don’t know which ones might have changed. Remember, too, that problems can potentially change status more than once, so just because it’s been marked as CLOSED once doesn’t mean it won’t become OPEN again later. This exchange is very wasteful, very slow and ultimately (with enough reports) may become de facto impossible.

    Asking for just the changes (good)

    So here’s a better way of doing it. We’ve actually been doing this for some months, and now seems the time to share.

    The client asks the server for just the updates on a regular basis, so any requests that have recently changed get updated on FixMyStreet automatically, usually just a few minutes later.

    Have you changed the status of any of service requests today?

    Yes, request 981276 was CLOSED at 3 o’clock (I filled in the pothole)

    Or, more practically for keeping FixMyStreet up to date:

    Have you changed the status of any of service requests in the last 15 minutes?

    Nope.

    This is handled by our extension to Open311, GET Service Request Updates. There’s also an optional equivalent call for putting updates into the server (POST Service Request Update), which would apply if the client changed the status after the service request had been submitted.

    Note that the server identifies the problem with its own reference (that is, 981276 is the council’s reference, not a FixMyStreet ID, for example). This is important because not all these requests necessarily came from this particular client. Remember that all service requests are available through the Open311 GET Service Requests call anyway (as shown above). So the server doesn’t send each service request back in its entirety: just its ID, the new status, when it changed, and a brief description.

    In practice the client wouldn’t usually ask for “today”. In fact, we typically send a request asking for any updates in the last 15 minutes, and then at the end of the day ask for the whole day’s updates, just to check none were missed.

    The technical bit

    From a client’s point of view, this is simply an extra call like others in the Open311 API. So it’s just a request over HTTP(S) for XML (or JSON, if required).

    We deliberately make the client poll the server for updates and pull them in, rather than expecting the server to push updates out. This frees the server from any obligation to track which clients (for there may be more than just one) care about which updates. The requests themselves are sent with unique IDs, allocated by the server, so the client can dismiss duplicates. It’s also robust in the event of connection failures, so if there are timeouts or retry logic, that’s for the clients to worry about, not the server. Basically, this is all to make it as light on the server as possible: the only real issue is that it must be able to provide a list of updates. This usually means adding a trigger to the database, so that when a problem’s status is updated a record of that update is automatically created. It’s the table of those “service request update” records that incoming requests are really querying.

    We have published our own recommendation of this mechanism, which FixMyStreet already implements, alongside the FixMyStreet codebase.

    Is that it?

    Yup, that’s it.

    This extension is in addition to the Open311 specification — it doesn’t break existing implementations in any way. Obviously this means FixMyStreet’s Open311 implementation is compatible with existing Open311 servers. But we hope that others working on Open311 systems will consider our extension so that clients are kept better informed of the status of the problems being fixed.

    Why are statuses so important that it is worth extending the Open311 spec?

    mySociety didn’t originally build FixMyStreet because we wanted to get potholes fixed. We built it because we wanted nervous, politically inexperienced people to know what it felt like to ask the government to do something, and to be successful at that. We wanted to give people the buzz of feeling like they have a bit of power in this world, even if the most tiny amount.

    If the government fixes a problem and the citizen doesn’t find out it’s a double loss. The citizen becomes disillusioned and weakened, and the government doesn’t get the credit it is due. Everyone loses. We think that Open311 is a key mechanism for making large numbers of people feel that the government does respond to their needs. It just needs a bit of an upgrade to do it better. We hope very much that the wider community tests and endorses our extensions, and it can be folded in to the next official version of the Open311 standard.