Today sees the official launch of FixMyStreet’s open source codebase as a proper tool that we hope people will want to deploy in cities and countries around the world. It is based on FixMyStreet.com which we believe is the most usable, most mature street problem reporting tool in the world, but which is only available to British users.
We’re shouting about this launch a bit because we need your help to make the service ever better. First, we need feedback from programmers about whether we’ve got the install process right – whether it’s as easy and clear as we want it to be. And for non-coders who want to get involved, we want to ask for help with the process of translating the site’s text into different languages.
Over the years there have been many copies of FixMyStreet set up in many countries, often using the site’s original name, but always written by developers from scratch. We’re delighted to have inspired people, but all too often the people trying to build copies have stumbled as they realise just how hard it is to build a tool like this with the polish that users expect. We think that people everywhere would be better off if they could have a local FixMyStreet that was really usable, and really connected to the right people.
So we’re very happy to be able to open up a codebase that has been extensively modified in the last year, to help users around the world manage easy, successful deployments. Steps we have taken include:
- Putting the translation text into Transifex, so that non-technical translators can get started whenever they feel like it
- Developing Amazon Machine Images so people who want to tinker can get started in the minimum possible time
- Rewriting the entire codebase in order to make it a less confusing installation
- Building a global version of our MapIt political boundaries web service, so you can get going without having to wrestle administrative data out of your government before you get started.
Plus with the help of the wonderful OpenStreetMap, you can get maps without licensing hassles too.
Calling it version 1.0 is our way of saying two things. First, that the tool still has a lot of evolution left to do, and a long way to go before it is as good as we want it to become. But more ambitiously, calling it 1.0 is also our way of saying that it’s no longer just a codebase dumped into Github. It’s a real open source project, which we plan to support, and which we hope will make a real difference in the lives of ordinary people. Check it out.
One of the key differences between the UK’s national parliament and its local governments is that Parliament produces a written record of what gets said – Hansard.
This practice – which has no actual legal power – still has a huge impact on successful functioning of Parliament. MPs share their own quotes, they quote things back to one-another, journalists cite questions and answers, and every day TheyWorkForYou sends tens of thousands of email alerts to people who want to know who said what yesterday in Parliament. Without freely available transcripts of Parliamentary debates, it is likely that Parliament would not be anything like as prominent an institution in British public life.
No Local Hansards
Councils, of course, are too poor to have transcribers, and so don’t produce transcripts. Plus, nobody wants to know what’s going on anyway. Those are the twin beliefs that ensure that verbatim transcripts are an exceptional rarity in the local government world.
At mySociety we think the time has come to actively challenge these beliefs. We are going to be building a set of technologies whose aim is to start making the production of written transcripts of local government meetings a normal practice.
We believe that being able to get sent some form of alert when a council meeting mentions your street is a gentle and psychologically realistic way of engaging regular people with the decisions being made in their local governments. We believe transcripts are worth producing because they show that local politics is actually carried out by humans.
The State of the Art Still Needs You
First, though – a reality check. No technology currently exists that can entirely remove human labour from the production of good quality transcripts of noisy, complicated public meetings. But technology is now at a point where it is possible to substantially collapse the energy and skills required to record, edit and publish transcripts of public meetings of all kinds.
We are planning to develop software that uses off-the-shelf voice recognition technologies to produce rough drafts of transcripts that can then be edited and published through a web browser. Our role will not be in working on the voice recognition itself, but rather on making the whole experience of setting out to record, transcribe and publish a speech or session as easy, fast and enjoyable as possible. And we will build tools to make browsing and sharing the data as nice as we know how. All this fits within our Components strategy.
But mySociety cannot ourselves go to all these meetings. And it appears exceptionally unlikely that councils will want to pay for official transcribers at this point in history. So what we’re asking today is for interest from individuals – inside or outside councils – willing to have a go at transcribing meetings as we develop the software.
It doesn’t have to be definitive to be valuable
Hansard is the record of pretty much everything that gets said in Parliament. This has led to the idea that if you don’t record everything said in every session, your project is a failure. But if Wikipedia has taught us anything, it is that starting small – producing little nuggets of value from the first day – is the right way to get started on hairy, ambitious projects. We’re not looking for people willing to give up their lives to transcribe endlessly and for free – we’re looking for people for whom having a transcript is useful to them anyway, people willing to transcribe at least partly out of self interest. We’re looking for these initial enthusiasts to start building up transcripts that slowly shift the idea of what ‘normal’ conduct in local government is.
Unlike Wikipedia we’re not really talking about a single mega database with community rules. Our current plans are to let you set up a database which you would own – just as you own your blog on Blogger or WordPress, perhaps with collaborators. Maybe you just want to record each annual address of the Lord Mayor – that’s fine. We just want to build something that suits many different people’s needs, and which lifts the veil on so much hidden decision making in this country.
Get in touch
The main purpose of this post is to tell people that mySociety is heading in this direction, and that we’d like you along for the ride. We won’t have a beta to play with for a good few months yet, but we are keen to hear from anyone who thinks they might be an early adopter, or who knows of other people who might want to be involved.
And we’re just as keen to hear from people inside councils as outside, although we know your hands are more tied. Wherever you sit – drop us a line and tell us what sort of use you might want to make of the new technology, and what sort of features you’d like to see. We’ll get back in touch when we’ve something to share.
The local press in Islington has just reported the accidental release of quite a bit of sensitive personal data by Islington council.
One of our volunteers, Helen, was responsible for spotting that Islington had made this mistake, and so we feel it is appropriate to set out a summary of what happened, to inform journalists and citizens who may be interested.
On 27th May a user of our WhatDoTheyKnow website raised an FOI request to Islington Borough Council. On the 26th June the council responded to the FOI request by sending three Excel workbooks. Unfortunately, these contained a considerable amount of accidentally released, private data about Islington residents. In one file the personal data was contained within a normal spreadsheet, in the two other workbooks the personal data was contained on four hidden sheets.
All requests and responses sent via WhatDoTheyKnow are automatically published online without any human intervention – this is the key feature that makes this site both valuable and popular. So these Excel workbooks went instantly onto the public web, where they seem to have attracted little attention – our logs suggest 7 downloads in total.
Shortly after sending out these files, someone within the the council tried to delete the first email using Microsoft Outlook’s ‘recall’ feature. As most readers are probably aware – normal emails sent across the internet cannot be remotely removed using the recall function, so this first mail, containing sensitive information in both plain sight and in (trivially) hidden forms remained online.
Unfortunately, this wasn’t the only mistake on the 26th June. A short while later, the council sent a ‘replacement’ FOI response that still contained a large amount of personal information, this time in the form of hidden Excel tabs. As you can see from this page on the Microsoft site , uncovering such tabs takes seconds, and only basic computer skills.
At no point on or after the 26th June did we receive any notification from Islington (or anyone else) that problematic information had been released not once, but twice, even though all mails sent via WhatDoTheyKnow make it clear that replies are published automatically online. Had we been told we would have been able to remove the information quickly.
It was only by sheer good fortune that our volunteer Helen happened to stumble across these documents some weeks later, and she handled the situation wonderfully, immediately hiding the data, asking Google to clear their cache, and alerting the rest of mySociety to the situation. This happened on the 14th July, a Saturday, and over the weekend mySociety staff, volunteers and trustees swung into action to formulate a plan.
The next working day, Monday 16th July, we alerted both Islington and the ICO about what had happened with an extremely detailed timeline.
The personal data released by Islington Borough Council relates to 2,376 individuals/families who have made applications for council housing or are council tenants, and includes everything from name to sexuality. It is for the ICO, not mySociety, to evaluate what sort of harm may have resulted from this release, but we felt it was important to be clear about the details of this incident.
All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.
Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.
As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.
Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.
We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.
We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.
The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.
From Britain to the World
For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.
MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.
If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.
About the Data
The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.
The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global. We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.
If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.
Commercial Licenses and Local Copies
MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.
This post was written by mySociety volunteer Peter Dixon, who is part of the FixMyTransport team.
There are many reasons nowadays for you to travel across the UK for business, with meetings and relocations being a key reason. Both have affected my personal circumstances within the past few months, so I felt it would be beneficial for me to show how I had used FixMyTransport and FixMyStreet to see more of the areas I am visiting or relocating to.
One of the biggest frustrations when arriving at a hotel on business is that you can be stranded in a strange town or city with no idea about what is in the area. Once you have factored in dinner, the rest of the time is spare and very few people enjoy being stranded in a hotel with only a few TV channels to entertain.
To discover an area, you need to have a look round and a purpose for making a journey can make it easier to undertake a gentle stroll.
When faced with this on a recent trip to the West Midlands, I used FixMyStreet and FixMyTransport to locate issues that were in the area around the hotel and planned a quick walk that allowed me to look at issues that had not been marked as fixed.
As a result of this walk, I was able to mark a couple of the reports as fixed and update some of the others. This doesn’t just limit you to walking, a cycle or a drive can be a great stress reliever for some and a reason to do them makes it so much easier.
I have recently moved house too, and both of the websites provided a great opportunity to explore my new local area. To relieve the boredom of endless housing estates, I used the two websites to find something to look at when having an explore and added new issues as I looked for the existing issues. It’s great to see things getting fixed and know that you have already added to your new community.
So when you are next in a new area and looking for something to do, log on to FixMyStreet or FixMyTransport and see what you can add to the local community.
Image credit: Checking the map by Shaun Dunmall.
On the 21st of February 2012 Alex Skene, representing mySociety’s Freedom of Information website WhatDoTheyKnow, appeared in front of the UK Parliament’s Justice Select Committee. The MPs on the committee were holding an evidence session as part of their post-legislative scrutiny of the Freedom of Information Act.
Video of the session can be viewed online via ParliamentLive.TV and the BBC’s Democracy Live. A transcript of the session will become available via TheyWorkForYou, typically these take a week or two to be produced.
Prior to the session WhatDoTheyKnow had submitted written evidence to the review making three main points:
- The scope of the act should be extended to cover a wider range of public bodies.
- Time limits should be introduced for public interest tests and internal reviews.
- There is a need for more proactive publication of information, and a culture of openness and transparency needs to continue to be nurtured and extended within the UK’s public sector
The committee appeared genuinely interested in finding out how FOI has performed to-date and how it can be improved.
Alex told the committee that FOI enables evidence based policy making and empowers citizens; he said the WhatDoTheyKnow.com website supercharges the provisions of the FOI Act making it easier for people to take advantage of the right to access information which it gives them.
Elfyn Llwyd MP raised the question of vexatious and frivolous requests through the medium of ghosts. Asked if requests about ghosts could ever be justified Alex told MPs that it was hard to draw the line between acceptable and unacceptable requests. He noted that one council had spent public money on an exorcism, so in that case there would be information held and an FOI request justified. He questioned if requests on ghosts were to be deemed unacceptable, what other areas might be excluded. UFOs? The MoD for a long period did have an office collating UFO reports, again there was public spending, and recorded information held, in this area. Homeopathy was also highlighted, that’s about as real as ghosts or UFOs, but again FOI requests about it must surely be permitted as significant amounts of taxpayers money are spent on it.
Maurice Frankel, the director of the Campaign for Freedom of Information, who was giving evidence alongside WhatDoTheyKnow took a stronger line. He described those who made FOI requests about ghosts as “idiots”; but also accepted it was hard, and undesirable, to try and outlaw requests on certain subjects. He added that such requests did not generally cost large amounts of money to deal with.
MPs on the committee appeared sympathetic to calls from the representatives of WhatDoTheyKnow and the Campaign for Freedom of Information to introduce stricter time limits. The need for time limits was brought into focus during the discussion of the time limits for prosecutions under S.77 of the Act (Offence of altering etc. records with intent to prevent disclosure), very few requests have gone through a response, and internal review, and the Information Commissioner within the time limit for launching a prosecution. An MP suggested making offences under S.77 triable in either a magistrates or a crown court so as to extend the time period while retaining consistency with the rest of the justice system.
When asked to comment on the idea of introducing fees for all FOI requests Alex said such proposals would be “devastating” and would deter many from making requests. Alex noted that the public had paid for the information in question already, via general taxation, and ought be able to access it.
When asked to comment on lobbying from universities to be exempted from FOI, Alex robustly defended their inclusion in the act, pointing to their role in controlling access to professions and awarding degrees. Maurice Frankel and Alex noted the universities’ argument that they were being funded by a decreasing fraction of public money wasn’t really relevant, as that is not the basis on which bodies are deemed to be covered by the Act.
Extending Coverage of FOI
The reach of FOI into commercial organisations carrying out work on behalf of public bodies was briefly discussed however notably there was little further discussion of extending the coverage of FOI, perhaps suggesting this may be a dedicated subject for future evidence session. This session was been described as the committee’s first, suggesting there will be more. At least one of these will presumably hear from the Information Commissioner.
The written evidence we submitted can be read on page 81 of the compendium of submitted evidence (PDF).
When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.
Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:
- It is entirely responsively designed, so that it works on the simplest of mobile web browsers from day one.
- All the lessons we learned from storing political data wrongly have been baked into this site (i.e we can easily cope with people changing names, parties and jobs)
- Every organisation, position and place in the system is now a proper object in the database. So if you want to see all the politicians who went to Nairobi University, you can.
- There is lots of clear information on how parliament functions, what MPs and committees do, and so on.
- It synthesizes some very complex National Taxpayer’s Association data on missing or wasted money into a really clear ‘scorecard‘, turning large sums of money into numbers of teachers.
The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.
Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.
If you haven’t got a penny,
A ha’penny will do,
If you haven’t got a ha’penny,
Then God bless you.
We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?
mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.
Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.
What’s behind the door? A letter to Santa.
If you can fit them down the chimney, here’s what we’re dreaming of:
More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.
We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.
Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.
Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?
A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.
Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?
FixMyTransport buy-in – from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.
Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.
One of the most popular features on TheyWorkForYou is the plain English, non-judgemental vote analyses on MP pages that say things like “voted strongly against introducing a smoking ban“. We call these ‘policies’, and they are authored by skilled people using the volunteer run PublicWhip website.
Making each one of these policies is a painstaking task, requiring good knowledge of how the Parliamentary voting system works, good writing skills, patience, and the strength of character not to let your own views about the issues cloud the analysis. It is of utmost importance to both mySociety and our users that these policies are fair and trustworthy.
Earlier this year we started to update the process by which we made new policies to make it even more rigorous, which we wrote about here. Marcus Fergusson and Stephen Young came onboard and did sterling work, but they have now moved on to greater things, and so we’re looking to recruit two to three new people to do this job. Uber volunteer Richard Taylor has been helping out recently, but this is really a job for two or more people.
You might very well ask ‘why two people, given the work is part time?’. The answer is that we really want every new policy to be cross-checked by two different people every time it is added or amended. This is to help eliminate possible mistakes, and prevent any unconscious biases.
We pay for this work on a piece work basis – £160 a time for a combination of one new policy authored, and one other policy double-checked. This money comes mainly from people making small donations, which I think helps keep everyone focussed on how important it is to get these right. We hope to add about two new policies a month, once the new team is up to speed.
If you’d like to be considered, please email firstname.lastname@example.org with ‘mspolicies’ in the subject line. Applications close 22nd November 2010.
We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.
- Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
- As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
- From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
- Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
- As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.