Until about two years ago I was quite actively involved in the Open Data movement. I sat in on the 2007 gathering in California where the first Open Data Principles were drafted, and later sat on the Transparency Board at the UK government.
I stopped being involved in early 2012 because I saw a couple of things happening. First, the Open Data baton had been picked up by dedicated, focused advocates like the Open Data Institute and the Open Knowledge Foundation, who could give 100% to fighting this fight (I always had to fit it around managing a growing organisation with other goals). And second I felt that the surge of relatively meaningful data releases in the country I live in (the UK) had pretty much come to an end. The real policy action and innovation will now happen in more rapidly-changing countries where transparency is a more visceral issue.
Still, despite walking away, I remained optimistic. It seemed more or less impossible to imagine that in twenty years’ time that there wouldn’t be quite a bit more Open Data around, especially in rich countries. But given the virtually-zero political gain to be had from this agenda in countries like the UK, where is said data actually going to come from?
Learning from Microsoft (really)
The more I thought about it, the more I realised that we’d already seen the answer in the form of Microsoft. Throughout the 1990s the .doc and .xls standard rose and took over governments around the world, even though there was never anything like a clear policy process that drove that decision.
There was certainly no high profile ‘Microsoft Government Partnership’ with international conferences and presidential speeches. Instead there was a safe, ‘no brainer’ product that governments bought to solve their problems, and these data standards came with it. The pressure on governments to do anything at all probably came from the fact that the private sector had widely adopted Office first.
I think that a recurrence of this phenomenon – change-through-replacing-old-computers – is where Open Data at real scale is going to come from. I think it’s going to come from old government computers being thrown away at their end-of-life and replaced with new computers that have software on them that produces Open Data more or less by default.
The big but
However, there’s a big BUT here. What if the new computers don’t come with tools that produce Open Data? This is where SayIt comes in, as an example of a relatively low-cost approach to making sure that the next generation of government IT systems do produce Open Data.
SayIt is a newly launched open source tool for publishing transcripts of trials, debates, interviews and so on. It publishes them online in a way that matches modern expectations about how stuff should work on the web – responsive, searchable and so on. It’s being built as a Poplus Component, which means it’s part of an international network of groups collaborating on shared technologies. Here’s JK Rowling being interviewed, published via SayIt.
But how does this little tool relate to the business of getting governments to release more Open Data? Well, SayIt isn’t just about publishing data, it’s about making it too – in a few months we’ll be sharing an authoring interface for making new transcripts from whatever source a user has access to.
We hope that having iterated and improved this authoring interface, SayIt can become the tool of choice for public sector transcribers, replacing whatever tool they use today (almost certainly Word). Then, if they use SayIt to make a transcript, instead of Word, then it will produce new, instantly-online Open Data every time they use it.
The true Open Data challenge is building brilliant products
But we can’t expect the public sector to use a tool like SayIt to make new Open Data unless it is cheaper, better and less burdensome than whatever they’re using now. We can’t – quite simply – expect to sell government procurement officers a new product mainly on the virtues of Open Data. This means the tough task of persuading government employees that there is a new tool that is head-and-shoulders better than Excel or Word for certain purposes: formidable, familiar products that are much better than their critics like to let on.
So in order for SayIt to replace the current tools used by any current transcriber, it’s going to have to be really, really good. And really trustworthy. And it’s going to have to be well marketed. And that’s why we’ve chosen to build SayIt as an international, open source collaboration – as a Poplus Component. Because we think that without the billions of dollars it takes to compete with Microsoft, our best hope is to develop very narrow tools that do 0.01% of what Word does, but which do that one thing really really well. And our key strategic advantage, other than the trust that comes with Open Source and Open Standards, is the energy of the global civic hacking and government IT reform sector. SayIt is far more likely to succeed if it has ideas and inputs from contributors from around the world.
Regardless of whether or not SayIt ever succeeds in penetrating inside governments, this post is about an idea that such an approach represents. The idea is that people can advance the Open Data agenda not just by lobbying, but also by building and popularising tools that mean that data is born open in the first place. I hope this post will encourage more people to work on such tools, either on your own, or via collaborations like Poplus.
Photo by Troy Morris (CC)
FixMyStreet, our site for reporting things like potholes and broken street lights, has had something of a major redesign, kindly supported in part by Kasabi. With the help of Supercool, we have overhauled the look of the site, bringing it up to date and making the most of some lovely maps. And as with any mySociety project, we’d really appreciate your feedback on how we can make it ever more usable.
The biggest change to the new FixMyStreet is the use of responsive design, where the web site adapts to fit within the environment in which it’s being viewed. The main difference on FixMyStreet, besides the obvious navigation changes, is that in a small screen environment, the reporting process changes to have a full screen map and confirmation step, which we thought would be preferable on small touchscreens and other mobiles. There are some technical details at the end of this post.
Along with the design, we’ve made a number of other improvements along the way. For example, something that’s been requested for a long time, we now auto-rotate photos on upload, if we can, and we’re storing whatever is provided rather than only a shrunken version. It’s interesting that most photos include correct orientation information, but some clearly do not (e.g. the Blackberry 9800).
We have many things we’d still like to do, as a couple of items from our github repository show. Firstly, it would be good if the FixMyStreet alert page could have something similar to what we’ve done on Barnet’s planning alerts service, providing a configurable circle for the potential alert area. We also are going to be adding faceted search to the area pages, allowing you to see only reports in a particular category, or within a certain time period.
Regarding native phone apps – whilst the new design does hopefully work well on mobile phones, we understand that native apps are still useful for a number of reasons (not least, the fact photo upload is still not possible from a mobile web app on an iPhone). We have not had the time to update our apps, but will be doing so in the near future to bring them more in line with the redesign and hopefully improve them generally as well.
The redesign is not the only news about FixMyStreet today
As part of our new DIY mySociety project, we are today publishing an easy-to-read guide for people interested in using the FixMyStreet software to run versions of FixMyStreet outside of Britain. We are calling the newly upgraded, more re-usable open source code the FixMyStreet Platform.
This is the first milestone in a major effort to upgrade the FixMyStreet Platform code to make it easier and more flexible to run in other countries. This effort started last year, and today we are formally encouraging people to join our new mailing list at the new FixMyStreet Platform homepage.
Coming soon: a major upgrade to FixMyStreet for Councils
As part of our redesign work, we’ve spoken to a load of different councils about what they might want or need, too. We’re now taking that knowledge, combining it with this redesign, and preparing to relaunch a substantially upgraded FixMyStreet for Councils product. If you’re interested in that, drop us a line.
Kasabi: Our Data is now in the Datastore
Finally, we are also now pushing details of reports entered on FixMyStreet to Kasabi’s data store as open linked data; you can find details of this dataset on their site. Let us know if it’s useful to you, or if we can do anything differently to help you.
On a mobile, you can see that the site navigation is at the end of the document, with a skip to navigation link at the top. On a desktop browser, you’ll note that visually the navigation is now at the top. In both cases, the HTML is the same, with the navigation placed after the main content, so that it hopefully loads and appears first. We are using display: table-caption and caption-side: top in the desktop stylesheet in order to rearrange the content visually (as explained by Jeremy Keith), a simple yet powerful technique.
If you have any technical questions about the design, please do ask in the comments and I’ll do my best to answer.
If you haven’t got a penny,
A ha’penny will do,
If you haven’t got a ha’penny,
Then God bless you.
We wish you all a merry and prosperous Christmas – and for those of you who are already feeling quite prosperous enough, may we point you in the direction of our charitable donations page?
mySociety’s work is made possible by donations of all sizes and from all sorts of people. Those donations help fund all the online projects we create; projects that give easy access to your civic and democratic rights. If that’s important to you, show your appreciation, and we promise we’ll make the best use of every penny.
Thank you for sticking with us through this month-long post. We hope you’ve found it interesting and we wish you the very merriest of Christmases.
What’s behind the door? A letter to Santa.
If you can fit them down the chimney, here’s what we’re dreaming of:
More publicly available data Of course, we were delighted to hear in Mr Osborne’s autumn statement that all sorts of previously-inaccessible data will be opened up.
We’re wondering whether this new era will also answer any of our FixMyStreet geodata wishes. Santa, if you could allocate an elf to this one, we’d be ever so pleased.
Globalisation …in the nicest possible way, of course. This year has seen us work in places previously untouched by the hand of mySociety, including Kenya and the Philippines. And we continue to give help to those who wish to replicate our projects in their own countries, from FixMyStreet in Norway to WhatDoTheyKnow in Germany.
Santa, please could you fix it for us to continue working with dedicated and motivated people all around the world?
A mySociety Masters degree We’re lucky enough to have a team of talented and knowledgeable developers, and we hope we will be recruiting more in the coming year. It’s not always an easy task to find the kind of people we need – after all, mySociety is not your average workplace – so we’ve come to the conclusion that it’s probably easiest to make our own.
Back in February, Tom started thinking about a Masters in Public Technology. It’s still something we’re very much hoping for. Santa, is it true you have friends in academic circles?
FixMyTransport buy-in – from everyone! Regular users of FixMyTransport will have noticed that there are different kinds of response from the transport operators: lovely, fulsome, helpful ones, and formulaic ones. Or, worse still, complete refusal to engage.
Santa, if you get the chance, please could you tell the operators a little secret? Just tell them what those savvier ones already know – that FixMyTransport represents a chance to show off some fantastic customer service. And with 25,000 visitors to the site every week, that message is soon spread far and wide.
I was just talking to someone in a local council about the fact that they’d opened up the location of 27,000 streetlights in their council area. They wanted to know if FixMyStreet could incorporate them so that problem reports could be more accurately attached.
This conversation reminded me that we’ve had an informal wish list of geodata for FixMyStreet for some time. What we need is more data that lets us send problems to the correct entity when the problem is not actually a council responsibility.
I’m just posting these up to see if anyone knows a guy who knows a girl who knows a dog who knows how to get hold of any of these datasets. In some vector data format, if possible, please!
- Canals and responsible authorities
- Supermarkets (esp car parks) and responsible companies
- Network Rail’s land
- Council owned land
- Land and roads controlled by the Highways agency
- Shopping malls
- National parks
- BT phone boxes (the original problem which inspired FixMyStreet)
So, do you know someone who might know someone who can help us improve FixMyStreet? And guess what, if we do add this to our web services, you’ll probably be able to query them too.
Note: This post is a work in progress, I need your help to improve it, especially with knowledge of non-English sites
I was recently in Washington DC catching up with mySociety’s soul-mates at the Sunlight Foundation. As we talked about what was going on in the field of internet-enabled transparency, it came clear to me that there are now more identifiable categories of transparency website than there used to be.
Identifying and categorising these types of site turns out to be surprisingly useful. First, it can help people ask “Why don’t we have anyone doing that in our country?” Second, it can help mySociety to make sure that when we’re planning ahead we don’t fail to consider certain options that be currently off our radar. Also, it gives me an excuse to tell you about some sites that you may not have seen before.
Anyway, enough preamble. Here they are as I see them – please give me more suggestions as you find them. As you can see there’s a lot more activity in some fields than others.
1. Transparency blogs & newspapers – At the technically simplest, but most manual labour-intensive end of the scale is sites, commercial and volunteer driven, whose owners use transparency to help them to write stories. Given almost every political blog does this a bit, it can be hard to name specific examples, but I will note that Heather Brooke is the UK’s pre-eminent FOI-toting journalist/blogger, and we’ve just opened a blog for our awesome volunteers on WhatDoTheyKnow to show their FOI skills to an as-yet unsuspecting public.
2. What Politicians do in their parliaments – These sites primarily include lists of politicians, and information about their primary activities in their assemblies, such as voting or speaking. This encompasses mySociety’s TheyWorkForYou.com, Rob McKinnon’s one man labour of love TheyWorkForYou in NZ, Italy’s uber-deep OpenPolis.it (6 layers of government, anyone?), Germany’s almost-un-typable Abgeordnetenwatch, Romania’s writ-wielding IPP.ro, Josh Tauberer’sGovTrack.us, plus the bonny bouncing babies OpenAustralia and Kildare Street (Ireland). Of special note here are Mzalendo (Kenya) who unlike everyone else, can’t reply on access to a parliamentary website to scrape raw data from, and Julian Todd’s UNDemocracy (International), that has to fight incredible technical barriers to get the information out.
3. Databases of questions and answers posed to politicians – These sites let people post politicians questions, and the publish the questions and answers. The Germans running Abgeordnetenwatch (Parliament Watch) seem to have had considerable success here, with newspapers citing what politicians say on their site. Yoosk has some politicians in the UK on it, too.
4. Money in politics – This comes in two forms, money given to candidates (MAPlight), and money bunged by politicians to their favourite causes (Earmark watch). In the UK, as far as I know, the Electoral Commission’s database remains currently unscraped, perhaps because the data is so ungranular.
6. Websites containing bills going through parliament, or the law as voted on – This includes the increasingly substantial OpenCongress in the US which saw major traffic during the Health Care debates, and the UK government’s own Acts database and Statute Law Database. Much of the legal database field, however, remains essentially private.
7. Services that create transparency as a side effect of delivering services – Our own sites lead the way here: FixMyStreet‘s public problem reports and WhatDoTheyKnow’s FOI archive are both created by people who aren’t primarily using the site to enrich it – they’re using it to get some other service.
8. Election websites – These come in many forms, but what they have in common is their desire to shed light on the positions and histories of candidates, whether incumbents or new comers. The biggest beast here is Stemwijzer (Netherlands), probably in relative terms the most used transparency or democracy site ever. However these sites are popular in several places, the big but highly labour intensive VoteSmart (US), Smartvote.ch (Switzerland), plus others. mySociety is shortly to start to recruit constituency volunteers to help with our take on this problem, keep an eye on this blog if you want to know more.
9. Political document archives – This is a new category, now occupied by Sunlight’s Partytime archive for invitation to political events, and TheStraightChoice, Julian Todd and Richard Pope’s wonderful new initiative for archiving election leaflets and other paper propoganda.
10. Bulk data – Online transparency pioneer Carl Malamud doesn’t do sites, he does data. Big globs zipped up and made publicly available for coders and researchers to download and process. The US government has now stepped into this field itself with Data.gov, doubtless soon to be followed by data.gov.uk.
Please don’t shoot me if I’ve missed anything here, the world is a big place. But I thought that was a useful and interesting exercise, and I hope you’ll both find it useful, and help me improve it too. Comment away.
Matthew’s just updated ScenicOrNot, the little game that we built to provide a ‘Scenicness’ dataset for Mapumental, to include a data dump of the raw data. The dump will update automatically on a weekly basis, but currently it contains averaged scores for 181,188 1*1km grid squares, representing 83% of the Geograph dataset we were using, or 74% of all the grid squares in Great Britain. It is, in other words, really pretty good, and, I think, unprecedented in coverage as a piece of crowd sourced geodata about a whole country.
It’s available under the Creative Commons Attribution Noncommercial 3 Licence, and we greatly look forward to seeing what people do with it.
To: Anyone thinking of running any reasonably developed country, any time soon.
The most scary thing about the Internet for your government is not pedophiles, terrorists or viruses, whatever you may have read in the papers. It is the danger of your administration being silently obsoleted by the lightening pace at which the Internet changes expectations. I’m not going to give examples of this change, others can do this far better than I. But you don’t need experts’ advice to tell which way the wind blows – if you can’t find any examples of changing expectations in your own life, driven by the internet, I can’t help you anyway: please point me to your successor.
This is a list of the top 5 major things any government of any developed nation should be doing in relation to the Internet, as I see it at the start of 2009. They are not in any order, and do not lack ambition – they are for the Next Government, after all.
Hire yourself some staff who know what the Internet really means for government, and fund a university to start training more who really understand both worlds: you’re going to need them. There just aren’t enough employed in any government anywhere yet to save you from being hopelessly outstripped by external progress. The citizen discontent resulting from massive shifts in expectation could wash your entire government away without you ever having anyone skilled enough to tell you why everyone was so pissed off. Your chances of truly reinventing what your government is are basically zero without such staff.
Free your data, especially maps and other geographic information, plus the non-personal data that drives the police, health and social services, for starters. Introduce a ‘presumption of innovation’ – if someone has asked for something costly to free up, give them what they want: it’s probably a sign that they understand the value of your data when you don’t.
Give external parties the right to interface electronically with any government or mainly public system unless it can be shown to create substantial, irrevocable harm. Champion the right fiercely and punish unjustified refusals with fines. Your starting list of projects should include patient-owned health records, council fault reporting services and train ticket sales databases. All are currently unacceptably closed to innovation from the outside, and obscurity allows dubious practices of all kinds to thrive.
Commission the world’s first system capable of large scale deliberation, and hold a couple of nation wide sessions on policy areas that you genuinely haven’t made your minds up on yet. When it is over, mail people who participated with a short, clear list of things you’ve done that you wouldn’t have done without them. Once you’ve made it work well, legislate it into the fabric of your democracy, like elections and referendums.
When people use your electronic systems to do anything, renew a fishing license, register a pregnancy, apply for planning permission, given them the option to collaborate with other people going through or affected by the same process. They will feel less alone, and will help your services to reform from the bottom up.
mySociety wants to see all these things happen. Get in touch if you are interested.
Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.
We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.
I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.
- TheyWorkForYou opens up access to conventional, representational democracy, between and during elections. In 10 years time, I asked for Parliament to publish all information about its work in a structured way, as hinted at in our Free Our Bills campaign. So it is much easier for everyone to help make new laws better.
- FixMyStreet is local control of the things people care about, a very practical democracy. In 10 years time I’d like to see all councils running their internal systems (planning, tree preservation orders… everything that isn’t about individuals) in public, so everyone can see and be reassured about what is being done, why and where.
- WhatDoTheyKnow shows the deep interest that there is by the public in the functioning of all areas of government. In 10 years time, I’d like to see document management systems in wide use by public authorities that publish all documents by default. Only if overridden for national security or data protection reasons would they be hidden.
Charles Armstrong, from the Circus Foundation, has written up the workshop.
Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort – Charles is going to work on it for another time.
Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).
The Office of Public Sector Information (snappy name, lads) has launched a simple new service where you can publicly lodge a request for some public sector data that you can’t get, but need for some reason. They’ll then act as your behind-the-scenes champions and attempt to lever it out of which ever bit of government is trying to keep hold of it for no good reason.
You can also read other people’s requests, which hopefully will help people realize how much good data there is out there, and leave comments suggesting further reasons why it might be a good idea to let it loose.
Full disclosure, this was my idea, as part of the Power of Information review, so I’m not neutral in wanting to see people get what they want through it.
mySociety has never run a campaign before today. And we’re not sure anyone’s ever run a campaign featuring a charismatic duck-billed platypus escaping from under the closing jaws of a Parliamentary portcullis.
Update 15.34 25/03/2008
Conservative Party leader David Cameron has just endorsed the campaign in a video.
Update 17.14 25/03/2008
Now kind words from techy Lib Dem MP Lynne Featherstone.
Update 11.38 1/04/2008
We’ve just recieved this fantastic endorsement from Nick Clegg, leader of the Liberal Democrats:
“Parliament belongs to the people. It’s time to open it up so people can find out what’s going on. mySociety has done a brilliant job in recent years in doing that – and it’s time to take this project to the next level and get information about the laws Parliament passes into the public domain.
“It takes a new MP months to figure out how the tortuous bills procedures work – so how we expect the voters to know what’s going on, I have no idea. The changes MySociety are calling for are vital so that every MP is fully accountable for the decisions they take on behalf of their constituents.
“I fully support the Free our Bills campaign, and will do all I can in Parliament to win this battle.”