Until about two years ago I was quite actively involved in the Open Data movement. I sat in on the 2007 gathering in California where the first Open Data Principles were drafted, and later sat on the Transparency Board at the UK government.
I stopped being involved in early 2012 because I saw a couple of things happening. First, the Open Data baton had been picked up by dedicated, focused advocates like the Open Data Institute and the Open Knowledge Foundation, who could give 100% to fighting this fight (I always had to fit it around managing a growing organisation with other goals). And second I felt that the surge of relatively meaningful data releases in the country I live in (the UK) had pretty much come to an end. The real policy action and innovation will now happen in more rapidly-changing countries where transparency is a more visceral issue.
Still, despite walking away, I remained optimistic. It seemed more or less impossible to imagine that in twenty years’ time that there wouldn’t be quite a bit more Open Data around, especially in rich countries. But given the virtually-zero political gain to be had from this agenda in countries like the UK, where is said data actually going to come from?
Learning from Microsoft (really)
The more I thought about it, the more I realised that we’d already seen the answer in the form of Microsoft. Throughout the 1990s the .doc and .xls standard rose and took over governments around the world, even though there was never anything like a clear policy process that drove that decision.
There was certainly no high profile ‘Microsoft Government Partnership’ with international conferences and presidential speeches. Instead there was a safe, ‘no brainer’ product that governments bought to solve their problems, and these data standards came with it. The pressure on governments to do anything at all probably came from the fact that the private sector had widely adopted Office first.
I think that a recurrence of this phenomenon – change-through-replacing-old-computers – is where Open Data at real scale is going to come from. I think it’s going to come from old government computers being thrown away at their end-of-life and replaced with new computers that have software on them that produces Open Data more or less by default.
The big but
However, there’s a big BUT here. What if the new computers don’t come with tools that produce Open Data? This is where SayIt comes in, as an example of a relatively low-cost approach to making sure that the next generation of government IT systems do produce Open Data.
SayIt is a newly launched open source tool for publishing transcripts of trials, debates, interviews and so on. It publishes them online in a way that matches modern expectations about how stuff should work on the web – responsive, searchable and so on. It’s being built as a Poplus Component, which means it’s part of an international network of groups collaborating on shared technologies. Here’s JK Rowling being interviewed, published via SayIt.
But how does this little tool relate to the business of getting governments to release more Open Data? Well, SayIt isn’t just about publishing data, it’s about making it too – in a few months we’ll be sharing an authoring interface for making new transcripts from whatever source a user has access to.
We hope that having iterated and improved this authoring interface, SayIt can become the tool of choice for public sector transcribers, replacing whatever tool they use today (almost certainly Word). Then, if they use SayIt to make a transcript, instead of Word, then it will produce new, instantly-online Open Data every time they use it.
The true Open Data challenge is building brilliant products
But we can’t expect the public sector to use a tool like SayIt to make new Open Data unless it is cheaper, better and less burdensome than whatever they’re using now. We can’t – quite simply – expect to sell government procurement officers a new product mainly on the virtues of Open Data. This means the tough task of persuading government employees that there is a new tool that is head-and-shoulders better than Excel or Word for certain purposes: formidable, familiar products that are much better than their critics like to let on.
So in order for SayIt to replace the current tools used by any current transcriber, it’s going to have to be really, really good. And really trustworthy. And it’s going to have to be well marketed. And that’s why we’ve chosen to build SayIt as an international, open source collaboration – as a Poplus Component. Because we think that without the billions of dollars it takes to compete with Microsoft, our best hope is to develop very narrow tools that do 0.01% of what Word does, but which do that one thing really really well. And our key strategic advantage, other than the trust that comes with Open Source and Open Standards, is the energy of the global civic hacking and government IT reform sector. SayIt is far more likely to succeed if it has ideas and inputs from contributors from around the world.
Regardless of whether or not SayIt ever succeeds in penetrating inside governments, this post is about an idea that such an approach represents. The idea is that people can advance the Open Data agenda not just by lobbying, but also by building and popularising tools that mean that data is born open in the first place. I hope this post will encourage more people to work on such tools, either on your own, or via collaborations like Poplus.
Photo by Troy Morris (CC)
mySociety ltd, our trading arm, has been commissioned by Parliament to conduct a strategic review of their digital service provision. We’ve spoken face to face to all kinds of people, and now we are moving on to the online component of our survey. This gives everyone a chance to take part – including you.
We would like suggestions about where the strengths lie, and where you think there are gaps in Parliament’s online provision. Whether you’re a member of the general public who occasionally uses Parliamentary digital services, or you have a professional connection, your views are welcome.
With this commission, we have an opportunity to help Parliament gain a deeper insight into its users, their needs and offer advice on the best way to serve them – so we jumped in with both feet.
To kick things off, we’ve been interviewing all kinds of users, internal and external. It’s been absolutely fascinating to speak directly to the people who use the many sectors of Parliament’s large online estate.
Now we need your views. If you can spare a minute, please visit our short survey here. It will remain open until Friday the 22nd of November.
Photo by Garry Knight (CC)
A commission from the Welsh Government has resulted in new functionality for Mapumental, which now has the capability to display multiple points and to produce RAW data compatible with GIS applications. Here’s how it happened.
How accessible is your nearest school, post office, or GP’s surgery? In Wales, that’s not always a simple question: the country’s mountainous landscapes, rural populations, and sometimes infrequent bus services can mean that those without cars are rather cut off from public service provision.
But of course, like any other authority, the Welsh Government has an obligation to quantify just how accessible their services are.
For many years, they have done so using a number of different methods. Some of these involve literally millions of point to point calculations – so, naturally, when Bill Oates, Head of Geography & Technology, Knowledge Services at the Welsh Government, approached Mapumental, he was keen to discover whether we could simplify things.
We were keen to try it, too – plotting multiple points would add a whole new slew of possibilities to Mapumental. Previously, Mapumental has been all about travel from a single point, and this functionality would bring new applications across all kinds of industries and users.
There’s only one way to find out
The sensible way forward was to pick a single service and see what we could do. One of the government’s open data sets showed positioning of all the secondary schools in the country, and would give us a very good indication of how manageable the task would be across all other provisions.
So we set ourselves this aim: to display the shortest transit time to get to any secondary school in Wales, from any point in that country.
This project was not like the map we made for the Fire Protection Association last year, with its postcode input and interactive sliders. It bore more relation to our static maps, but with the additional dimension that the single map would have multiple points plotted on it. Each point would display its own associated journey times, and where travel to one school was quicker than to another, it would have to override the data of the school that was further away.
And here’s the (very pretty) result
Transit times by public transport to secondary schools in Wales, with an arrival time of 9:00am.
Time bands are in 15-minute increments, with red areas being those where schools are accessible within a 15-minute journey (the centres of the red dots therefore also represent the positions of the schools).
Purple areas are those where journey time is between 1.75 and 2 hours, and the colours in between run in the order you see bottom right of the map. White areas (much of which are mountainous and sparsely-populated) are outside the two-hour transit time.
But there’s more – data for GIS
Plotting all the schools on a single map required quite a bit of modification to Mapumental, but there was another important part of the project that also had to be worked on, if the output was to meet all the needs of the Welsh Government.
They needed to be able to export the raw transit time data to their own GIS tools – the tools that they use to feed into official statistics. This allows the transit time data to be combined with other datasets, such as population density, for in-depth analysis.
We added a feature which allows Mapumental to produce what is known as a ‘raster grid’ output – basically, an enormous matrix that gives every pixel on the map a travel time value. To do this, we used the open source GRASS format.
Bill Oates is keen to see where this project can go:
“I’m really excited at the prospect of combining the power of Mapumental with our open data, and fully understanding how accessible Welsh public services are by public transport.”
To him, the benefits are clear:
“Mapumental’s approach is significantly quicker than our current methods, so this work will help save us time as well as providing a more engaging output.
“We hope that future work with mySociety will give us a sustainable approach to calculating the accessibility of local shops, hospitals, post offices and other services on an ongoing basis to help ensure that we’re meeting the needs of our citizens.”
We’re looking to build on our success, and offer this service to others – initially on request but via our API as soon as we can. We’ll keep you posted as to our progress.
You can see multiple-point mapping in action, on our Mapumental Property project – now the tool allows house-hunters to take more than one person’s commute into consideration when choosing where to live.
Who might use Mapumental?
Now that Mapumental can plot transit times from multiple points, and provide RAW data for GIS applications, we have great potential for use by anyone interested in travel and accessibility. That could be in central and local government strategy, town planning, architectural consultancy, transport provision, large enterprises looking to save on parking, or start-ups in the green transport space…to name but a few.
Could Mapumental help you with your mapping needs? If so, please do drop us a line at firstname.lastname@example.org.
Photo of Welsh school bus (bws ysgol) by Aqwis (CC)
The last few weeks since the US election have seen an explosion in articles and blog posts about how Obama’s tech team pulled out the stops in their race against the Republicans. It’s been an exciting time to learn about the new techniques dreamed up, and the old ones put to the test.
For those of us who develop non-partisan services to help people report broken street lights or make Freedom of Information Requests, such stories certainly seem unimaginably glamorous: I don’t think any of my colleagues will ever get hugged by Barack Obama!
But it has also been an interesting time to reflect on the difference between choosing to use tech skills to win a particular fight, versus trying to improve the workings of the democratic system, or helping people to self-organise and take some control of their own lives.
At one level there’s no competition at all: the partisan tech community is big and economically healthy. It raises vast amounts of cold hard cash through credit card payments (taking a cut to pay its own bills) and produces squillons of donors, signers, visitors, tweeters, video watchers and so on. The non-partisan tech community is much smaller – has fewer sustainable organisations, and with the exception of some big online petitions, doesn’t get the same sort of traffic spikes. By these metrics there’s absolutely no doubt which use of tech is the most important: the partisan kind where technology is used to beat your opponent, whether they are a political candidate, a policy, company, or an idea.
But I am still filled with an excitement about the prospects for non-partisan technologies that I can’t muster for even the coolest uses of randomized control trial-driven political messaging. The reason why all comes down to the fact that major partisan digital campaigns change the world, but they don’t do it in the way that services like eBay, TripAdvisor and Match.com do.
What all these sites have in common – helping people sell stuff they own, find a hotel, or a life partner – is that they represent a positive change in the lives of millions of people that is not directly opposed by a counter-shift. These sites have improved the experience of selling stuff, finding hotels and finding life partners in ways that don’t attract equal and opposite forces, driven by similar technologies.
This is different from the case of campaigning tech: here a huge mailing list is pitted against another even huger mailing list. Epic fund-raising tools are pitched against even more epic fund-raising tools. Orca vs Narwhal. Right now, in US politics, the Democrats have a clear edge over the technology lined up against them, and I totally understand why that must feel amazing to be part of. But everything you build in this field always attracts people trying to undo your work by directly opposing it. There is something inate to the nature of partisanship which means that one camp using a technology will ultimately attract counter-usage by an opposing camp.
This automatic-counterweighting doesn’t happen with services that shift whole sectors – like TripAdvisor did. In the hotel-finding world, the customer has been made stronger, the hotel sector weaker, and the net simply doesn’t provide tools to the hotel industry to counter what TripAdvisor does.
It is this model – the model of scaleable, popular technology platforms that help people to live their lives better – that I aspire to bring to the civic, democratic and community spheres in my work. Neither I nor mySociety has yet come up with anything even remotely on the scale of a TripAdvisor, but there remains the tantalising possibility that someone might manage it – a huge, scaleable app of meaningful positive impact on democratic, civic or governance systems*. Our sites are probably about as big as it gets so far, and that’s not big enough by far.
If someone does manage to find and deliver the dream – some sort of hugely scalable, impactful non-partisan civic or democratic app & website, it is unlikely that the net will instantly throw up an equal and opposite counterweight. There is a real possibility that the whole experience of being a citizen, the whole task of trying to govern a country well will be given a shot in the arm that won’t go away as soon as someone figures out how to oppose it. I’m not talking Utopia, I’m just talking better. But what motivates me is that it could be better for good, not just until the Other Team matches your skills.
And that – in rather more words than I meant to use – is why I am still excited by non-partisan tech, and why I really hope that some of the awesome technologists who worked in the political campaigns of 2012 get involved in our scene.
I’m on Twitter if anyone wants to talk about this more.
* You can certainly make an argument that Twitter and Facebook sort-of represent non-partisan democracy platforms that have scaled. But some people disagree vehemently, and I don’t want to get into that here.
A number of non-departmental government bodies / quangos have been named as being up for abolition, merger, privatisation or absorption into parent departments, as part of the Coalition Government’s Spending Review, due this autumn. This has been widely dubbed in the press as a “bonfire of the quangos“. The list of quangos up for review is still being compiled by the government, and there have been a number of clarifications, amendments and retractions as further details come to light.
The Telegraph has obtained and published today a leaked list of 177 quangos up for abolition, plus a further 200 that are still being reviewed.
This is a great opportunity to highlight that mySociety’s Freedom of Information site WhatDoTheyKnow covers nearly all of these little-known bodies that spend public money (we currently have just over 3,800 public authorities listed on the site). Given their impending doom, there is little time left to find out what they spent public funds on, as only their most important records will be transferred to the National Archives or successor bodies for permanent storage. The remainder are likely to be shredded, or deleted, as only “records identified as valuable for future administrative need” are kept.
You can see our annotated list of the Telegraph’s list here – our volunteers have added links to most of the bodies’ pages on WhatDoTheyKnow, so you can more easily make your final FOI requests to them…
Current government policy in relation to the Internet can broadly be summarised as occupying three areas:
1. Getting people online (broadband access, and lessons for people who don’t have the skills or interest)
2. Protecting people from bad things done using the Internet (terrorism, child abuse, fraud, hacking, intellectual property infringement)
3. Building websites for departments and agencies.
The government does all these things primarily because it believes that the Internet boosts the economy of the UK, and that IT can reduce the cost of public services whilst increasing their quality. Together, these outweigh the dangers, meaning it doesn’t get banned. Gordon Brown’s recent speech at Google was an exemplar of this mainly economically driven celebration of the Internet’s virtues, telling audience members that your industry is driving the next stage of globalisation”.
The first challenge for the government is to understand that whilst these beliefs are true, they are only a minor part of the picture. Tellingly, Browns’ speech contained almost no language that couldn’t have been used to explain the positive impact of electrification or shipping containers.
The way in which the Internet Is not like Electrification or Shipping Containers
The Internet has been relentlessly undermining previous practices in the running of businesses, dating, parenting, spying, producing art and many other areas. So, however, did electrification and shipping containers. From cheaper raw materials, to cheaper cars to have sex in the back of, economic and social change has always been driven by technological change.
What is different is the way in which the Internet changes social and economic practices – the vector of attack. In the 20th century, advancement of human welfare went hand in hand with the rise of companies that used economies of scale to deliver better goods and services for customers. Technology effectively made it possible and much easier to be a big, highly productive company, to gather expertise and capital together and to target markets for maximum yields.
Now take a look for a moment at Wikipedia, MoneySavingExpert, Blogger or Match.com – all big websites, all doing different things. Each one, however, is in its own way is reducing the ability of large, previously well functioning institutions to function as easily.
These services are reducing traditional institutions ability to charge for information, seize big consumer surpluses, limit speech or fix marriages. It has, in other words, become harder to be a big business, newspaper, repressive institution or religion. Nor is this traditional ‘creative destruction’ going on in a normal capitalist economy: this isn’t about one widget manufacturer replacing another, this is about a newspaper business dying and being replaced by no one single thing, and certainly nothing recognisable as a newspaper business.
This common pattern of more powerful tools for citizens making life harder for traditional institutions is, for me, a cause for celebration. However, I am not celebrating as a libertarian (which I am not) I celebrate it because it marks a historic increase in the freedom of people and groups of people, and a step-change in their ability to determine the direction of their own lives.
How the government can be on the side of the citizen in the midst of the great Internet disruption
Disruption like this is scary for any institution, which will tend to mean that as a public entity which interfaces with other institutions the temptation will be to hold back the sea, not swim with it. Government must swim with the tide, though, not just to help citizens more but to avoid the often ruinous tension of a citizenry going one way and a government going another. There are various things government can do to be on the right side.
1. Accept that any state institution that says “we control all the information about X” is going to look increasingly strange and frustrating to a public that’s used to be able to do whatever they want with information about themselves, or about anything they care about (both private and public). This means accepting that federated identity systems are coming and will probably be more successful than even official ID card systems: ditto citizen-held medical records. It means saying “We understand that letting train companies control who can interface with their ticketing systems means that the UK has awful train ticket websites that don’t work as hard as they should to help citizens buy cheaper tickets more easily. And we will change that, now.”
2. Seize the opportunity to bring people together. Millions of people visit public sector websites every day, often trying to achieve similar or identical ends. It is time to start building systems to allow them to contact people in a similar situation, just as they’d be able to if queuing together in a job centre, but with far more reach and power. This does open the scary possibility that citizens might club together to protest about poor service or bad policies, but given recent news, if you were a minister would you rather know about what was wrong as soon as possible, or really late in the day (cf MPs‘ expenses, festering for years)?
3. Get a new cohort of civil servants who understand both the Internet and public policy, and end the era of signing huge technology contracts when the negotiators on the government’s side have no idea how they systems they are paying for actually work. Coming up with new uses of technology, or perceiving how the Internet might be involved with undermining something in the future is an essential part of a responsible policy expert’s skill-set these days, no matter what policy area they work in. It should be considered just as impossible for a new fast-stream applicant without a reasonably sophisticated view of how the Internet works to get a job as if they were illiterate ( a view more sophisticated than generated simply by using Facebook a lot, a view that is developed through tuition ). Unfashionably, this change almost certainly has to be driven from the center.
4. Resist calls from institutions of all sorts to change laws to give them back the advantages they previously had over citizens, and actively appoint a team to see where legislation is preventing possible Internet-enabled challenges to institutions that could do with shaking up. At the moment, this is mostly seen in the music and video fields, but doubtless it will occur in more fields in the next decade, many of them quite possibly less sexy but more economically and socially significant than a field containing so many celebrities.
5. Spend any money whatsoever on a centrally driven project to cherry pick the best opportunities to ‘be on the side of the citizen’ and drive them through recalcitrant and risk averse departments and agencies. Whilst UK government is spending £12-13bn a year on IT at the moment, almost none of that is being spent on projects which I would describe as fitting any of the objectives described above. And the good news, for a cash strapped era, is that almost anything meaningful that the government can do on the Internet will cost less than even the consulting fees for one large traditional IT project.
There are, obviously, more reasons why the Internet isn’t like electrification or shipping containers. But keeping the narrative simple is always valuable when proposing anything. The idea that a wave is coming that empowers citizens and threatens institutions makes government’s choice stark – who’s side do we take? History will not be kind to those that take the easy option.
To: Anyone thinking of running any reasonably developed country, any time soon.
The most scary thing about the Internet for your government is not pedophiles, terrorists or viruses, whatever you may have read in the papers. It is the danger of your administration being silently obsoleted by the lightening pace at which the Internet changes expectations. I’m not going to give examples of this change, others can do this far better than I. But you don’t need experts’ advice to tell which way the wind blows – if you can’t find any examples of changing expectations in your own life, driven by the internet, I can’t help you anyway: please point me to your successor.
This is a list of the top 5 major things any government of any developed nation should be doing in relation to the Internet, as I see it at the start of 2009. They are not in any order, and do not lack ambition – they are for the Next Government, after all.
Hire yourself some staff who know what the Internet really means for government, and fund a university to start training more who really understand both worlds: you’re going to need them. There just aren’t enough employed in any government anywhere yet to save you from being hopelessly outstripped by external progress. The citizen discontent resulting from massive shifts in expectation could wash your entire government away without you ever having anyone skilled enough to tell you why everyone was so pissed off. Your chances of truly reinventing what your government is are basically zero without such staff.
Free your data, especially maps and other geographic information, plus the non-personal data that drives the police, health and social services, for starters. Introduce a ‘presumption of innovation’ – if someone has asked for something costly to free up, give them what they want: it’s probably a sign that they understand the value of your data when you don’t.
Give external parties the right to interface electronically with any government or mainly public system unless it can be shown to create substantial, irrevocable harm. Champion the right fiercely and punish unjustified refusals with fines. Your starting list of projects should include patient-owned health records, council fault reporting services and train ticket sales databases. All are currently unacceptably closed to innovation from the outside, and obscurity allows dubious practices of all kinds to thrive.
Commission the world’s first system capable of large scale deliberation, and hold a couple of nation wide sessions on policy areas that you genuinely haven’t made your minds up on yet. When it is over, mail people who participated with a short, clear list of things you’ve done that you wouldn’t have done without them. Once you’ve made it work well, legislate it into the fabric of your democracy, like elections and referendums.
When people use your electronic systems to do anything, renew a fishing license, register a pregnancy, apply for planning permission, given them the option to collaborate with other people going through or affected by the same process. They will feel less alone, and will help your services to reform from the bottom up.
mySociety wants to see all these things happen. Get in touch if you are interested.
…so what of the future?
First, I am more convinced than ever that mySociety offers something quite unique, and something must survive if technology is going to be best applied on the side of the citizen. Despite the explosion of so called web 2.0 technologies being adapted by newspapers, government and other media companies, the tools mySociety builds remain unique. They don’t just involve repurposing generic new communications tools like blogs, they involve conceptualising how technology can empower people from first principles. Nobody else is in the UK even attempts to build services like WhatDoTheyKnow or TheyWorkForYou, they’re just too different from what’s out there to copy. And when we do build them, they get copied across the world – one of the things I didn’t expect five years ago is that I’d be celebrating tonight with Rob McKinnon, the man who took TheyWorkForYou and made it work in New Zealand, and being toasted from Australia via Twitter. But we know from the continued influence of newspapers, some born in the 19th century, that political media needs longevity to gain the reach and legitimacy required to transform whole systems and to challenge the expectations of whole populations. mySociety needs to work out how to be here not just in 6 months, but in 20 years.
To do this, however, we must do something about our funding. mySociety remains deeply financially insecure, and if we’re to celebrate our 6th birthday, let alone our 10th something urgent has to happen.
Next, we need to admit that we’ve shifted the culture of government internet usage less than we might have hoped over the last five years. Nevertheless, I honestly believe that a relatively minor shake-up at relatively low cost can see a massive step change in the way that government delivers services online, the way that it talks to citizens, and the way that it makes information available. But so long as the cult of outsourcing everything computer related continues to dominate in Whitehall, and so long as experts like Matthew and Francis are treated as suspicious just because they understand computers, little is going to change. Government in the UK once led the world in its own information systems, breaking Enigma, documenting an empire’s worth of trade. And then it fired everyone who could do those things, or employed them only via horribly expensive consultancies. It is time to start bringing them back into the corridors of power.
In one way that’s great for mySociety’s reputation that government progress has been so slow – even on a bankruptcy budget mySociety will continue to at least appear to out-innovate the entire UK government. But from a public welfare perspective it’s a tragic farce.
What we want from the government is technologies that empower and uplift, not depersonalise and degrade. mySociety wants to be part of this change, and I hope we don’t have to wait until a new government comes in to have a decent shot at slaying some of the shiboleths that stand in our way to decent reform.
Last, but not least, I want as many of you as possible to be part of making mySociety’s vision of easier, more accessible, more responsive democracy the minimum that people expect, not the best they can hope for. This will take lots of volunteers, and lots of funding funding and ideas and newspaper stories. It’ll take lots of brilliant coding and better design. It will take political leaders who understand that the internet is the big, unique chance their generation has to shake things up and get into the history books.
And, more than anything else, I want to do it with you people. I want to do it with mySociety.
All mySociety’s servers are named after British food food and drink – tea, cake, pimms etc. A couple of weeks ago we finally set up haggis.ukcod.org.uk as a server for little unofficial projects of all sorts. It is shiny, brand new, bullet fast and looking for people in the mySociety volunteer community to love it.
The first user is one of our most regular volunteers, Sam Smith, who has migrated and upgraded his never-quite-officially-launched site TheGovernmentSays.com. If you like the way TheyWorkForYou can email you when politicians talk about words or phrases of interest to you, then you will most likely find this indespensible – the same functionality but for government press releases and news from across the public sector.
If you’d be interested in an account on Haggis in order to do some mySocietyish work, just let us know.