Today, we’re sharing research conducted on the impact of online Freedom of Information technology, including our own platform Alaveteli.
Researchers Savita Bailur and Tom Longley spent three months gathering first-person experiences, analysing data and assessing existing literature to answer this question:
“In what circumstances, if any, can the Freedom of Information tools mySociety builds be shown to have measurable impacts on the ability of citizens to exert power over underperforming institutions?”
You can read their findings here:
1. Literature review [PDF]
The research was conducted in three parts: first, Tom and Savita reviewed existing literature on the impact of FOI, particularly FOI online, to form a baseline of existing knowledge in the area.
They went on to interview people who run, or ran, FOI sites in 27 different countries. They used the resulting transcripts for qualitative research, pulling out common themes to help them draw conclusions.
Finally, they were able to use these insights to create a list of critical success factors for those implementing FOI (especially Alaveteli) websites.
Why did we conduct this research, and why now?
Alaveteli has had a period of intense growth over the last three years – but it would be irresponsible of us to continue its promotion without assessing its true worth and impact.
This is best learned from the people who are at the coalface – the implementers (as Tom and Savita mention in the final research, a fuller study would have allowed them to include government workers and the sites’ end users, too, but that’s perhaps something for the future).
Alaveteli was created with the best intentions – to allow anyone, anywhere to put questions to the people and institutions in power – but it is important to assess whether those intentions have been realised.
We need to ensure that we have spent our efforts and our funders’ money responsibly, and that we are not wasting resources by making poor decisions.
mySociety’s Head of Research, Dr Rebecca Rumbul, says, “This report confirms that the basic model does work, with the UK site WhatDoTheyKnow.com operating as a well-used civic resource with thousands of users per month.
“Whilst the research shows that our partners implementing Alaveteli in their own countries are demonstrably up to the technical challenge of running these sites, it identifies the importance of governmental relations and receiving the right support in the early stages of implementation.
“We now hope to build on this research to better understand how to maximise the use and effectiveness of our platforms around the world in empowering citizens to engage with governments and decision-makers.”
The research was made possible by a grant from the Open Society Foundations.
As the literature review confirms, this type of study has never been done before – and with practitioners speaking to the researchers from within many different cultural backgrounds and political regimes (they interviewed implementers of 20 Alaveteli instances, from Australia to Uruguay), we are in a unique position to take a global view on the subject. For a fully-rounded picture, the study also spoke to implementers of seven sites running non-Alaveteli FOI software.
Of the experience, Savita and Tom say “We were so impressed by the dedication and determination of all the implementers in wanting to raise awareness of FOI and seeing Alaveteli as the platform to do this (even taking into account constructive criticism). The research experience was also great.”
The end result? Take a look for yourself – if you have the slightest interest in online democratic technologies or government-to-citizen information sharing on an international basis, it’s compelling reading.
What we’ll take away
There are learnings for us here, although it was great to hear such consistent praise for the Alaveteli platform and the community that has been created around it.
mySociety’s Director Tom Steinberg said, “We will certainly be looking carefully at the recommendations that have come from this report.
“This will include decisions about how to share best practice across the Alaveteli community, and not just in the technical areas.
“We’ll also be looking hard at the issue of how to ensure consistency in the analytics that are collected by different sites. And we very much hope to return to the subject in a couple of years’ time, when today’s new sites have become established, in order to conduct a follow-up piece of research.”
What impact do mySociety sites actually have? We could lose a lot of sleep over this important question – or we could do something concrete, like conducting academic research to nail the answers down for once and for all.
As slumber enthusiasts, we went for the research option – and, to help us with this commitment we’ve recently taken on a new Head of Research, Rebecca Rumbul. Watch this space as she probes more deeply into whether our tools are making a difference, both in the UK and abroad.
Even before Rebecca came on board, though, we had set a couple of research projects in motion. One of those was in partnership with the University of Manchester, funded by the ESRC, which sought to understand what impact our core UK sites (FixMyStreet, WriteToThem, TheyWorkForYou and WhatDoTheyKnow) have on their users, and specifically on their level of political engagement.
Gateways to participation
It’s perhaps worth mentioning that, while our sites appear, on the face of it, to be nothing more than a handy set of tools for the general citizen, they were built with another purpose in mind. Simply put, each site aims to show people how easy it is to participate in democracy, to contact the people who make decisions on our behalf, and to make changes at the local and national levels.
Like any other online endeavour, we measure user numbers and transaction completions and time spent on site – all of that stuff. But one of the metrics we pay most attention to is whether users say they are contacting their council, their MP or a public body for the first time. Keeping track of this number ensures that we’re doing something to open democratic avenues up to people that haven’t used them before.
But there are plenty more questions we can ask about the impact we’re having. The University of Manchester study looked into one of them, by attempting to track whether there was a measurable change in people’s political activity and engagement after they’ve used one of our sites. On Monday, researchers Rachel Gibson, Marta Cantijoch and Silvia Galandini presented their findings to an attentive audience at King’s College London.
The project has taken a multi-pronged approach, asking our users to complete questionnaires, participate in online discussions, or keep a 12-week diary about political and community engagement (thanks very much to you, if you were one of the participants in this!). The result was a bunch of both qualitative and quantitative data which we’ll be able to come back to and slice multiple ways in the future – Gibson says that they haven’t as yet managed to analyse all of the free text diaries yet, for example.
In itself this study was interesting, because not much research has previously been conducted into the impact of digital civic tools – and yet, as we know from our own international activities, people (not least ourselves) are launching sites all over the world based on the premise that they work.
Some top-level conclusions
The research will be published in full at a future date, and it’s too complex to cover all of it within the confines of a short blog post, but here are just a few of the takeaway findings:
- A small but quantifiable uplift in ‘civic participation’ was noticed in the period after people had used our sites. This could include anything from working with others in the local community to make improvements, to volunteering for a charity.
- No change was found in the level of political influence or understanding that people judged themselves to have. This was a surprise to the researchers, who had thought that users would feel more empowered and knowledgeable after contacting those in power, or checking up on their parliamentary activity.
- As with our research back in 2011, the ‘average’ user of mySociety sites was found to be white, above middle-aged, and educated to at least degree level. Clearly this is a userbase which we desperately need to expand, and we’ll be looking carefully – with more research and some concentrated outreach efforts – at how we can do that.
- Users tended to identify themselves as people who already had an interest in politics. Again, here is an area in which we can improve. Of course, we’re happy to serve such users, but we also want to be accessible to those who have less of a baseline interest.
- Many users spoke of community action as bringing great satisfaction. In some cases, that was getting together in real life to make improvements, but others saw something as simple as reporting graffiti on FixMyStreet as an action that improved the local area for everyone.
Thanks to the University of Manchester researchers for these insights and for presenting them so engagingly. We’ll update when the full research is available.
That year saw many debates in Parliament on topics that have since become very familiar: the question of whether the tax on cigarettes should be raised; whether cigarettes should be advertised on television, whether smoking should be allowed in public places, and whether warnings should be printed on packets.
Rich and fascinating stuff for any social historian – and it’s all on TheyWorkForYou.com.
Hansard is an archive
Hansard, the official record of Parliament, is a huge historic archive, and whatever your sphere of interest, it is bound to have been debated at some point.
Browsing through past debates is a fascinating way of learning what the nation was feeling: worries, celebrations, causes for sorrow – all are recorded here.
How to use TheyWorkForYou to browse historic debates
TheyWorkForYou contains masses of historic information: House of Commons debates back to 1935, for example, and details of MPs going back to around 1806. You can see exactly what the site covers here.
There are various ways to search or browse the content. Start with the search box on the homepage – it looks like this:
You can do a simple search right from this page, or choose ‘more options’ below the search box to refine your search.
We’ll look at those advanced options later, but let’s see what happens when you input a simple search term like ‘smoking’.
Here (above) are my search results, with my keyword helpfully highlighted.
By default, search results are presented in reverse chronological order, with the most recent results first. If you are particularly interested in historical mentions, you may wish to see the older mentions first.
That’s easy – just click on the word ‘oldest’ after ‘sorted by date’:
You’ll notice a few other options here:
- Sort by relevance orders your results with the most relevant ones first, as discerned by our search engine. This will give you those speeches with the most mentions of your keyword ahead of those where it is only mentioned once or twice.
- Show use by person displays a list of people who have mentioned your keyword, with the most frequent users at the top. This can be fascinating for games such as “who has apologised the most?” or “who has mentioned kittens most often?”
Click through any of the names, and you’ll see all the speeches where that person mentioned your keyword.
That’s a good start – but what if there are too many search results, and you need some way to refine them? You’ll notice from my screenshots above that there are (at the time of writing) over 10,000 mentions of smoking.
That’s where Advanced Search comes in. You can access it from a few places:
- The ‘more options’ link right next to the search box on search results pages (see image below)
- The ‘more options’ link below the search box on the homepage (see image below)
- Or just navigate directly to our dedicated Advanced Search page (see image below)
Whichever way you arrive at it, the Advanced Search page helps you really get to the content you’re interested in.
The pink box on the right gives you some tips for effective searching.
For example, just as with Google, you can search for exact phrases by putting your search term within quotation marks. Otherwise, your results will contain every speech where all your words are mentioned, even if they’re not together. For phrases like “high street”, this could make a real difference.
Even if you are only searching for a single word, you can put it in quotation marks to restrict the use of ‘stemming’ – so, for example, a search for the word house will also return results containing houses, housing and housed, unless you put it in quotation marks.
You can exclude words too: this can be useful for minimising the number of irrelevant results. So, for example, you might want to find information about the town of Barking, but find that many of your results are debates about dogs. Simply enter the search term “barking” -dogs. The minus sign excludes the word from your search.
In the main body of the page, you’ll also see options to restrict your search to within certain dates, or a specific speaker, or a department, section (eg Scottish Parliament or Northern Ireland Assembly) and even political party.
Get stuck in
The best way to see what you can find is to dig in and give it a go. If your search doesn’t work for you the first time, you can always refine it until it does.
Let us know if you find anything interesting!
We’re always busy at mySociety. Running projects in the UK, and helping international partners get started on their own projects – it takes time and energy. It’s easy to get swallowed up in the day-to-day logistics and never take a step back.
But it’s important to make sure that our projects are actually having positive impacts. To that end, we’ve instigated a number of research projects – and you may have seen our recent ad for a Head of Research, now happily filled and coming on board soon.
One piece of research very near completion involves a review of FOI online technologies around the world, including our Alaveteli platform.
Researchers Savita Bailur and Tom Longley have focused on three areas: a literature review to see what research is already out there, in-depth interviews with people who have installed FOI technologies in many different countries, and the compilation of a list of critical success factors. You can read more about them, and their approach in their introduction over on the Alaveteli blog.
We’ll be publishing their final report in full as soon as it’s ready, but here’s an interim update from Savita.
We’re more than half way through the research on the impact of technologies on FOI around the world.
A literature review of the publications we have found on impact of FOI technologies is now in draft form, and we will share that soon.
The practitioner review is coming along. So far we’ve spoken to people who run Alaveteli installations in the European Union, Australia, Bosnia, Canada, Czech Republic, Guatemala, Hungary, Israel, Italy, Kosovo, Liberia, Macedonia, New Zealand, Romania, South Africa, Spain, Tunisia, Uganda, Ukraine and Uruguay.
We’ve also spoken to non-Alaveteli FOI site implementers such as Acceso Inteligente in Chile, MuckRock and IFOIA in the USA, Open Data Georgia, and FragdenStaat in Germany.
Thanks to all of you who kindly gave your time so far – there’s certainly a lot happening out there and some very inspiring, dedicated people! We still have a few more people to talk to, but thought we’d give a quick progress report on findings so far:
- FOI websites are definitely making the process of FOI requests much easier. They help citizens and journalists organise their requests. In publishing all exchanges, they make FOI requests more useful for other people who might be searching for similar information.
- However, the concept is still new – it’s even seen as exotic and possibly rather rude in places where email is regarded as an informal form of communication – so the first step seems to be public (and government) awareness raising. Without it, there is a risk of “transparency theatre” as one respondent delicately put it.
- Users vary (everyday “citizen” users and/or activists) but very few sites have much demographic information about their users.
- One common finding from Australia to Canada and in-between is that the use by journalists of Alaveteli is not as extensive as expected, because it takes too long and they fear losing the competitive scoop (as all exchanges are publicly available immediately)
- Implementers of FOI sites are mostly funded solely by grants, in-kind contributions or not at all. Some are managing to make a living from running FOI sites. Passion and dedication is a very common theme across all sites!
- By and large, obstacles to realising effective FOI exist everywhere. There is evidence that governments are unfamiliar with FOI (e.g. Tunisia), can be obstructive (e.g. Canada or more specifically the Quebec site, where online FOI requests were originally considered invalid because they were not “in writing”) or openly unwelcoming (e.g. Germany, where a copyright law prohibits sharing of FOI information, or Hungary, where civil society organizations working on FOI have faced intimidation).
- Hunting down the email addresses of who to contact in public institutions in the first place is one of the most challenging tasks an implementer can face (e.g. Uruguay). Many aren’t published, are inactive, personal emails or simply respond with “quota full” messages. Conversely, some government officials have started asking for their email to be included in the FOI site.
- Some “vexatious” or facetious requests actually raise the profile of FOI, e.g. a New Zealand request asking if the prime minster was a “shapeshifting reptilian alien” created great publicity for the site
- In Australia, RightToKnow used FOI to find out what a government ministry thought about the RightToKnow site itself. If you can find more of these examples, please get in touch.
- FOI implementers face difficult decisions of whether to collaborate with (e.g. Uganda and Canada) or confront (e.g. Spain) government.
- Alaveteli implementers consider the Alaveteli support community one of the most valuable aspects of the software and there is often a regional/cultural assistance, e.g. Spain’s Tuderechoasaber assisting Guatemala in starting Guateinformada, or Uganda working with neighbours. However, the code needs to be more customisable.
- Sustainability is still an issue. This is particularly the case for implementers in “developed” countries who may find fundraising for FOI work more challenging than in countries where donors are investing heavily in transparency and accountability (e.g. World Bank in Uganda).
… and there are many more interesting findings to follow!
All the interviews are now being transcribed (the interviewees having agreed to being recorded) and we are analysing them through themes in qualitative software.
We have more interviews over the next fortnight. The literature review, report of findings and brief strategy document will be out in September – stay tuned.
It is a cliché for any manager to say that they are proud of their team, and mildly nausea-inducing to listen to anyone who goes on about it too long. However, the purpose of this post is to argue that the world would benefit from a new kind of post-graduate Masters programme – something that is hard to do without describing the virtues of the type of people who should come out of it. So please bear with me, and keep a sick bag to one hand.
mySociety’s core development team is very, very good. But they’re not just good at turning out code. Louise Crow, for example, has a keen eye for things that will and won’t make a difference in the offline world, as well as the skills to build virtually whatever she can think of. And the exact same thing is true of the whole coding team: Duncan, Matthew, Edmund and Dave in the current team, plus Francis, Chris and Angie before them.
mySociety didn’t give these people their raw talent, nor the passion to be involved with projects that make a difference. What it has given them, though, is the chance to spend a lot of time talking to each other, learning from their triumphs and their mistakes, and listening to users. This space and peer-contact made them into some of the world’s few genuine experts in the business of conceptualising and then delivering digital projects that deliver new kinds of civic and democratic benefits.
So, why am I sitting here unashamedly blowing my colleagues trumpets like this? (I don’t have these skills, after all!) Well, in order to point out that there are quite simply far too few people like this out there.
Too few experts
“Too few for what?” you may well ask. Too few for any country that wants to be a really great place to live in the 21st century, is my answer.
There is barely a not-for-profit, social enterprise or government body I can think of that wouldn’t benefit from a Duncan Parkes or a Matthew Somerville on the payroll, so long as they had the intelligence and self-discipline not to park them in the server room. Why? Because just one person with the skills, motivation and time spent learning can materially increase the amount of time that technology makes a positive contribution to almost any public or not-for-profit organisation.
What they can do for an organistion
Such people can tell the management which waves of technology are hype, and which bring real value, because they care more about results than this week’s craze, or a flashy presentation. They can build small or medium sized solutions to an organisation’s problems with their bare hands, because they’re software engineers. They can contract for larger IT solutions without getting ripped off or sold snake oil. And they can tell the top management of organisations how those organisations look to a digital native population, because they come from that world themselves.
And why they don’t
Except such experts can’t do any of these things for not-for-profit or public institutions: they can’t help because they’re not currently being employed by such bodies. There are two reasons why not, reasons which just may remind you of a chicken and an egg.
First, such institutions don’t hire this kind of expert because they don’t know what they are missing – they’re completely outside of the known frame of reference. Before you get too snarky about dumb, insular institutions, can you honestly say you would try to phone a plumber if you had never heard that they existed? Or would you just treat the water pouring through the ceiling as normal?
Second, these institutions don’t hire such experts because there just aren’t enough on the market: mySociety is basically the main fostering ground in UK for new ones, and we greedily keep hold of as many of our people as possible. Hands off my Dave!
Which leads me to the proposal, a proposal to create more such experts for public and non-profit institutions, and to make me feel less guilty about mySociety hoarding the talent that does exist.
Describing the Masters in Public Technology
The proposal is this: there should be a new Masters level course at at least one university which would take people with the raw skill and the motivation and puts them on a path to becoming experts in the impactful use of digital technologies for social purposes. Here’s how I think it might work.
In the first instance, the course would only be for people who could already code well (if all went well, we could develop a sister course for non-coders later on). Over the course of a single year it would teach its students a widely varied curriculum, covering the structure and activities of government, campaigns, NGOs and companies. It would involve dissecting more and less impactful digital services and campaigns, like biology students dissect frogs, looking for strengths and weaknesses. It would involve teaching the basics of social science methodologies, such as how to look for statistical significance, and good practice in privacy management. It would encourage good practice in User Experience design, and challenge people to think about how serious problems could be solved playfully. It would involve an entire module on explaining the dos and don’t of digital technology to less-literate decision makers. And most important, it would end with a ‘thesis’ that would entail the construction of some meaningful tool, either alone or in collaboration with other students and external organisations.
I would hope we could get great guest lecturers on a wide range of topics. My fantasy starter for 10 would include names as varied in their disciplines as Phil Gyford, David Halpern, Martha Lane Fox, Ben Goldacre, Roz Lemieux, William Perrin, Jane McGonigal, Denise Wilton, Ethan Zuckerman, as well as lots of people from in and around mySociety itself.
What would it take?
I don’t know the first thing about how universities go about creating new courses, so having someone who knew about that step up as a volunteer would be a brilliant start!
Next, it would presumably take some money to make it worth the university’s time. I would like to think that there might be some big IT company that would see the good will to be gleaned from educating a new generation of socially minded, organisation-reforming technologists.
Third, we’d actually need a university with a strong community of programmers attached, willing and ready to do something different. It wouldn’t have to be in the UK, either, necessarily.
Then it would need a curriculum, and teaching, which I would hope mySociety could lead on, but which would doubtless best be created and taught in conjunction with real academics. We’d need some money to cover our time doing this, too.
And finally it would need some students. But my hunch is that if we do this right, the problem will probably be fending people off with sticks.
I’m genuinely not sure – I hope this post sparks some debate, and I hope it provokes some people to go “Yeah, me too”. Maybe you could tell me what I should do next?