As this post goes live, mySociety Head of Research Rebecca Rumbul is out in Mexico, hosting a session at the Open Government Partnership Global Summit.

She’s presenting our latest paper, which seeks to provide a firm foundation for future impact research, by asking the most basic questions about who actually uses civic technology and why.

The headline findings

It’s not a difficult read, but if you’re looking for some easy takeaways, we got ’em. Try these for starters:

• Civic technology is used by a wide spectrum of individuals, but there are big differences between countries, especially when comparing developed countries to developing countries.
• Generally, more men than women use civic tech (although this isn’t the case in the USA).
• In the USA and UK, civic tech users tend to be above the age of 45 (over 70%) and well educated: to degree level or higher.
• In Kenya and South Africa, civic tech users tend to be under the age of 45, and more individuals without higher educational qualifications participate through these platforms.
• Comparative to population of each participating country, users from ethnic minorities are under-represented.
• Confidence in civic tech websites is very high. In each country surveyed, an overwhelming majority of individuals (over 85%) believed that these websites help them to hold the government to account, and believed that the government would behave differently if they were unable to see the information contained on these sites. On average 97% of users would use these sites again.

Now go and read

At the heart of open data is the belief that information is empowering and allows citizens to hold their governments to account. The idea that transparency promotes good governance is so widely accepted that few people dare question it. But how solid is the evidence linking open data initiatives to intended impact? Do transparency measures necessarily lead to policy change? What about the impact and effectiveness of Civic Tech in general?

mySociety has been at the forefront of talking about impact and just last March organized the first Impacts of Civic Technology Conference addressing these questions. The conference brought together practitioners, activists and scholars who discussed various ways to define and measure impact and make civic technologies more effective.

Civic Tech is a rapidly growing field and covers a wide range of initiatives from open data groups to crowdsourcing and service delivery platforms and parliamentary monitoring websites. According to a Knight Foundation report, $431 million has been invested in Civic Tech from January 2011 to May 2013 and the sector has grown at the rate of 24% per year since 2008.

As the interest and funding in Civic Tech continues to grow, it is important to start thinking about ‘impact’ in more concrete terms. The focus on Civic Tech so far has been more geared towards how to get people involved and make the technology more accessible. When impact studies have been done, they have tended to focus on the adoption, awareness or quality of the platforms rather than the actual impact of the initiatives on people and communities.

The default position seems to be that once bureaucratic or technical problems are addressed, these technologies will inevitably produce favorable results. Very often organizations and practitioners fail to see that this is not always the case. The danger when we don’t think about impact in concrete terms is it might lead us to confuse output with outcomes. For instance, poorly thought out open data initiatives might lead to transparency for transparency’s sake. Open data while hugely important is not a one-stop solution for governance problems and corruption and often it requires more than freeing data to push government and institutions to carry out transformative actions.

We have also only recently begun to think about the unintended consequences of civic technologies. One of the criticisms for Civic Tech, especially in the Global South, has been that they often end up empowering the already empowered. It is argued that social and economic inequalities are further entrenched when these technologies are made available without considering the Digital Divide in these countries. Understanding impact also means that these questions are explored in more depth and detail.

The novelty of Civic Tech as a field means that in many cases, the kind of information needed to conduct good impact assessment is not available. In developing countries, where infrastructure, access and adoption continue to be problems, impact studies simply do not top the list of an organization’s priorities. It does not help that most findings are based on self-reporting and anecdotal evidence making verification rather complex. The difficulty of measuring change, however, should not deter Civic Tech organizations from prioritizing impact. There is great value in the work that Civic Tech activists are doing around the world but it pays to know in more certain terms just how their works have been changing things if and when they are. Empirical findings are crucial but so are qualitative insights.

We have to understand the mechanisms and processes of how and in what institutional contexts Civic Tech works in order to achieve better results and avoid unintended outcomes and wasted projects. And that means among other things, establishing metrics and theories of change, finding links between activities and results and producing robust evidence. The bottom line is that Civic Tech is not an end in itself but a means to propel meaningful action and without a clear and sustained focus on results it can be very easy to forget that.

If you are a Civic Tech organization or researcher carrying out or planning to do an impact evaluation of your work, what have been your experiences been so far? What can Civic Tech organizations teach each other about defining and measuring impact? We would love to hear your perspectives.

Rubeena Mahato is working on a digital democracy project at mySociety as an Open Society Rights and Governance Grantee. The views are of the author.