How do you pin down the intangible?
More specifically, how do we understand something as nebulous as why people visit our websites? For better or worse, Google Analytics can’t provide brain scans to show our users’ motivations, so the only solution is to ask them.
Between May and August of this year, if you visited TheyWorkForYou, you might have seen a pop-up inviting you to answer a few questions. Some users also gained new site features. It was all part of a concerted drive to understand more about why you use the site, and part of our wider research programme funded by The William and Flora Hewlett Foundation.
Many thanks if you were one of those who responded. In all, the survey generated a couple of a million data points, which has certainly given our resident data-cruncher, Nick, plenty to wade through.
Why do people use TheyWorkForYou?
We know why we run TheyWorkForYou: it aims to make Parliament—often perceived as ‘not for the likes of me’—more accessible for everyone. But what we don’t know is whether or not we are achieving that aim.
You see, with our other sites, WhatDoTheyKnow, FixMyStreet or WriteToThem, there is a set path that indicates a successful visit. The user lodges an FOI request, reports a problem, or writes to their representatives.
TheyWorkForYou, on the other hand, does not promote an action. There are actions you can take, like signing up for email alerts, or adding an annotation to a debate—but these are secondary to the site’s primary function, which is simply to present information. There’s no measurable metric that can tell us how informed people are after they’ve read one of our pages, nor what they do with that information. Hence the survey.
People hate pop-ups
One of our first findings came as no surprise: yes, we hate pop-ups too*! And yet, we wanted to keep users on the site while they answered our questions, so that’s the interface we used.
75.7% of people dismissed the pop-up when it appeared. We’d probably have done the same. We’d like to meet the remaining 24.3% of people and shake their hands.
The pop-up asked users what they would go on to do with the information they’d found on TheyWorkForYou. And most people gave the same answer they would if a shop assistant had asked: in fact, 75.2% said they were ‘just browsing’, with no further plans.
That makes us really curious as to what provoked that browsing: are they regular visitors, or were they sent to TheyWorkForYou via a tweet or Facebook reminder? Maybe one of our email alerts was the trigger. Google Analytics will allow us to see which site referred visitors to us, but of course we can’t match that data with individual motivations.
A further 10.9% of users had come with the intention of contacting their MP: TheyWorkForYou’s MP pages feature a prominent button inviting you to do that, and clicking on it takes you to our site WriteToThem.
6.6% of people were using the site for work purposes. Only 3.1% of people had plans to share content via social media, a statistic we’d like to increase if we can.
During the research period, some users (not necessarily those who had seen the pop-up) saw an addition to person pages and debate pages: a “where next” button, which led to some suggestions on how to use the information they’d found on the site.
This button was not widely used: just 0.1% of people clicked it. As so often with research findings, this result leads to more questions: did people already know what they wanted to do? Did they not want to do anything? Were they, perhaps, annoyed at the suggestion they might need guidance?
Or was the button just poorly positioned? We may run some further testing to find out if a more prominent placing leads to more clicks.
Of those few people who did click through for advice, the most popular options were to use WriteToThem to contact their MP about what they’d just read, or to contact the Parliamentary Ombudsman—it seems that on the few occasions when people do welcome a next step, they do want to take action.
Follow the news
Meanwhile, we were also taking a concentrated look at our site analytics. The test period coincided with the Labour party leadership contest and David Cameron’s cabinet announcements, and we saw very clearly how current affairs have an effect on which pages people visit.
John Whittingdale (the new Culture Secretary)’s voting record and Jeremy Corbyn’s profile page were far and away the most visited pages by several thousand (they’re responsible for the spike on the left of the graph that you see below). Also popular were the profile pages of the other Labour candidates, and Corbyn’s voting record.
But the other pages visited were very diverse: an extreme example of the long tail effect.
We reckon it takes more than a few seconds to take in a parliamentary debate, so one of the things we measure on Google Analytics is how many people stick around for more than 7 seconds, and again, for more than ten seconds.
Actually, we get pretty good results for the 7-second cut-off. Recent work we did on the design of the site’s debate pages had an effect that was a real cause for celebration: a massive 46% drop in the bounce rate (ie people who leave the site after viewing just one page)
But only 38% then stick around long enough for the second trigger, which doesn’t indicate as much involvement as we might have hoped for.
What have we learned?
At the end of this experiment, we reckon we’ve learned four main things:
- We need to carry on thinking about how to encourage more engagement with TheyWorkForYou’s content
- Current affairs are a huge driver to the site, and we can build on this via social media—and especially by encouraging our users to share the content they’ve found
- We need to conduct more experiments to see whether people genuinely don’t want advice about how to use the information they’ve found, or whether they would take it if it was more prominently flagged up
- There’s also a motivation to dig in deeper to motivations: what made people come to the site, if they are ‘just browsing’? What exactly are they doing ‘for work’? And if they want to contact their MP, what was the trigger for that?
If you have thoughts about any of these, as regards your own usage of TheyWorkForYou, please do feel free to comment below. And if you’d like to find out more about our research programme there’s plenty of information here, along with news of our annual research conference, TICTeC.
*And we know who to blame.
Image: The U.S. Printing Co., Russell-Morgan Print, Cincinnati & New York [Public domain] via Wikimedia Commons.
How many people visit mySociety’s websites?
That’s a question we don’t ask ourselves as much as many other organisations. Much of our current funding is dependent on transactions (that is, the number of people using the site to complete an action such as making an FOI request, writing to a politician, or signing up to receive emails when their MP speaks), and rightly so, since that is a better measure of the sites’ actual effectiveness.
All the same, visitor numbers* do tell us about things like how much public awareness there is of what we do, and which of our sites is more visible than the others, so it’s good to take a proper look now and again.
Which of our UK sites is most visited?
By far our most popular site in terms of visitor numbers is our Freedom of Information site WhatDoTheyKnow. With over 4.5 million visitors 2014-15, it’s had three times more users than its closest competitor, TheyWorkForYou.
As well as allowing users to submit FOI requests, WhatDoTheyKnow also puts the responses into the public domain, so that the information becomes openly available. Every request receives, on average, twenty readers, meaning that transactions do not show the whole picture for this site.
WhatDoTheyKnow’s user numbers are also rising steadily. It’s up 8% on last year, and March 2015 was its highest month for unique users since its launch in 2008, at 470,509.
Which is least visited?
This dubious honour goes to WriteToThem, which nonetheless welcomed 457,209 visitors during the year, either helping them to write to their representatives, or simply showing them who those representatives were.
This was still a decent 11% rise on the previous year, despite a real rollercoaster where some months dipped substantially from the previous year.
Which made the most gains in the last year?
FixMyStreet saw the biggest percentage change, with a 21% rise in visitor numbers compared to the previous year; we talked a bit more about that in a recent blog post. WhatDoTheyKnow had the highest rise in actual visitor numbers: over 360,000 up on 2013-14.
Which fell by the most in the last year?
TheyWorkForYou saw a 12% drop in visitor numbers year on year (and also the biggest drop in real terms)—disappointing, but something we hope to rectify with the new voting pages, an ongoing process of rolling redesign, and some grassroots outreach.
How much effect do external events have on visitor numbers?
We already know that, as you’d expect, when Parliament is on holiday, MPs, debates and legislation aren’t in the news, and TheyWorkForYou visitor numbers fall. There’s also a weekly pattern for all our sites, where far fewer people use them at the weekends, presumably indicating that lots of our users access them from work.
It’s too early to say exactly what effect the election has had on our sites: as I write, people are eagerly checking out the voting records of newly-appointed cabinet ministers on TheyWorkForYou.
One thing we know for sure is that fewer people will have been using WriteToThem, because there have been no MPs to write to for the last few weeks. We’ve removed the “write to your MP” links from TheyWorkForYou, which always drove a good deal of WriteToThem’s traffic.
FixMyStreet enjoyed a boost back in June, when it was featured on the Channel 4 programme ‘The Complainers’—and the nice thing is, user numbers never receded back to their previous levels after the programme was over. Maybe people just need to use FixMyStreet to see how useful it is.
How many people visit mySociety’s UK websites in total?
This is a difficult figure for us to produce with accuracy, because we don’t trace whether you’re the same person visiting a number of our different sites.
However, the aggregate total of visitors to all our UK sites (WriteToThem, TheyWorkForYou, FixMyStreet and WhatDoTheyKnow) for 2014-15 is 6,983,028. Thanks very much if you were one of them 🙂
How can I help?
Glad you asked! If you find mySociety sites useful, you can help us spread the word by telling friends, sharing the URLs with any groups you are a member of, posting on Facebook or Twitter, or writing to your local paper.
We have a number of materials for FixMyStreet which can be found here; we hope to create similar materials for our other sites too, and we’ll make sure we announce it on here when we do.
* Note: all references to ‘users’ refer to unique users within the period discussed. So, users in a year means individual people who may have visited any number of times over that year, but are only counted once; same with monthly users.
At mySociety we like transparency – it’s baked into most of our projects.
TheyWorkForYou attempts to make it easier to find out what your MP has been doing in Parliament. WhatDoTheyKnow tries to make it easier to find out what’s going on inside other public bodies. FixMyStreet and the upcoming FixMyTransport also use transparency to help get problems resolved.
We think transparency is a good thing for many reasons, but one of its rarely mentioned virtues is how valuable transparency can be for the people within the organisations which are transparent.
Transparency can be useful because it means people outside an organisation can make critical, constructive suggestions about how you can improve, and it lowers the odds that people in one part of your own organisation will be ignorant of the activities of people in other parts.
We were not highly prescriptive in our instructions, and we certainly didn’t ask Tobias to ‘discover’ pre-determined findings. All we did was ask Tobias to find out who was coming to the sites, what they were doing, and whether or not the sites could be considered to be succeeding. We didn’t do it for a PR stunt: we did it so we could learn from our mistakes, and so that we could share those learnings with others who might benefit.
His detailed, quantitative analysis holds the sites up to mySociety’s own stated aims, for the first time. And we’ve published both documents, in full, below.
Swings and Roundabouts
It was great to discover that we have, indeed, attained some of our goals by running these sites. For example, one of the reasons we set up TheyWorkForYou and WriteToThem was to make representatives accessible to people who were newcomers to the democratic process. It was therefore heartening to read that 60% of visitors to TheyWorkForYou had never previously looked up who represents them, and two in five users of WriteToThem have never before contacted one of their political representatives.
But, as you would expect with any properly neutral evaluation, it’s not all good news. Our sites aim to reach a wide range of people, but compared to the average British internet user, WriteToThem users are twice as likely to have a higher degree and a higher income. It also seems that users are disproportionately male, white, and over 35. These figures and many more are available within these highly readable papers – Tobias did a terrific job in gathering and analysing a huge amount of data, and then making it easy to understand.
These reports are rich with data, from how visitor numbers boomed during the MPs’ expenses scandal to which MPs most people sign up to receive alerts about. You can also read how a budget airline almost brought a site to its knees in 2007; what part Joanna Lumley plays in our history; and how many visits to TheyWorkForYou actually come from within Parliament itself.
TheyWorkForYou and WriteToThem have inspired many people around the world to set up similar (and not so similar) sites inspired by the vision of using the Internet to lower barriers to democracy. However, until now we’ve never seen a really clear-eyed assessment of what seems to work, and what doesn’t.
If you’re at all interested in using the Internet to engage people with democratic systems, Tobias Escher’s excellent research papers will make a compelling read. Thank you Toby!
…and do come back and tell us what you found interesting.
We hope to publish two evaluation reports like this at the start of each new year from now on. Next year’s sites will probably be FixMyStreet and WhatDoTheyKnow. Do get in touch if you’d like to input!