When working with data that you didn’t set out to gather you have to be careful to think about what the data actually means, rather than what it seems to be saying. As an example, one of the “interesting” side effects of FixMyStreet is a database of places people have reported dog poop (or “dog fouling” as it tends to be called academically). We now have over 20,000 locations across the UK where nature’s call has both been heard, and reported.

My first thought when learning about this data was “that’s a lot of dog poop!” but it turns out 20,000 dog poops is not a lot of dog poop at all. There are an estimated 8.5 million dogs in the UK, assuming (on average) each one poops once a day, they’ll produce over 3.1 billion poops a year.

So actually, 20,000 poops over nine years is nothing compared to the amount of pooping going on. But just because our data is a drop in the bucket doesn’t mean we can’t learn interesting things from it. The first question to ask is if we have a representative sample of where all this dog fouling is going on. The answer, sadly, is no. But the reasons for that answer raise further questions – which is interesting!

When you map the location of dog poo complaints in England against the Index of Multiple Deprivation [1], you get this:

This tells us that reports about dog fouling are roughly parabolic – there are more in areas in the middle than those that are either very deprived or very not.

This is interesting because when Keep Britain Tidy actually went out into the world and checked (p. 14), they found this:

This graph tells a very different story, where dog fouling gets worse the more deprived the area. But why is this? And why doesn’t our data tell the same story?

One reason we would expect more dog poop in the most deprived areas is that the most deprived areas are more urban. Taking the same IMD deciles and using the ONS’s RUC categories to apply a eight point ‘ruralness’ scale (where 1 is ‘Urban major conurbation’ and 8 is ‘Rural village and dispersed in a sparse setting’) lets us see the average ‘ruralness’ of each decile. While this reflects that deprivation is spread across urban and rural areas – the most deprived areas tend to be more urban.

As urban areas have fewer natural places to dispose of dog waste, and the most deprived areas are more urban, we would expect the most deprived areas to have more dog fouling.[2] We also know that measures that contribute to IMD scores (such as crime levels) are related to trust and social cohesion in an area.[3] When social cohesion is lower, we would expect more dog fouling because owners feel less surveyed and are less concerned with the opinion of neighbours. The real world increase reported by the Keep Britain Tidy survey supports these relationships.

The drop off in our reported data compared to the real world can be explained by features of the general model for understanding FixMyStreet reports — some measures of deprivation are correlated with increased reports (because they relate to more problems) and others with decreased reports (because they hurt the ability or inclination of people to report). We would also expect areas with worse deprivation to have fewer reports because of disengagement with civic structures.

Quickly checking the English dog fouling data (so only 17,103 dog poops) against the same model confirms that significant relationships exist for the same deprivation indexes as the global dataset with the largest effect size of a measure of deprivation being for health – as health deprivation in an area goes up, reports of dog fouling increase.

What this tells us is that our dog data (and probably our data more generally) is clipped in areas of the highest deprivation. We’re not getting as many reports as the physical survey would suggest and so our data has very real limits in identifying the areas worse affected by a problem.

This is a lesson in being careful about interpreting datasets you pick up off the ground – if you used this data to conclude the most deprived areas had a similar dog poop problem to the least deprived areas you would be wrong. Because we have an independent source of the real world rate of problems, we can see there is a mismatch between distribution in reports and reality. Using this independent data of ‘actual problems’ for one of our categories makes us more aware that there is negative pressure on reports in highly deprived areas.

If you’d like to learn more about the history of dealing with dog poo on the street (and who wouldn’t want to learn more about that!) – I’ve very generously gone into more detail here.

[1]: An index that combines thirty-seven indicators from seven domains (income, health, crime, etc) to provide a single figure for an area that is indicative of its level of deprivation relative to other areas.

[2]:This is relative. Rural areas still have problems with bagged dog poo (“the ghastly dog poo bauble” hanging from branches – as MP Anne Main put it). There is also a risk to the health of cows from dog fouling in farmland – so there are unique rural dog poo problems.

[3]: Ross et al. found “People who report living in neighborhoods with high levels of crime, vandalism, graffiti, danger, noise, and drugs are more mistrusting. The sense of powerlessness, which is common in such neighborhoods, amplifies the effect of neighborhood disorder on mistrust.”

Header image: https://www.flickr.com/photos/scottlowe/3931408440/

I’m just a few weeks into my position of Research Associate at mySociety and one of the things I’m really enjoying is the really, really interesting datasets I get to play with.

Take FixMyStreet, the site that allows you to report street issues anywhere in the UK. Councils themselves will only hold data for the issues reported within their own boundaries, but FixMyStreet covers all local authorities, so we’ve ended up with probably the most comprehensive database in the country. We have 20,000 reports about dog poop alone.

Now if you’re me, what to do with all that data? Obviously, you’d want to do something with the dog poop data. But you’d try something a bit more worthy first: that way people won’t ask too many questions about your fascination there. Misdirection.

How does it compare?

So, starting with worthy uses for that massive pile of data, I’ve tried to see how the number of reports in an area compares against other statistics we know about the UK. Grouping reports into ONS-defined areas of around 1,500 people, we can match the number of reports within an area each year against other datasets.

To start with I’m just looking at English data (Scotland, Wales and Northern Ireland have slightly different sets of official statistics that can’t be combined) for the years 2011-2015. I used population density information, how many companies registered in the area, if there’s a railway station, OFCOM stats on broadband and mobile-internet speeds, and components from the indices of multiple deprivation (various measures of how ‘deprived’ an area is, such as poor health, poor education prospects, poor air quality, etc) to try and build a model that predicts how many reports an area will get.

The good news: statistically we can definitely say that some of those things have an effect! Some measures of deprivation make reports go up, others make it go down. Broadband and mobile access makes them go up! Population density and health deprivation makes them go down.

The bad news: my model only explains 10% of the actual reports we received, and most of this isn’t explained by the social factors above but aspects of the platform itself. Just telling the model that the platform has got more successful over time, which councils use FixMyStreet for Councils for their official reporting platform (and so gather more reports) and where our most active users are (who submit a disproportionate amount of the total reports) accounts for 7-8% of what the model explains.

What that means is that most reasons people are and aren’t making reports is unexplained by those factors. So for the moment this model is useful for building a theory, but is far from a comprehensive account of why people report problems.

Here’s my rough model for understanding what drives areas to submit a significantly higher number of reports to FixMyStreet:

• An area must have a problem

Measures of deprivation like the ‘wider barriers to housing deprivation’ metric (this includes indicators on overcrowding and homelessness) as well as crime are associated with an increase in the number of reports. The more problems there are, the more likely a report is — so deprivation indicators we’d imagine would go alongside other problems are a good proxy for this.

• A citizen must be willing or able to report the problem

Areas with worse levels of health deprivation and adult skills deprivation are correlated with lower levels of reports. These indicators might suggest citizens less able to engage with official structures, hence fewer reports in these areas.

People also need to be aware of a problem. The number of companies in an area, or the presence of a railway station both increase the number of reports. I use these as a proxy for foot-traffic – where more people might encounter a problem and report it.

Population density is correlated with decreased reports which might suggest a “someone else’s problem” effect – a slightly decreased willingness to report in built-up areas where you think someone else might well make a report.

• A citizen must be able to use the website

As an online platform, FixMyStreet requires people to have access to the website before they can make a report. The less friction in this experience makes it more likely a report will be made.

This is consistent with the fact that an increased number of slow and fast home broadband connections (and fast more than slow ones) increases reports. This is also consistent with the fact that increased 3G signal in premises is correlated with increased requests.

Reporting problems on mobile will sometimes be easier than turning on the computer, and we’d expect areas where people more habitually use mobiles for internet access to have a higher number of reports than broadband access alone would suggest. If it’s slightly easier, we’d expect slightly more – which is what this weak correlation suggests.

Other factors

Not all variables my model includes are significant or fit neatly into this model. These are likely working as proxy indicators for currently unaccounted for, but related factors.

I struggle, for instance, to come up with a good theory why measures of education deprivation for young people are associated with an increase in reports. I looked to see if there was a connection between an area having a school and having more reports on the basis of foot-traffic and parents feeling protective over an area – but I didn’t find an effect for schools like I did for registered companies.

So at the moment, these results are a mix of “a-hah, that makes sense” and “hmm, that doesn’t”. But given that we started with a dataset of people reporting dog poop, that’s not a terrible ratio at this point. Expanding the analysis into Scotland and Wales, analysing larger areas, or focusing on specific categories of reports might produce models that explain a bit more about what’s going on when people report what’s going wrong.

I’ll let you know how that goes.

—

Image: Dave_S (CC-by/2.0)

In the period between the May 2015 general election and the June 2016 EU referendum WhatDoTheyKnow sent 1,022 FOI requests that matched our EU keywords. These were generated by 641 unique requesters.

Looking at these requesters: 79% of them made just a single request, and 96% made four or less. The remaining 25 users made 25% of all EU requests — with three users making more than 20 requests each.

For the year leading up the election there was an average of 55 users making 75.6 EU-related requests between them each month.

If we split this into two halves (the last half of 2015 and the first half of 2016), the average number of users per month had increased by 20 in 2016 compared to the second half of 2015 — with a peak in both users and requests in the month before the referendum and a decline in the immediate run-up.

So people had more questions to ask once the referendum was more in the public eye. But maybe that’s just reflecting wider trends across the board. Can we state with certainty that this change was referendum-related?

Let’s move on to the second question: What happened after the referendum?

After the referendum

Comparing the three months before the referendum with the three months after it, we see users and requests are up in the post referendum period.

	EU-related requests	Users making EU-related requests
Pre-referendum	310	216
Post-referendum	332	252

Looking month-by-month, we can see this is mostly an immediate spike followed by a drop-off:

In fact when we looked week-by-week, we could see the largest spike was in the week following the vote. This gives us some definite hints that it was the referendum that was driving this.

But to make extra sure that this increase really was referendum-related, we compared these changes to the overall WhatDoTheyKnow trends at the time.

The number of requests made across the platform increased between the two periods (17,246 increased to 19,120) — but there was also a decrease in the number of unique users making requests (4,850 decreased to 4,721).

This means the post-referendum increase in EU requests was counter to the general flow – and we can use a statistical test (chi-square) to confirm that the difference in users making EU requests is sufficiently different from the overall direction of users to reject the idea they are being driven by the same trend (p < 0.01 for those that want to know) .

So we can say there is a real difference before and after the referendum: people were asking government for more for more EU-related information after the referendum than before it.

 Notes

*First appearance in an FOI request: May 2015!

**Obvious Complaint: But Alex! Aren’t some of those a bit broad? And the answer is yes! In fact we discarded ‘immigration’ and ‘migration’ as keywords because when separated from other keywords, these were mostly requests for information about immigration rules relevant to the requester (although that said, a similar post-referendum peak appears when we looked at these ‘immigration’ requests in isolation. There were just too few to make as big a deal out of the change).

‘EU’ as a keyword will similarly be catching requests that have nothing to do with the EU, as EU law is so integrated that appeals to directives or other obligations can make an appearance in requests to just about any public body on just about every topic.

While the global count of ‘EU related requests’ might be inflated by this, a change relative to the population of all requests (like the one we found) should be robust — assuming that non EU-related requests that mention the EU are not distributed differently to non EU-related requests that don’t. This seems reasonable and so for the sake of this blog post — let’s say that’s so.

—
Image: Speedpropertybuyers.co.uk (CC by/2.0)