Hansard Society report on MPs Online

The Hansard Society have just published a report entitled MPs Online: Connecting with Constituents. I’m only going to talk about one part, the part that mentions the mySociety project WriteToThem in a section on MPs’ use of email.

We’re surprised and disappointed to see our methodology for collecting data on how well MPs respond to constituency mail being called “unreliable”, especially from a paper that makes a number of simple mistakes of its own in just a few lines on one page.

  • On page 5, they state that WriteToThem has been “tracking responsiveness to emails via their website for three years”. Most importantly for the theme of the report, we don’t just send emails – we send faxes to a number of MPs (and other representatives) who do not accept or want messages via email.
  • The figures given for survey responses in the table are backwards; 2007 and 2005′s figures should be interchanged – how could we get more survey responses than messages (again, not necessarily emails) sent? :-)
  • They claim there is “no quantification of the response categories provided” – the raw data used to automatically generate these categories is given in the adjacent column (“very high” simply means a response percentage of 80% or more, for example; our code is all open source).
  • We exclude MPs with very small sample sizes, and take a range of steps to make sure the data is not abused.
  • We have four years of statistics now, not three; our stats for 2008 were published nearly six months ago.

The Hansard Society, to the best of my knowledge, never got in touch with us to request any clarification or ask about our data or methodology, which we would have been more than happy to supply.

2 Comments

  1. One thing I’d really like to see is a proper analysis of MPs constituency email, letters and faxes. It is much more complex than just counting.

    Firstly, it should measure spam, and exclude it from figures.

    Secondly, it should separately measure bulk email. e.g. Generated by lobbying companies, or by NGOs, sending mass emails.

    Thirdly, it should measure other bulk messages the MP receives at their email or postal address. e.g. mailing lists they have deliberately signed up to.

    Fourthly, it should account for whether messages are from constituents or not.

    That would leave letters written by hand by constituents themselves.

    I have a vested interested for wanting to measure this, as WriteToThem prevents any messages which aren’t from constituents, or are sent in bulk or are spam.

    It would be interesting to also know how many emails from constituents MPs lose due to spam filters putting them in the wrong place.

  2. I’ve been emailing with Andy Williamson from the Hansard Society. They say they did try and get in touch, but we can’t find any record at our end of receiving anything. Ah well, these things happen.

    On my main concern of our methodology being unreliable (I’ll restate that I’m not saying anything about the report or its conclusions, this was just a small reference to us), he writes: “By ‘unreliable’ I mean that the information we had did not make it sufficiently clear how robust the stats were, it’s not to say that they weren’t simply that we couldn’t say so – hence to us and any other academic using that data it was unreliable. It’s not a criticism, just a statement of fact based on what we had.” – my reply, with which he agreed, was “To me, a statement of fact would be that you could not comment on the reliability of the statistics as we hadn’t provided you with enough information (as you’ve just said), which sounds totally fair. Calling the statistics unreliable implies to me that you have drawn a conclusion based on the data – ie. you have confirmed that it *is* unreliable, not that you just don’t know, which is not what you now state. Thanks for clearing that up. :-)”

    Someone sent me this report last week as if it were new, and the report didn’t contain a publish date, so I missed that it was actually out last February – that would explain why it doesn’t have our 2008 data :) Sorry about that.