One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.
So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.
I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline – something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.
The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.
If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.
We’re busy as bees, lots of things happening, increasingly many of which are commercial, and we can’t talk about until they’re released.
Commercial? But you’re a charity! Yes – but just as Oxfam have a trading subsidiary company which runs the second hand clothes shops, we have a trading subsidiary company that sells services relating to the websites that we make (structural details here).
Everything from other small charities to large media companies are buying our services – which range from customised versions of FixMyStreet, through to strategic consulatancy. If you’ve got something that you think we might be able to help with, email Hello@mysociety.org – easier to talk to than us geeks.
Meanwhile we’re cracking on with our free services for the public, which are increasingly funded by this commercial work.
TheyWorkForYou recently launched a Scottish version, thanks to volunteer Mark Longair, and Matthew. More goodies in store as the Free Our Bills campaign unfolds. We’ve started a sprint to get a photo for every MP’s page. If you work for or are an MP or have copyright of a photo of one that we’re missing, then email it to us.
WhatDoTheyKnow is getting lots of polishing – the new site design that Tommy has been working on is nearly ready. Today I just turned on lots of new email alerts and RSS feeds, so you can get emailed, for example, when a new request is filed to a particular public body, or when a request is successful.
Our super ace volunteers have been busy adding public authorties to the site, and we now have 1153 in total. We’re getting a steady trickle of good requests (pretty graph) coming in. Blogs such as Blind man’s buff and confirm or deny are sorting the wheat from the chaff. Do blog about and link to any interesting requests that you see!
Other things in the works are a much needed revamp of www.mysociety.org, some interesting things on GroupsNearYou, and no doubt squillions of other things. I’ll let Matthew post up anything I’ve missed 🙂
Matthew and I have been sitting next to each other today looking at the outputs of his lovely new custom built conversion tracking system, designed to ensure that the optimal number of users who just come to one of our services as a one off get signed up to something else longer lasting.
I’ve been banging on for ages about how government should seize on cross selling people who’ve just finished using one online service into using another of a more democratic nature, so it seems worth spelling out some of the lessons.
First, there’s some interesting data from the last few weeks, since our newest conversion tracking infrastructure has been running in its nice new format.
One of the adverts randomly served to users of WriteToThem (after they’ve finished sending their letter) encourages them to sign up to TheyWorkForYou email alerts – the service people use to get emailed whenever their MP speaks in Parliament. The advert features a slogan of encouragement, and a pre-populated email form containing the user’s email, and a ‘Subscribe me’ button. This advert was shown to 2328 users last month, of whom 676 became TheyWorkForYou email subscribers, which is a pretty cool 29.04% conversion rate. However, we also showed another advert for the same service, to the same WriteToThem users, which also had the same button and text, but which hid the form (and their address). That was shown to 2216 users of whom 390 signed up, a more modest 17.6%. So the impact of simply showing an email box with the users email address in it, versus hiding it, was worth 10% more users. Why? Go figure!
So now we’ve canned the advert that hides the address form, and instead we’re comparing two different adverts both of which feature the pre-populated signup form, but which use different words. It’s probably too early to judge, but the new ad appears to have a very similar conversion rate suggesting it might be hard to squeeze many more subscribers out of this page. We’ll keep trying though!
Another thing we learned of interest was that monthly subscribers to email alerts on TheyWorkForYou were down year on year in the month before we added this new advertising & conversion tracking system, even though the total number of visitors were clearly up on the same month last year. This appears to suggest that two things are happening. First RSS is catching on, so some users who would previously have got email alerts are subscribing to RSS feeds instead. Second, it suggests that the TheyWorkForYou user audience might have been getting more saturated with regulars – proportionally fewer new users coming (although more visitors in absolute terms) so fewer people signing up to get alerts. The cross marketing and conversion tracking seems to have reversed that trend, which is awesome.
We also advertise several different services to people who just finish signing up to get email alerts on TheyWorkForYou itself. We’ve just noticed that a full 25% of people shown the advert to sign up for HearFromYourMP proceed to sign up. We’ve therefore just decided to dump other adverts shown on TheyWorkForYou (such as advertisements for other sorts of TheyWorkFor you email alert) and concentrate on just cross selling HearFromYourMP. A back of the envelope calculation suggests that by just advertising this one site from the completion page we should get an extra 10,000 subscribers to HearFromYourMP this year on top of the organic growth. Not bad for a few minutes analysis, and a number likely to make a fair few more MPs post messages to their patiently waiting constituents.
One last interesting thing (at least to me) is how some more demanding services are a much harder sell than others to users. So asking people to make new groups on GroupsNearYou.com or report a problem in a street on FixMyStreet tend to result in more traditional online marketing scale conversion rates of 0.1% to 2%. Still worth doing, and so we compare different versions of those ads too, to try and eke up those rates for these sites that arguably have more tangible, direct impacts on people and communities.
It will be a challenge for mySociety’s future to work out how to trade off impact against scale of service use – are 10 HearFromYourMP subscribers worth one pothole that doesn’t get fixed? Answers on a postcard…
Just a quick post to keep those of of you interested in mySociety in the loop with our activities at the moment.
New Things You Can Use Now
1. Email or RSS alerts when people report problems in your ward or your council via FixMyStreet. Ideal for councillors, people on resident’s associations, or anyone just concerned about what’s breaking and being fixed in the area right near their home.
Have a go – it’s ace when the mail comes dropping in from just down your road.
2. The Queen on TheyWorkForYou
Is this the first monarch with her own RSS feed? Would anyone really care if she was?
New Projects Coming Up
We have three major projects under way at the moment, and unusually
only two of them involve us building websites.
1. The Freedom of Information Filer and Archive website is under construction. Aiming to make it easier to make freedom of information requests, and easier for people to find what other people have found out, this is being build mainly by Francis. We’re having lots of discussions about design and features right now, and if you have anything to contribute please either get in touch or leave your ideas on the wiki page.
2. Local Email Groups Near You – an attempt to record the location of hyper local email groups and local forums and websites and to share that information on lots of other sites. Why go blindly hunting for advice on a plumber if there’s already an email list that covers your street? This is going to be a rare international project for us, so if you’re outside the UK and interested in community Internet usage, please get in touch.
3. The 90 Day Project – mySociety’s first lobbying exercise, trying to encourage parliament to take some steps to improve the way it publishes information, and to improve the tools that MPs have to handle mail from their constituents.
There’s lots and lots more too, but we can’t blow all our surprises in one go, can we?
Whereas new sites are lovely, and I talk about Neighbourhood Fix-It improvements further down, there’s still quite a bit of work that needs to go into making sure our current sites are always up-to-date, working, and full of the joys of spring. Here’s a bit of what I’ve been up to recently, whilst everyone else chats about database upgrades, server memory, and statistics.
The elections last week meant much of WriteToThem has had to be switched off until we can add the new election results – that means the following aren’t currently contactable: the Scottish Parliament; the Welsh Assembly; every English metropolitan borough, unitary authority, and district council (bar seven); and every Scottish council. The fact that the electoral geography has changed a lot in Wales means there will almost certainly be complicated shenanigans for us in the near future so that our postcode lookup continues to return the correct results as much as possible.
Talking of postcode lookups, I also noticed yesterday that some Northern Ireland postcodes were returning incorrect results, which was caused by some out of date entries left lying around in our MaPit postcode-to-area database. Soon purged, but that led me to spot that Gerry Adams had been deleted from our database! Odd, I thought, and tracked it down to the fact our internal CSV file of MLAs had lost its header line, and so poor Mr Adams was heroically taking its place. He should be back now.
A Catalan news article about PledgeBank brought a couple of requests for new countries to be added to our list on PledgeBank. We’re sticking to the ISO 3166-1 list of country codes, but the requests led us to spot that Jersey, Guernsey and the Isle of Man had been given full entry status in that list and so needed added to our own. I’m hoping the interest will lead to a Catalan translation of the site; we should hopefully also have Chinese and Belarussian soon, which will be great.
Neighbourhood Fix-It update
New features are still being added to Neighbourhood Fix-It.
Questionnaires are now being sent out to people who create problems four weeks after their problem is sent to the council, asking them to check the status of their problem and thereby keep the site up-to-date. Adding the questionnaire functionality threw up a number of bugs elsewhere – the worst of which was that we would be sending email alerts to people whether their alert had been confirmed or not. Thankfully, there hadn’t yet been any such alert, phew.
Lastly, the Fix-It RSS feeds now have GeoRSS too, which means you can easily plot them on a Google map.
Since the petition system went out properly on Wednesday, we’ve been absolutely buried in an avalanche of changes, fixes, feature additions and massive massive amounts of email. I thought that you might be interested as to what sort of stuff has happened in the first two days:
- Email has taken over our lives. Matthew has responded to over 200 emails since yesterday morning, and I was up at 4am last night just trying to cope with the rate of incoming of mail. Francis, who’s now in Canada, then heroically took up the baton and responded to mail all (UK) night! Many if not most of these mails are giving us suggestions, as well as bug reports, problems with email and bits of praise and the odd conspiracy theory.
- Changes made to cope with expats and overseas military personnel.
- Phoned Hotmail to stop their system from eating 95% of the confirmation messages being sent to Hotmail accounts!
- Redesigned the automated mails no10 get telling them there’s a new petition (they’ve had over 500 of these mails, so they need to be clear and easy to read!)
- Made the rejected petitions system more granular, so that if a petition has to be rejected, and part of it has to be hidden (say, if it is libellous), then it only hides that bit, not the whole thing. Maximum transparency is the goal, you see.
- New options added to sort the list of all petitions in different ways, by number of signatures being the most asked for.
- Limited the length of “more info” fields so people can only write long rants, rather than really really long rants 🙂
- Special cased people with AOL accounts, so that their, erm, nonstandard email clients can actually cope with the confirmation links.
- Made several fixes to the processes involved in sending out confirmation mails.
- Made RSS changes and improvements.
- Updated various bits of text, like providing examples of what “party political” means. The BBC initially wrote that this meant no pledges mentioning controversial issues like Iraq, which was grabbing quite the wrong end of the stick about the nature of the rules. Now we have some complaining emails saying we’re being too liberal!
- Compiled a big list of user suggestions and fixes on the wiki.
- Made the rejection criteria in the Ts&Cs actually match the ones in the admin interface.
- Installed a stats packages to watch what’s going on.
- Added facility to search petitions
- Improved/fixed logging
- Added link and text pointing to the open source code.
I’ve probably missed some – I’m sure Matthew, Chris, Francis and Ben will let me know!
Today mySociety launches a pair of complementary services sprinkling some of our democratic pixie dust on the House of Lords:
Between them, these new services let you:
- Identify a Lord who is interested in an issue of interest to you.
- Write to any Lord you want via WriteToThem.com
- View individual Peer profile pages including attendance at votes, most recent speeches, rebelliousness and more.
- Get custom email alerts every time a certain Lord speaks, or when a word or phrase is spoken by anyone in the Lords. Over 5000 people will be mailed with unique updates today alone.
- Produce league tables of which Lord or which MP has spoken words or phrases the most (handy for identifying which members show a public interest in which issues issues).
- Search, read and annotate Lords Hansard back to 1999
Collectively we think that these tools should help pour light into the activities and workings of the House of Lords, and help members of the public develop productive relationships with the peers who vote on issues of importance to all of us.
So, we’ve been a bit quiet on this blog, but naturally busy. I just did my invoice and timesheet for last month, and remembered how bitty it has been. In one day I often do things to 3 websites, and that is just CVS commit messages – no doubt I handled emails for more. This makes it quite hard to summarise what has been happening, and also quite hard to measure how much time we spend maintaining each website.
We’ve recently made a London version of PledgeBank, which I’ll remind Tom to explain about on the main news blog. It is a PledgeBank “microsite”, with a special query for the front page and all pledges page that shows only pledges in Greater London. Which is conveniently almost exactly a circle radius 25km with centre at 51.5N -0.1166667E. I worked that out by dividing the area (found on the Greater London Wikipedia page) by pi and taking the square root And rounding up a bit.
Yesterday we launched a new call for proposals – head on over, and tell us your ideas for new civic websites. It is another WordPress modification, but this time to the very blog that you’re reading now. The form for submitting proposals I made anew, It creates a new WordPress low-privileged user by directly inserting into the database, and then calls the function wp_insert_post to create a post by them in a special category. The rest of the blogging software then trivially does comments, RSS, search, email alerts and archiving.
Meanwhile, Chris has written some monitoring software for our servers, to alert us of problems and potential problems. Perl modules do the tests, things like enough disk space and that web servers that are up. I’ve been tweaking it a bit, for example adding a test to watch for long-running PostgreSQL queries which indicate a deadlock. We’ve got a problem in the PledgeBank SMS code which causes deadlocks sometimes, which we’re still debugging.
So, HassleMe launched today (despite mostly having been written just before Christmas). Good work by Etienne getting it all together. Today Matthew and I have been working on adding “instant-messenger” functionality to the site, which turns out to be a bit painful. Right now it seems like the most robust solution will be to use bitlbee, a proxy which allows you to interact with the various and wretched instant messenger protocols through the less varied and marginally less wretched IRC protocol.
Integrating a website with instant messenger is an interesting problem. I’m not yet sure how much of the experience of building sites which send and receive email will carry over. We’ll see….
Lots of countries gradually loading into one of our servers. There’s 220Mb of data comprising 227 countries, with about 5,000,000 places altogether. With a global population of about 6 billion, that means the average “place” has 1,200 people living in it. For each place we have the latitude and longitude. (All this data comes from the US military)
Try it out by signing up for a local alert in any country. Let us know if you find any bugs, or have any problems or suggestions to make. Also, if you want access to this gazetteer as a web service, send us a mail.
Currently it’s up to Uruguay, it’ll be a bit longer before we’ve finished the alphabet. It takes quite a while partly because of the volume of data and indices being built, partly because for places with the same name as each other it hunts for nearby towns to disambiguate, and partly because we didn’t optimise the perl script. It won’t run very often.