1. Your chance to test out Projects, our newest tool for FOI

    Are you investigating, researching or gathering large quantities of data through Freedom of Information requests? Perhaps you’re a journalist, academic or NGO. We’re looking people based in the UK who’d like to try out our new ‘Projects’ feature for WhatDoTheyKnow Pro.

    Projects allows you to crowdsource the extraction of data from multiple (or batch) FOI requests made to multiple authorities. You can set up a project with a brief description of what it is and what you are hoping to achieve, and some tasks that volunteers can complete to help you with this aim (like categorising responses, or answering questions about the data released).

    Once that’s done, you can set it up to invite volunteers, who can help you to extract all the information you need from the released responses.

    You’ll be able to download your volunteers’ input as a spreadsheet, meaning analysis of the data is much quicker and easier — so you can get on with the task of forming conclusions and writing up your findings.

    What we’ll need from you

    Projects is still in its nascent stage, so we need feedback from our testers. This will help us improve the service and tailor it to users’ needs, based on real life use cases.

    Right now, we handle the setup and importing of the requests you want to work on manually (that is, our developers have to do it) — but we’re working on improving this aspect, and your feedback will be crucial in shaping the direction our development takes. We’re also looking for general comments, once you’ve used the service, on what’s useful and what’s missing; what you tried to do but couldn’t, and what made things easier for you.

    If this sounds interesting, please get in touch at pro_team@whatdotheyknow.com. We look forward to hearing from you!

    Image: Jessica Lee

  2. FixMyTransport is close to launch – and we need your help to make it fantastic

    Photo of display board showing delayed trains, by Glenn Scott

    It’s been a while since we updated you on the progress of our next major project, FixMyTransport, but we’re still working hard behind the scenes. As you may recall, FixMyTransport will deal with public transport problems – delayed trains, vandalised stations, overcrowded buses, you name it. It’ll put problems in the public arena, while also reporting them directly to the relevant transport operator. Read more about the project here.

    We will shortly be arriving at our final destination

    Things are going to get exciting very soon. As launch date approaches, we’ll be starting a closed beta (mid July), rapidly followed by a full open public beta launch (end of July). During the closed beta we want to get as much feedback as possible from future users of the site, as well as pressure groups, transport operators, and anyone else who has anything valuable to contribute.

    If you would like to be invited to beta test, and weren’t one of our alpha testers, please email us on team@fixmytransport.com. Alpha testers will, of course, be invited to test again.

    Mind the (data) gaps

    We got extremely useful feedback from our alpha testers, and a wealth of crowdsourced data from our community. Thanks to their efforts we now have contact details for the operators of about 50% of the routes in the UK. However, this leaves a lot of operators where we don’t know how to get in touch.

    We really need your help to get them! If you can spare a few minutes, visit our spreadsheet and see if you can fill in any of the missing details.

    The more contact details we can get hold of, the better experience FixMyTransport will offer to our users. As well as publishing passengers’ reports on the site, FixMyTransport sends them directly to the operators too, helping to get the issue fixed.

    So, we especially need the email addresses for operators’ customer services departments. Finding these may be as simple as visiting the operators’ websites, or it may require a bit of sleuth-work on your part.  If advanced Googling gets you nowhere, we’ve found that simply phoning head office can get results.

    Incidentally, the main operators are near the top of the sheet – those are the ones that will benefit the most users, although obviously the nearer completion we get, the better.

    You’ll notice that the spreadsheet now includes a non-obligatory column for your name: this is to offer a small incentive. If you want to, tag your entries and at the end we’ll be offering goodies to the top contributors. Depending on your preference, this might be one of our highly sought-after mySociety hooded tops (they’re snuggly!), or a chance to become more involved in the project.

    Those who helped in the first iteration, please note that although this sheet looks different, your details have been retained and indeed have been extremely useful as we build the site. Also –  if  you have already been a major contributor during our previous rounds of testing and data collecting, please holler so that we can give you proper credit.

    Hold tight, please

    Not long now… we hope you’re as excited as we are.

  3. Announcing Brief Encounters: mySociety’s first Prequel Site

    FixMyTransport is the most challenging project mySociety has ever tried to build. It’s so ambitious that we’re taking the unusual move of breaking off part of the problem and stress-testing it in the form of the new mini-site Brief Encounters, which has gone live today. It was built by Louise Crow, or Crowbot, as we know her, with design support from Dave Whiteland.

    Brief Encounters is not, as the name might suggest, mySociety’s long awaited attempt at a dating site. Instead it’s a place where people can share whimsical stories about unusual things that happened them them, or other people, on public transport. We hope you’ll have a go, read some examples and then contribute your own.

    You might be thinking that a whimsical story site doesn’t sound very mySocietyish – and you’d be right. Brief Encounters is actually a technology test-bed to help us crack a new design and data problem: how do you make it as easy as possible for users to pinpoint a specific bus stop, or train route, or a ferry port, as easily as possible? There are over 300,000 such beasties, and nobody has ever really tried to build an interface that makes it easy to find each one quickly and reliably.

    So, what we want from you, dear readers, is three fold. We want:

    1. Stories – the more hilarious or sob-inducing the better
    2. Feedback on the user experience – how can we make finding a route or node easier?
    3. Feedback on any data problems you find, ie “My bus stop is missing” – we’re going to have to patch our data with your help, there’s just no other way

    For those of you tech minded, the project is built in Ruby and uses the NaPTAN dataset of stations, bus stops and ferry terminals, the National Public Transport Gazetteer database of towns and settlements in the UK, and the National Public Transport Data Repository of sample public transport journeys, from 2008. The first two datasets are free of charge, and the third one mySociety pays for.

    Lastly, kudos must go to the hyper-imaginative Nicky Getgood who suggested we collect stories on FixMyTransport, as well as problem reports.

  4. WriteToThem all at once

    I’m happy to say that with a lot of help from Francis and Matthew I’ve just rolled out a long-requested addition to WriteToThem. You can now use the site to write one message and send it to all your representatives in a multi-member constituency – so, for example, you can send a message to all your MEPs at once. A nice side project along the way has been getting the test suite for WriteToThem running happily on a mySociety server, and adding a few more tests. Run, tests, run! Good tests.

  5. The user is always right (curse them)

    An oft discussed but infrequently implemented feature of good web design is what we call conversion tracking. This is where you randomly offer users one of two or more versions of the same page, and then compare their relative merits at getting the users to do whatever your page is there to help them do.

    A couple of weeks ago we finally started to test our first working version of conversion tracking on the homepage of HearFromYourMP. We provided two versions – the super elegent, minimalist version that had a simple, bold heading and almost nothing else, and a more verbose, text heavy version explaining what the site did.

    As congenital advocate of super-simple sites, I was really hoping the public would go for the less wordy version. Did they? No, of course not – the conversion rate from visitor to subscriber was 16.5% for the simple version, and 23% for the more text filled version. Slightly gallingly, this means we might have forfeited thousands of signers since our launch in November.

    So, as painful as it was, we’ve now abandoned the minimalist version, and we’re comparing two more text heavy versions. You can see the process in action if you go to HearFromYourMP and hit refresh a few times.

    Over time we’ll role this out to all of our sites and a never ending ‘winner stays on’ competition will start between different versions of important pages. Maybe we could even start taking bets? 🙂

  6. We’re all too busy…

    …testing stuff for the launch on Monday to post anything more!

  7. Boiling boiling hot

    Today is the sort of muggy, hot day, which makes you believe that old story. The one about the British having an empire because of the bad weather. They could actually concentrate, plot and do under the clouds, rather than just dozing in a field, lazing in the sun. How true it is I’m not sure, but it definitely seems quiet everywhere today. I’ve got fewer emails than ever, and I’m still amazed I’ve managed to think all day at all.

    I’m leaving PledgeBank behind for a bit, as most of the obvious bugs are fixed, and features added. Earlier this week, and at the end of last, I put in local email alerts. Now if you sign up to local alerts you’ll get mailed at most once a day when a new pledge is made within 25km of you. PledgeBank really is having a proper beta test, with about 100,000 visitors since launch a month ago. This has been fantastic, the feedback has made it a very stable and hopefully more usable site. Good software is software which is used lots, with a virtuous feedback loop from users into making it better.

    So this morning I went back to WriteToThem. We still haven’t updated county councils after the elections. So I’ve been writing some three way merging code, to import changed council data from GovEval. It is merging, rather than just loading, because we’ve also altered the data. This was to make the ward names match up consistently with those from our mapping data.

    Things are working pretty well for councils where neither GovEval or us have added or deleted any representatives. There are unique identifiers, and very few clashes. Only one throughout all the data which has been caused by one of us editing the ward name, the other editing the councillor name, and so the repersentative really being neither.

    The hard case is additions and deletions. Obviously, I’d like to keep our work of mySociety additions. But this is no use, as eventually one day they’ll be wrong. The councillor we thought should be there no longer will be. So how do I detect when?

  8. More PledgeBank

    So, it’s my turn to write something here. Well, this’ll be short. As Francis mentions below, PledgeBank launched today, and everything’s gone reasonably smoothly, with the exception of some tedious PHP bug we haven’t tracked down yet. With any luck the new version of PHP will fix it, so there won’t be hours of painful debugging to do. (I could bore you for hours with my opinions of PHP — actually, I could probably shorten that quite a lot if I were allowed to swear, but this is a family-friendly ‘blog — but let’s just say that fixing PHP problems Is Not My Favourite Job.)

    Instead I’ll say what I’m doing right now, which is beginning to add geographical lookup to PledgeBank. At the moment we ask pledge authors for a country (though it can either be “UK” or “global” at the moment), and, if they’re in the UK, a postcode. The idea is to “georeference” (i.e. look up the coordinates of) the postcode, though we don’t actually do that yet. So I’m modifying the database a bit to store coordinates (as a latitude/longitude, so that we don’t have to write a separate case for every wacky national coordinate grid) and generalise the notion of “country” so that we can let non-Brits actually put in their own countries when they create pledges.

    Other things we’ve discovered today:

    • People are confused by the “(suspicious signer?)” link next to signatures on each pledge page — several people thought that we were reporting our suspicions of the signer. You probably think that’s stupid, but if so that’s only because you’re familiar with sites that have this sort of retroactive moderation button everywhere. Actually it’s us that’s being stupid and we’re going to remove it until we have a better way to implement it — at the moment we think we’re mostly on top of the occasional joke/abusive signature.
    • People are confused by the pledge signature confirmation mail, which currently reads (for instance),

      Please click on the link below to confirm your signature on the pledge at the bottom of this email.


      The pledge reads:

      ‘Phil Booth will refuse to register for an ID card and will donate £10 to a legal defence fund but only if 10,000 other people will also make this same pledge.’

      — the PledgeBank.com team

      We got several emails from people saying “your site has got my name wrong — I’m not Phil Booth”. The point is that Phil Booth wrote the pledge, so it’s in his name; the email reflects that. But that’s not obvious to the signer, and since the only name in the body of the email isn’t theirs, they think it’s got it wrong and complain. (To be fair, only three out of ~1,100 did, but that’s still bad.) This is the sort of problem we need user testing to spot: none of us saw anything wrong with the text when we were testing it. So we need to reword that.

    • We’ve had several people email to say that they’d like to do versions of PledgeBank in their own countries, and we’d like to hear from anyone interested in localising the mySociety projects who has time, expertise or even just opinions to donate. If that’s you, please get in touch!

    And probably some other stuff, but I said this post would be short….

  9. Thinking of PledgeBank uses

    PledgeBank is now well and truly in testing, so we’re spending lots of time finding people to help us test it. One idea Chris just had is to try out a money pledge, but a simple one which doesn’t need any more code. For example “I’ll pay £10 towards a £750 cow at sendacow.co.uk if 74 other generous people will too”.

    We’re thinking of ideas for what charity would be suitable, add your own to PledgeBankPossibleUses on the wiki. It needs to be a capital project, that you couldn’t give less towards and it make sense, so that the pledge aspect works. Of course, if you’ve got any other ideas for using PledgeBank, add them there as well, or email us.

    Today I’m working my way through bugs. Right now I’m improving the error messages when confirmation tokens are bad, or when they are activated twice. There are several places this needs checking – for pledge creation, signing and for announce messages.

  10. Testing SMS code

    Today I’m expanding the test harness to thoroughly test out all the SMS code. Adding in SMS creates all sorts of edge cases to check out, and indeed quite a few common cases. For example, it needs to make a pledge with both SMS and email signers, and test SMS conversion to email. And do both failure and success for all those cases. It gets even more complicated when you allow for SMS delivery reports.

    To do this, I’ve got small bits of script which pretend to be C360, our SMS aggregator. This is much cheaper than using real SMS messages every time I run the test script 🙂 The pretend scripts talk C360’s API, and fake telling our scripts as if incoming SMS messages arrive. They also store up outgoing SMS from our scripts, and check all the correct messages have been sent out to users.

    At the moment Chris is making some crazy Perl stuff to override the time() function in the SMS daemon, so when the test harness fakes the advance of time, it notices.

    If I get bored of all this, I’ll tidy up and test the new comments code Chris started the other day.