1. Calling anyone at Technorati

    Please pass this on if you have any first or second-hand contact with Technorati:

    “Hello!

    Can we please have an API key with a high or unlimited daily query cap so that we can help users of TheyWorkForYou to see what discussions about Parliament are going on across the web?

    thanks,

    Everyone at mySociety”

  2. PledgeBank Facebook application disabled

    Unfortunately, I’ve had to disable the PledgeBank Facebook application. It used to let you sign and share pledges from within Facebook.

    Facebook recently changed their platform (again!), breaking our code for sending success/failure messages. Obviously, it is no good signing up to a pledge if you don’t get informed when it succeeds.

    I tried to fix it, but couldn’t work out how to do so quickly. We don’t have the time and money at the moment to chase after this, so I’ve disabled the application entirely. Links to PledgeBank pages on Facebook now redirect to pledgebank.com.

    Hopefully it’ll be back one day – do send us emails if you miss it (or money if you have a large pledge that really needs it!). I think there may be a better solution with a simpler interface – the current application tried too hard to reimplement all of PledgeBank within Facebook. And besides, we should be supporting OpenSocial now it exists. It’s an open standard, Facebook isn’t.

    Technical details: We used infinite session keys to send notifications from cron jobs. Quite reasonably, this no longer works. However, I couldn’t find out what to use instead. I think Facebook should respect backwards compatibility of its APIs a lot more, and if it breaks it they should give clear instructions about what to use instead. This does put me off ever wanting to develop anything on their platform again.

  3. Relentlessly into autumn

    I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.

    mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.

    Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages – causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).

    Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.

    (the autumn photo thanks to Nico Cavallotto)

  4. Highlighting the current speech

    Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?

    I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. 🙂

    Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.

    The highlighting works the other way – as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! 🙂 ).

    Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.

    Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse 🙂

    Previous articles

    1. The Flash player
    2. Seeking
    3. Highlighting the current speech
  5. Richard Pope’s GroupsNearYou.com launches

    Say hello to GroupsNearYou.com.

    The Short Geeky Explanation

    GroupsNearYou.com is an entirely user generated API-queryabledatabase of the location and nature of local online communities, irrespective of the platform they are hosted on. A piece of the programmable web, in short, with local community building focus. The syndicated community information can easily be layered onto a map, for example.

    The Business Problem it Solves

    Do you run a site that tells people stuff about their local area? Do you suspect that there might be quite a lot of internet enabled community activity going on in local areas that you’d like to tell your users about? Use our feeds.

    The Social Problem It Solves:

    There’s a proven real world social value to people belonging to very local email lists and other forms of local online community. However there is no eBay or Craiglist or other market dominant player in the local online community world, instead there’s a myriad of google groups, yahoo groups, Facebook & other YASN groups, extremely old school CCed email lists, online forums and so on. As a consequence of not having one big simple place to go to find and join local groups (many of which are not even on the web for Google to find) far fewer people ever find out about and join their local online groups. GroupsNearYou.com is about getting more people to join groups, groups that are not hosted by us, and (hopefully) mainly discovering them via uses of our syndicated info on sites that aren’t run by us. It’s a piece of pure internet infrastructure, with a positive social bent.

    Who did it?

    Astonishingly, the project was almost entirely built by a volunteer, Richard Pope assisted in design by another brilliant volunteer Denise Wilton. Their only reward is a highly sought after mySociety Hoodie, plus the love and gratitude of all our users.

    Richard has been an amazingly dedicated volunteer for mySociety and on his own projects for over two years, and deserves the reputation he is rapidly gaining as one of the world’s truly great civic minded web innovators. The project was funded by the UK’s Government’s Ministry of Justice who have been trying to run experiments of different kinds in the realm of electronic democracy. Their money will go to help improve and grow the site, rather than building it, which is a very interesting funding model in its own right. The several hundred groups already in the system are mainly added by users of WriteToThem.com

    Next steps

    Almost all the groups listed in the database are in the UK at the moment, and they’re all from users of our other sites. We’re interested in working with anyone who runs sites that might want to either take information out of it, or put information into it (Hello email list/social network providers!)

    Anyway, it’s dead simple really, just a little brick in the internet wall, albeit one that I hope will help a few more people meet their neighbours and improve their communities.

  6. Today is Thursday

    Lots has happened since I last posted, which was months ago, before Chris died.

    Luckily for us, in the last few months Chris had only been working one day a week for us, so it hasn’t been as difficult in practical terms as it could have been. There were various mySociety things running in his flat, such as the WriteToThem fax server, which had to be set up quickly elsewhere.

    We miss Chris’s expertise most days (only yesterday I was swearing at gnuplot). Matthew is twice as much for me to handle; he works so quickly, he has hard questions to ask at a ferocious rate that I can’t keep up with. I think before Chris used to handle most of them, so it was much easier for me.

    I highly recommend Chris Lightfoot’s obituary by Martin Keegan. Also, Chris’s obituary on Last Word, which was on Radio 4 on 23rd March, is well worth listening to.

    We have a few new members of staff.

    Keith Garrett is working for us now, mainly tending our servers, but he’s also been working on the E Petitions site. He’s trying to bully us into documenting all our internal processes so it’s easier for new people.

    Heather Cronk has started working for us in the US, evangelising PledgeBank. This is funded by the Omidyar Foundation. As well as getting the word out about PledgeBank in the states, this is extra good for us, as she’s forcing us to give PledgeBank the TLC that it deserves.

    Deborah Kerr is now doing customer support for us part time. She’s been busy with Neighbourhood Fix-it which has had lots of traffic and attention the last few weeks.

    OK, back to the present.

    Last week Ben Campbell and I gave a seminar at Technology for a Small Nation in Llandudno. We split into two groups, I got people to add Google maps to an HTML page on their website, and Ben got people to call the TheyWorkForYou API from PHP. It went down well, very satisfying to do practical exercises, and answer all the niggling questions (how to install a testing webserver on Windows, how to use FTP) which are the real things that stop people doing what they want to do.

    And Thursday? Just a reminder, unless you’re not apathetic, that there’s an election today.

  7. Back in the South for now

    I seem to keep ending up in the North of England on work this Autumn, which is good as that’s where I’m from.

    The Liverpool event on Friday went really well. Over the day, we probably had about 10 people altogether. It ended up more like a rolling seminar series – we all introduced and talked about what we were up to. Of course, Matthew on the TheyWorkForYou API, Tom on what mySociety doing these days. Also Julian (who does lots of the Parliament screen scraping that TheyWorkForYou depends on) talked about using Public Whip for collaborative research on Parliamentary votes. We spent a while talking about Fredom of Information with Steve Wood, who lives in Liverpool.

    Ben (noii) told us about Tad Hirsch at MIT, who has done lots of interesting projects, which are inspiring for mySociety style project ideas. I’m sure we can do a lot with automatic routing of voice calls to volunteers, and with text message lists. Another Ben showed us an early version of some great work he is doing extracting information about newspapers and journalists. Thanks very much to Aidan from Blue Fountain who hosted us in the beautiful India building. We’ll be having two more events to promote the TheyWorkForYou API, and talk about other mySociety matters, later in the year. One in London, one somewhere else, suggestions welcome.

    mySociety developers in the Lake District

    Earlier in September, all of us went to work together in the Lake District (as mentioned in my last post). It’s important to meet up for a solid chunk of time in the year when we spend most of it spread out across the Internet. Matthew took lots of photos, one of which is above (Tom, Louise, Matthew, Chris, Francis and Anna from left to right). It might look from the photo like we were brainstorming solutions to thorny problems, clearing back logs of customer support, writing code, and creating todo lists as long as your arm. But actually we just climbed mountains, hung out by tarns, drank beer and had a good time.

    And next week I’m back off up North, to Betws-y-Coed in Wales this time. To give a talk to Bloc about TheyWorkForYou.

  8. What we’re up to

    Much of my August seems to have been absorbed with maintenance tasks.

    For example, Chris and I spent a few days tightening up WriteToThem’s privacy. I made sure the privacy statement correctly describes what happens with backup files, and failed messages. I reduced the timeouts on how long we keep the body of failed messages. I made sure we delete old backup files of the WriteToThem database. I wrote scripts to run periodically to check that no bugs in our queueing demon can accidentally mean we keep the body of messages for longer than we say. I added a cron job to delete Apache log files older than a month for all our sites. As AOL know to their cost, the only really private data is deleted data.

    Earlier in the month, I handled some WriteToThem support email for the first time in ages. We get a couple of hundred messages a week, which Matthew mainly slogs through. It’s good for morale to do it, as we get quite a lot of praise mail. It is also hard work, as you realise how complicated even our simple site and the Internet are, and it leads to fixing bugs and improving text on the site. I made a few improvements to our administration tools, and things like the auto-responder if people reply to the questionnaire, to try and reduce the amount of support email, and make it easier to handle.

    I did some more work on the geographically cascading pledges (like this prototype one), but I’m still not happy with them. In the end, I realised that it is the structure of wording of the pledge that is the key problem. Our format of “If will A but only if N others will B” just isn’t easily adapted to get across that the pledge applies separately in different geographically areas. Working out how to fix that is one of the things we’ll brainstorm about in the Lake District (see below).

    The last couple of days I’ve been configuring one of our new servers who is called Balti, and getting the PledgeBank test harness working on it. Until now, it has only been run on my laptop. This is partly heading towards making a proper test harness for the ePetitions site, running on a server so we properly test nothing can be broken before deploying a new version.

    Matthew has wrapped up the TheyWorkForYou API now, and is working on Neighbourhood Fixit next. Chris has been doing lots more performance work for the ePetitions site. And he’s been making some funky monitoring thing to detect PostgreSQL database lock conflicts, which we get occasionally and are hard to debug

    Tom’s in Berlin at the moment, he gave a talk last night, and I think has been to see some people from Politik Digital. As we’ve been discussing on the mySociety email list, there’s an EU grant we’re likely to apply for in collaboration with them.

    On Friday, we’re all going to the Lake district for a week, with some of the trustees and volunteers intermittently. We very conveniently and cheaply all work from home, so it’s good and necessary to meet up for a more sustained period of time at least once a year. Last year we were in Wales.

  9. Northern Folks: Come to our developers event in Liverpool

    On Friday September 29th mySociety will be holding a volunteers hacking day in Liverpool. Everyone is invited, and we can even cover some train fares, plus a round or three in the pub. To be eligible for the train fare support, though, you Must Live North of the Watford Gap.

    The main theme will be to hack around with the new TheyWorkForYou API , and we’ve got a few hundred quid to cover costs from the day as part of the funding from the Department of Constitutional Affairs (the majority of the money has covered Matthew’s costs in building it from scratch).

    The event is at Blue Fountain

    The timing is 10AM to 7PM, followed by adjournment to some sort of public house. For those of you with work commitments, please consider coming in the evening.

    Venue kindly provided by mySoc friend Aidan McGuire. The meeting will also be a chance to meet key TheyWorkForYou and Public Whip volunteer Julian Todd, whose reclusive Liverpudlian lifestyle means we don’t get to see him often 🙂

  10. Battle your way to Sedgefield!

    It’s a Saturday, so I’m not going to talk about anything serious like our new TheyWorkForYou API. Matthew has just told you all about that much better than I could.

    No instead, obviously what you need is a game to play. And none of those new-fangled flash things. You need a good old text game. And that buzzing feeling of revenge for overpowering MPs of your choice.

    Welcome to MP fight!
    ====================
    
    Battle your way to Sedgefield! A game, by Francis Irving
    Built using the TheyWorkForYou API http://www.theyworkforyou.com/api
    Type 'q', 'quit' or 'exit' at any time to finish.
    
    Enter your postcode.
    >
    

    Play MP fight! Start good old telnet (type “telnet” in Start | Run on Windows, or use the command line on Macs or Linux), connect to seagrass.goatchurch.org.uk on port 646. Or try Chris‘ web version, although it just doesn’t have the same retro charm.

    Post your experience score and how far you got in the comments below. If it’s too easy, try a friend’s postcode. Maybe in Scotland. It’s hard to get out of Scotland.