Recently I gave a talk at a conference where I told a group of local government officials that FixMyStreet was built not just to provide cleaner streets for their citizens, but also to force the hands of councils to procure and contract internal IT systems fit for the 21st century. In particular I pointed out that companies like Google seek to have people use their service from any site, any browser and device – they don’t just demand that everyone goes to www.google.com. And, I said, it’s only through building nice interfaces (APIs) that you can become an organisation that realises the benefits for yourself and other organisations from taking this ‘we’re happy to interoperate with anyone’ approach.

Less than three weeks later Michael Houlsby from East Hants council has single-handedly built an external facing API for their faults and problems database. So now FixMyStreet posts problems in that council direct into their database, without them first being translated into emails.

This is fantastic, especially as Michael clearly knocked it together in his spare time, and helps confirm what we’ve said before – if government builds nice interoperable APIs people like mySociety will use them to improve citizens’ experiences, whist simultaniously keeping everyone’s unnecessary workloads and expenses to a minimum. Plus it shows that if your IT supplier tells you you need to sign a new five or six figure contract to add an API to a CRM system you’ve already bought – you’re being jerked around.

Hats off to Michael – you’re a great example of a pro-active public servant using your skills to make government both better and more efficient.

Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.

We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.

I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.

• TheyWorkForYou opens up access to conventional, representational democracy, between and during elections. In 10 years time, I asked for Parliament to publish all information about its work in a structured way, as hinted at in our Free Our Bills campaign. So it is much easier for everyone to help make new laws better.
• FixMyStreet is local control of the things people care about, a very practical democracy. In 10 years time I’d like to see all councils running their internal systems (planning, tree preservation orders… everything that isn’t about individuals) in public, so everyone can see and be reassured about what is being done, why and where.
• WhatDoTheyKnow shows the deep interest that there is by the public in the functioning of all areas of government. In 10 years time, I’d like to see document management systems in wide use by public authorities that publish all documents by default. Only if overridden for national security or data protection reasons would they be hidden.

Charles Armstrong, from the Circus Foundation, has written up the workshop.

Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort – Charles is going to work on it for another time.

Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).

One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.

So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.

I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline – something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.

The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.

If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.

I’ve just come back from a hectic week away – visiting friends and working while there. It’s good teleworking, I worked in a welsh valley on Monday, Tuesday and Wednesday staying at my friend Ben’s house.

Ben has been working for the Media Standards Trust, on their fab new site Journalisted. It’s a kind of a TheyWorkForYou-style site, but which gathers info and stats about journalists (as opposed to MPs). I’ve added links to it from TheyWorkForYou, so if you browse to an MP page who is also a journalist you can easily look up what they wrote.

Hopefully this will put some much needed scrutiny on journalists – rewarding the good ones, who are both allowed the time to properly research stories, and are energetic enough to do so. Now, what will be the next Great British Institution to have some sunlight on it…

Ben and I whizzed to London on the train to catch Steve Coast’s talk last night. He was as usual, excellent. We also had some great questions – it was interesting to hear a detailed account of the financial side of Open Street Map. Afterwards we had Italian food and beer. mySociety’s next disruptive tech talk is at the end of November. It’s by Jason Kitcat of the Open Rights Group talking about electronic voting.

I’m glad to be back home after a week away.

At mySociety we’re always very lucky to meet and spend time with some extremely diverse and impressive people.

We thought it would be great to share a bit of that good fortune by holding some talks from some of our favourite thinkers, and to have an excuse to meet more people in the wider mySociety community face to face.

To that end, we’re holding four talks in London this autumn (location TBD but almost certainly a centralish pub). Each link below goes to an Upcoming page where you can sign up to let us keep track of numbers and how big a venue we need.

4/10/2007 – Stefan Magdalinski, net-political troublemaker extraordinaire

1/11/2007 – Steve Coast, founder of Open Street Map

CANCELLED 29/11/2007 – Jason Kitcat, e-voting expert

12/12/2007 – Peter Wainman, IT-specialist solicitor and blogger

We look foward to seeing you there.

Last Friday Matthew and I went to the Ordnance Survey’s UK Geospatial Mash-up day. And a splendid time was had by all. Really this post is just a placeholder for a link to a copy of my presentation slides (not quite what I delivered, I’m afraid), but if anyone was there and has any questions they weren’t able to put to us in person, or wasn’t there and wished they had been, then the comments section awaits….

Much of my August seems to have been absorbed with maintenance tasks.

For example, Chris and I spent a few days tightening up WriteToThem’s privacy. I made sure the privacy statement correctly describes what happens with backup files, and failed messages. I reduced the timeouts on how long we keep the body of failed messages. I made sure we delete old backup files of the WriteToThem database. I wrote scripts to run periodically to check that no bugs in our queueing demon can accidentally mean we keep the body of messages for longer than we say. I added a cron job to delete Apache log files older than a month for all our sites. As AOL know to their cost, the only really private data is deleted data.

Earlier in the month, I handled some WriteToThem support email for the first time in ages. We get a couple of hundred messages a week, which Matthew mainly slogs through. It’s good for morale to do it, as we get quite a lot of praise mail. It is also hard work, as you realise how complicated even our simple site and the Internet are, and it leads to fixing bugs and improving text on the site. I made a few improvements to our administration tools, and things like the auto-responder if people reply to the questionnaire, to try and reduce the amount of support email, and make it easier to handle.

I did some more work on the geographically cascading pledges (like this prototype one), but I’m still not happy with them. In the end, I realised that it is the structure of wording of the pledge that is the key problem. Our format of “If will A but only if N others will B” just isn’t easily adapted to get across that the pledge applies separately in different geographically areas. Working out how to fix that is one of the things we’ll brainstorm about in the Lake District (see below).

The last couple of days I’ve been configuring one of our new servers who is called Balti, and getting the PledgeBank test harness working on it. Until now, it has only been run on my laptop. This is partly heading towards making a proper test harness for the ePetitions site, running on a server so we properly test nothing can be broken before deploying a new version.

Matthew has wrapped up the TheyWorkForYou API now, and is working on Neighbourhood Fixit next. Chris has been doing lots more performance work for the ePetitions site. And he’s been making some funky monitoring thing to detect PostgreSQL database lock conflicts, which we get occasionally and are hard to debug

Tom’s in Berlin at the moment, he gave a talk last night, and I think has been to see some people from Politik Digital. As we’ve been discussing on the mySociety email list, there’s an EU grant we’re likely to apply for in collaboration with them.

On Friday, we’re all going to the Lake district for a week, with some of the trustees and volunteers intermittently. We very conveniently and cheaply all work from home, so it’s good and necessary to meet up for a more sustained period of time at least once a year. Last year we were in Wales.

I haven’t posted on here for six months; quality over quantity is my motto. 🙂 In that time I have answered a lot of user support email (we’re up to Ticket 5741 in RT, the subject of my last post, and quite a lot of that isn’t spam or out of office replies…) and made sure that Hansard has parsed successfully into TheyWorkForYou’s database every morning. I get all the fun jobs. 😉 Oh, I also quit my day job to work for mySociety basically full time, and wrote or helped write WriteToThem Lords, TheyWorkForYou Lords, various features and improvements to our other sites, and much else I’ve probably forgotten. It’s been fun.

Today, I gave a talk (my first!) at LUGRadio Live 2006 on mySociety in general, with various open source related ramblings in particular. I wrote the presentation in Slidy, Dave Raggett’s HTML slide generation software, which means anyone should be able to see it online at https://www.mysociety.org/2006/lugradio-live/ – I’ll try and add some speaking notes to explain some of the slides in more detail. The talks were all recorded, though the audio of the talk before mine didn’t work – I’m not sure if mine did, or if so, where they’re going to be uploaded, but I’ll post here if I find out.

Lastly, as a minor rant, my laptop’s power supply just stopped working yesterday – annoying when you hoped to use it for a talk the next day. I had bought it because my old one had become very frayed and liable to cut out randomly; I’ve had to switch back to that one (cutting out sometimes is better than nothing), and it’s proving very annoying. Dell using differently shaped plugs for all their models is rather annoying too. Grr.

PledgeBank has had a great first month-and-a-half with over 20,000 signatures on over 500 pledges. The task has now started of trying to get word about PledgeBank out to those little grassroots organisations that would really benefit from it.

To do this, we need your help. We’ve created a presentation (PDF) specifically aimed at small grassroots organisations, and we’d like you to let organisations near you know about it. All you have to do is google for charties and community groups in your area, and send them the link – why not lend us a hand right this minute?