I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.

mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.

Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages – causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).

Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.

(the autumn photo thanks to Nico Cavallotto)

“People often say they could run Britain better than the political parties. A web-based revolution may give them the chance”

Nice article in the Sunday Times today mentioning lots of our sites and others.

It’s the first full working day for the new facility to annotate Freedom of Information (FOI) requests on WhatDoTheyKnow, and people have been hard at it.

Mr Ormerod points out that private information isn’t necessarily so private if someone has died, so perhaps the exemption the MOD used shouldn’t apply.

Trevor R Nunn has posted three annotations (e.g. this one) to show that his three FOI requests are being treated as one. The annotations facility is great for handling edge cases like this, which don’t happen often enough to be worth explicitly adding to the code, but need some mention.

And finally Edward Betts has processed the list of post boxes retrieved by FOI into a more structured data format, and posted up a link to it. Exactly the kind of collaboration I love to see!

And that’s just this morning!

One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.

So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.

I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline – something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.

The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.

If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.

We’re busy as bees, lots of things happening, increasingly many of which are commercial, and we can’t talk about until they’re released.

Commercial? But you’re a charity! Yes – but just as Oxfam have a trading subsidiary company which runs the second hand clothes shops, we have a trading subsidiary company that sells services relating to the websites that we make (structural details here).

Everything from other small charities to large media companies are buying our services – which range from customised versions of FixMyStreet, through to strategic consulatancy. If you’ve got something that you think we might be able to help with, email Hello@mysociety.org – easier to talk to than us geeks.

Meanwhile we’re cracking on with our free services for the public, which are increasingly funded by this commercial work.

TheyWorkForYou recently launched a Scottish version, thanks to volunteer Mark Longair, and Matthew. More goodies in store as the Free Our Bills campaign unfolds. We’ve started a sprint to get a photo for every MP’s page. If you work for or are an MP or have copyright of a photo of one that we’re missing, then email it to us.

WhatDoTheyKnow is getting lots of polishing – the new site design that Tommy has been working on is nearly ready. Today I just turned on lots of new email alerts and RSS feeds, so you can get emailed, for example, when a new request is filed to a particular public body, or when a request is successful.

Our super ace volunteers have been busy adding public authorties to the site, and we now have 1153 in total. We’re getting a steady trickle of good requests (pretty graph) coming in. Blogs such as Blind man’s buff and confirm or deny are sorting the wheat from the chaff. Do blog about and link to any interesting requests that you see!

Other things in the works are a much needed revamp of www.mysociety.org, some interesting things on GroupsNearYou, and no doubt squillions of other things. I’ll let Matthew post up anything I’ve missed 🙂

Some of the work we do at mySociety these days is policy related, and happens behind the scenes. I’m conscious that we haven’t been blogging here like we did in the early days, and that is partly because advice and consultancy often have to be confidential.

Two speeches, both of which mention TheyWorkForYou, were recently given by senior members of the UK’s two main opposing parties. They’re both worth reading, and will set you thinking about how much further mySociety’s work can be taken.

First a recent speech by Tom Watson MP, a Cabinet Office Minister on “Transformational Government”. He talks about the massive change we are living through, in terms of how IT can and will improve Government.

Less than a decade ago, people were just recipients of information, they got what they were given when they were given it. Today, the most successful websites are those that bring together content created by the people who use them (Tom Watson)

Second a speech by David Cameron, Leader of the Opposition, which talks a lot about open knowledge [link removed Oct 20013 as no longer available]. Can local government be transformed by better information? TheyWorkForYou for local councils?

We will require local authorities to publish this information – about the services they provide, council meetings and how councillors vote – online and in a standardised format. (David Cameron)

Residents turn to web in lane fight describes a set of problems reported on FixMyStreet.

A NARROW mountain lane has been damaged and turned into an “international playground” for 4x4s and satnav-guided lorries, angry villagers claimed yesterday.

The interesting thing is the claim that making the reports public really helped pressurise the authority into fixing them

one [resident] claimed the local authority had now been “embarrassed” into action by the complaints on fixmystreet.com

This project became Mapumental. Please visit that site for details of our travel-time maps services.
The work was funded and supported by the Department for Transport.

See also: the main travel-time maps report.

——————

Our newly released travel time maps are currently shooting round the internet. It was great fun making them, and you might like to have a go too – there are plenty of public datasets you could overlay on the same base maps, using the same flash app (source code). There are a few notes about how we made them on the page itself, and the associated real time page. For a far more interesting view of the development process, read Tom Carden from Stamen’s account.

The most interesting blog post I’ve seen to come from this is Whitehall staff have no life by Simon Dickson, who was inspired by the maps to think about the destruction of social capital caused by commuting. “Whitehall staff on all but the highest salaries can’t expect to live anywhere near their work, and hence can’t expect to have any kind of a social (capital) life.”

It’s been a long year.

A friend just pointed me to this by Danny Hillis from the Whole Earth Catalog, Winter 2000.

Like much of my generation, I grew up believing that I should try to “change the world,” presumably for the better. But I didn’t know how to do this. Looking at how other people have changed the world I concluded there are five ways of doing it:

• Some people change the world by imposing their will on it.
• Some people change the world by discovering a truth.
• Some people change the world by changing people’s minds.
• Some people change the world by creating things of great beauty.
• Some people change the world by making new tools for change.

Danny Hillis goes on to say that the last one, the making of new tools for change, is the one that appeals to him the most. I think my friend was just flattering me, as that is what mySociety tries to do.

Chris, who I started this year with and have sadly ended it without, would, I think, have been on the surface deeply cynical about even the last one. He’d have sad that tools could be used for evil – indeed, part of the point of campaigns that he took up from his heart, such as no2id, is to point out how computers are just such a tool that can be used for evil.

Nevertheless, he worked tirelessly to make other tools (e.g. WriteToThem), so that we could all use them for good. Hopefully, you can pick the tools that you make, choosing ones that maybe a few will use unwisely, but for which the many will make it up by using them wisely.

Happy Christmas!

May your New Year be full of will imposed judiciously, truth that both aches and thrills, minds changed to be more enlightened, beauty that is great, and more tools that everyone can use wisely for change.

On our servers we only install software from Debian packages, or our own software with install scripts from our own CVS. This at first seems a bit mad, especially to Ruby on Rails people who love their gems. But it’s a sane way of managing lots of servers (we’ve got 7 Debian servers, and 2 FreeBSD servers to run at the moment).

Of course, you could install packages on them from CPAN, from Ruby Gems, by compiling them yourself and putting them in /usr/local. But you’d have to have another system for each packages system to keep track of what you’d installed and what version, and to worry about security updates. And you’d lose some of the benefits of dependency checking.

Most of our servers are, inevitably, still running Debian Sarge (the latest and greatest when we started them a few years ago). We’re going to gradually upgrade them to Debian Etch, but it is going to take a while. In the fast moving world of Rails this isn’t particularly helpful, so you have to backport packages. I couldn’t find any, so have made some myself.

You can find packages for Rails 1.2.5-1 on Sarge in our Debian package repository. Yeah, still an old version for you people “living on the edge”, but it’s the one in Etch (the latest Debian stable), and is way better than 0.13.1-1 that we had before 🙂