Help us to make more
useful things.
Donate to mySociety
Public authorities have now sent back 100 Excel files in response to FOI requests on WhatDoTheyKnow.
The nice thing is, that if somebody bothered to use a spreadsheet, it must contain useful, factual, numerical data across either time or space. Everything from job advert expenditure in Kings Lynn council, to school budgets in the Western Isles.
Have a mine.
P.S. Don’t forget to click “Track things matching ‘filetype:xls’ by email” to be emailed when there are new spreadsheets to look at :)
One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.
So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.
I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline - something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.
The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.
If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.
The epic task of manually matching each of the 42,019 video clips of MPs was started way, way back, ooh, about 12 whole weeks ago. Two days ago the Number 1 rated volunteer timestamper in our league table, Abi Broom, completed the last clip in our database, bringing her personal tally to 8,543 clips.
Abi Broom, No1 timestamping league table champ
Last night we went out and met with Abi and Robert Whittakker, one of the other super-timestampers who had done over 2,000 himself.
As a result of their efforts, and those of hundreds of other volunteers, we have put all the video that we have of the House of Commons sitting over the last year online, next to the text of the debates. The many thousands of people per day who visit TheyWorkForYou can, as a direct consequence of this work, now see video of most of the debates for the last year. When people embed clips on their own sites, that’ll also be thanks in part to our volunteers.
Robert Whittaker, down in 5th place with a measly 2047 clips to his name
When Parliament starts again in the Autumn there’ll be another 300-400 clips a day to do, but we have a feeling the only problem doing them will be who gets to them first.
In the meantime, we’ll soon be working on another game-like toy to help create more data. Hint - it might have something to do with GroupsNearYou.
It’s the first full working day for the new facility to annotate Freedom of Information (FOI) requests on WhatDoTheyKnow, and people have been hard at it.
Mr Ormerod points out that private information isn’t necessarily so private if someone has died, so perhaps the exemption the MOD used shouldn’t apply.
Trevor R Nunn has posted three annotations (e.g. this one) to show that his three FOI requests are being treated as one. The annotations facility is great for handling edge cases like this, which don’t happen often enough to be worth explicitly adding to the code, but need some mention.
And finally Edward Betts has processed the list of post boxes retrieved by FOI into a more structured data format, and posted up a link to it. Exactly the kind of collaboration I love to see!
And that’s just this morning!
A lot has been happening with PledgeBank in the U.S. over the past few months, but an exciting podcast just went up that’s a great synthesis of some of that work. Idealist just launched a podcast featuring PledgeBank, and also telling the story of one of our favorite pledge campaigns. You can hear the podcast here.
You can also see more from the Bakul Foundation on their new website. Good things are happening, my friends…please help spread the word about PledgeBank to those folks you know who want to do incredible things like what the Bakul Foundation has done!
Last week I gave my first presentation by video conference. It was to the intriguing Circus Foundation, who are running a series of workshops on new democracy. It came about because I was a bit busy and tired to travel from Cambridge into London. Charles Armstrong, from the Circus Foundation, suggested that I present over the Internet.
We used Skype audio and video, combined with GoToMeeting so my laptop screen was visible on a projector to an audience in London. Apparently my voice was boomed round the room. It was a slightly odd experience, more like speaking on the radio. However, I had a good serendipitous one to one chat while we were setting up, with Jonathan Gray from OKFN.
I was asked to give a quick overview of mySociety, as a few people in the audience hadn’t heard of us, and also to talk about how I saw the future of democracy. I talked about three of our sites, and what I’d like to see in each area in 10 years time.
Charles Armstrong, from the Circus Foundation, has written up the workshop.
Downsides of the video conferencing were that I couldn’t hear others speak, as they didn’t have the audio equipment. I had to take questions via Charles. This meant I also couldn’t participate in the rest of the evening, or easily generally chat to people. All very solvable problems, with a small amount of extra effort - Charles is going to work on it for another time.
Of course this also all saves on carbon emissions (cheekily, taking off my mySociety hat for a moment, sign up to help lobby about that).
Wow! The video timestamping site on TheyWorkForYou has been live for just under ten days, but you’ve already managed to timestamp almost 14,000 clips - that’s almost 40% of our entire video archive!
Thank you for all the good work - and especially to our top timestampers. David Jones, Alex Hazell and Lee Maguire are currently the top three in the overall rankings, but there are five more people who have timestamped more than 500 clips, another seventeen people who have done 100 clips or more, and more than 100 people who’ve done anything from 1 to 100 clips. And of course, there’s also a fair few anonymous people who haven’t yet registered, so their individual contributions to the “anonymous” total of 3349 clips are not recorded on the league tables. Remember, we’ll be handing out prizes to the top timestampers, so get registered before you timestamp your next video!
We’re starting to collect a list of notable clips that we can use to compile a “best of parliament” video gallery - if you would like to nominate a particular speech, please leave a comment below or send an email to team@theyworkforyou.com - just tell us the name of the MP speaking, and the URL of the page where this speech appears on theyworkforyou.com. We’ll put the best of them together and publish a list later this summer.
We’re busy as bees, lots of things happening, increasingly many of which are commercial, and we can’t talk about until they’re released.
Commercial? But you’re a charity! Yes - but just as Oxfam have a trading subsidiary company which runs the second hand clothes shops, we have a trading subsidiary company that sells services relating to the websites that we make (structural details here).
Everything from other small charities to large media companies are buying our services - which range from customised versions of FixMyStreet, through to strategic consulatancy. If you’ve got something that you think we might be able to help with, email Karl. He’s easier to talk to than us geeks.
Meanwhile we’re cracking on with our free services for the public, which are increasingly funded by this commercial work.
TheyWorkForYou recently launched a Scottish version, thanks to volunteer Mark Longair, and Matthew. More goodies in store as the Free Our Bills campaign unfolds. We’ve started a sprint to get a photo for every MP’s page. If you work for or are an MP or have copyright of a photo of one that we’re missing, then email it to us.
WhatDoTheyKnow is getting lots of polishing - the new site design that Tommy has been working on is nearly ready. Today I just turned on lots of new email alerts and RSS feeds, so you can get emailed, for example, when a new request is filed to a particular public body, or when a request is successful.
Our super ace volunteers have been busy adding public authorties to the site, and we now have 1153 in total. We’re getting a steady trickle of good requests (pretty graph) coming in. Blogs such as Blind man’s buff and confirm or deny are sorting the wheat from the chaff. Do blog about and link to any interesting requests that you see!
Other things in the works are a much needed revamp of www.mysociety.org, some interesting things on GroupsNearYou, and no doubt squillions of other things. I’ll let Matthew post up anything I’ve missed :)
mySociety has been asked to do some work on an online petition system for Number 10 Downing street. mySociety is delighted to have been asked to bring our experience to the task, and aims to develop a new tool as easy to use, popular and trustworthy as our other non-partisan democractic web sites. In particular we are pleased to have jointly designed a system in which moderation is extremely transparent, and which we hope can be used as an exemplar of good practice elsewhere.
Francis and Matthew have just started coding, and we plan to have a public beta running later in the year. All the code that is generated by mySociety (and which isn’t specific to the No10 site, like logos) will of course be available under open source licences, and we will be delighted to talk to other organisations who are interested in petition systems themselves.
We sometimes have incorrect or out of date addresses for sending Freedom of Information requests to. Now anyone can check our addresses. Click “view FOI email address” on the page for any authority, and enter two of those squiggly words to prove you are not a robot.
If you are using WhatDoTheyKnow, and suspect problems with a request, please do check the address we are using is correct. If you are from an authority, or work closely or know a particular authority, please also check the address.
Over a thousand Ebbsfleet United supporters have used PledgeBank to raise enough money to buy a striker Michael Gash from Cambridge City.
This is an excellent example of why you should never pre-determine exactly how people are going to use your site!
Last week at the SustainIT eWell-Being Awards, we picked up an award for FixMyStreet. The judges said it was “[a]n excellent example of an independent website which empowers the general public in their dealings with their local council. It is a relatively simple application, yet highly effective and replicable.” One example the accompanying Independent supplement mentioned was “a community in Great Yarmouth which joined forces through FixMyStreet to clear their local unused railway track. The site made possible a dialogue between community members and the council’s community development worker, who organised a “clear up” day where locals could get involved with rectifying the situation, with tools, insurance and even a barbeque provided.” It’s great to see that sort of thing happening on the site, and also great to be recognised in this way.
In a spirit of celebration (though more to celebrate the endorsements the campaign has received), TheyWorkForYou now covers the Scottish Parliament - see the TheyWorkForYou news for more information.
FixMyStreet has a lot of RSS feeds. There’s one for every one-tier council (170), one for every ward of every one-tier council (another 5,044), two for every two-tier (county and district) council (544), and two for every ward of every two-tier council (20,296) – two per two-tier council because you might want either problems reported to one council of a two-tier set-up in particular, or all reports within the council’s boundary.
Then there’s an RSS feed every 162m across Great Britain in a big grid, returning all reports within a radius of that point, the radius by default being automatically determined by that point’s population density, but customisable to any distance if preferred. That’s, at a very rough approximation assuming Great Britain is a rectangle around its extremities, which it’s not, 19 million RSS feeds, lots of which will admittedly be very similar. :)
Every single one of those feeds can be subscribed to by email instead if that’s preferable to you, and are all accessible through a simple interface at http://www.fixmystreet.com/alert.
However, none of these RSS feeds was suitable for the person who emailed from a Neighbourhood Watch site and said that all they had was a postcode and they wanted to display a feed of reports from FixMyStreet. Given you could obviously look up a FixMyStreet map by postcode, it did seem odd that I hadn’t used the same code for the RSS feeds. Shortly thereafter, this anomaly was fixed, and if you now go to a URL of the form http://www.fixmystreet.com/rss/pc/postcode you will be redirected to the appropriate local reports feed for that postcode (I could say that adds another 1.7 million RSS feeds to our lot, but given they’re only redirects, that’s not strictly true). And after a couple more emails, I also added pubDate fields to the feeds which should make displaying in date order easier.
It’s great to see our RSS feeds being used by other sites – other examples I’ve recently come across include Brent Council integrating FixMyStreet into their mapping portal (select Streets, then FixMyStreet), or the Albert Square and St Stephen’s Association listing the most recent Stockwell problems in their blog sidebar. If you’ve seen any notable examples, do leave them in the comments.
Residents turn to web in lane fight describes a set of problems reported on FixMyStreet.
A NARROW mountain lane has been damaged and turned into an “international playground” for 4×4s and satnav-guided lorries, angry villagers claimed yesterday.
The interesting thing is the claim that making the reports public really helped pressurise the authority into fixing them
one [resident] claimed the local authority had now been “embarrassed” into action by the complaints on fixmystreet.com
Super WhatDoTheyKnow volunteer John Cross has made an interesting petition about Freedom of Information and publicly owned companies
“We the undersigned petition the Prime Minister to support a change to the law to make companies owned two thirds or more by public authorities subject to the Freedom of Information Act 2000.”
The petition goes on to explain (in more details at the bottom right of the petition page) that the situation is quite comical at the moment. If a company is owned by one local authority, then it is subject to FOI, but if it is jointly owned by two then it isn’t. This makes little sense, and it is also very important, as private companies owned by authorities often do important work.
This week has been quite bitty. I’ve been doing more work on the Freedom of Information site, have been getting into the swing of Ruby on Rails. Once you’ve learnt its conventions, it is quite (but not super) nice.
As far as languages are concerned, Ruby seems identical in all interesting respects to Python. It’s like learning Spanish and Italian. Both are super languages. Ruby has nice conventions like exclamation marks at the end of function names to indicate they alter the object, rather than return the value (e.g. .reverse!). But then Python has a cleaner syntax for function parameters. It is swings and roundabouts.
Rails has lots of ways of doing things which we already have our own ways of doing for other sites. The advantage of relearning them, is that other people know them too. So Louise was able to easily download and run the FOI site, and make some patches to it. Which would have been much harder if it was done like our other sites. Making development easier is vital - for a long time I’ve wanted a web-based cleverly forking web application development wiki. But while I dream about that, Rails packaging everything you need to run the app in a standard way in one directory that quite a few people know how to use, helps.
Other things… I’ve been helping Richard set up GroupsNearYou on our live servers, it should be ready for you to play with soon. It looks super nice, and is easy to use. I’ve had some work to do with recruitment. And catching up on general customer support email for TheyWorkForYou and PledgeBank. I’ve also been updating the systems administration documentation on our internal wiki, so others can work out how to run our servers.
I haven’t posted here for a while - in late May / early June I was on holiday in China for a month. After 18 months of hectic mySociety with only a break for a week, I needed it!
Since I got back, I’ve been busy on the no. 10 petitions site, which in the tag-team coding way that we do, Matthew has now taken over for a bit. And I’m doing something I’m hesitatingly calling “geocascading pledges”.
I’m not quite sure what to call it yet. It lets you make one pledge, but whose target counts separately in different towns. When someone signs up, they also choose a place. Rather than having to make a separate pledge for everywhere in the world, you can just make one.
Here’s an example pledge, which is also real (more on that later), so do sign it if you like. And then post your thoughts in the comments - it needs a bit of polishing yet.
So, I’ve just had a shower and I’m waiting for Matthew and Tom to turn up. As time goes on, mySociety seems to get more geographically disparate, and I look forward to meeting my coworkers. Then for 1pm we’ll be heading to CB2 for the mySociety developers meeting. Feel free to come along any time afternoon or evening, whatever your skills or interest in mySociety.
I haven’t posted on here for ages, since October. I’ve been away on holiday quite a lot, and when I’ve not been away I’ve been busy, partly with systems administration. We’ve set up lots of servers in the last month for the E Petitions site. When you go from 3 servers to 7 servers, there’s another step change in sorting out systems administration tools. For example, I had to change the monitoring script so every server wouldn’t monitor every other. And I had to work out the quirks and bugs in the system we have for storing config files for different classes of server in CVS. Because we only had one class of server before.
I’ve also had to learn lots about server monitoring and load balancing. Things have slowed down a bit now, to maybe 10 hits per second. But a few weeks ago the road pricing petition was often getting 50 hits per second. I’ve never worked on a site with that level of traffic before. You find all the bugs in your code, all the missing indices in PostgreSQL, all the badly tweaked FastCGI parameters. I’ve been sucking knowledge off Chris like a sponge, so tools like strace and vmstat begin to become instinctive. Seemingly nobody offers a book or a course which teaches this stuff well - every server setup is different, everyone knows different ways to tune and profile. But maybe you can tell me different in the comments.
Louise has been busily working away on lots of things. Amongst that is a major change to WriteToThem, to let you write to all the members in a multi-member constituency in one go. The last day or two, she’s been installing the WriteToThem test code on one of our servers, when it has only run on my laptop before. This will be fantastic - hopefully can get Matthew to be bolder about making changes to WriteToThem, if he has a test script he can easily run (getting Matthew to be bold isn’t normally a problem, but he seems mildly less bold when it comes to the WriteToThem queue daemon).
Tom and I have also been busy on a second travel maps report for the DfT. More on that soon. Lots of running screen scraping jobs on TransportDirect which take days. On the subject of Tom, he seems to have got expert at “stacking meetings” next to each other. In one day last week he had 7 meetings!
A few Christmases ago Chris Lightfoot and Etienne Pollard built a little throwaway website called HassleMe.co.uk as a one-day coding challenge. The purposes of HassleMe was to let people nag themselves into doing things that needed doing by arranging to be sent automated emails.
The genius of HassleMe lay in the timing of the responses. Instead of it mailing users exactly 5 days or 2 weeks after they subscribed, it mails people approximately when they ask. As a consequence you never know when the mail’s going to arrive, making it strangely more effective as a nag.
After Chris died the site sat running on his server for a couple of years, and mySociety didn’t have access to it. Last week Francis migrated the site to our own machines with the help of Pete Stevens from Mythic Beasts. When Francis looked at the database he saw that there were 16,000 hassles that users had agreed could be made public. However, the feature to show them was never built.
Now it has been added on this page, and makes for a fascinating, often foul mouthed insight into what it is that people need to motivate themselves to do. There are thousands of examples, each stranger than the next, so just hit refresh for more. And please do paste and favourites as comments to this post.
PS And as always, if you like any of mySociety’s services, or even just find the comments amusing, please donate - we’re a charity!
Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?
I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. :-)
Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.
The highlighting works the other way - as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! :) ).
Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.
Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse :-)
So, there are now over 600 petitions in the petitions system, and we’re getting a steady stream of appeals from our users to add categories.
I’m posting to ask how you all think we should handle this. It seems to me that there are a few options:
More than just thinking about the overall philsophy I’d also appreciate thoughts on design. When you come to the homepage, how should the category system be presented to you? Tricky stuff, and I’d really appreciate your thoughts.
TheyWorkForYou now finds whenever an old version of Hansard is referenced (which they do by date and column number, e.g. Official Report, 29 February 2008, column 1425) and turns the citation into a link to a search for the speeches in that column on that date. This only really became feasible when we moved server, upgraded Xapian, and added date and column number metadata (among others), allowing much more advanced and focussed searching - the advanced search form gives some ideas. Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech, much like our video matching (nearly 80% of our archive done!). :)
Kudos to Google and Yahoo! for spotting this change within a couple of days, as they’re now so busy crawling everything for changes that they’re slowing the whole website down… ;-)
Matthew and I have been sitting next to each other today looking at the outputs of his lovely new custom built conversion tracking system, designed to ensure that the optimal number of users who just come to one of our services as a one off get signed up to something else longer lasting.
I’ve been banging on for ages about how government should seize on cross selling people who’ve just finished using one online service into using another of a more democratic nature, so it seems worth spelling out some of the lessons.
First, there’s some interesting data from the last few weeks, since our newest conversion tracking infrastructure has been running in its nice new format.
One of the adverts randomly served to users of WriteToThem (after they’ve finished sending their letter) encourages them to sign up to TheyWorkForYou email alerts - the service people use to get emailed whenever their MP speaks in Parliament. The advert features a slogan of encouragement, and a pre-populated email form containing the user’s email, and a ‘Subscribe me’ button. This advert was shown to 2328 users last month, of whom 676 became TheyWorkForYou email subscribers, which is a pretty cool 29.04% conversion rate. However, we also showed another advert for the same service, to the same WriteToThem users, which also had the same button and text, but which hid the form (and their address). That was shown to 2216 users of whom 390 signed up, a more modest 17.6%. So the impact of simply showing an email box with the users email address in it, versus hiding it, was worth 10% more users. Why? Go figure!
So now we’ve canned the advert that hides the address form, and instead we’re comparing two different adverts both of which feature the pre-populated signup form, but which use different words. It’s probably too early to judge, but the new ad appears to have a very similar conversion rate suggesting it might be hard to squeeze many more subscribers out of this page. We’ll keep trying though!
Another thing we learned of interest was that monthly subscribers to email alerts on TheyWorkForYou were down year on year in the month before we added this new advertising & conversion tracking system, even though the total number of visitors were clearly up on the same month last year. This appears to suggest that two things are happening. First RSS is catching on, so some users who would previously have got email alerts are subscribing to RSS feeds instead. Second, it suggests that the TheyWorkForYou user audience might have been getting more saturated with regulars - proportionally fewer new users coming (although more visitors in absolute terms) so fewer people signing up to get alerts. The cross marketing and conversion tracking seems to have reversed that trend, which is awesome.
We also advertise several different services to people who just finish signing up to get email alerts on TheyWorkForYou itself. We’ve just noticed that a full 25% of people shown the advert to sign up for HearFromYourMP proceed to sign up. We’ve therefore just decided to dump other adverts shown on TheyWorkForYou (such as advertisements for other sorts of TheyWorkFor you email alert) and concentrate on just cross selling HearFromYourMP. A back of the envelope calculation suggests that by just advertising this one site from the completion page we should get an extra 10,000 subscribers to HearFromYourMP this year on top of the organic growth. Not bad for a few minutes analysis, and a number likely to make a fair few more MPs post messages to their patiently waiting constituents.
One last interesting thing (at least to me) is how some more demanding services are a much harder sell than others to users. So asking people to make new groups on GroupsNearYou.com or report a problem in a street on FixMyStreet tend to result in more traditional online marketing scale conversion rates of 0.1% to 2%. Still worth doing, and so we compare different versions of those ads too, to try and eke up those rates for these sites that arguably have more tangible, direct impacts on people and communities.
It will be a challenge for mySociety’s future to work out how to trade off impact against scale of service use - are 10 HearFromYourMP subscribers worth one pothole that doesn’t get fixed? Answers on a postcard…
PledgeBank is mySociety’s second project. The purpose of PledgeBank is to get people past a barrier which strikes down endless good plans before they can are carried out - the fear of acting alone. It allows anyone to say “I’ll do X if other people also do X”, for example “I’ll write to my councillor if 5 other people on my street do the same”. However, there is no scale to big or too small, it could equally be used to say “I’ll start recycling if 10,000 other people in Britain also start”.
Pledgebank developement will start soon, and we are now looking for partner organisations, large and small, who would like to use Pledgebank when it launches. If you run a charity or other organisations, or if you know anyone who does, please do email us.
Ian has used PledgeBank to start a residents association for a new block of flats in Cambridge, UK.
We’re going to be adding lots of features and major design improvements to our Freedom of Information site WhatDoTheyKnow.com in the next few weeks. One thing we want to add is little explanatory videos helping describe how to make the best request possible.
Today we’re launching the hastily named WhatDoTheyKnow Video Challenge. We want you to make short videos (max 2 mins) in which you explain in a clear and friendly way how to file successful FOI requests. We’re not expecting Hollywood production values, just a friendly face and a good explanation would do fine. If you can do funny, splendid.
Instructions:
1. Record your vid
2. Upload to your video hosting venue of choice
3. Post the link as a comment to this post
We’ll send out a coveted mySociety hoodie to anyone who makes anything we’d seriously consider using (unless bazillions of people enter, of course - do you want to bankrupt us?). We don’t sell our coveted mySociety hoodies, they’re only for people who’ve done something useful for mySociety so they’ll mark you out as a pillar of the community the first time you walk down the street.
Which are yours?
Bakul-Library - it’s just awesome, that they used PledgeBank, that they got a 1000 people, and that they’re going to make a new library in India because of it.
deathpenalty - Not because it was effective, and not because I agree with it. But because I disagree with it, and because of the reaction it got from left-wingers not used to seeing other points of view. PledgeBank is for anyone.
walkout - PledgeBank offline. These school students got together to march against war (whatever or yippeee), but just LOOK at how many text message sign ups there were. Tom saw in London an illegally flyposted flyer.
First1000 - I mean, they may be crazy libertarians, but ONE THOUSAND PEOPLE are going to move house. OK loads won’t move. But hundreds will.
tunepiano - It’s about fun and joy, not just politics and worthiness (bit of a worthy example to make that point I know, but it is a piano!)
Google have launched a new UK politics site at google.co.uk/politics. mySociety were delighted to be asked to build a TheyWorkForYou widget for this site. There’s no doubt that this sort of modular re-purposing of our information is going to happen a lot more in the future, and it’s great to start out with the best of possible partners.
There’s a lot left to do, but Francis Irving’s brilliant new mySociety Freedom of Information site is now live. You can file requests to central government departments (most of the them), and browse what other people have been requesting (already fascinating). It doesn’t have a name yet, nor any slick design, nor half the features we want it to have, but it works and it gets things done.
And dammit, people, that’s what mySociety’s all about. Can we explain it any better?
Last night was the annual New Statesman New Media Awards, held in Westminster Abbey’s College Gardens. mySociety were finalists in two categories, Modernising Government and Contribution to Civic Society, with both Number 10 petitions and FixMyStreet nominated in both. Also, two other projects we host, PlanningAlerts and The Government Says, were both finalists in the Information & Openness category.
It was a lovely evening, seeing some people I haven’t seen for some time and meeting new people too. We ended up winning in both our categories - the Number 10 petitions site in Modernising Government, and FixMyStreet in Contribution to Civic Society, which is obviously fantastic for everyone involved. The judges were impressed at the open source nature of the petitions site, and the “deceptive simplicity” of FixMyStreet. This is now the third year in a row we’ve won the Civic Society award - TheyWorkForYou won in 2005, and WriteToThem in 2006, so we’re obviously doing something right. :)
It’s a shame that Chris could not be with us, but his mother did attend to see the projects he worked on recognised.
Thanks and congratulations to all the other winners and finalists.
I’m very pleased to announce that the petitions system we’ve built for 10 Downing Street has gone live today.
I’m very grateful for the hard and often inspired work put into this by Chris Lightfoot and Matthew Somerville, as well as the civil servants who have helped to build a petitions system which I believe is in a real class of its own.
The most notable features are:
1. Petitions are accepted and published, regardless of the political slant of the petition. However, if they break the Ts&Cs (a petition that doesn’t actually ask for any action, for example) then they are put on a special rejected petitions page: they don’t just vanish. We think this transparency feature is probably unique.
2. The site is being launched in beta, and will change over time. This might seem too commonplace to note for many of you, but it reflects a willingness to see a public IT service evolve in response to users, not simply fulfil a contract agreed in advance. mySociety exists partly to spread good practice in the public sector, and we think this is a nice example of that in action.
3. The code, including Chris’s amazing high-load optimised engine, is all open source.
Any questions? Come into our chat channel at www.irc.mysociety.org or mail us at team@mysociety.org.
Francis has been furiously adding new features to our Freedom of Information website WhatDoTheyKnow ever since it launched earlier this year. He’s just added one of the most important missing features, the ability to leave annotations or comments on FOI requests.
This is especially useful for providing plain English summaries of what information in a response was actually interesting, or to discuss refusals to supply information and what to do with them. To add one just go to a request page and scroll to the bottom, just like adding a comment on a blog post.
So, whether you’ve made a request in the past, or you’re just interested in helping out, get annotating.
I’ve had the good fortune to talk with a few folks over the past several days who have echoed many of the things that we’ve been learning at PledgeBank over the past several months and years, and I thought I’d share…
First, I chatted yesterday with Jason Dick, who blogs at A Small Change. The blog focuses on nonprofit fundraising, and in our conversation we chatted about the tendency of nonprofits to be more than a bit behind the curve when it comes to adoption of new ideas/technologies…but that’s old news. We also chatted about the organizations that have decided to take a bit of a chance on online fundraising, and that have done some incredible things. We’ve seen the same thing happen at PledgeBank — a small organization (or a not-yet-formed organization) trusts its supporters enough to put some modicum of responsibility in their hands, and gets a tremendous response.
I also chatted yesterday with Peter Dietz of Social Actions and the impressive lot behind the Social Actions Mashup (selected as a finalist in the NetSquared competition, btw). There was great conversation about the value in aggregating information, what the nonprofit sector can gain from the syndication of social actions, and how we can all work together to create more ways for more people to do more good. Really.
Finally, I had the pleasure of reconnecting with an old friend, Susannah Cowden, who is now working with Be the Change. As an organization that’s still forming and growing, there’s significant pressure to both be different and be innovative — I hear the same struggle from a lot of organizations in the U.S. While there are many folks who have every intention of trusting their supporters to act on their behalf and equipping them with the tools to do so, there are very few who actually do it.
I wanted to capture these conversations because these themes are not at all unique. We all know that nonprofits move slowly when it comes to technology. We all know that those willing to take a chance are, more often than not, rewarded. And we all know that there’s a delicate balance between controlling your organization’s message (especially if your tax status is 501(c)3 in the U.S.) and empowering your organization’s members.
What I think these conversations illuminated for me is the need for someone to cut through all of these lessons and to find ways to make real stuff happen, with real impact. That’s what mySociety is about: giving people “simple, tangible benefits in the civic and community aspects of their lives.” And (I hope) that’s what PledgeBank enables. Holler (heather at pledgebank dot com) if you’re interested in working together to create a larger platform for collective community action.
We’re delighted to announce that mySociety has been generously awarded $100,000 by Omidyar Network, a mission-based investment group committed to enabling individual self-empowerment. The purpose of the investment, our largest to date from a philanthropic source, is to spread the use of PledgeBank.com to a host of community and grassroots groups in the US.
Omidyar Network invested in PledgeBank because it encourages collective effort by enabling users to rally the support of others through their own commitment to take action.
From starting new organizations, to giving blood, to volunteering to mentor others, Pledgebank has enabled people to do things they wouldn’t have done otherwise. Just this morning we were sent these pictures of a protest in Brazil that was organised using PledgeBank.
What has been missing to date, though, has been any resource to spread the use of the tool to traditionally offline groups. Whether these are schools, community groups, NGOs, churches or neighbourhood watch groups, many could benefit from PledgeBank, but few have had the opportunity to hear about it. Therefore, PledgeBank is looking to hire an outreach coordinator who will travel around the US meeting groups, listening to their goals, and explaining how PledgeBank can help them, for free.
mySociety will be recruiting the outreach coordinator in the New Year. If you’d like to get in touch, perhaps to pre-book an appointment to meet or talk with the outreach coordinator, or because you know someone who could do the job well, please drop us a line to team@mysociety.org.
One of mySociety’s most below-the-radar projects is GroupsNearYou.com , a project to build a web service that eventually will allow other websites, such as FixMyStreet, to tell their users “Look! There’s a local email list here. Why not join it and discuss what you can do to stop those phoneboxes being smashed up?”"
However, in order for GroupsNearYou to become a useful web service for mySociety and the rest of the geospatial Internet, it really needs a good pile of pre-existing groups adding from across the globe. To help with this process Richard Pope has built a little game, rather in the spirit of our video timestamping game. To play it involves trying to identify which Yahoo Groups (and soon others, like Google Groups) cover which areas on the ground.
Reasons we think you should have a play include:
Thanks Richard!
Some lovely new volunteers have been working their socks off to add the Scottish Parliament to TheyWorkForYou. Yay!
Unfortunately, the server that TheyWorkForYou sits on is almost full, so we can’t launch their hard work. Boo!
TheyWorkForYou isn’t an externally funded project, and we need funding from other sources to keep it growing and improving. So if the season has filled you with generosity of spirit, why not drop us a few pennies to pay for some upgrades? Any extra beyond what we need will go into the general pot to keep mySociety running and the developers from starving.
Unfortunately, I’ve had to disable the PledgeBank Facebook application. It used to let you sign and share pledges from within Facebook.
Facebook recently changed their platform (again!), breaking our code for sending success/failure messages. Obviously, it is no good signing up to a pledge if you don’t get informed when it succeeds.
I tried to fix it, but couldn’t work out how to do so quickly. We don’t have the time and money at the moment to chase after this, so I’ve disabled the application entirely. Links to PledgeBank pages on Facebook now redirect to pledgebank.com.
Hopefully it’ll be back one day - do send us emails if you miss it (or money if you have a large pledge that really needs it!). I think there may be a better solution with a simpler interface - the current application tried too hard to reimplement all of PledgeBank within Facebook. And besides, we should be supporting OpenSocial now it exists. It’s an open standard, Facebook isn’t.
Technical details: We used infinite session keys to send notifications from cron jobs. Quite reasonably, this no longer works. However, I couldn’t find out what to use instead. I think Facebook should respect backwards compatibility of its APIs a lot more, and if it breaks it they should give clear instructions about what to use instead. This does put me off ever wanting to develop anything on their platform again.
So, PledgeBank launched today. Already, you can sign pledges to have trees planted to compensate for your CO2 emissions, help clean up the banks of the River Taff, support local retailers, and one or two, err, more politically controversial ones. So, take a look, and sign — or, better, create — some pledges!
If you enter your postcode on TheyWorkForYou and it’s Scottish or Northern Irish, you’re now presented with your MSPs and MLAs as well as your MP, which makes sense given the site covers their Parliament and Assembly respectively. :-) You also get an extra tab in the navigation linking through to Your MSPs or MLAs. In order to do this, I needed a quick way of determining if a postcode was Northern Irish or Scottish. Northern Ireland was easy, as all postcodes there begin with BT. I assumed Scotland was also easy, which turned out to be true apart from the TD postcode area that straddled the border like a mail-sorting Niagara Falls. After some very dull investigation, I eventually worked out that e.g. most of TD15 is in England, but (amongst others) TD15 1X* is in Scotland, except for TD15 1XX which is apparently back in England. The final result was the postcode_is_scottish() function in postcode.inc, which (hopefully) correctly determines if a given postcode is Scottish or not - perhaps someone else will find it useful.
My house mate just said that his friend, who is at sixth form college, just had a PSHE (personal, social and health education) lesson in which they studied the website TheyWorkForYou.com.
Apparently it is good and I should go to it.
I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.
mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.
Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages - causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).
Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.
(the autumn photo thanks to Nico Cavallotto)
Say hello to GroupsNearYou.com
The Short Geeky Explanation
GroupsNearYou.com is an entirely user generated API-queryable database of the location and nature of local online communities, irrespective of the platform they are hosted on. A piece of the programmable web, in short, with local community building focus. Check this lovely example of how easily the syndicated community information can be layered onto a map, for example.
The Business Problem it Solves
Do you run a site that tells people stuff about their local area? Do you suspect that there might be quite a lot of internet enabled community activity going on in local areas that you’d like to tell your users about? Use our feeds.
The Social Problem It Solves:
There’s a proven real world social value to people belonging to very local email lists and other forms of local online community. However there is no eBay or Craiglist or other market dominant player in the local online community world, instead there’s a myriad of google groups, yahoo groups, Facebook & other YASN groups, extremely old school CCed email lists, online forums and so on. As a consequence of not having one big simple place to go to find and join local groups (many of which are not even on the web for Google to find) far fewer people ever find out about and join their local online groups. GroupsNearYou.com is about getting more people to join groups, groups that are not hosted by us, and (hopefully) mainly discovering them via uses of our syndicated info on sites that aren’t run by us. It’s a piece of pure internet infrastructure, with a positive social bent.
Who did it?
Astonishingly, the project was almost entirely built by a volunteer, Richard Pope assisted in design by another brilliant volunteer Denise Wilton. Their only reward is a highly sought after mySociety Hoodie, plus the love and gratitude of all our users.
Richard has been an amazingly dedicated volunteer for mySociety and on his own projects for over two years, and deserves the reputation he is rapidly gaining as one of the world’s truly great civic minded web innovators. The project was funded by the UK’s Government’s Ministry of Justice who have been trying to run experiments of different kinds in the realm of electronic democracy. Their money will go to help improve and grow the site, rather than building it, which is a very interesting funding model in its own right. The several hundred groups already in the system are mainly added by users of WriteToThem.com
Next steps
Almost all the groups listed in the database are in the UK at the moment, and they’re all from users of our other sites. We’re interested in working with anyone who runs sites that might want to either take information out of it, or put information into it (Hello email list/social network providers!)
Anyway, it’s dead simple really, just a little brick in the internet wall, albeit one that I hope will help a few more people meet their neighbours and improve their communities.
It is with great sadness that I must report the death of Chris Lightfoot, mySociety’s first developer and a good friend to all of us. He was found by friends at his flat on February 11th. The main announcement can be read in this post on his blog.
Chris was perhaps the pre-eminent example so far of what polymath means in the Internet age. His contributions to the world are more than just a formidable legacy of computer code of the very highest quality, for mySociety and many others. They also include substantial contributions to applied statistics, geographic information systems, economics and a range of public policy issues from identity cards to speed cameras.
Everything Chris did in these fields combined an incredulity-inducing array of technical and analytical skills with a wickedly funny, savage turn of phrase. To understand what a remarkable intellectual outlier he was, simply sift through his blog and marvel at the quantity of primary research and original coding that went into it. Documenting and exploring his work would provide material for many years of research, and yet all this was accomplished by the age of 28.
Within mySociety he was involved right from the start through the development of WriteToThem, HearFromYourMP and PledgeBank, building some amazing underpinning geographic and political web services like Gaze, MaPit and DaDem. These components make all our sites work and make a raft other tools and sites possible in the future.
For the last three or four months he was working at another employer, Media Molecule four days a week, but still helped the full time staff with the petitions work. The last major thing he built for us was the system that serves up the maps for Neighbourhood Fix-It, a site which was only just soft launched before he died, but of which he was apparently fond for its WriteToThem-like habit of getting simple things done that mattered to normal people.
Building mySociety’s major sites involved mighty team efforts, something which can obscure even huge invididual talent. So perhaps the sort of work for which Chris will be be most remembered is his wonderfully individualistic, virtuoso forays into scholastic areas in which he had no formal training. He wandered into differing disciplines, made a mark, and wandered on again like a giant that had no idea he’d just trodden on a village. The political survey work he did both hugely illuminates our understanding of our own political world, whilst raising the question “how come none of the professional political analysts thought of this?” And his travel-time maps should make everyone in government wonder if they’re sitting on information which could be reused to such amazing, potentially life changing effect.
Chris’ intellect and appetite for knowledge was surpassed by only one aspect of his character: his integrity. If you’ve ever wondered why WriteToThem goes to such lengths to protect users’ data it is largely because of his rock solid belief in the dignity and social indispensibility of privacy. Chris was an energetic campaigner in this field, notably for No2ID, who have posted a tribute.
It doesn’t stretch the truth an inch to say that with his death the whole of the UK’s citizenry, not just his family, friends and colleagues, will be worse off. Rest in peace, Chris.
The “um’s” and “uh’s” on the interview are embarrassing, to say the least, but PledgeBank just got its first podcast coverage. Check out my NetSquared interview here. I’m hoping this gets us a bit of exposure in the U.S.
And I’ll do better next time with all the incidental sounds… :-)
The latest “Giving Carnival” question is being organized this month by Peter Deitz, the brains behind Social Actions.
Peter’s question: “Is person-to-person fundraising dead, or just getting started?�?”
I’ll give my brief answer. Person-to-person fundraising is either dead or slowly dying, but that could be good news. Let me explain.
We’ve all probably had some experience with person-to-person fundraising — me, I sold Girl Scout Cookies. Now, I wasn’t selling Girl Scout cookies because I deeply cared about raising money for the organization or because I was interested in sharing the story of the Girl Scouts with those whose doorbells I rang. I was in it for the badges and stuffed unicorns that I could win by selling the most boxes of cookies. Person-to-person fundraising has reached a point, I think, at which there is such commodification of an organization’s story that there is little meaning left in the actual transaction…just stuffed unicorns.
Person-to-person action, however, is a different story — and I think that’s where there is tremendous potential. Take, for instance, what’s happening on PledgeBank. Individuals create a pledge to do “something,” but must make asks of their social network (friends, family, co-workers, etc.) in order to make that “something” happen. Rather than operating via a “transactional” outlook, it’s actually necessary to tell the story of why this “something” is important. There are few drive-by fundraisers on the site — you have to interact with folks and let them know why they should sign up to this social contract with you in order to meet your pledge’s target number of signers.
Similarly, I recently chatted with David Stoker, who’s working with Ashoka’s Citizen Base Initiative. They’re addressing the problem of superficial interactions by encouraging organizations to rally a fan base of individuals who support them…much like a sports team. The analogy is lost when you start imagining people painting their faces on behalf of the Red Cross, but the idea of organizations looking to their network for more than just a donation is certainly compelling.
Don’t get me wrong — I still buy and love Girl Scout cookies. But if I had to place bets on where the future of online organizing lies, it’s on peer-to-peer action that requires interaction, that illuminates an individual’s values, and that is part of a meaningful narrative arc. Thin Mint, anyone?
No lolling about in the sun for us, as we follow an endless chain of projects through the hot months. Inured to hasslebot, we’ve not been posting to this blog much. Instead, busy working on, or soon about to work on:
And that’s without mentioning Neighbourhood Fix-it and the call for proposals. Later in the year. Have I missed anything?
Have a good weekend!
mySociety has never run a campaign before today. And we’re not sure anyone’s ever run a campaign featuring a charismatic duck-billed platypus escaping from under the closing jaws of a Parliamentary portcullis.
Update 15.34 25/03/2008
Conservative Party leader David Cameron has just endorsed the campaign in this video.
Update 17.14 25/03/2008
Now kind words from techy Lib Dem MP Lynne Featherstone.
Update 11.38 1/04/2008
We’ve just recieved this fantastic endorsement from Nick Clegg, leader of the Liberal Democrats:
“Parliament belongs to the people. It’s time to open it up so people can find out what’s going on. mySociety has done a brilliant job in recent years in doing that - and it’s time to take this project to the next level and get information about the laws Parliament passes into the public domain.
“It takes a new MP months to figure out how the tortuous bills procedures work - so how we expect the voters to know what’s going on, I have no idea. The changes MySociety are calling for are vital so that every MP is fully accountable for the decisions they take on behalf of their constituents.
“I fully support the Free our Bills campaign, and will do all I can in Parliament to win this battle.”
Thanks Nick!
Well, I doubt that this will end the mashup trend going on out in the interwebs, but I thought folks might be interested in seeing this…
NetSquared (based in the U.S.) has launched their newest summer contest, the N2Y3 (that’s NetSquared Year Three) Mashup Challenge. You can see the 100+ projects that have been submitted here. One of mySociety’s projects, PledgeBank, is featured in one of the submissions: Social Actions. OK, the name isn’t super-sexy. But the idea is. Peter Deitz is developing a way to lead any given user (an individual or an organization) through the process of selecting a social action platform. Do you want to raise money? Do you need to integrate with a specific CRM? Do you need an online donation processing tool? Do you need a widget for your site? This mashup with combine 29+ (the list keeps growing) “action” tools (including PledgeBank) in that wizard, helping the average Joe or Jane figure out which tool would work best for them.
Of course, in order to move forward in the competition for mentoring and money, Peter needs your vote. To vote for this mashup (and at least four more — NetSquared is smarter than to just let everyone vote for one), just create a free account on the site and add at least five projects to your ballot. There are some really cool ones out there, so browse around a bit. The polls opened on Monday at 8am PST, and they will be closing on Friday at 5pm PST. The 20 mashup proposals with the most votes will attend the annual NetSquared Conference in San Jose, May 27 & 28, 2008. During the conference, the mashup creators will have a chance to pitch their projects to funders, foundations, and fellow nonprofit tech professionals.
As they say in Chicago, vote early and vote often!
A WhatDoTheyKnow user Tom Taylor has posted a cool query to the Royal Mail - he wants a list of where all the postboxes in the UK are (presumably so he can build a ‘find your nearest post box’ web site).
After some delay Colin Young of the Royal Mail responded with a list in a PDF file. However, whilst the list is pretty long, it only contains the postcode location of each postbox, not an actual coordinate that can be plotted on a map. So neither he, nor anyone else, can build a postbox finder service.
Just think about that for a second. The Post Office doesn’t know where its Post Boxes are. Whoda thunk? Good use of WhatDoTheyKnow.com, Tom!
Our video is streamed via progressive HTTP, using lighttpd and mod_flv_streaming. This works by having keyframe metadata at the start of the FLV (Flash video) file (we add ours using yamdi as that doesn’t load the whole file into memory first), which maps times within the video to byte positions within the file. When someone drags the position slider, or presses a skip button, the player actually changes the source of the video to something like file.flv?start=<byte position> which starts a new download from that point in the video. This means you can seek to parts of the video not yet downloaded, which is definitely a required feature.
The video is split up into programme chunks, according to BBC Parliament’s schedule, so each Oral Questions will (approximately) be its own video chunk, and the main debates will be a couple of chunks. By default, the video player will show a screengrab from the start of the video, as that’s all that’s available when it first loads (you have to load the start of the FLV file to fetch the keyframe metadata in order to move anywhere else :) ). I wanted the player to show a relevant screengrab before you hit Play, so came up with the slightly messy workaround of setting the volume to 0, seeking and playing the video for under a second in order to start it from the new point and show the video, then stopping it and resetting the volume. It works most of the time :-)
Some of our video chunks have jumps in them, due to problems in downloading the original WMV stream. The timestamping interface has a link for people to let us know of such problems, so
