mySociety blog » Developers

“People often say they could run Britain better than the political parties. A web-based revolution may give them the chance”

Nice article in the Sunday Times today mentioning lots of our sites and others.

It’s the first full working day for the new facility to annotate Freedom of Information (FOI) requests on WhatDoTheyKnow, and people have been hard at it.

Mr Ormerod points out that private information isn’t necessarily so private if someone has died, so perhaps the exemption the MOD used shouldn’t apply.

Trevor R Nunn has posted three annotations (e.g. this one) to show that his three FOI requests are being treated as one. The annotations facility is great for handling edge cases like this, which don’t happen often enough to be worth explicitly adding to the code, but need some mention.

And finally Edward Betts has processed the list of post boxes retrieved by FOI into a more structured data format, and posted up a link to it. Exactly the kind of collaboration I love to see!

And that’s just this morning!

One of the special pieces of magic in TheyWorkForYou is its email alerts, sending you mail whenever an MP says a word you care about in Parliament. Lots of sites these days have RSS, and lots have search, but surprisingly few offer search based email alerts. My Mum trades shares on the Internet, setting it to automatically buy and sell at threshold values. But she doesn’t have an RSS reader. So, it’s important to have email alerts.

So naturally, when we made WhatDoTheyKnow, search and search based email alerts were pretty high up the list, to help people find new, interesting Freedom of Information requests. To implement this, I started out using acts_as_solr, which is a Ruby on Rails plugin for Solr, which is a REST based layer on top of the search engine Lucene.

I found acts_as_solr all just that bit too complicated. Particularly, when a feature (such as spelling correction) was missing, there were too many layers and too much XML for me to work out how to fix it. And I had lots of nasty code to make indexing offline - something I needed, as I want to safely store emails when they arrive, but then do the risky indexing of PDFs and Word documents later.

The last straw was when I found that acts_as_solr didn’t have collapsing (analogous to GROUP BY in SQL). So I decided to bite the bullet and implement my own acts_as_xapian. Luckily there were already Xapian Ruby bindings, and also the fabulous Xapian email list to help me out, and it only took a day or two to write it and deploy it on the live site.

If you’re using Rails and need full text search, I recommend you have a look at acts_as_xapian. It’s easy to use, and has a diverse set of features. You can watch a video of me talking about WhatDoTheyKnow and acts_as_xapian at the London Ruby User Group, last Monday.

TheyWorkForYou now finds whenever an old version of Hansard is referenced (which they do by date and column number, e.g. Official Report, 29 February 2008, column 1425) and turns the citation into a link to a search for the speeches in that column on that date. This only really became feasible when we moved server, upgraded Xapian, and added date and column number metadata (among others), allowing much more advanced and focussed searching - the advanced search form gives some ideas. Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech, much like our video matching (nearly 80% of our archive done!). :)

Kudos to Google and Yahoo! for spotting this change within a couple of days, as they’re now so busy crawling everything for changes that they’re slowing the whole website down… ;-)

If you enter your postcode on TheyWorkForYou and it’s Scottish or Northern Irish, you’re now presented with your MSPs and MLAs as well as your MP, which makes sense given the site covers their Parliament and Assembly respectively. :-) You also get an extra tab in the navigation linking through to Your MSPs or MLAs. In order to do this, I needed a quick way of determining if a postcode was Northern Irish or Scottish. Northern Ireland was easy, as all postcodes there begin with BT. I assumed Scotland was also easy, which turned out to be true apart from the TD postcode area that straddled the border like a mail-sorting Niagara Falls. After some very dull investigation, I eventually worked out that e.g. most of TD15 is in England, but (amongst others) TD15 1X* is in Scotland, except for TD15 1XX which is apparently back in England. The final result was the postcode_is_scottish() function in postcode.inc, which (hopefully) correctly determines if a given postcode is Scottish or not - perhaps someone else will find it useful.

Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?

I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. :-)

Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.

The highlighting works the other way - as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! :) ).

Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.

Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse :-)

Previous articles

1. The Flash player
2. Seeking
3. Highlighting the current speech

Our video is streamed via progressive HTTP, using lighttpd and mod_flv_streaming. This works by having keyframe metadata at the start of the FLV (Flash video) file (we add ours using yamdi as that doesn’t load the whole file into memory first), which maps times within the video to byte positions within the file. When someone drags the position slider, or presses a skip button, the player actually changes the source of the video to something like file.flv?start=<byte position> which starts a new download from that point in the video. This means you can seek to parts of the video not yet downloaded, which is definitely a required feature.

The video is split up into programme chunks, according to BBC Parliament’s schedule, so each Oral Questions will (approximately) be its own video chunk, and the main debates will be a couple of chunks. By default, the video player will show a screengrab from the start of the video, as that’s all that’s available when it first loads (you have to load the start of the FLV file to fetch the keyframe metadata in order to move anywhere else :) ). I wanted the player to show a relevant screengrab before you hit Play, so came up with the slightly messy workaround of setting the volume to 0, seeking and playing the video for under a second in order to start it from the new point and show the video, then stopping it and resetting the volume. It works most of the time :-)

Some of our video chunks have jumps in them, due to problems in downloading the original WMV stream. The timestamping interface has a link for people to let us know of such problems, so that we can mark the relevant speeches as missing video and not have them be offered to future timestampers. One valiant volunteer, Tim, let us know about two such videos, but with the added oddity that if you let them play, they would happily carry on past their “end” point, but this made timestamping those speeches quite difficult.

I started investigating, firstly noting that both videos should have been 6 hours long, but were both listed as 1:20:24, which I thought was a bit of an odd coincidence. After reading the FLV file specification, it turned out that 32-bit millisecond timestamps in FLV are split into two - first the low 24 bits, then the high 8 bits. 2^24 = 16,777,216, which in milliseconds is 4 hours, 39 minutes, 37 seconds, which is pretty much exactly what the two videos’ durations were short by! All the timestamps in our FLV files were not setting the high byte, so after 4:39:37, they were wrapping round to 0 (and thus 6 hours became 1:20:24ish).

Our video processing consists of four major steps - the downloading script uses ffmpeg to convert each 75 minute chunk from WMV to MPEG; then nightly processing uses ffmpeg again to convert the right bits of these MPEG files to FLV, mencoder to join the relevant FLV files into one FLV chunk, then yamdi to add the metadata. My first try at a solution was to alter yamdi to increment the high byte itself, which fixed the duration display and let you seek to high times, but when you tried to go to e.g. 5 hours, the video started playing from the right point but the video thought it was playing from 20 minutes in. This would obviously confuse timestamping!

As the FLV files produced by ffmpeg were all under 75 minutes long, they couldn’t have the problem. It turned out we were running an old version of mencoder, and updating that and converting all our long video files fixed the problem. Phew :-)

Join us later today for my third short technical talk on TheyWorkForYou video, where I’ll explain how our Flash application talks to the HTML and vice-versa to enable the “Watch this” and highlighting of speeches.

1. The Flash player
2. Seeking
3. Highlighting the current speech

TheyWorkForYou video timestamping has been launched, over 40% of available speeches have already been timestamped, and (hopefully) all major bugs have been fixed, so I can now take a short breather and write this short series of more technical posts, looking at how the front end bits I wrote work and hang together.

Let’s start with the most obvious feature of video timestamping - the video player itself. :) mySociety is an open-source shop, so it was great to discover that (nearly all of) Adobe Flex is available under the Mozilla Public Licence. This meant I could simply download the compiler and libraries, write some code and compile it into a working SWF Flash file without any worries (and you can do the same!).

Writing a Flex program is split into three main areas - MXML that lays out your application, defines any web services you’re using and so on; CSS to define the style of the various components; and ActionScript to deal with things like events, or talking to the JavaScript in the parent HTML. My code is probably quite shoddy in a number of places - it’s my first application in Flex :-) - but it’s all available to view if you want to take a peek, and it’s obviously running on the live TheyWorkForYou site.

To put a video component in the player is no harder than including an <mx:VideoDisplay> element - set the source of that, and you have yourself a video player, no worrying about stream type, bandwidth detection, or anything else. :) You can then use a very useful feature called data binding to make lots of things trivial - for example, I simply set the value of a horizontal slider to be the current playing time of the video, and the slider is then automatically in the right place at all times. On the downside, VideoDisplay does appear to have a number of minor bugs (the most obvious one being where seeking can cause the video to become unresponsive and you have to refresh the page; it’s more than possible it’s a bug in my code, of course, but there are a couple of related bugs in Adobe’s bug tracker).

As well as the buttons, sliders and the video itself, the current MXML contains two fades (one to fade in the hover controls, one to fade them out), one time formatter (to format the display of the running time and duration), and three web services (to submit a timestamp result, delete a mistaken timestamp, and fetch an array of all existing timestamps for the current debate). These are all called from various places within the ActionScript when certain events happen (e.g. the Now button or the Oops button is clicked).

Compiling is a simple matter of running mxmlc on the mxml file, and out pops a SWF file. It’s all straightforward, although a bit awkward at first working again with a strongly-typed, compiled language after a long time with less strict ones :-) The documentation is good, but it can be hard to find - googling for [flex3 VideoDisplay] and the like has been quite common over the past few weeks.

Tomorrow I will talk about moving around within the videos and some bugs thrown up there, and then how the front end communicates with the video in order to highlight the currently playing speech - for example, have a look at last week’s Prime Minister’s Questions.

1. The Flash player
2. Seeking
3. Highlighting the current speech

I only met Harry Metcalfe a few weeks ago, when he was volunteering for the Open Rights Groups.

Since then he’s dazzled us with his completely single handed production of TellThemWhatYouThink, a site which draws most central government consultations into one site. The main citizen benefits from this is that you can get email or RSS alerts when a department decides to hold a consultation on an issue that you care about. As per usual it’s also starting with horrid nonstandard data in a zillion formats and turning it into nice structured data for everyone else to play with too.

It’s not a mySociety project, he’s just using our friends and family server, but he’s made it look more like a mySociety site than even we could manage. Kudos Harry!

PS Harry, along with both of the last two new volunteers to do major pieces of coding for mySociety, is in the third year of his PhD. I think I can see a pattern emerging…

Chris Lightoot died a year ago today (or yesterday, by a few minutes).

I’m just sitting here reading the very first emails I ever got from him, back in 2003. Within the first few mails he’s invented and hacked up the idea that is now Richard Pope’s PlanningAlerts.com, coded and developed the idea that persuaded YouGov to donate vast amounts of free polling data to form PoliticalSurvey2005.com (a wider understanding of which would greatly help in the US election if the methodology was only applied there) and in this post he’s foreseen the Google maps mashup craze and offered it on a plate to the Ordnance Survey to pioneer, two years before Google started.

The invention and brilliance comes so thick and fast reading these mails that I now realise that I’d persuaded myself over the year that I’d mis-remembered quite how insanely creative he was, trying to correct for rose-tinted lenses. But he was a proper, bona fide, no-holds-barred cantankerous genius. Most days I think about Chris at least once: I try to make sure we live up to his standards (he wouldn’t have tolerated my use of ‘But’ at the start of the last sentence, for example). Reading these mails tonight drives home the scale of what we all lost, amongst our friends, on the Internet and in society at large. It aches to contemplate.

You may remember that back in 2006 mySociety published some maps showing how long it took to commute places via public transport.

We’ve just made some more which have some lovely new features we reckon you’ll probably like a lot.

If you’d like to see more maps like this in your area, please ask your local transport authority to get in touch with us, or nudge these people :)

PS As always, Francis Irving remains a genius.

This is the second in a short series of interviews with people building and running some of the most exciting internet and democracy projects in Europe.

Adrian Moraru from the IPP in Romania set the BerlinInAugust unconference abuzz with occasional gasps at the uncompromising relentlessness of their approach, which included suing to obtain the mobile phone numbers of all the politicians with handsets provided on the public purse. Below you can hopefully see why they got people excited…

——-

What is the organisation you work for?

Our organization, the Institute for Public Policy, is an independent think tank based in Bucharest. We have a permanent staff of 12 people plus a pool of external experts and part time collaborators that we work with on project based relationship. This external group may number as many as 50 in a year and range from former public officials, to politicians, independent experts, journalists, students, young researchers and academics. We work in numerous areas but we specialise in local government, parliament and the ministries.

What is the main purpose of the site(s) that you run?

The main purpose is to give people with a specialised, professional interest in politics an easy way to access facts and statistics about the way MPs are working & voting, as well providing information for the general public.

Can you tell us about some of the unusual ways you ensure that your vote attendance information is accurate?

Sure, it’s easy. In our parliament the attendance is recorded based on a attendance register at the entrance of the plenary hall. However, it is common for some MPs to sign on behalf of their colleagues and/or friends. So in order to expose the size of this phenomenon we decided to keep track of MP’s real attendance in a more accurate way.

Some politicians have legitimate exemptions, which we record, be we also wanted an accurate record of how many of them are present when votes happen. So lets say you have 20 votes in a day. If the name of the MP Mr. X shows up only in 14 of them then he is present only 70%. Furthermore, if, say, only 204 voted out of a possible 322, we deduce from our database the 118 who didn’t show up, and add that to their record.

We have used video cameras from time to time in order to combat the practice of multiple voting. This is happens because of our voting system in Parliament is based on electronic voting stations placed on your bench were MPs identify themselves with a smart card (aka voting card) before pushing a button corresponding with their voting choice.

Politicians have 10 seconds to do so once the vote is initiated. Some MPs use these 10 seconds to vote once with their own cards and then once with the cards of colleagues who are, for example, out at lunch. This is a widespread practice.

We have the plenary sessions broadcasted live and also available recorded on
the Parliament website. We suggested that the during voting that a camera record the activity int he whole chamber. We therefore exposed a few cases of this multiple voting, although not much has happened as a result yet.

You also collect information about politician’s travel. How did you get that? What does it tell your users?

We get it through our freedom of information laws. But is not that easy to get hold of. Sometimes we even have to go on court to get it, and sometimes even when we do it comes on paper, not in electronic formats, which is obviously harder to re-use.

What it shows is where an MP went, when, why, how much it cost, how long they stayed and so on. From this we can help people establish whether they think it was strictly necessary for an MP to visit French Guineau to see the launching of an Ariane V rocket, and we can provide the most popular country destination by political parties.

Do you ever face claims that the effects you have on politicians aren’t entirely positive? If so, how do you respond?

Well this is not a consolidated democracy, you know. MPs are not as nice as yours. So, yes large parts of the databases hurts a lot of them a great deal. Let’s just say we are not scared. But on the other hand we strive to get the best data and to present it in a non aggressive, non biased way using the best algorithms.

You are good at using the law to obtain information. Can you tell us a bit about your approach, and what information you’ve obtained through the courts?

This is a very distinct topic. We always ask for information via our Freedom of Information act, using a special format of letter which cannot be completely ignored. We have lawyers following the flow of requests together with an office manager and we sue every time we do not get an answer, have our request denied or find that information we’ve been provided with is incomplete.

We ask for a lot. A lot! Usually we fight for data that exposes bad practices and most of the things involving expenses or money. It is here where there is a lot to hurt bad politicians by exposing how unwisely some of them are spending the money.

What other projects around the world excite you the most, and why?

Tough one. None. I like opensecrets.org and votesmart.org but that’s it. I do not believe in moving participatory democracy online in our life time. Instead I think we should be looking for ways to open up government and make it more transparent using the internet. In my opinion we are not even at 10% of the way to what we can ultimately do, either in Romania or elsewhere. We can think also about real interactivity in the future.

What’s next for you and IPP on the Internet?

Who knows?

—

That’s it for the moment. Please post any questions for Adrian in the comments below, and I’ll see if I can update this accordingly.

At mySociety we’re always very lucky to meet and spend time with some extremely diverse and impressive people.

We thought it would be great to share a bit of that good fortune by holding some talks from some of our favourite thinkers, and to have an excuse to meet more people in the wider mySociety community face to face.

To that end, we’re holding four talks in London this autumn (location TBD but almost certainly a centralish pub). Each link below goes to an Upcoming page where you can sign up to let us keep track of numbers and how big a venue we need.

4/10/2007 - Stefan Magdalinski, net-political troublemaker extraordinaire

1/11/2007 - Steve Coast, founder of Open Street Map

CANCELLED 29/11/2007 - Jason Kitcat, e-voting expert

12/12/2007 - Peter Wainman, IT-specialist solicitor and blogger

We look foward to seeing you there.

Hello, all. Just wanted to let you know about another podcast interview I did about PledgeBank…this time on the 501c3Cast. This podcast is geared toward folks in the nonprofit/philanthropic world (the tax code classification for many nonprofits in the U.S. is “501(c)3″) to spread the word about new tools, ideas, and conversations in the sector. You can find the podcast here (though, be forewarned, it’s a bit long…).

A few weeks ago mySociety and Politik Digital held a small unconference in Berlin. The idea was to get together some of the best practioners building and running democracy websites across Europe, regardless of their size or status.

I’ll try to write this up more fully soon, but for the moment I wanted to share some email interviews I did with some of the participants after the event. The first is with Guglielmo Celata from the Italian group D.E.P.P. We first came across them a couple of years ago when they borrowed some code from PublicWhip.org.uk (the independent volunteer vote analysis project run by Julian Todd and mySociety senior developer Francis Irving) for their website OpenPolis.

Anyway, enough for the context - D.E.P.P have some great, boundary-pushing work coming up and I thought people in the English speaking community would want to know.

—

What is the organisation you work for?

The name of the Association is D.E.P.P., that stands for Electronic Democracy and Public Partecipation.
It’s a relatively small group of people (four) who work on e-partecipation projects with local administrations (the municipality of Rome and the Regione Lazio, for examples). We also have an self-financed project, named Openpolis, to map politicians, their charges, their declarations, both at a national and at a local level. More on this later.

What is the main purpose of the site(s) that you run?

We have a project named eDem 1.0 which has been so far installed twice: municipiopartecipato.it focuses in enabling e-participation of local communities on the “participatory budget”; and edem-regione on the budget of the Regione Lazio (the link points to an alpha version).

I think the participatory budget for the local community is far more interesting. The site shows a list of issues categorized by theme and territory. Registered users can vote up issues and make them emerge as important. Issues are created by the users. Users can also create proposals related to issues and vote them. The integration with Google Maps, allows user to see how issues and proposals distribute in their territory; it makes the user interface immediate (and of course makes the site sooo stylish).

The proposals emerging as the most voted are approved and follow a workflow to be actually financed and implemented.

Online activities and offline physical assemblies (which exist), are linked together by a group of paid people, called enablers. They take care of moderating both offline and online activities, too.

The other project has almost the same features, but applied to the budget document of the Regione Lazio. Of course, the issues here are not created by the citizens, being the chapters (or sections) of the official budget document. The citizens can create and rate proposals, but such proposals are never going to be implemented.

This happens a lot, administrators are intersted in e-participation projects, but they want to reduce the possibility of issues emerging directly from citizens, and€ of course they try to change the nature of the project from a participative one, into a consultative one. A kind of Poll 2.0, if one wants to be cynical.

Can you tell us about your next site, the one you showed us in Berlin?

Openpolis is a project to gather informations on our political class and make them transparent. How they vote once elected, what laws they propose, their charges in institution, political parties and private organisations, public declarations, financial interests, judicial positions etc. The aim of the project is to revive the bond between the citizens and their representatives. We would like to give individuals or organized group of citizens, a set of tools to enable them to perform lobbying activities.

We want to work both at a national level and at a very local level, and to do this we plan to allow users to create part of the content on the site, and hope this way to create communities, wiki-style.
However, the site is not a wiki, since content has to be well-structured; we want to export statistics and make analysis on data added by users.

You are planning to combine information gathered from formal sources, and submitted by users. Can you tell us where you’re getting the formal information from, and how you are going to handle the information submitted by users?

We have different levels, and correspondingly different sources. At a national level, we are harvesting the official web sites of the Camera and Senato (the two houses of national representatives) and the web site of European Parliament. At local levels we rely on official biographical data from the Ministero degli Interni (Interior Ministry). We double check politician’s data for the 20 major cities in italy, but of course can’t possibly dream of doing that for the 109 provinces and 8100 municipalities.

For data on charges, declarations, financial interests and judicial positions, and for a complete double check on details and biographical data, we plan to leverage the community of users. The more users, the more data and verification.

Of course, data inserted by the users must be always connected to sources (i.e a web link, a reference to a book, an article in a newspaper, or a radio or television program). Data will be verified by moderators, and the community of moderators will grow on trust basis (using a karma-based system, so that when a subscribed user reach a certain treshold of trust, he is proposed as moderator to the board of administrators). We all know that this part is a real challenge and that handling a community online is a daunting task, but, hey, let’s try.

Users can be banned and content can be censored (after publishing), but any banning or censorship will be performed transparently, so that anyone, in any moment will be able to know the reason why a user was banned.

Do you ever face claims that the effects you have on politicians aren’t entirely positive? If so, how do you respond?

We actually have not yet started, but we do plan on receiving a lot of such claims. Of course we are trying to create something that the politicians should use, as well, so the most interested and active users should be the politician themselves.

Are there any other features of your site that you think are unusual or unique?

We plan to release an API, in order to make integration of our data and analysis possible directly from other web applications. Starting from RSS feed, to a proper API, it should be possible to integrate pieces of our applications directly into people’s blog or other similar applications.

What other projects around the world excite you the most, and why?

Well, of course the TheyWorkForYou project was a real kick off, we just thought: “wow, we have to do that here in Italy!” Then I really appreciate the work at GovTrack.us, especially from the technical standpoint, for the innovative way of using RDF and the Semantic Web approach.
Here in Italy, a project I forgot to mention in Berlin is: http://fainotizia.radioradicale.it.

FaiNotizia means Make Your Own News, it’s a project by Radio Radicale, an historical radio broadcast of the Italian Radical Party. It provides one of the first citizens journalism website in Italy and we plan to integrate with them in the future.

Do you use the law to help you get information? If so, how have you gone about it, and what have you obtained?

We haven’t so far, every information that we gathered was publicly available, we just wrote tons of parser code.

We plan to push the release of data on financial interests and judicial positions, though. Those data are public, but poorly accessible (no electronic format, no scanning, phtos or copies possible). This will require some legal actions or some fantasy to get them. We’ll see.

——

So there we are. If you’ve any further questions or clarifications, just post a comment here and I’ll update this post with Guglielmo’s help.

I’ve been working on PledgeBank quite a bit recently. As well as adding survey emails asking whether signers have done their pledge, and a feature for people to contact a pledge’s creator, I’ve been fixing numerous bugs that have sprung up along the way. For starters, people on the Isle of Man and the Channel Islands now get a much more helpful error if they try and enter their postcode anywhere on the site, rather than the confusing postcode not recognised they were getting previously.

Other errors I found turned out not to be with our code. The PledgeBank test suite (that we run before deploying the site to check it all still works) was throwing lots of warnings about “Parsing of undecoded UTF-8 will give garbage” when it got to the testing of our other language pages. Our code wasn’t doing anything special, and there were multiple places the warning came from - upgrading our libwww-perl removed one, and I’ve submitted bug reports to CPAN for the rest (having patched our copies locally - hooray for open source).

The Perl warnings were at least understandable, though. While tracking down why the site was having trouble sending a couple of emails, I discovered that we had a helper function splitting very long words up to help with word-wrapping - which when applied to some Chinese text was cutting a UTF-8 multibyte character in two and invalidating the text. No problem, I think, I simply have to add the “/u” modifier to PHP’s regular expression so that it matches characters and not bytes. This didn’t work, and after much playing had to submit a bug report to PHP - apparently in PHP “non-space character followed by non-space character” isn’t the same as “two non-space characters in a row”…

The “um’s” and “uh’s” on the interview are embarrassing, to say the least, but PledgeBank just got its first podcast coverage. Check out my NetSquared interview here. I’m hoping this gets us a bit of exposure in the U.S.

And I’ll do better next time with all the incidental sounds… :-)

I’m still busy beavering away at the Facebook / PledgeBank integration. It all works now, but will take a bit more polishing to get just right. Matthew is, I think adding surveys to PledgeBank. So it finds out later if people have or have not done their pledge. Or is he updating to a new version of BoundaryLine at the moment, so our postcode lookup on WriteToThem and everywhere else gets better? Hard to keep track when he does so much at once.

Keith is upgrading our internal documentation, so new people at mySociety can learn how to keep things going. Heather is stalking all of America, finding people to use and promote PledgeBank. Tom is on a much deserved holiday, after seemingly a zillion meetings per day for months.

There’s lots of ongoing maintenance for all our sites. We’re lucky that large chunks of our customer support email are done by volunteers (thanks Anna, Louise, Tim and Tomski/James) and by Debbi (yay Debbi!). Much of this is routine - changing pledge text, updating council email addresses, giving MPs posting links for HearFromYourMP, putting new MP photos up on TheyWorkForYou etc. A lot of it is unique - handling new translations, answering questions from MPs and Lords about their voting record. I’ll let the others give some more examples of the kind of thing we answer.

Speaking of which, do you know any good web developers who would like to work for mySociety? If so, put them in touch.

A couple of weekends ago when it was still sunny, a group of 20 or so mySociety developers, trustees, and volunteers went away together to a farmhouse in Warwickshire (thanks to everyone especially Tim Morley and Tom Loosemore for their help). This was not only an opportunity for people like me to finally meet all those I’ve been emailing for months if not years, but also to discuss various things about mySociety.

It was an excellent weekend – we learnt lots of new things, like how UKCOD and mySociety have developed over the last 10 years(!), Rob’s excellent TheyWorkForYou.co.nz, and Richard’s PlanningAlerts.com. We also discussed what mySociety’s core aims and principles should be - here are some thoughts:

Aims

1) Build sites that build civic value, using the internet natively as a medium and that scale elegantly
2) Build sites that are easy to use for those without experience
3) Build sites that are focused on meeting one simple need
4) mySociety should become self-sustaining, financially and staff-wise

Principles for developing mySociety services and products

1) Build things that meet people’s needs, and that they can’t express yet
2) Do one thing really, really, really well (brand on one thing)
3) Treat the entire world as a creative canvas (plug-ins, widgets, etc.)
4) Do not attempt to do everything yourself; use other people’s content
5) Back success, get rid of failure
6) The web is a conversation; join in
7) Any website is only as good as its worst page
8) Make sure your content can be linked to forever
9) Your granny will never use Second Life
10) Maximize roots to content; optimize your site to run high on Google
11) One size does not fit all – users should know they’re on your site
12) Accessibility is not an optional extra
13) Let people paste their content on their own sites
14) Link to discussions on the web, not necessarily host them
15) Personalization should be unobtrusive and coherent

And some more thoughts:

1) Only use html and CSS
2) Ensure accessibility
3) Ensure usability
4) Make it work across the spectrum – screen readers to mobile phones
5) Build things that don’t require key “stick in the muds�? to do anything
6) Don’t ever build anything that might become an empty cupboard, or if you do, make it very easy for people to fill that cupboard.
7) Don’t rely on network effect, but do seek out network effect
8) Engineer serendipity
9) Help users connect with other users
10) Set the bar high for privacy

However, we still have some challenges ahead: we need to think about how to make the most of our existing sites, and had a very good session on how to improve PledgeBank’s outreach; we also need to engage better with both our current and potential volunteers; and, of course, move towards becoming financially self-sustaining to keep up our good work without always relying on grants.

And finally, because we like tangible actions, we launched the UKCOD site on Saturday night too.

So what happens next? Well some of the things have already happened, like Matthew and others transforming FixMyStreet and Francis developing some widgets. We’ll also see what the new PM wants to do with e-petitions (keep it, apparently, which is good), and how the e-democracy landscape is changing. And, soon we hope, we’ll give this site a bit of a facelift.

But we still have much to do, and the weekend wasn’t long enough to get through everything we wanted. So here are a few more things to chew over.

• Have you wanted to volunteer for mySociety but found it difficult, e.g. the tasks were too technical, or didn’t really know where to start?
• Is there something you want to know about mySociety, or our sites, but not been able to find?
• How can we improve our existing sites?
• Do you know any nice millionaires with some spare cash burning a hole in their pockets, and they just don’t know what to do with it?

Let us know why and we’ll try to do something about it.

The last batch of councillor data arrived this morning, thanks very much to GovEval, so pratically every council (bar all Scottish councils and the 17 English councils that had boundary changes, for which we’re just awaiting a new version of Boundary-Line, Ordnance Survey’s product that says where constituency boundaries are) should now be contactable again through WriteToThem.

I’ve been doing some work on helping people promote our sites and the things on them - spurred by a request from a user who was holding a street party, we’ve made some posters and flyers for FixMyStreet (thanks to volunteer Ayesha Garrett for designing them), and we’ve started providing online tools to promote pledges on PledgeBank, including an up-to-date status image or text of a particular pledge, alongside the established, more offline, flyers.