Just a quick post to keep those of of you interested in mySociety in the loop with our activities at the moment.

New Things You Can Use Now

1. Email or RSS alerts when people report problems in your ward or your council via FixMyStreet. Ideal for councillors, people on resident’s associations, or anyone just concerned about what’s breaking and being fixed in the area right near their home.

Have a go – it’s ace when the mail comes dropping in from just down your road.

2. The Queen on TheyWorkForYou

Is this the first monarch with her own RSS feed? Would anyone really care if she was?

New Projects Coming Up

We have three major projects under way at the moment, and unusually
only two of them involve us building websites.

1. The Freedom of Information Filer and Archive website is under construction. Aiming to make it easier to make freedom of information requests, and easier for people to find what other people have found out, this is being build mainly by Francis. We’re having lots of discussions about design and features right now, and if you have anything to contribute please either get in touch or leave your ideas on the wiki page.

2. Local Email Groups Near You – an attempt to record the location of hyper local email groups and local forums and websites and to share that information on lots of other sites. Why go blindly hunting for advice on a plumber if there’s already an email list that covers your street? This is going to be a rare international project for us, so if you’re outside the UK and interested in community Internet usage, please get in touch.

3. The 90 Day Project – mySociety’s first lobbying exercise, trying to encourage parliament to take some steps to improve the way it publishes information, and to improve the tools that MPs have to handle mail from their constituents.

There’s lots and lots more too, but we can’t blow all our surprises in one go, can we?

A couple of weekends ago when it was still sunny, a group of 20 or so mySociety developers, trustees, and volunteers went away together to a farmhouse in Warwickshire (thanks to everyone especially Tim Morley and Tom Loosemore for their help). This was not only an opportunity for people like me to finally meet all those I’ve been emailing for months if not years, but also to discuss various things about mySociety.

It was an excellent weekend – we learnt lots of new things, like how UKCOD and mySociety have developed over the last 10 years(!), Rob’s excellent NZ TheyWorkForYou, and Richard’s PlanningAlerts.com. We also discussed what mySociety’s core aims and principles should be – here are some thoughts:

Aims

1) Build sites that build civic value, using the internet natively as a medium and that scale elegantly
2) Build sites that are easy to use for those without experience
3) Build sites that are focused on meeting one simple need
4) mySociety should become self-sustaining, financially and staff-wise

Principles for developing mySociety services and products

1) Build things that meet people’s needs, and that they can’t express yet
2) Do one thing really, really, really well (brand on one thing)
3) Treat the entire world as a creative canvas (plug-ins, widgets, etc.)
4) Do not attempt to do everything yourself; use other people’s content
5) Back success, get rid of failure
6) The web is a conversation; join in
7) Any website is only as good as its worst page
8) Make sure your content can be linked to forever
9) Your granny will never use Second Life
10) Maximize roots to content; optimize your site to run high on Google
11) One size does not fit all – users should know they’re on your site
12) Accessibility is not an optional extra
13) Let people paste their content on their own sites
14) Link to discussions on the web, not necessarily host them
15) Personalization should be unobtrusive and coherent

And some more thoughts:

1) Only use html and CSS
2) Ensure accessibility
3) Ensure usability
4) Make it work across the spectrum – screen readers to mobile phones
5) Build things that don’t require key “stick in the muds?? to do anything
6) Don’t ever build anything that might become an empty cupboard, or if you do, make it very easy for people to fill that cupboard.
7) Don’t rely on network effect, but do seek out network effect
8) Engineer serendipity
9) Help users connect with other users
10) Set the bar high for privacy

However, we still have some challenges ahead: we need to think about how to make the most of our existing sites, and had a very good session on how to improve PledgeBank’s outreach; we also need to engage better with both our current and potential volunteers; and, of course, move towards becoming financially self-sustaining to keep up our good work without always relying on grants.

And finally, because we like tangible actions, we launched the UKCOD site on Saturday night too.

So what happens next? Well some of the things have already happened, like Matthew and others transforming FixMyStreet and Francis developing some widgets. We’ll also see what the new PM wants to do with e-petitions (keep it, apparently, which is good), and how the e-democracy landscape is changing. And, soon we hope, we’ll give this site a bit of a facelift.

But we still have much to do, and the weekend wasn’t long enough to get through everything we wanted. So here are a few more things to chew over.

• Have you wanted to volunteer for mySociety but found it difficult, e.g. the tasks were too technical, or didn’t really know where to start?
• Is there something you want to know about mySociety, or our sites, but not been able to find?
• How can we improve our existing sites?
• Do you know any nice millionaires with some spare cash burning a hole in their pockets, and they just don’t know what to do with it?

Let us know why and we’ll try to do something about it.

mySociety is pleased to announce the winner of our 2006 call for proposals, plus our thoughts on the best runners up, and various other lessons.

Winner

Our winner, and the next major site we are planning to build is the Freedom of Information Filer and Archive; a searchable, readable, googlable user-created archive of FOI requests and their responses. Think of a combined TheyWorkForYou and WriteToThem.com for FOI requests and their responses, and you’ll have our vision.

This idea was actually submitted twice, once by Phil Rodgers and once by Francis Irving (a mySociety coder).

We believe the idea is especially powerful in a form extended somewhat beyond that submitted. We think that the best way to build a top quality archive is to simultaniously build the best possible “File an FOI request” tool, and then publish both the requests and the responses made through it in the archive. From the private desire to easily file FOI requests we hope that we can generate the public benefit of an easy to use archive.

We asked our community of users and friends to list their top three projects, and the FOIFA was named more often than any other single project as the winner: 9 out of a total of 22 people who left a comment expressing their preferences. The core team, core volunteers and trustees agreed with the users, and so we have a winner.

Logistics

mySociety will start building the system in early 2007. We will try to fund it in two ways. First, we will approach donors, most probably foundations, to see if they are interested in supporting it. Secondly, we will see if we can set aside some surplus from contract work, such as branded versions of the other sites. And lastly, we’ll work with any volunteers who are willing to dig in.

Our initial estimate is that the site will take 120 full time developer days to design, build and launch to beta, for a total cost of about £25,000 including servers, management time, gathering of contact details, buying of sweets, motivational calendars and so on. The cost of running it thereafter are hard to gauge at this point, and will depend on usage patterns and the final spec we settle on.

In the run up to building and launching it we’ll gladly talk to anyone who wants to be involved, including public sector agencies who we hope might use this system to publish responses to requests made via other channels.

Runner Ups

In no particular order, these are some of the other ideas that had some legs. We’re putting them here to suggest to the world that there might be something well worth exploring here.

1. A to B travel, by Murray, is a sort of collaborative journey planner, where people share information on journeys that they’ve made. Unfortunately, too much of this site is already done by the big and expensively run government site Transportdirect.info , but it has nice ideas that are worth someone doing. In particular the idea of local knowledge and general comments on different journeys is an excellent, and Seat61.com shows that there is some considerable appetite for journeys explained in a human form. Often I don’t want the fastest journey from A to B, I might want the best view, or the most pleasant form of transport, or the one that can be broken somewhere notable.

We’ve also come up with a feature that this site could add. It is the idea of registering to express an interest not in a specific journey, but in a general journey: “I go from Manchester to London a few times a year, and I might want to share a car in future”.

2. Get Out! by Mary Reid. This proposal was about building a site that would contain a user build database of places to go in the UK that would contain something nice and easy to do if you had an hour or two to spare and wanted to get out of the house.

We’ve felt for a while that there is a great problem with knowledge of local activities being hopelessly fractured across the UK Internet, spread across a million different sites and so worth much less than the sum of its parts. A site that could become a reference place to store interesting things to see, and a reference place to find them could be excellent indeed. Maybe a rebuild and extension of our little back o’ the envelope site YourHistoryHere.com?

3. Write To Your Newspaper by Francis Irving (again)

This proposal was about a site that makes it much easier to write to local newspapers. It is undoubtedly a good thing, but it simply didn’t beat the FOI archive because we felt the demand and public benefit just wasn’t as great as for FOIFA. One of mySociety’s volunteers has actually already written some code in this area, and we certainly think it should go further.

4. TheyWantToWorkForYou by Seb Bacon – a site where people could find out prospective politicians rather than current ones was voted for by a few people. We think it would be a good idea for such a system to exist, but the scale problem is enormous. With 20,000 current councillors, just imagine how many candidates there are at each election, and the massive problem of trying to get them to give structured views. What is missing here really is a strong motivation for candidates to go to a certain site and enter info themselves – it just doesn’t exist, and probably couldn’t without the major backing of someone like a big newspaper,or the BBC. NB, we also feel strongly that such a site would have to be permanent, and not just run at elections.

Lessons Learned from Running the Call

Just some thoughts about the process, really here for anyone else who might be planning to run a call like this and who stumbles across us via Google.

1. First time round, in 2003, the call for proposals got 250+ proposals, whereas this time it had more like 100, even though mySociety has moved from completely unknown to somewhat better known. Clearly despite BBC and Guardian coverage, we did something not as well this time. This might simply have not been hammering every list and person we could with personally crafted emails, or it could have just been blind chance.

2. We should have determined and published the judging process before the call for proposals was put out. Nobody seems to have been especially upset by our drawn out and ill-planned selection process, but it would have meant we would have made our decision much more quickly.

3. We should have set a timetable for all parts of the process.

4. We should have made some sort of web based voting gadget to engage people slightly more with the deciding process (despite knowing that online voting is mostly bunk, of course).

5. We could have made a shortlist and then asked the authors to do more work in polishing up their ideas.

If you’ve any further questions about the call for proposals, or the Freedom of Information Filer and Archive get in touch with us at hello@mysociety.org

I mentioned the other day “a syndicated version of PledgeBank for someone’s large global warming campaign later in the year”. In the early hours of this morning, Global Cool soft launched.

The idea is that people can use the Global Cool website as a central place to manage reducing or offsetting their carbon emissions, and recording and competing for how much they have done so. There’ll be tools like carbon counters, systems to track when you change electricity company, and so on. PledgeBank comes into this as another set of actions Global Cool people can take, both making and signing pledges which reduce their CO2 footprint.

Global Cool Foundation UK, the new charity behind it, only got their funding recently, and so as Gavin explains have had to build the site in phenomenal time. We got login intergration with PledgeBank working in the early hours of this morning. It works, but there are some missing features, and it still needs lots of polishing. The integration is needed so later you can be credited for signing and doing a pledge on the main part of the Global Cool site.

The “microsites” file in the PledgeBank source code is even more complicated now, with authentication hooks. Basically, the PledgeBank part of Global Cool runs on a subdomain of their domain, being pledge.global-cool.com. This is so it can read the Global Cool cookies for authentication. The pledges themselves are in the same database, so they can be shared with the rest of the site later.

We can happily syndicate or theme PledgeBank in all sorts of ways for your site. We charge for this via our trading arm. Any profit (not really any yet!) going back into running the main PledgeBank service for free, and to run mySociety.

The man responsible for the mySociety sites’ distinctive and ultra-usable look has just dumped his day job to become a full time member of the mySociety core development team. He’s going to be doing lots of good things to TheyWorkForYou, which has just come under the mySociety wing, as well as helping our swelling volunteer ranks to achieve the things they want to achieve. Come and help us and some of his usability mojo might just rub off 😉

We’re all busy giving WriteToThem some tender loving care. Everything from installing the latest versions of BoundaryLine and CodePoint to generating yearly statistics we’d like to be as fun and pretty as this. Watch out for more announcements as the week goes past.

The new servers are mostly running happily now. Amongst other things we can much more easily make developer sandboxes. So if you want a quiet place to hack some improvements to any of our sites, let us know.

So, HassleMe launched today (despite mostly having been written just before Christmas). Good work by Etienne getting it all together. Today Matthew and I have been working on adding “instant-messenger” functionality to the site, which turns out to be a bit painful. Right now it seems like the most robust solution will be to use bitlbee, a proxy which allows you to interact with the various and wretched instant messenger protocols through the less varied and marginally less wretched IRC protocol.

Integrating a website with instant messenger is an interesting problem. I’m not yet sure how much of the experience of building sites which send and receive email will carry over. We’ll see….

It’s very quiet today. I’ve been updating the projects page now that HearFromYourMP is truly launched. This means we’ve built all the original launch projects, except GiveItAway (more about what is happening with that another time). And it’s over a year since mySociety began. I feel exhausted, and definitely need to take some holiday – I was going to go to Egypt last month, but moved house instead.

This week was usability week. Or was that last week, I’ve got confused. Anyway, Tom made about a million tickets with little usability tweaks to all the sites. Most of these have been put in now, but there’s still lots to do. Good software is about polish polish polish polish, and more polish. I could spend another few years just polishing these sites without making any more.

I’ve also been doing a bit of work on “cobranding”. An ugly marketing term, but there you go. We’ve had a Cheltenham version of WriteToThem for some time. As well as local government, we’re also looking at campaigns groups. So we’ve done a version for AnimalAid, which they’ll be using from their website soon. Apart from the logo and colours, cobranding has some benefits for the user. It’ll make them think they haven’t changed website, and be less disconcerting. In particular, it’ll take them back to other campaigns actions when they’re done.

So, I left regular readers on a geographical cliffhanger last week in my search for a decent gazetteer of the whole world which we can use to let pledge creators tell us where their pledges apply to (and to let people to search for pledges near them). No doubt you expect me to now say that I’ve done this and that this marvellous new feature is now up and running on PledgeBank.

Sadly not.

The best data I’ve been able to find is the GEOnet Names Server dumps from the US Department Of Knowing What Places Are Called (or “National Geospatial Intelligence Agency”, as they call themselves). They maintain a big database of all the places in the world (except for the United States, which task falls to the US Geological Survey), mostly, as I understand it, for military applications. Presumably the idea here is that if some US soldier finds himself sharing Hicksville, Iraq, with something he wants to blow up, he can whip out his satellite-telephone, dial 1-800-US-AIR-FORCE (“You Call: We Bomb”) and, once he’s outwitted the phone menu and call-center staff, can arrange an air-strike without having to know anything tedious like his coordinates (“I’m sorry, could you repeat that, please. Do you mean Hicksville, Iraq, or Hicksville, Alabama?”).

Now, when I last looked at this data, it was full of random and quite significant errors (~5km, for the locations of villages in England — much larger than we’d expected from the coordinate transforms from WGS84 to OSGB36 and the National Grid). For its intended application I suppose this just comes down to a question of how big a bomb you’re prepared to use; for PledgeBank, this is irritating, but not fatal, since all we want is approximate location data which is good enough to let users look for things in the same general area as them.

(There are alternative gazetteers but those I’ve looked at are either proprietary, derived from the GNS data, much smaller than GNS, share its problems while adding new ones of their own, or several of the above. That said, I remain open to alternative suggestions.)

So, the plan is simple: grab all the GNS data (717MB of it), import it into a big database, then let people select their country and type in their (nearest) town, and look up coordinates from that. What could be simpler? It turns out that they’ve even abandoned their rather quaint practice of inventing their own characters sets for everything, and now use UTF-8 for (most of) the fields in the database.

Unfortunately, at this point there’s a more serious problem. If somebody in the UK, say, types “Cambridge” as their location, then probably they’re talking about Cambridge, Cambridgeshire; but there’s a small chance that they might be talking about Cambridge, Gloucestershire (population ~1,700), and we’d look like total muppets if we confused the two. Generally, place names have a habit of nonuniqueness; for instance in the GNS data for the UK we have,

Now, ideally we’d disambiguate these by asking which one of those they meant, using the name of the enclosing administrative region or some other piece of information the user might be expected to know as a qualifier. Sadly, though GNS nominally has this kind of hierarchical structure (see the ADM1 and 2 fields in this list), in practice many placenames are coded without any information on enclosing geographical region, but with ADM1 set to, for instance, “00 — United Kingdom (general)”.

(As an aside, we don’t actually need this stuff for the UK particularly, because we can ask users for their postcodes and do coordinate lookups from that. But the reason I’m starting by looking at the UK part of the gazetteer is that I know the UK’s geography better than that of, say, Congo or France or somewhere, so it’s easier to see what processing steps are required. Plus, privacy-conscious users may prefer to name a town rather than give their actual postcode, since this makes it much harder for us to lock on to them with our orbiting mind-control LASER satellites.)

So, my current job is to invent some plausible heuristic for annotating nonunique place-names, either by trying to guess the administrative region in which they live (probably not usually practical) or adding other qualifiers such as “near Gloucester” or whatever. I suspect this won’t work all that well, but it only has to be good enough. After all, I’m not going to be using the results to bomb anyone….

We’re looking for ways to make it easier for volunteers to get involved in mySociety. Like everything in real life this is mostly a question of openness and policy, but there are also a few technical steps we think would make life easier. One of these is to make it easier for us to hand over a test web server to a volunteer or a group of volunteers to develop code on, play with and generally break. At the moment that’s quite hard to do, because we use apache and all our sites are hosted on one machine (yes really — computers are fast and memory is cheap, though in day-to-day life you’d never notice that, because most of the IT industry is involved in developing “technology” — meaning, “programs that don’t work yet” — that are designed to make your computer slow again: Microsoft Windows, Java, modern web browsers, etc. etc.). Apache is monolithic and if one user breaks the configuration of their test site they can bring down all the sites hosted on the machine. Also, apache isn’t very good at crossing security boundaries (arguably that’s a fault of UNIX generally), so unless we’re prepared to give all the volunteers root (not acceptable for policy reasons) they need to hassle us to get things done (not acceptable to them). Indeed, to save time and admin hassle, IVotedForYouBecause was developed and is hosted elsewhere.

So the idea is to strip away all this crap by running lots of instances of apache, and giving one to any group of volunteers who want to play with one of our sites — all running under their own unprivileged UID — and then direct requests from the outside world through to the appropriate internal apache server via a public-facing proxy. The design I’m envisaging looks something like this:

[Basically pointless block diagram – server hosted on has gone]

(Actually it’s not clear to me that that diagram conveys anything you won’t have understood anyway, but there we go.) For the front-end server I’m using Squid, which is balky and overcomplicated, but supports one very handy feature which is invaluable in this setup: external URL rewriting scripts, which can be used to redirect requests that come through the cache to other resources. The classic application of this is to redirect requests for advertising and other pointless content to local resources so that they don’t take up bandwidth or break your web browser; in this case we’ll be using it to rewrite requests for certain publically-visible URLs (“http://fred.pledgebank.com/…” or whatever) into internal URLs which route to individual users’ apache servers (“http://127.0.0.1:8001/…”). One of the nice things about this is that it preserves the Host: header, and (with a further small hack) apache can be persuaded to pretend that requests weren’t proxied at all, so any back-end stuff that needs to know clients’ IP addresses (such as logging, etc.) can be used unmodified. On top of this, squid will cache responses (assuming that we aren’t lazy about the headers we emit on our own content), which may speed things up a bit for certain sites, though I suspect (with little evidence, and none I’m prepared to bore you with now) this won’t be very useful in practice for the types of sites we’re building.

Another attractive feature of this scheme is that it means that we’re not tied to apache: we could use lighttpd or something, if we wanted to. I doubt that a technical reason to do that will arise in practice, but every minute I spend fighting apache configurations is a minute closer to chucking the bloody thing and picking another web server.

So, it’s the usual story: you start off trying to work around the brokenness in one bit of software, and then all sorts of exciting possibilities suddenly open up. At least, that’s one way to look at it.