mySociety’s sites are all open-source, which means that anyone can take our code and build their own sites with it. That’s been a core mySociety principle from the very beginning, but as time has passed, we’ve realised that we could have made the process easier.

Until recently, you had to be pretty techie to use our code. But now, under the banner DIY mySociety, we’re actively working to lower the bar. Firstly, and most importantly, we’re in the process of rewriting much of our code so that it’s nearer ‘plug and play’ status than previously. Then, we’re backing it up with documentation in the form of easy-to-read handbooks, and supportive communities.

If you’ve ever thought of replicating a mySociety-style site in your own neck of the woods, take a peek at DIY mySociety. We’ll be regularly updating with news and advice.

Image by Kenny Louie, used with thanks under the Creative Commons licence.

When TheyWorkForYou was built by a group of volunteer activists, many years ago, it was a first-of-a-kind website. It was novel because it imported large amounts of parliamentary data into a database-driven website, and presented it clearly and simply, and didn’t supply newspaper-style partisan editorial.

These days dozens of such sites exist around the world. But today sees the launch of a rather-special new transparency site: Mzalendo, covering the Parliament of Kenya.

Mzalendo (which means ‘Patriot’ in Swahili) has been around for a few years too, as a blog and MP data website founded by volunteer activists Conrad and Ory. However, over the last few months mySociety’s team members Paul, Jessica and Edmund, plus the team at Supercool Design have been helping the original volunteers to rebuild the site from the ground up. We think that what’s launched today can stake a claim to being a true ‘second generation’ parliamentary monitoring site, for a few reasons:

• It is entirely responsively designed, so that it works on the simplest of mobile web browsers from day one.
• All the lessons we learned from storing political data wrongly have been baked into this site (i.e we can easily cope with people changing names, parties and jobs)
• Every organisation, position and place in the system is now a proper object in the database. So if you want to see all the politicians who went to Nairobi University, you can.
• There is lots of clear information on how parliament functions, what MPs and committees do, and so on.
• It synthesizes some very complex National Taxpayer’s Association data on missing or wasted money into a really clear ‘scorecard‘, turning large sums of money into numbers of teachers.

The codebase that Mzalendo is based on is free and open source, as always. It is a complete re-write, in a different language and framework from TheyWorkForYou, and we think it represents a great starting point for other projects. Over the next year we will be talking to people interested in using the code to run such sites in their own country. If this sounds like something of interest to you, get in touch.

Meanwhile, we wish Ory and Conrad the best of luck as the site grows, and we look forward to seeing what the first users demand.

Just a week after WhatDoTheyKnow’s big, round number, FixMyStreet also passed a significant milestone.

200,000 reports have been sent through FixMyStreet since its launch in February 2007. It currently sends an average of 250+ messages about potholes, broken streetlights, and other problems to local authorities each day. So far this month, we’ve processed just over 5,000 reports.

Those reports are the work of over 87,000 people, 52% of whom had never before reported an issue to the council. That statistic is important to us: we aim to make it easy to access civic rights, especially for people doing so for the first time.

FixMyStreet.com is a site with a simple premise, and it hasn’t changed greatly since 2007 – though it is currently undergoing a facelift, bringing it more in line with today’s design expectations. Last year we introduced user accounts and zoomable maps, along with a few tweaks here and there.

Like other mySociety projects, FixMyStreet is, of course, built on open code, so that it can be replicated by anyone with a little technical knowledge. The FixMyStreet interface is already up and running in Norway, and soon, the Philippines will see trials of their own version – proving that the model can work in very different infrastructures. Meanwhile, the basic FixMyStreet concept has been replicated in Brazil, New Zealand, and South Korea. Here in the UK, some councils have bought FixMyStreet to embed into their own websites.

FixMyStreet sends reports to the council, and also publishes them online – so each report is read by many people. This simple system helps them find out more about their local community, and what the council are doing to get things fixed.

Uneven paving stones and malfunctioning pelican crossings may not be the stuff of high drama, but against expectations, FixMyStreet does make for fascinating reading sometimes. Take a look at this page if you’d like to see some of the more unusual reports. And if you’d like some insight into some of the issues our developers deal with, you might like to read Matthew Somerville’s solution to the dog poo problem. It’s all glamour at the cutting edge of FixMyStreet.

[Note, November 2014: Petition Your Council has now been retired]

Local petitions can be highly effective, and we think that making them easier to create is in the public interest. Many councils have petitions facilities buried deep within their websites,  most often, very deeply. In fact it brings to mind Douglas Adams’ quote about important council documents being “on display on the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard'”.

Our most recent mini-project is an attempt to make it as easy as possible to find your local council’s e-petitioning site, if they have one. PetitionYourCouncil.com (you’ll notice we stuck to our tried and tested format for site names, there) is a way of finding every council e-petitioning website we know about.

Our original motivation for building the site was that we, along with other suppliers, have supplied online e-petitioning sites to numerous councils ourselves – it’s one of the ways in which we fund our charitable activities. Having delivered these sites, we later noticed that many of them are left under-used and in some cases, not used at all: only because people don’t know about them. We hate to think of councils spending money on a splendid resource that could be improving democratic processes for their citizens – and those citizens never knowing that they exist. In particular, we owe Dave Briggs thanks for pushing us into action with this blog post.

And yes, in case you’re wondering, PetitionYourCouncil links to every council petitions site, not just the ones we made.

The site was built by mySociety developer Edmund von der Burg using Django, jQuery, Google maps and Mapit, and like most mySociety projects, it’s open source. There’s a bit more detail on the About page. Please do try it out, and let us know what you think.

At mySociety we love our site FixMyStreet – it’s the epitome of a web tool that gives simple tangible benefits whilst generating a little accountability at the same time. Reports through the site were up 40% last year, so it’s clear that users quite like it too.

FixMyStreet has been copied in many different countries, which makes everyone in mySociety very happy, too, even apparently appearing in a slide deck the White House uses to show innovative services. However, it turns out that the cheerfully minimalist, almost wantonly unfashionable user-interface has an unfortunate down side: most people who copy the site look at it, think “That looks easy!” and then cheerfully start coding their own clone.

Deceptive simplicity

Alas – the very simplicity that makes the site good hides the fact that making a site like FixMyStreet really work well is actually way harder than it looks. What will you do when a government email inbox fills up? What about when administrative boundaries change, due to an election or restructuring? How do you know you’re not scaring users away with careless wording? All the hard-won lessons from these questions have been baked into the FixMyStreet codebase, and we’re only too keen to talk to people about them.

We were therefore particularly pleased when the Nowegian Unix User’s Group (NUUG) came to us to ask if we could help improve FixMyStreet to make it easier for them to install. Over the last month mySociety Senior Developer Matthew Somerville has been working hard with Petter Reinholdtsen and Christer Gundersen of NUUG, and here’s what they’ve managed in just a handful of weeks.

• The launch of a Norwegian FixMyStreet called Fiksgatami, covering nearly every corner of Norway’s 300,000 square kilometers.
• Problems reported anywhere within Norway will be correctly directed to any of the 400+ responsible municipalities, thanks to Petter and Christer’s amazing data sourcing skills.
• As a necessary side-effect of developing this, Norway now has a free, public administrative web service gazeteer – http://mapit.nuug.no. If Norway is anything like the UK this will soon become an indispensable service for many other web sites and mobile tools.
• The standard mapping is now OpenStreetMap, pulled together by the brilliant Norwegian OpenStreetMap community*. We couldn’t take a technological step backward, and so whilst the site uses OpenLayers if you have JavaScript, the map continues to work just fine without as well.
• The open source FixMyStreet codebase has been upgraded to make it easier to translate into other languages, easier to use different mapping with, and easier to install. These efforts will continue, as we realise this has been one reason why others have made their own versions.
• All this has been done without forking, so various major upgrades we have planned for the UK version will be exportable later in the year.

NUUG’s Fiksgatami is the epitome of what makes civic open source at its best so unmatchably good. It was developed incredibly quickly: just a month to create what is effectively a fully fledged, best-of-breed nationwide e-government service – albeit an unofficial one. Thanks to the hard work of the public servants who fix problem reports, it will make small but meaningful improvements to the lives of a lot of people in Norway.  And it has made the free FixMyStreet codebase better and easier for other people to use to help them do the same thing in other countries.

I know that at mySociety we are all looking forward to working with NUUG again. And I hope that this story inspires others to look at our code, and to work with mySociety to make FixMyStreet a service that can help everyone who would benefit from it.

* We’ll be rolling out updated mapping (including OSM) and more in the UK, soon.

I’m very pleased to announce that mySociety’s upgraded point and postcode lookup service, MaPit, is public and available to all. It can tell you about administrative areas, such as councils, Welsh Assembly constituencies, or civil parishes, by various different lookups including name, point, or postcode. It has a number of features not available elsewhere as far as I know, including:

• Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! 🙂
• Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
• Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
• History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

When a bit of government forwards or attaches emails using Outlook, they get sent using a special, strange Microsoft email format. Up until now, WhatDoTheyKnow couldn’t decode it. You’d just see a weird attachment on the response to your Freedom of Information request, and probably not be able to do anything with it.

Peter Collingbourne got fed up with this, and luckily for us, he can code too. He forked our source code repository, and made a nice patch in his own copy of it.

He then told us about it, and I merged his changes into the main WhatDoTheyKnow code, tested them out on my laptop, then made them live. It all work perfectly first time. Peter even added the new dependency on vpim to WhatDoTheyKnow conf/packages.

Now if you go to an Outlook attachment on WhatDoTheyKnow,such as this one you’ll just see the files, and be able to download them, and view them as HTML as normal. They’ll also get indexed by the search (although I need to do a rebuild for that for it to work with old requests).

Thanks Peter!

If you want to have a go making an improvement to a mySociety site, you can get the code for most of them from our github repositories. For some sites, there’s an INSTALL.txt file explaining how to get a development environment set up. Let us know if you do anything – even incremental improvements to installation instructions are really useful. And new, useful, features like Peter’s are even more so.

Matthew’s just updated ScenicOrNot, the little game that we built to provide a ‘Scenicness’ dataset for Mapumental, to include a data dump of the raw data. The dump will update automatically on a weekly basis, but currently it contains averaged scores for 181,188 1*1km grid squares, representing 83% of the Geograph dataset we were using, or 74% of all the grid squares in Great Britain. It is, in other words, really pretty good, and, I think, unprecedented in coverage as a piece of crowd sourced geodata about a whole country.

It’s available under the Creative Commons Attribution Noncommercial 3 Licence, and we greatly look forward to seeing what people do with it.

FixMyStreet has a lot of RSS feeds. There’s one for every one-tier council (170), one for every ward of every one-tier council (another 5,044), two for every two-tier (county and district) council (544), and two for every ward of every two-tier council (20,296) – two per two-tier council because you might want either problems reported to one council of a two-tier set-up in particular, or all reports within the council’s boundary.

Then there’s an RSS feed every 162m across Great Britain in a big grid, returning all reports within a radius of that point, the radius by default being automatically determined by that point’s population density, but customisable to any distance if preferred. That’s, at a very rough approximation assuming Great Britain is a rectangle around its extremities, which it’s not, 19 million RSS feeds, lots of which will admittedly be very similar. 🙂

Every single one of those feeds can be subscribed to by email instead if that’s preferable to you, and are all accessible through a simple interface at http://www.fixmystreet.com/alert.

However, none of these RSS feeds was suitable for the person who emailed from a Neighbourhood Watch site and said that all they had was a postcode and they wanted to display a feed of reports from FixMyStreet. Given you could obviously look up a FixMyStreet map by postcode, it did seem odd that I hadn’t used the same code for the RSS feeds. Shortly thereafter, this anomaly was fixed, and if you now go to a URL of the form http://www.fixmystreet.com/rss/pc/postcode you will be redirected to the appropriate local reports feed for that postcode (I could say that adds another 1.7 million RSS feeds to our lot, but given they’re only redirects, that’s not strictly true). And after a couple more emails, I also added pubDate fields to the feeds which should make displaying in date order easier.

It’s great to see our RSS feeds being used by other sites – other examples I’ve recently come across include Brent Council integrating FixMyStreet into their mapping portal (select Streets, then FixMyStreet), or the Albert Square and St Stephen’s Association listing the most recent Stockwell problems in their blog sidebar. If you’ve seen any notable examples, do leave them in the comments.

One of the nicest things about being involved with mySociety is seeing people in other countries starting similar organisations and building similar websites. After a Skype conversation with another eager hopeful last night, I thought I’d blog a bit about the things I think are most important to know if you’re just starting up. Here goes.

1. Absolutely the most totally essential thing is to be an organisation of amazing, politically minded coders, not an organization employing or contracting good coders*. Their skills are your lifeblood, their ideas your bread and butter, and finding the best civic hackers in your country and building your organization around them is the only path to success. And that means they should be making most of the day to day decisions, not you, you ignorant, arts-degree-clutching clot.

2.Ask the public what they think you should build. Not only will that give you access to more ideas than you have yourselves, it’ll engage people with you. Also, it’ll help you focus on the vital business of building sites that users want, not that YOU want.

3. Keep your cost base low, and put all the money you have into looking after your core staff and being nice to volunteers. Work on building a community of volunteers, even if most of them are really just friends rather than people putting in lots of time. Avoid renting offices, avoid non-essential non-coder staff, get people to donate serving infrastructure and bandwidth. Because building and running democratic websites is a fundamentally new area of human endeavor (not like blogging which at least has an analogy in journalism) there are basically no pre-existing funding streams for the type of work you’re about to do. You will have to create the buzz around yourselves that will lead to people wanting to fund you, and it will probably take years, if you get there at all (mySociety hasn’t quite done this yet, even at 5).

4. Ensure that the core of what you build can struggle on by even if your whole organization collapses. That means being open source, putting energy into sites that are as automated as possible, and making people excited about being volunteers.

5. If you aren’t pissing off at least some people all the time, you’ve probably been captured by the establishment.

6. Take whatever your first website plan is and remove 90% of the features you want. Then build it and launch it and your users will tell you which features they actually wanted instead. Build them and bask in the warm glow of appreciation.

*yes, yes I know I can’t code for toffee, and I’ve got an arts degree, but I’m still a geek, honest. (Proudest moment, working out that a batch of PCI network cards were unreliable because they’d come from the factory flashed with the EPROMs for the wrong hardware, and fixed it.)