Today, we are using the phrase “Alaveteli upgrade” rather a lot – and not just because it’s such a great tongue-twister. It’s also a notable milestone for our open-source community.
Alaveteli is the software that underlies WhatDoTheyKnow, our Freedom of Information website. The code can also be deployed by people in other countries who wish to set up a similar site. If you’re a ‘front-end user’, someone who just uses WhatDoTheyKnow to file or read FOI requests, this upgrade will go unnoticed… assuming all goes well at our end, that is. But if you’re a developer who’d like to use the platform in your own country, it makes several things easier for you.
Alaveteli will now be using the Rails 3 series – the series we were previously relying on, 2, has become obsolete. One benefit is that we’re fully supported by the core Rails team for security patches. But, more significant to our aim of sharing our software with organisations around the world, it makes Alaveteli easier to use and easier to contribute to. It’s more straightforward to install, dependencies are up-to-date, code is clearer, and there’s good test coverage – all things that will really help developers get their sites up and running without a problem.
Rails cognoscenti will be aware that series 4.0 is imminent – and that we’ve only upgraded to 3.1 when 3.2 is available. We will be upgrading further in due course – it seemed sensible to progress in smaller steps. But meanwhile, we’re happy with this upgrade! The bulk of the work was done by Henare Degan and Matthew Landauer of the Open Australia Foundation, as volunteers – and we are immensely grateful to them. Thanks, guys.
Image credit: Sashi Manek (cc)
The Open Democracy Advice Centre (ODAC) in South Africa will be using mySociety’s Alaveteli software in their latest project – and, with a bit of match-making from mySociety, the preparation period has been rigorous.
Alaveteli is our open source Freedom of Information platform. It underlies our own UK-based WhatDoTheyKnow, and right-to-know sites around the world. Alaveteli sites make it easy for citizens to ask questions of those bodies who operate under Freedom of Information law and, significantly, they automatically publish all responses.
Before any coding or implementation began, we got ODAC together with the “Governance Collaboratory”, an initiative from the d.school in Stanford University that seeks to apply the “design thinking” approach to projects that intend to make government more open, more effective, and more accountable. We’ve observed quite a few Alaveteli installs, but while we’re always on hand to offer support and answer development queries, we’ve never prepared the ground quite like this.
Gabriella Razzano of ODAC welcomed Jeremy Weinstein and Jenny Stefanotti, both from the d.school, to Cape Town for an intensive few days of assessing how the design thinking approach could shape the project. Two staff from mySociety also went along — Paul (our Head of International Projects) and Dave (one of our developers) — because we’re keen to understand how the d.school’s approach might improve the way we go about building our new projects.
Now, at mySociety we already know a thing or two about building civic systems that engage with the public, because we have considerable experience in the field. We are expert at combining user experience and current tech to create simple, usable interfaces. We conduct usability tests, we apply A/B testing, and we think hard about what our analytics tell us. But actually much of this is reactive, iterative design: it’s being applied after the core product has already been built.
Design thinking challenges this approach by suggesting that the user on which initial designs are often based is purely imaginary. As a result, the site inevitably includes the assumptions and prejudices of its creators. This won’t necessarily lead to a bad design — especially if the creators are benign and experienced — but it must fail, by definition, to account for the unexpected things that may motivate or concern actual users. The design thinking process attempts to change this by approaching the initial problem in a prescribed way and following a process that isolates genuine, existing requirements. This includes, in design thinking terms, processing the initial interviews into empathy maps from which requirements emerge, and which themselves become features that are rapidly prototyped in isolation from other parts of the system.
This is uncomfortable for those of us used to building loose iterations from the bottom up and refining them later. It means introducing empathy and rapid, offline prototyping much earlier in the process than we’d normally expect. Certainly in the commercial world it’s common for a company to prototype against their target consumers early on. But for civic projects such as mySociety’s, it’s often much harder to identify who the users will be, for the impressive yet overwhelming reason that often we are building our platforms for everybody. This can lead to generalisations which may miss specific issues that could make a huge difference to some users.
The d.school advocates a “learning by doing” way of teaching, so the days we spent in Cape Town were a busy mix of practice as well as theory. We interviewed people who had a variety of reasons to want to make Freedom of Information requests, including an activist who’s already used South Africa’s Freedom of Information legislation to make requests regarding housing projects, the head of a rape crisis centre, and law students who may well become a nation’s most empowered activists. From these interviews we isolated specific needs, which at this stage were nearly all unconnected to any digital or web requirement. Jeremy and Jenny then led us through the process of rapid, analogue prototyping intended to address those needs.
Inevitably we could only scratch the surface in the few days we had available, but we hope ODAC will be able to apply the process to the development of their project, just as we aim to use it to benefit the work on ours.
Image credit: Procavia capensis (Rock-dwelling Hyrax or Dassie) by Arthur Chapman, released under CC BY-NC-SA on Flickr.
They tell visitors that dassies such as these live atop Table Mountain. We went up there and saw none. Similarly, Freedom of Information requests exist in South Africa under the Promotion of Access to Information Act 2000 (PAIA), but most people have never seen one — fewer than 200 PAIA requests were made nationally in 2012. This tenuous comparison allows us to illustrate the blog post with a cute picture of fuzzy mammals.
If so, mySociety has some news of an offer that may interest you.
As of May 2013 we will be offering free technical time from mySociety’s developers to a limited number of people and organisations who want to get versions of Alaveteli (Freedom of Information requests) and FixMyStreet (street problem reports) working, anywhere in the world.
All you have to do to be considered is to send us a message expressing an interest in gaining our support, and telling us a bit about you and what you hope to achieve.
If you’re selected, we’ll help modify the software to make sense in relation to your own region’s laws or local authority’s systems, and we’ll even host the service if that is a problem.
More importantly, we’ll help to explain how the software works at a technical level, so you or a local developer can really understand how the open source code works, and how to make changes to it.
This service from mySociety is worth thousands of dollars a time. We are offering it because we think it is important to support people who have the enthusiasm, but perhaps not the means, to run a service like FixMyStreet or WhatDoTheyKnow.
In order to qualify, you must be a group or an individual who can show us that you have a desire to run online civic and democratic projects like FixMyStreet or WhatDoTheyKnow in the long term, and that you have access to some kind of web developer skills. You can be anywhere in the world.
What does commitment mean? Nothing impossible, but there are a couple of requirements.
You need long-lasting enthusiasm. We’ll be looking to make sure that you understand the ongoing time and energy commitments a project like this will involve. To put it frankly, we don’t want to invest in a project that may close down after a few months. So, we’ll want to have a chat to ensure that you really know what you’re getting into.
You need access to a web developer – at least sometimes. While these kinds of sites do, to some extent, run themselves, some work will always be necessary to keep them running smoothly*. And while our developers will help you get your site off the ground, you will need your own developer too, both at set-up, and as the site continues to run.
But don’t let that put you off – we also want to hear from you even if you haven’t yet got a group in place. The important thing is that you have the desire and the motivation to drive a project to completion.
Interested? Drop us a line now and let’s talk. Don’t forget to tell us what country, city or region you’re interested in covering, and what resources you can contribute to making your site into a success.
* See the following resources to understand what sort of work is involved in running a civic or democratic website:
Photo by Ken Hawkins (CC)
Alaveteli (the software that runs WhatDoTheyKnow) is capable of being translated into any language, and we’ve finally switched on the ability to use the website in Welsh today. Many apologies for the long wait as this has been on our to-do list for well over 2 years…
As you can see, we don’t yet have a complete Welsh translation, and it’s just a start: we’ve done the help pages, and around 6% of the rest. To take a look at what’s been done, just click the “Cymraeg” link at the top of any page.
We’d love it if you could help us get to 100% by adding translations (or correcting any mistakes we’ve made!) at Transifex. You can read more about working with translations for Alaveteli, here and here, or just get in touch if you need a helping hand getting started or have any further questions.
And finally, a massive thank you & diolch to the translators who have already helped us get this far!
Summer may seem like a long time ago, but despite the cold outside, we’ve been looking back over our participation in Google’s Summer of Code project. It’s almost enough to warm us up!
This post is an attempt to record the process from our point of view. We hope it will be useful for other organisations considering participating next year, and for students who want to know more about how the scheme works.
What is Google Summer of Code?
It’s a programme sponsored by Google’s philanthropic arm, giving students the chance to experience real-life coding on open source software.
The scheme is open to students all over the world, who are then paired up with open source organisations like us. The students gain paid work experience and mentoring; the organisations gain willing workers and some fresh new perspectives; the world gains some more open source code to use or develop further.
Everyone’s a winner, basically.
2012 was our first year on the programme: once we had been accepted on the scheme, we were given two student slots – the maximum allowed for a first-time organisation.
Given mySociety’s wide suite of codebases, there were several projects that could have benefited. We listed all our ideas, and let people apply for the ones they found appealing.
Goodness, there were a lot of applicants! It was very heartening to discover that there is such an enthusiastic community of young coders all around the world – even if it did take us a long time to sift through them all and make our choices.
You might remember our post back in May, when we announced that we’d made our choices. We were delighted to get working with Dominik from Germany and Chetan from India.
As things turned out, our students ended up working on a project that wasn’t even on our original list: PopIt, our super-easy ‘people and positions’ software.
That’s because once we spoke to our chosen students, we realised they had the skills that could really help us forge ahead with this project – and once we discussed it with them, they were keen. So PopIt it was.
Germany and India are a bit of a commute away, but fortunately development work can be managed remotely. We know this particularly well at mySociety: our core team work from home and are scattered across the UK.
The only difference here was the 6+ hour time difference between us and India: it was important to be rigorous about checking in at times when Chetan would be awake!
We communicated via IRC (instant chat), email, and occasionally Skype, and it all worked well.
Edmund, the team member chosen to be mentor, broke the required tasks down into big pieces so that the students would have realistic work units of several days each.
What was achieved
PopIt is primarily a tool for helping people create and run parliamentary monitoring websites (like TheyWorkForYou) with minimal coding knowledge/effort, though we anticipate that it will have many other uses too.
Our students spent the first half of the summer learning and improving the PopIt codebase. Once they were confident in it, they created their own sites using PopIt as a datasource to test the API, and, hopefully, create a valuable reference resource for the community.
Dominik added a migration tool to PopIt, which lets you upload data as a CSV. This means that you can start a site with a database of names, positions and dates at its heart – within seconds.
Chetan created an image proxy that lets us serve images in a smart way that makes sense for APIs. His test site was for Indian representatives (here’s the code).
Neither site is being maintained now, which just confirms that it is harder to run a site than to start it. This is not a failing, though. The creation of these sites, along with Chetan and Dom’s feedback, helped us understand where improvements needed to be made. In the course of one summer, PopIt became much more mature.
Looking back on the Summer of Code
Edmund attended a follow-up ‘mentors’ summit’ at the Googleplex in California – he found it very helpful to compare notes with other organisations and find out what had worked best for them all, and he made some good contacts too.
Assuming we get the chance again, would we participate in 2013? Our experience was very positive, but as yet we are undecided, purely because of the fluid nature of our workflow: we don’t yet know whether time and resources will permit.
Obviously, we have enjoyed great benefits from the scheme, but that has depended on quite a bit of input from our side, and we need to be sure that we can ensure that happens again.
Edmund has compiled a list of advice, from the practical (ask students to treat the placement like a full-time job; test coding skills before acceptance) to the desirable (a weekly blog post from participants; make sure you over-estimate the time you’ll spend mentoring). If you’re thinking of participating next year, he’d be happy to pass on his tips for ensuring that you, and your assigned students, get the best out of the Google Summer of Code. Just drop him a line.
I am Duncan Parkes, a developer for mySociety, a non-profit full of web geeks. One of the things we try to do well here is to take complicated data and turn it into really usable tools – tools which are attractive to people who aren’t web (or data) geeks.
For some considerable time I’ve been working on Mapumental – a project that is about turning public transport timetable data into pretty, interactive maps featuring isochrones, shapes that show people where they can live if they want to have a commute of a particular time. You can play with the new version we just launched here. That particular map shows the commuting options to where the Queen lives. Slide the slider for full effect.
There are a couple of hard problems that need solving if you want to build a service with an interactive journey times overlay like this. You need to be able to calculate a *huge* number of journeys extremely quickly, and you need to be able to make custom map layers so that it all looks nice. But what I think might be most interesting for you is the way in which the contours get rendered on top of the maps.
It all started about three years ago, when the first version of the app – co-developed with the geniuses at Stamen – used Flash/Flex to draw contours on the maps, and to let people play with them. You can still play with a couple of versions of that technology from way back in 2007, that is, unless you’re using an iPad or iPhone, which of course don’t do Flash.
What was going on inside this Flash app was as follows. We needed to show the user any one of hundreds of different combinations of journey times (5 minutes, 12 minutes, 56 minutes, etc) depending on where they set the slider. Sending each one from the server as a tiled map overlay would be dead slow. Even Google – who have chosen to send new tiles each time – end up with a service which is surprisingly slow (try choosing a different time on this map).
With some help from Stamen, we decided that the way of making it possible to show many different contours very quickly was send the client just one set of tiles, where each tile contained all the data for a variety of journey times. What does that mean? Simple: each colour in the tile represented a different number of minutes travelling on the map. So a batch of pixels that are colour X, all show places that are 15 minutes from the centre of the map.
So, in this old Flash system, when you slide the slider along, the Flash app makes some of the coloured pixels opaque, and the others transparent. It was, in short, a form of colour cycling, familiar to lovers of 8 and 16 bit computer games.
However, from about 2010 onwards, the march of iOS spelt the end of Flash. And that meant that we couldn’t launch a shiny new site based on this technology, as lovely as it was. We had to work out some approach that would use modern web standards instead.
The Death of Flash Makes Life Difficult – for a while
How do we replicate the experience of dragging a slider and seeing the map change like in the original Mapumental demo, but without Flash? One of the things that made the original Mapumental nice to use was how smooth the image changes were when you dragged the slider. Speed really matters to create that sort of organic effect that makes the demo so mesmerising.
So as we started to tackle the question “How do we make this work in a post-Flash world?”. And the first thought was “Let’s do away with those map tiles, filled with all that journey time data!”. After all – why send any tiles to a modern browser, if it can just render nice shapes on the fly?
So we had a go. Several goes. At first we tried rendering SVG circles around each public transport stop – but that was too slow, particularly when zoomed out. Then we tried rendering circles in Canvas, and whilst that was OK in sparsely populated places it sucked in the cities, where people would actually want to use it.
Back to Colour Cycling – Using Web Standards
So, I had a bit of a look at the waterfall. It seems to work by holding in memory a structure which has all the pixels which change and all the colours they should change to and when. This works beautifully for the waterfall picture, but only a limited number of the pixels in that image actually change colour, and the image is quite small. For a full screen web browser with a big map in, this didn’t seem promising, although I’d love to see someone try.
Unfortunately, there is no way to change the palette of an image that you’ve put on the canvas. In fact, there’s no way to change the palette of an HTML img element: all you can do is assign it a new src attribute.
But this gets back to the original problem – we don’t want to download new mapping for every different position on the time slider. We definitely can’t afford to have the client downloading a new image source for every tile whenever the slider is moved, so we had to find a way to make that src at the client end and get that into the src attribute.
The Breakthrough – Data URIs and Base64 encoding
So we started trying data URIs. For those of you not familiar, these allow you to put a whole object into your HTML or CSS, encoded in Base64. They’re commonly used to prevent pages having to make extra downloads for things like tiny icons.
My new plan was that the client, having downloaded each palette-based image, would make a Base64 encoded version of it, which it could then use to build a version with the right palette and assign this as a data URI of the tile.
So in summary, what we built does this:
- The server calculates the journey times and renders them to palette-based tiles.
- It sends these to the client, encoded in Base64, and with the initial bits up to the palette and transparency chunks removed.
- At the client end, we have a pre-prepared array of 255 ‘starts’ of PNGs that we combine with the later parts of the ’tiles’ from the tile server to make data URIs.
- When you drag the slider it combines the appropriate ‘start’ of a PNG with the bulk of the tile that has been downloaded from the server, and assigns that to the src attribute of the tile.
And that’s how the nice overlays on Mapumental work. But as so often in coding, the really interesting devil is in the detail – read on if you’re interested.
Diving into Base64 and the PNG file format – The Gnarly Bits
So – why are there 255 of these ‘starts’ of these PNGs, and what do I mean by a ‘start’ anyway?
PNG files are divided up into an 8 byte signature (the same for every PNG file) and a number of chunks, where each chunk consists of 4 bytes to tell you its length, 4 bytes of its name, some data, and 4 bytes of cyclic redundancy check. In this case, what I call a ‘start’ of a PNG is the 8 byte signature, the 25 byte of the IHDR chunk, and the PLTE (palette) and tRNS (transparency) chunks. The PLTE chunk has 12 bytes of overhead and 3 bytes per colour, and the tRNS chunk has 12 bytes of overhead and 1 byte per colour.
Base64 encoding is a way of representing binary data in text so that it can be used in places where you would normally expect text – like URIs. Without going into too much detail, it turns groups of 3 bytes of binary gumpf into 4 bytes of normal ASCII text without control characters in it, which can then be put into a URI.
Why do we have 255 colours, rather than the maximum 256 which are available in a palette? Because we need the break between the end of the tRNS chunk and the start of the IDAT chunk in the PNG file to align with a break between groups of three bytes in the Base64 encoded image. We need the length of these starts to be a multiple of 3 bytes in the original PNG format, which translates into a multiple of 4 bytes in the Base64 encoded version, so we can cut and shut the images without corruption.
Which just goes to show that even though web GIS technologies may feel like they are approaching a zenith of high level abstraction, there’s still some really gnarly work to be done to get the best out of current browsers.
Since its launch in 2005, WriteToThem has always covered all parts of the United Kingdom, and the Northern Ireland Assembly was the first body added to TheyWorkForYou after the UK Parliament, in late 2006. So whilst we certainly have not ignored Northern Ireland, it had always been an irritant of mine (and a cause of infrequent emails) that FixMyStreet only covered Great Britain.
This was due to the way it had originally been funded and set up, but those issues were in the past, due to a myriad of changes both internal and external, and it was now more a case of being able to find the resources to implement the necessary work. Late last year, mySociety worked with Channel 4 on the website for their series of programmes on The Great British Property Scandal. This used, in part, code similar to FixMyStreet to let people report empty homes, and it was required to work in all parts of the UK. So as part of that process, code was written or generalised that let aspects of FixMyStreet like the maps and place name lookup work for Northern Ireland locations.
It’s taken a few months since then to allocate the time, but we’ve now been able to take the code written back then, add various other bits, and incorporate it into FixMyStreet – which now covers the 26 councils of Northern Ireland, and the central Roads Service. Issues such as potholes, graffiti, and broken street lighting can be reported to Antrim or Newry and Mourne as easily as Aberdeen or Wyre Forest, and just as in the rest of the UK you can sign up for alerts based around your location or to your council.
We’ve mentioned Components before on this blog – they’re modules which you can slot into your website, and which should save you a lot of time and effort. Today, we’re pleased to announce that a fundamental Component is ready for use – MapIt Global.
This Component will match geographical points to administrative areas anywhere in the world. So for example, you can use it on sites like FixMyStreet, where we ask the user for a zipcode/postcode, and then automatically knows which council to send their report to.
mySociety’s Director Tom has written an in-depth blog post about MapIt Global. You’ll want to read it if you’re thinking of building a site or app for reporting street faults, for contacting elected representatives, for parliamentary monitoring… or just maybe you have ideas for a type of website that we haven’t even thought of. We look forward to seeing how MapIt Global will be used.
Questions? Thoughts? We’d love to hear them, either on this post or on Tom’s.
All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.
Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.
As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.
Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.
We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.
We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.
The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.
From Britain to the World
For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.
MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.
If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.
About the Data
The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.
The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global. We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.
If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.
Commercial Licenses and Local Copies
MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.
Version 0.6 of Alaveteli, our Right to Know platform, has just been released. Some of the major features of the new release are for users, some for Admin:
- We’re all so used to the concept of getting updates on a ‘wall’ or a ‘stream’, thanks to Facebook and Twitter – and now users can do the same on Alaveteli sites.
- It’s difficult to moderate all unsuitable requests when you are running a high-traffic site – but now you can tap into the power of the crowd, with ‘report this request’ buttons (and a moderation list in Admin).
- The back-end looks extra smart now, thanks to some nifty code “built for and by nerds” by Twitter (Bootstrap).
- And – one for developers – Alaveteli is now using Bundler wherever possible.
Seb’s also written a round-up of the most interesting changes and bug-fixes. If you’re running an Alaveteli site, or just thinking about it, you should head over there for a read.
Image by Caterpiya.