1. A wish list for geodata on FixMyStreet

    I was just talking to someone in a local council about the fact that they’d opened up the location of 27,000 streetlights in their council area. They wanted to know if FixMyStreet could incorporate them so that problem reports could be more accurately attached.

    This conversation reminded me that we’ve had an informal wish list of geodata for FixMyStreet for some time. What we need is more data that lets us send problems to the correct entity when the problem is not actually a council responsibility.

    I’m just posting these up to see if anyone knows a guy who knows a girl who knows a dog who knows how to get hold of any of these datasets. In some vector data format, if possible, please!

    • Canals and responsible authorities
    • Supermarkets (esp car parks) and responsible companies
    • Network Rail’s land
    • Council owned land
    • Land and roads controlled by the Highways agency
    • Shopping malls
    • National parks
    • BT phone boxes (the original problem which inspired FixMyStreet)

    So, do you know someone who might know someone who can help us improve FixMyStreet? And guess what, if we do add this to our web services, you’ll probably be able to query them too.

  2. Mapping points and postcodes to areas

    I’m very pleased to announce that mySociety’s upgraded point and postcode lookup service, MaPit, is public and available to all. It can tell you about administrative areas, such as councils, Welsh Assembly constituencies, or civil parishes, by various different lookups including name, point, or postcode. It has a number of features not available elsewhere as far as I know, including:

    • Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! 🙂
    • Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
    • Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
    • History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

    If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

    Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

  3. What are the two sorts of Cloud infrastructure called?

    I’ve been doing lots of research around “cloud computing” recently, so we can change how Mapumental works and take it out of private beta.

    One thing that’s struck me is that there doesn’t seem to be a proper, industry standard name to distinguish what to me are two fundamentally different sorts of “cloud computing”. I’m focusing here entirely on cloud services for programmers (let’s leave what it means to end users or businesses for another day).

    Here are my own names and descriptions of them:

    1) Cloud hardware server provision (Cloud HSP)
    Low level APIs for making and destroying (virtual) servers, and loading machine images onto them. e.g. Amazon Elastic Compute Cloud, Rackspace Cloud Servers, Eucalyptus’s EC2 bits. Basically, what Eucalyptus v 1.5 can do and what libcloud should do. (By analogy, this is the assembly language of cloud computing)

    2) Cloud developer service provision (Cloud DSP) A service that a developer accesses with one name and a simple API, and behind the scenes it scales for him, automatically. e.g. Amazon Queue Service, Rackspace Cloud Files. (By analogy, this layer is the C programming language of cloud computing)

    [as an aside, Google AppEngine is an interesting one. It is definitely in the Cloud DSP category, but I think it is larger than that – it is a whole set of APIs all in that category. Something like Google DataStore is a single Cloud DSP, albeit one apparently only accessible within AppEngine apps]

    It’s possible to use a Cloud HSP (assembly language), along with a bunch of your own software or open source software, to build new Cloud DSPs (C code). Right now this is pretty hard – even quite well known open source distributed datasbases like CouchDB still need scripting to even make them replicate. The code that makes and destroys servers and gives the service one name, needs manually stringing with quite new bits of wire (things like scalr and Wackamole).

    For this reason, I’m reluctant for mySociety to get into the “making our own Cloud DSP out of Cloud HSP” game. It feels to me like a suck of time, and like we wouldn’t be able to guarantee without lots of careful and expensive testing that it would scale. I’m more tempted to use the commercial Cloud DSP services where possible, even though they are proprietary. But use them via our own abstraction layer, so we can change as we need to. Of course, we have some C++ code (the public transport route finder), so will have to use the Cloud HSP API to get that going, perhaps with Amazon’s Auto Scaling. But it can jolly well use AQS and S3 to talk to other services.

    So, what do you think about the names Cloud HSP/DSP? Are there already existing names for the distinction that I’m making? Is it a useful distinction for you? Can you think of better names?

  4. What does it take to get FixMyStreet to post reports directly into a council CRM? One good public servant.

    Recently I gave a talk at a conference where I told a group of local government officials that FixMyStreet was built not just to provide cleaner streets for their citizens, but also to force the hands of councils to procure and contract internal IT systems fit for the 21st century. In particular I pointed out that companies like Google seek to have people use their service from any site, any browser and device – they don’t just demand that everyone goes to www.google.com. And, I said, it’s only through building nice interfaces (APIs) that you can become an organisation that realises the benefits for yourself and other organisations from taking this ‘we’re happy to interoperate with anyone’ approach.

    Less than three weeks later Michael Houlsby from East Hants council has single-handedly built an external facing API for their faults and problems database. So now FixMyStreet posts problems in that council direct into their database, without them first being translated into emails.

    This is fantastic, especially as Michael clearly knocked it together in his spare time, and helps confirm what we’ve said before – if government builds nice interoperable APIs people like mySociety will use them to improve citizens’ experiences, whist simultaniously keeping everyone’s unnecessary workloads and expenses to a minimum. Plus it shows that if your IT supplier tells you you need to sign a new five or six figure contract to add an API to a CRM system you’ve already bought – you’re being jerked around.

    Hats off to Michael – you’re a great example of a pro-active public servant using your skills to make government both better and more efficient.

  5. Calling anyone at Technorati

    Please pass this on if you have any first or second-hand contact with Technorati:

    “Hello!

    Can we please have an API key with a high or unlimited daily query cap so that we can help users of TheyWorkForYou to see what discussions about Parliament are going on across the web?

    thanks,

    Everyone at mySociety”

  6. PledgeBank Facebook application disabled

    Unfortunately, I’ve had to disable the PledgeBank Facebook application. It used to let you sign and share pledges from within Facebook.

    Facebook recently changed their platform (again!), breaking our code for sending success/failure messages. Obviously, it is no good signing up to a pledge if you don’t get informed when it succeeds.

    I tried to fix it, but couldn’t work out how to do so quickly. We don’t have the time and money at the moment to chase after this, so I’ve disabled the application entirely. Links to PledgeBank pages on Facebook now redirect to pledgebank.com.

    Hopefully it’ll be back one day – do send us emails if you miss it (or money if you have a large pledge that really needs it!). I think there may be a better solution with a simpler interface – the current application tried too hard to reimplement all of PledgeBank within Facebook. And besides, we should be supporting OpenSocial now it exists. It’s an open standard, Facebook isn’t.

    Technical details: We used infinite session keys to send notifications from cron jobs. Quite reasonably, this no longer works. However, I couldn’t find out what to use instead. I think Facebook should respect backwards compatibility of its APIs a lot more, and if it breaks it they should give clear instructions about what to use instead. This does put me off ever wanting to develop anything on their platform again.

  7. Relentlessly into autumn

    I’m enjoying the weather at the moment, seems to be sunnier than the summer, but cool with an atmospheric autumnal taste in the air.

    mySociety is changing as ever, leaping forward in our race to try and make it easier for normal people to influence, improve or replace functions of government. More on this as it happens.

    Meanwhile, I’ve been continuing to hack away at WhatDoTheyKnow. A little while ago Google decided to deep index all our pages – causing specific problems (I had to tell it to stop crawling the 117th page of similar requests to another request), and also ones from the extra attention. There have been quite a few problems to resolve with authority spam filters (see this FOI officer using the annotation function), and with subtle and detailed privacy issues (when does a comment become personal? if you made something public a while ago, and it is now a shared public resource, can you modify it or take it down?).

    Right, I’ve got to go and fix a bug to do with the Facebook PledgeBank app. It’s to do with infinite session keys, and how we send messages when a pledge has completed. Facebook seem to change their API without caring much that applications have to be altered to be compatible with it. This is OK if the Facebook application is your core job, but a pain when you just want your Facebook code to keep running as it did forever.

    (the autumn photo thanks to Nico Cavallotto)

  8. Highlighting the current speech

    Debate pages that have at least one timestamped speech (such as the previously mentioned last week’s Prime Minister’s Questions) have a video fixed to the bottom right hand corner (if your browser is recent enough) showing that debate. While playing the video, the currently playing speech is highlighted with a yellow background, and you can start watching from any timestamped speech by clicking the “Watch this” link by any such speech. So how does all that work?

    I’m very proud of this feature, I wasn’t sure it would be possible, and it’s very exciting. 🙂

    Flash has an ExternalInterface API, where JavaScript can call functions in the Flash, and vice-versa. When the video player loads, it requests an XML list from the server of all speech GIDs and timestamps for the current debate (here’s the file for the above debate). So when someone clicks a “Watch this”, it calls a moveVideo function in main.mxml with the GID of the speech, which loops through all the speeches and moves to the correct point if possible.

    The highlighting works the other way – as the video is playing, it checks to see which speech we’re currently in, and if there’s been a change, it calls the updateSpeech function in TheyWorkForYou’s JavaScript, which finds the right row in the HTML and changes the class in order to highlight it. Quite straightforward, really, but it does make following the debate very simple and highlights the linking between the video and the text, all done by our excellent volunteers (join in! 🙂 ).

    Talking of our busy timestampers, I’ve also been busy making improvements (and fixing bugs) to the timestamping interface to make things easier for them. As well as warnings when it looks like two people are timestamping the same debate at the same time, various invisible things have been changed, such as using other people’s timestamps to make the start point for future timestamps on the same day more accurate. I also added a totaliser, using the Google Chart API, for which you simply have to provide image size and percentage complete.

    Approaching 45% of our entire archive of video timestamped, with the totaliser approaching the chartreuse 🙂

    Previous articles

    1. The Flash player
    2. Seeking
    3. Highlighting the current speech
  9. Richard Pope’s GroupsNearYou.com launches

    Say hello to GroupsNearYou.com.

    The Short Geeky Explanation

    GroupsNearYou.com is an entirely user generated API-queryabledatabase of the location and nature of local online communities, irrespective of the platform they are hosted on. A piece of the programmable web, in short, with local community building focus. The syndicated community information can easily be layered onto a map, for example.

    The Business Problem it Solves

    Do you run a site that tells people stuff about their local area? Do you suspect that there might be quite a lot of internet enabled community activity going on in local areas that you’d like to tell your users about? Use our feeds.

    The Social Problem It Solves:

    There’s a proven real world social value to people belonging to very local email lists and other forms of local online community. However there is no eBay or Craiglist or other market dominant player in the local online community world, instead there’s a myriad of google groups, yahoo groups, Facebook & other YASN groups, extremely old school CCed email lists, online forums and so on. As a consequence of not having one big simple place to go to find and join local groups (many of which are not even on the web for Google to find) far fewer people ever find out about and join their local online groups. GroupsNearYou.com is about getting more people to join groups, groups that are not hosted by us, and (hopefully) mainly discovering them via uses of our syndicated info on sites that aren’t run by us. It’s a piece of pure internet infrastructure, with a positive social bent.

    Who did it?

    Astonishingly, the project was almost entirely built by a volunteer, Richard Pope assisted in design by another brilliant volunteer Denise Wilton. Their only reward is a highly sought after mySociety Hoodie, plus the love and gratitude of all our users.

    Richard has been an amazingly dedicated volunteer for mySociety and on his own projects for over two years, and deserves the reputation he is rapidly gaining as one of the world’s truly great civic minded web innovators. The project was funded by the UK’s Government’s Ministry of Justice who have been trying to run experiments of different kinds in the realm of electronic democracy. Their money will go to help improve and grow the site, rather than building it, which is a very interesting funding model in its own right. The several hundred groups already in the system are mainly added by users of WriteToThem.com

    Next steps

    Almost all the groups listed in the database are in the UK at the moment, and they’re all from users of our other sites. We’re interested in working with anyone who runs sites that might want to either take information out of it, or put information into it (Hello email list/social network providers!)

    Anyway, it’s dead simple really, just a little brick in the internet wall, albeit one that I hope will help a few more people meet their neighbours and improve their communities.

  10. Today is Thursday

    Lots has happened since I last posted, which was months ago, before Chris died.

    Luckily for us, in the last few months Chris had only been working one day a week for us, so it hasn’t been as difficult in practical terms as it could have been. There were various mySociety things running in his flat, such as the WriteToThem fax server, which had to be set up quickly elsewhere.

    We miss Chris’s expertise most days (only yesterday I was swearing at gnuplot). Matthew is twice as much for me to handle; he works so quickly, he has hard questions to ask at a ferocious rate that I can’t keep up with. I think before Chris used to handle most of them, so it was much easier for me.

    I highly recommend Chris Lightfoot’s obituary by Martin Keegan. Also, Chris’s obituary on Last Word, which was on Radio 4 on 23rd March, is well worth listening to.

    We have a few new members of staff.

    Keith Garrett is working for us now, mainly tending our servers, but he’s also been working on the E Petitions site. He’s trying to bully us into documenting all our internal processes so it’s easier for new people.

    Heather Cronk has started working for us in the US, evangelising PledgeBank. This is funded by the Omidyar Foundation. As well as getting the word out about PledgeBank in the states, this is extra good for us, as she’s forcing us to give PledgeBank the TLC that it deserves.

    Deborah Kerr is now doing customer support for us part time. She’s been busy with Neighbourhood Fix-it which has had lots of traffic and attention the last few weeks.

    OK, back to the present.

    Last week Ben Campbell and I gave a seminar at Technology for a Small Nation in Llandudno. We split into two groups, I got people to add Google maps to an HTML page on their website, and Ben got people to call the TheyWorkForYou API from PHP. It went down well, very satisfying to do practical exercises, and answer all the niggling questions (how to install a testing webserver on Windows, how to use FTP) which are the real things that stop people doing what they want to do.

    And Thursday? Just a reminder, unless you’re not apathetic, that there’s an election today.