1. Installing FixMyStreet and MapIt

    A photo of some graffiti saying "SIMPLE"

    Photo credit: duncan on Flickr

    One of the projects we’ve been working on at mySociety recently is that of making it easier for people to set up new versions of our sites in other countries.  Something we’ve heard again and again is that for many people, setting up new web applications is a frustrating process, and that they would appreciate anything that would make it easier.

    To address that, we’re pleased to announce that for both FixMyStreet and MapIt, we have created AMIs (Amazon Machine Images) with a default installation of each site:

    You can use these AMIs to create a running version of one of these sites on an Amazon EC2 instance with just a couple of clicks. If you haven’t used Amazon Web Services before, then you can get a Micro instance free for a year to try this out.  (We have previously published an AMI for Alaveteli, which helped many people to get started with setting up their own Freedom of Information sites.)

    Each AMI is created from an install script designed to be used on a clean installation of Debian squeeze or Ubuntu precise, so if you have a new server hosted elsewhere, you can use that script to similarly create a default installation of the site without being dependent on Amazon:

    In addition, we’ve launched new sites with documentation for FixMyStreet and MapIt, which will tell you how to customize those sites and import data if you’ve created a running instance using one of the above methods.

    These documentation sites also have improved instructions for completely manual installations of either site, for people who are comfortable with setting up PostgreSQL, Apache / nginx, PostGIS, etc.

    Another notable change is that we’re now supporting running FixMyStreet and MapIt on nginx, as an alternative to Apache, using FastCGI and gunicorn respectively.

    We hope that these changes make it easier than ever before to reuse our code, and set up new sites that help people fix things that matter to them.

    Photo credit: duncan

  2. New MapIt Global: An Administrative Boundaries Web Service for the World

    Introducing MapIt Global

    All of us at mySociety love the fact that there are so many interesting new civic and democratic websites and apps springing up across the whole world. And we’re really keen to do what we can to help lower the barriers for people trying to build successful sites, to help citizens everywhere.

    Today mySociety is unveiling MapIt Global, a new Component designed to eliminate one common, time-consuming task that civic software hackers everwhere have to struggle with: the task of identifying which political or administrative areas cover which parts of the planet.

    As a general user this sort of thing might seem a bit obscure, but you’ve probably indirectly used such a service many times. So, for example, if you use our WriteToThem.com to write to a politician, you type in your postcode and the site will tell you who your politicians are. But this website can only do this because it knows that your postcode is located inside a particular council, or constituency or region.

    Today, with the launch of MapIt Global , we are opening up a boundaries lookup service that works across the whole world. So now you can lookup a random point in Russia or Haiti or South Africa and find out about the administrative boundaries that surround it. And you can browse and inspect the shapes of administrative areas large and small, and perform sophisticated lookups like “Which areas does this one border with?”. And all this data is available both through an easy to use API, and a nice user interface.

    We hope that MapIt Global will be used by coders and citizens worldwide to help them in ways we can’t even imagine yet. Our own immediate use case is to use it to make installations of the FixMyStreet Platform much easier.

    Thanks OpenStreetMap!

    We’re able to offer this service only because of the fantastic data made available by the amazing OpenStreetMap volunteer community, who are constantly labouring to make an ever-improving map of the whole world. You guys are amazing, and I hope that you find MapIt Global to be useful to your own projects.

    The developers who made it possible were Mark Longair, Matthew Somerville and designer Jedidiah Broadbent. And, of course, we’re also only able to do this because the Omidyar Network is supporting our efforts to help people around the world.

     

    From Britain to the World

    For the last few years we’ve been running a British version of the MapIt service to allow people running other websites and apps to work out what council or constituency covers a particular point – it’s been very well used. We’ve given this a lick of paint and it is being relaunched today, too.

    MapIt Global is also the first of The Components, a series of interoperable data stores that mySociety will be building with friends across the globe. Ultimately our goal is to radically reduce the effort required to launch democracy, transparency and government-facing sites and apps everywhere.

    If you’d like to install and run the open source software that powers MapIt on your own servers, that’s cool too – you can find it on Github.

    About the Data

    The data that we are using is from the OpenStreetMap project, and has been collected by thousands of different people. It is licensed for free use under their open license. Coverage varies substantially, but for a great many countries the coverage is fantastic.

    The brilliant thing about using OpenStreetMap data is that if you find that the boundary you need isn’t included, you can upload or draw it direct into Open Street Map, and it will subsequently be pulled into MapIt Global.  We are planning to update our database about four times a year, but if you need boundaries adding faster, please talk to us.

    If you’re interested in the technical aspects of how we built MapIt Global, see this blog post from Mark Longair.

    Commercial Licenses and Local Copies

    MapIt Global and UK are both based on open source software, which is available for free download. However, we charge a license fee for commercial usage of the API, and can also set up custom installs on virtual servers that you can own. Please drop us a line for any questions relating to commercial use.

     Feedback

    As with any new service, we’re sure there will be problems that need sorting out. Please drop us an email, or tweet us @mySociety.

  3. Google Summer of Code

    A Day Spent Lying in the Grass, Watching the Clouds, by Elaine Millan

    Ah, summer: walks in the park, lazing in the long grass, and the sound of chirping crickets – all overlaid with the clatter of a thousand keyboards.

    That may not be your idea of summer, but it’s certainly the ways ours is shaping up. We’re participating in Google’s Summer of Code, which aims to put bright young programmers in touch with Open Source organisations, for mutual benefit.

    What do the students get from it? Apart from a small stipend, they have a mentored project to get their teeth into over the long summer hols, and hopefully learn a lot in the process. We, of course, see our code being used, improved and adapted – and a whole new perspective on our own work.

    Candidates come from all over the world – they’re mentored remotely – so for an organisation like mySociety, this offers a great chance to get insight into the background, politics and technical landscape of another culture. Ideas for projects that may seem startlingly obvious in, say, Latin America or India would simply never have occurred to our UK-based team.

    This year, mySociety were one of the 180 organisations participating. We had almost 100 enquiries, from countries including Lithuania, India, Peru, Georgia, and many other places. It’s a shame that we were only able to take on a couple of the many excellent applicants.

    We made suggestions for several possible projects to whet the applicants’ appetite. Mobile apps were popular, in particular an app for FixMyTransport. Reworking WriteToThem, and creating components to complement MapIt and PopIt also ranked highly.

    It was exciting to see so many ideas, and of course, hard to narrow them down.

    In the end we chose two people who wanted to help improve our nascent PopIt service. PopIt will allow people to very quickly create a public database of politicians or other figures. No technical knowledge will be needed – where in the past our code has been “Just add developers”, this one is “Just add data”. We’ll host the sites for others to build on.

    Our two successful applicants both had ideas for new websites that would use PopIt for their datastore, exactly the sort of advanced usage we hope to encourage. As well as making sure that PopIt actually works by using it they’ll both be creating transparency sites that will continue after their placements ends. They’ll also have the knowledge of how to set up such a site, and in our opinion that is a very good thing.

    We hope to bring you more details as their projects progress, throughout the long, hot (or indeed short and wet) summer.

    PS: There is a separate micro-blog where we’re currently noting some of the nitty gritty thoughts and decisions that go into building something like PopIt. If you want to see how the project goes please do subscribe! The Components mailing list is also a good way of staying in touch.

    Google Summer of Code

    Top image by Elaine Millan, used with thanks under the Creative Commons licence.

  4. FixMyStreet’s been redesigned

    FixMyStreet, our site for reporting things like potholes and broken street lights, has had something of a major redesign, kindly supported in part by Kasabi. With the help of Supercool, we have overhauled the look of the site, bringing it up to date and making the most of some lovely maps. And as with any mySociety project, we’d really appreciate your feedback on how we can make it ever more usable.

    The biggest change to the new FixMyStreet is the use of responsive design, where the web site adapts to fit within the environment in which it’s being viewed. The main difference on FixMyStreet, besides the obvious navigation changes, is that in a small screen environment, the reporting process changes to have a full screen map and confirmation step, which we thought would be preferable on small touchscreens and other mobiles. There are some technical details at the end of this post.

    Along with the design, we’ve made a number of other improvements along the way. For example, something that’s been requested for a long time, we now auto-rotate photos on upload, if we can, and we’re storing whatever is provided rather than only a shrunken version. It’s interesting that most photos include correct orientation information, but some clearly do not (e.g. the Blackberry 9800).

    We have many things we’d still like to do, as a couple of items from our github repository show. Firstly, it would be good if the FixMyStreet alert page could have something similar to what we’ve done on http://planningalerts.barnet.gov.uk/, providing a configurable circle for the potential alert area. We also are going to be adding faceted search to the area pages, allowing you to see only reports in a particular category, or within a certain time period.

    Regarding native phone apps – whilst the new design does hopefully work well on mobile phones, we understand that native apps are still useful for a number of reasons (not least, the fact photo upload is still not possible from a mobile web app on an iPhone). We have not had the time to update our apps, but will be doing so in the near future to bring them more in line with the redesign and hopefully improve them generally as well.

    The redesign is not the only news about FixMyStreet today

    As part of our new DIY mySociety project, we are today publishing an easy-to-read guide for people interested in using the FixMyStreet software to run versions of FixMyStreet outside of Britain. We are calling the newly upgraded, more re-usable open source code the FixMyStreet Platform.

    This is the first milestone in a major effort to upgrade the FixMyStreet Platform code to make it easier and more flexible to run in other countries. This effort started last year, and today we are formally encouraging people to join our new mailing list at the new FixMyStreet Platform homepage.

    Coming soon: a major upgrade to FixMyStreet for Councils

    As part of our redesign work, we’ve spoken to a load of different councils about what they might want or need, too. We’re now taking that knowledge, combining it with this redesign, and preparing to relaunch a substantially upgraded FixMyStreet for Councils product. If you’re interested in that, drop us a line.

    Kasabi: Our Data is now in the Datastore

    Finally, we are also now pushing details of reports entered on FixMyStreet to Kasabi’s data store as open linked data; you can find details of this dataset on their site. Let us know if it’s useful to you, or if we can do anything differently to help you.

    Technical details

    For the web developers amongst you – we have a base stylesheet for everyone, and another stylesheet that is only included if your browser width is 48em or above (an em is a unit of measurement dependent on your font size), or if you’re running Internet Explorer 6-8 (as they don’t handle the modern CSS to do this properly, we assume they’ll want the larger styles) using a conditional comment. This second stylesheet has slight differences up to 61em and above 61em. Whilst everything should continue to work without JavaScript, as FixMyStreet has done with its map-based reporting since 2007, where it is enabled this allows us to provide the full screen map you can see at large screen sizes, and the adjusted process you see at smaller resolutions.

    We originally used Modernizr.mq() in our JavaScript, but found that due to the way this works (adding content to the end of the document), this can cause issues with e.g. data() set on other elements, so we switched to detecting which CSS is being applied at the time.

    On a mobile, you can see that the site navigation is at the end of the document, with a skip to navigation link at the top. On a desktop browser, you’ll note that visually the navigation is now at the top. In both cases, the HTML is the same, with the navigation placed after the main content, so that it hopefully loads and appears first. We are using display: table-caption and caption-side: top in the desktop stylesheet in order to rearrange the content visually (as explained by Jeremy Keith), a simple yet powerful technique.

    From a performance point of view, on the front page of the site, we’re e.g. using yepnope (you can get it separately or as part of Modernizr) so that the map JavaScript is downloading in the background whilst you’re there, meaning the subsequent map page is hopefully quicker to load. I’m also adding a second tile server today – not because our current one isn’t coping, it is, but just in case something should happen to our main one – we already have redundancy in our postcode/area server MapIt and our population density service Gaze.

    If you have any technical questions about the design, please do ask in the comments and I’ll do my best to answer.

  5. Technical look at the new FixMyStreet maps

    This post explains how various aspects of the new FixMyStreet maps work, including how we supply our own OS StreetView tile server and how the maps work without JavaScript.

    Progressive enhancement

    During our work on FiksGataMi (the Norwegian version of FixMyStreet) with NUUG, we factored out the map code (for the Perlmongers among you, it’s now using Module::Pluggable to pick the required map) as FiksGataMi was going to be using OpenStreetMap, and we had plans to improve our own mapping too. Moving to OpenLayers rather than continuing to use our own slippy map JavaScript dating from 2006 was an obvious decision for FiksGataMi (and then FixMyStreet), but FixMyStreet maps have always been usable without JavaScript, utilising the ancient HTML technology of image maps to provide the same functionality, and we wanted to maintain that level of universality with OpenLayers. Thankfully, this isn’t hard to do – simply outputting the relevant tiles and pins as part of the HTML, allowing latitude/longitude/zoom to be passed as query parameters, and a bit of maths to convert image map tile clicks to the actual latitude/longitude selected. So if you’re on a slow connection, or for whatever reason don’t get the OpenLayers JavaScript in some way, the maps on FixMyStreet should still work fine. I’m not really aware of many people who use OpenLayers that do this (or indeed any JavaScript mapping API), and I hope to encourage more to do so by this example.

    Zooming

    We investigated many different maps, and as I wrote in my previous blog post, we decided upon a combination of OS StreetView and Bing Maps’ OS layer as the best solution for the site. The specific OpenLayers code for this (which you can see in map-bing-ol.js is not complicated (as long as you don’t leave in superfluous commas breaking the site in IE6!) – overriding the getURL function and returning appropriate tile URLs based upon the zoom level. OpenLayers 2.11 (due out soon) will make using Bing tiles even easier, with its own seamless handling of them, as opposed to my slight bodge with regard to attribution (I’m displaying all the relevant copyright statements, rather than just the one for the appropriate location and zoom level which the new OpenLayers will do for you). I also had to tweak bits of the OpenLayers map initialisation so that I could restrict the zoom levels of the reporting map, something which again I believe is made easier in 2.11.

    OpenStreetMap

    Having pluggable maps makes it easy to change them if necessary – and it also means that for those who wish to use it, we can provide an OpenStreetMap version of FixMyStreet. This works by noticing the hostname and overriding the map class being asked for; everything necessary to the map handling is contained within the module, so the rest of the site can just carry on without realising anything is different.

    OS StreetView tile server

    Things started to get a bit tricky when it came to being ready for production. In development, I had been using http://os.openstreetmap.org/ (a service hosted on OpenStreetMap’s development server) as my StreetView tile server, but I did not feel that I could use it for the live site – OpenStreetMap rightly make no reliability claims for it, it has a few rendering issues, and we would probably be having quite a bit of traffic which was not really fair to pass on to the service. I wanted my own version that I had control over, but then had a sinking feeling that I’d have to wait a month for something to process all the OS TIFF files (each one a 5km square) into millions and millions of PNG tiles. But after many diversions and dead ends, and with thanks to a variety of helpful web pages and people (Andrew Larcombe’s guide [dead link removed] to his similar install was helpful), I came up with the following working on-demand set-up, with no pre-seeding necessary, which I’m documenting in case it might be useful to someone else.

    Requests come in to our tile server at tilma.mysociety.org, in standard OSM/Google tile URL format (e.g. http://tilma.mysociety.org/sv/16/32422/21504.png. Apache passes them on to TileCache, which is set up to cache as GoogleDisk (ie. in the same format as the URLs) and to pass on queries as WMS internally to MapServer using this layer:

    [sv]
    type=WMS
    url=path/to/mapserv.fcgi?map=os.map&
    layers=streetview
    tms_type=google
    spherical_mercator=true

    MapServer is set up with a Shapefile (generated by gdaltindex) pointing at the OS source TIFF and TFW files, meaning it can map tile requests to the relevant bits of the TIFF files quickly and return the correct tile (view MapServer’s configuration – our tileserver is so old, this is still in CVS). The OUTPUTFORMAT section at the top is to make sure the tiles returned are anti-aliased (at one point, I thought I had a choice between waiting for tiles to be prerendered anti-aliased, or going live with working but jaggedy tiles – thankfully I persevered until it all worked :) ).

    Other benefits of OpenLayers

    As you drag the map around, you want the pins to update – the original OpenLayers code I wrote used the Markers layer to display the pins, which has the benefit of being simple, but doesn’t fit in with the more advanced OpenLayers concepts. Once this was switched to a Vector layer, it now has access to the BBOX strategy, which just needs a URL that can take in a bounding box and return the relevant data. I created a subclass of OpenLayers.Format.JSON, so that the server can return data for the left hand text columns, as well as the relevant pins for the map itself.

    Lastly, using OpenLayers made adding KML overlays for wards trivial and made those pages of the site much nicer. The code for displaying an area from MaPit is as follows:

        if ( fixmystreet.area ) {
            var area = new OpenLayers.Layer.Vector("KML", {
                strategies: [ new OpenLayers.Strategy.Fixed() ],
                protocol: new OpenLayers.Protocol.HTTP({
                    url: "/mapit/area/" + fixmystreet.area + ".kml?simplify_tolerance=0.0001",
                    format: new OpenLayers.Format.KML()
                })
            });
            fixmystreet.map.addLayer(area);
            area.events.register('loadend', null, function(a,b,c) {
                var bounds = area.getDataExtent();
                if (bounds) { fixmystreet.map.zoomToExtent( bounds ); }
            });
        }

    Note that also shows a new feature of MaPit – being able to ask for a simplified KML file, which will be smaller and quicker (though of course less accurate) than the full boundary.

  6. PetitionYourCouncil.com: making local council petitioning easier

    PetitionYourCouncil.com from mySociety

    Local petitions can be highly effective, and we think that making them easier to create is in the public interest. Many councils have petitions facilities buried deep within their websites,  most often, very deeply. In fact it brings to mind Douglas Adams’ quote about important council documents being “on display on the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard’”.

    Our most recent mini-project is an attempt to make it as easy as possible to find your local council’s e-petitioning site, if they have one. PetitionYourCouncil.com (you’ll notice we stuck to our tried and tested format for site names, there) is a way of finding every council e-petitioning website we know about.

    Our original motivation for building the site was that we, along with other suppliers, have supplied online e-petitioning sites to numerous councils ourselves – it’s one of the ways in which we fund our charitable activities. Having delivered these sites, we later noticed that many of them are left under-used and in some cases, not used at all: only because people don’t know about them. We hate to think of councils spending money on a splendid resource that could be improving democratic processes for their citizens – and those citizens never knowing that they exist. In particular, we owe Dave Briggs thanks for pushing us into action with this blog post.

    And yes, in case you’re wondering, PetitionYourCouncil links to every council petitions site, not just the ones we made.

    The site was built by mySociety developer Edmund von der Burg using Django, jQuery, Google maps and Mapit, and like most mySociety projects, it’s open source. There’s a bit more detail on the About page. Please do try it out, and let us know what you think.

  7. Helping voters in the devolved elections

    As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.

    To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).

    Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.

    Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.

    There don’t appear to be any central official lists of candidates in these elections. http://www.amnesty.org.uk/content.asp?CategoryID=11993 has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations :)

    Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15″ to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.

  8. New features on MaPit

    We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

    • Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
    • As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
    • From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
    • Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
    • As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
    • Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

    Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

  9. Mapping points and postcodes to areas

    I’m very pleased to announce that mySociety’s upgraded point and postcode lookup service, MaPit, is public and available to all. It can tell you about administrative areas, such as councils, Welsh Assembly constituencies, or civil parishes, by various different lookups including name, point, or postcode. It has a number of features not available elsewhere as far as I know, including:

    • Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! :-)
    • Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
    • Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
    • History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

    If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

    Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

  10. April Fools’ Day Council changes

    They could perhaps have picked a better day, as it was quite serious – at the stroke of midnight on the 1st of April, 37 district councils and 7 county councils in England ceased to exist, replaced by 9 new unitary authorities. This means people in Durham, Northumberland, Cornwall, Shropshire, Wiltshire, Chesire, and Bedfordshire only have one principal local authority to deal with now. The Wikipedia article on the changes has more information on the background to this change.

    Obviously this meant some work for WriteToThem and FixMyStreet, both of which require up-to-date local council information. Our database of voting areas, MaPit, has “generations”, so we can keep old areas around for various historical purposes. So firstly, I created a new generation and updated all the areas that weren’t affected to the new generation. Next, six of the new unitary authorities (all the counties except Cheshire and Bedfordshire, plus Bedford) share their boundaries and wards with the coterminous councils they’re replacing, so for them it was a simple matter of updating those councils to be unitary authorities.

    That left Bedfordshire and Cheshire. I created areas for the three new councils (Cheshire West and Chester, Cheshire East, and Central Bedfordshire), and transferred across the relevant wards from the old county councils – basically a manual process of working out the list of correct ward IDs. april2009-update.sql has the gory SQL details if you care.

    WriteToThem was now dealt with, but FixMyStreet needed a little more work. The councils that no longer existed had understandably disappeared from the all reports table, so I had to modify the function that fetches the list of councils to optionally return historical areas so they could be included. And lastly, FixMyStreet needs a way of mapping a point on a map to the relevant council. For this, it needs to know the area covered by a council, which was missing for the new authorities I’d manually created. Thankfully, each of the three new authorities are made up of the areas of either 2 or 3 district councils (e.g. Cheshire East is the area covered by Congleton, Macclesfield, and Crewe and Nantwich), so I just had to write a script that stuck those areas together to create the area of the new council. april2009-construct-new.pl. It all seems to work, and I’m sure our users will be in touch if it doesn’t :)

    So goodbye to Alnwick, Bedfordshire, Berwick-upon-Tweed, Blyth Valley, Bridgnorth, Caradon, Carrick, Castle Morpeth, Cheshire, Chester, Chester-le-Street, Congleton, Crewe and Nantwich, Derwentside, Durham City, Easington, Ellesmere Port and Neston, Kennet, Kerrier, Macclesfield, Mid Bedfordshire, North Cornwall, North Shropshire, North Wiltshire, Oswestry, Penwith, Restormel, Salisbury (which is getting a new town council), Sedgefield, Shrewsbury and Atcham, South Bedfordshire, South Shropshire, Teesdale, Tynedale, Vale Royal, Wansbeck, Wear Valley, and West Wiltshire. RIP.