mySociety blog » archive for 2005

Things have been quiet here recently, but are now getting busy again. Tom’s back from America, Chris is back from holiday, I’m better after being ill for most of last week.

Earlier in the week we finally managed to load new county boundaries into MaPit. So WriteToThem once again has county councils working. Please try it out with your postcode. Let us know of any problems.

This required lots of work from Chris, because a new version of BoundaryLine (from Ordnance Survey) has not yet been released with the updated boundaries. He’s done it using lists of the district council wards which make up the county electoral divisions.
These lists were taken from the Statutory Instruments. This has covered most postcodes, but there are still some where the boundaries were specificed in text (walk along this river etc.) rather than wards. And we don’t have those.

The last couple of days I’ve been turning on lots of things to automate updating of WriteToThem. A cron job now grabs new data on councillors from GovEval once a day, and merges their changes with any changes we’ve made.

It’s automatically emailing GovEval with user submitted corrections to councillor data (the “Have you spotted a mistake in the above list?” link on WriteToThem). Hopefully this will create a virtuous feedback loop of ever improving data quality goodness. Or at least let us keep up with council by-elections without having to lift a finger.

Finally I’ve made it send a mail once a week to the mailing list where WriteToThem admins (mostly volunteers) hang out. This describes what needs doing – such as missing contact details to gather, or messages in the queue which need human attention.

Next up, wiring up the new screenscrapers Richard and Jonathan contributed last week, so the Welsh and London Assemblies automatically update…

Hello everyone,

This Tuesday I am heading to North America for a couple of weeks to spread the news about PledgeBank and mySociety. I’d love to meet as many people as possible who are interested in mySociety, or who might become so, and I’m travelling all over the continent to do so.

I’m visiting now because we’ve quietly been doing lots of work to make our most popular site yet,PledgeBank, a truly international site with the ability to create, search and subscribe to pledges in any one of over 2 million cities, towns and villages around the world. We’ve also engineered the site so that volunteer translators can help us translate the whole site into other languages just by editing one easy text file. This is the first time a mySociety project has been truly international, and we’re keen to share our free services beyond the borders of the UK.

We’ve not done any publicity in the US yet because we know from experience that it is was an essential part of the highly successful UK launch that we had a variety of strong, clear and practical pledges ready to go from day one. Discussing potential US and Canada based pledges and building informal partnerships is what this is all about.

So, if you are in the US or Canada, or you know someone who is and who would be interested in meeting, please get in touch with tom@mySociety.org

A very quick post to announce the launch of a public interface to our Gaze web gazetteer service. The motivation behind Gaze is collecting location information from users without using maps (a clunky approach with poor accessibility and licensing problems) or postcodes (which do not have universal coverage and have privacy issues as well as licensing problems). Instead the idea is to use place names to identify locations, even in the presence of ambiguity, alternate names, etc. We do this by providing a search service over a large gazetteer (2.2 million places and 3 million names), and supplying additional contextual information to disambiguate common place names. The API is very simple, with one major function and two other supporting ones.

Anyway, without further ado, here is the API. Internally we use one based on RABX, but we’ve done a special “RESTful” API for everyone else. All requests should be HTTP GETs; all parameters must be in UTF-8; and all responses are in UTF-8 plain text or comma-separated values. All calls should be passed to the URL,

http://gaze.mysociety.org/gaze-rest

selecting a particular function by specifying the HTTP parameter f, for instance

http://gaze.mysociety.org/gaze-rest?f=get_find_places_countries

Available functions are:

get_country_from_ip

Parameters:

ip

IPv4 address of a host, in dotted-quad format

Guess the country of location of a host from its IP address. The result of this call will be an ISO country code, followed by a line feed; or, if it was not possible to determine a country, a line feed on its own.

get_find_places_countries

No parameters.

Return the list of countries for which the find_places call has a gazetteer available. The list is returned as a list of ISO country codes followed by line feeds.

find_places

Parameters:

country

ISO country code of country in which to search for places

state

state in which to search for places; presently this is only meaningful for country=US (United States), in which case it should be a conventional two-letter state code (AZ, CA, NY etc.); optional

query

query term input by the user; must be at least two characters long

maxresults

largest number of results to return, from 1 to 100 inclusive; optional; default 10

minscore

minimum match score of returned results, from 1 to 100 inclusive; optional; default 0

Returns in CSV format (as defined by this internet draft) with a one-line header a list of the following fields:

name

name of the place described by this row

in-qualifier

blank, or the name of an administrative region in which this place lies (for instance, a county)

near-qualifier

blank, or a list of nearby places, separated by commas

latitude

WGS-84 latitude of place in decimal degrees, north-positive

longitude

WGS-84 longitude of place in decimal degrees, east-positive

state

blank, or containing state code for US

score

match score for this place, from 0 to 100 inclusive

Enjoy! Questions and comments to hello@mysociety.org, please.

Update: we’ve now added the facilities for discovering population densities and “customary proximity” (as discussed in this post) to Gaze. The additional APIs are documented here.

mySociety yesterday launched a pair of Back o’ The Envelope projects based on Google Maps.

Placeopedia.com — Connect Wikipedia articles with the places they represent

YourHistoryHere.com — Share local and geographic history and trivia.

There are a few things to say about both projects:

1 – As is normal with mySociety projects the code for these projects (excepting Google maps) is open source. We hope that by providing a ready-made annotation system, people will find it easier to make their own publicly-authored layers of information.

2 – Both sites syndicate their data under open source licenses, and in a location-queryable fashion. This is really important, as it allows for all types of nice local history to be syndicated to tourism sites, local community discussion boards, blogs and so on.

3 – We’re calling them ‘Back o’ the Envelope’ to contrast them to the big, polished and time consuming projects we run like PledgeBank.com and WriteToThem.com.

Hello World,

mySociety and TheyWorkForYou are holding another of our not-so-legendary drinks evenings next Thursday, 15th September. The venue is downstairs at the Narrowboat in Islington from 7PM.

http://maps.google.com/maps?q=N1+8PZ&spn=0.018184,0.032616&t=k&hl=en

The idea is to give people updates on our projects, ply them with drinks, and then trap them in nets and drag them back to our ‘volunteers and friends’ dungeon.

We must keep an eye on numbers, so please let us know if you’re coming, or Frowning Will Ensue.

all the best,

Tom Steinberg

Lots of countries gradually loading into one of our servers. There’s 220Mb of data comprising 227 countries, with about 5,000,000 places altogether. With a global population of about 6 billion, that means the average “place” has 1,200 people living in it. For each place we have the latitude and longitude. (All this data comes from the US military)

Try it out by signing up for a local alert in any country. Let us know if you find any bugs, or have any problems or suggestions to make. Also, if you want access to this gazetteer as a web service, send us a mail.

Currently it’s up to Uruguay, it’ll be a bit longer before we’ve finished the alphabet. It takes quite a while partly because of the volume of data and indices being built, partly because for places with the same name as each other it hunts for nearby towns to disambiguate, and partly because we didn’t optimise the perl script. It won’t run very often.

Back from holiday in Edinburgh last week, where amongst other things I went to see NTK-recommended Coelcanth about an alternate reality England which didn’t get deforested, and where tree climbing is a national sport. It was excellent.

Currently my goal is to do stuff for the launch of PledgeBank in the US. At the moment I’m adding signup for local alerts in other countries. Then there are quite a few small URL and text tweaks do to do. For example, displaying US and global pledges only on the front page of the US site.

The hard question about all this is how to present all the slices of PledgeBank data that there are. By country, by local place, by language, by category. I’m completely ducking it by just worrying about there being a UK and a US site. This is often a good approach when your head is tangled by over complex abstract software requirements – focus in and solve the part of the problem that’s actually needed next.

Plenty more trees to climb later.

Today, I have been working on YourConstituencyMailingList (which will thankfully not be its final title, I keep spelling Constituency wrong myself :) ). The database now has knowledge of which member of a YCML is the MP, and records comments made by that person, so that they can appear differently in the thread of comments. Also comment email alerts for when new comments are posted to a particular message; I guess I should add RSS feeds of comments, messages, and so on too. A Welsh translation of PledgeBank is coming along nicely, hope to have that up as soon as it’s ready.

PledgeBank has had a great first month-and-a-half with over 20,000 signatures on over 500 pledges. The task has now started of trying to get word about PledgeBank out to those little grassroots organisations that would really benefit from it.

To do this, we need your help. We’ve created a presentation (PDF) specifically aimed at small grassroots organisations, and we’d like you to let organisations near you know about it. All you have to do is google for charties and community groups in your area, and send them the link – why not lend us a hand right this minute?

So, it’s back to electoral geography for me, this time to get the new county and county electoral-division boundaries live on WriteToThem. This is a prerequisite for getting mail to county councillors working again after the election on May 5th, so we’re already three months behind the times. But more generally, electoral boundaries are revised all the time to account for changes of population within each ward, constituency and so forth; and at most (local and national) elections some set of boundary changes takes effect. So to keep WriteToThem running we need to incorporate such updates routinely.

The way we handle electoral geography in general is to start with Ordnance Survey’s Boundary Line product, which, for each administrative or electoral area in Great Britain gives a polygon identifying that region. We then take a big list of all the postcodes in Britain (CodePoint) and figure out which polygons they lie in. Then when somebody comes along to WriteToThem and types in their postcode, we can figure out which ward, constituency etc. they are in, and tell them appropriate things about their representatives. (Technically this is a lie, of course, because postcodes represent regions, not points — we use the centroids of those regions — and each such region isn’t guaranteed to lie either wholly within or without all electoral and administrative regions. Unfortunately there isn’t a lot we can do about this beyond throwing our hands up and saying “oops, sorry”, so that’s what we do.)

As an aside, outside Great Britain — that is, in Northern Ireland, we don’t have the same sort of data so instead we rely on another field in the CodePoint data which gives, for each postcode centroid, the ONS ward code for the ward in which that point lies. From that ward code you can find the enclosing local authority area, local electoral area — in Northern Ireland local councils are elected by STV over multimember regions, rather than by first-past-the-post as in Great Britain — and constituency. Happily it turns out that all of those other regions are composed of whole numbers of wards; this happy state of affairs does not necessarily prevail elsewhere.

Now, twice a year, a new edition of Boundary Line is issued, taking account of recent changes in electoral geography. Usually this happens in May and October, though the schedule has been known to slip. In principle this should be easy to deal with: load up the new copy of Boundary Line, pass all the postcodes through it, and hey presto.

Life, of course, is rarely that simple, and this isn’t one of those occasions. When the boundaries of a region don’t change between one year and the next, we don’t want to make any alteration to that region in our database (which uses ID numbers to identify each area). More specifically, when a new revision of Boundary Line comes along, we want to ensure that — let’s say — Cambridge Constituency in the new revision is identified with Cambridge Constituency in the old version. Now, in principle, this should be easy, because each area in the data set, in the words of the manual,

… carries a unique identifier AI; this is the same identifier that was supplied in the previous specification of Boundary- Line. The same AI attribute is associated with every component polygon forming part of an administrative unit, irrespective of the number of polygons.

Now, the first time that we did this, we worked from a copy of the Boundary Line data supplied in the form of “ShapeFiles” (a format used in various proprietary GIS systems, and with which our local government partners were able to supply us without having to order it specially from Ordnance Survey). Unfortunately in the ShapeFile version, the allegedly unique administrative area IDs were, in fact, not unique. After discussion with Ordnance Survey it was concluded that this was a problem which affected the translation of the data from NTF (“National Transfer Format”, their own preferred format) into ShapeFile; and that the problem would be fixed in the next release.

So, taking no chances, we decided we’d work from the NTF format in future, since that seems to be closer to the authoritative source of the data, and anyway the ShapeFile format isn’t at all well-documented (for instance, many of the field names for the metadata about each area differ from those described in the manual for Boundary Line). So I’ve written code to parse the (slightly bonkers, natch) NTF files and modified our import scripts to use this code, with a view to then being able to keep up-to-date with future boundary revisions without too much trouble.

You will not be surprised, therefore, to hear that this has not worked out exactly as planned. Unfortunately it appears that the May 2005 NTF release of Boundary Line suffers exactly the same problems of non-uniqueness as did the previous ShapeFile release. So unless some cleverer solution presents itself, I’ll have to revive the hack we intended to use with the ShapeFile data — try to construct unique IDs for areas from their geometry, and hope that the exact coordinates of the polygon vertices for unchanged areas do not change between revisions. We shall see. But right now I’m mostly worrying about why my parser script runs out of memory on my 1GB computer after reading a couple of hundred megs of input data.

Today I’m fiddling with the pledge creation form again, to fit all the new location stuff Chris has done in better. We’ve swapped over from last week, like tag team wrestlers. Chris is busy getting the boundary line data for new County boundaries into WriteTothem.

We’re looking for someone to intern with us for what’s left of this grim grey summer. I’ve just sent this job description to the Interns network, but I thought I’d paste it here to see if it pops up in anyone’s RSS feeds. Come and work with us – you know you want to!

———–

mySociety is a charitable organisation which has built some of the best known political web tools in the country, such as ww.WriteToThem.com , an award winning service that lets anyone in the
UK write to any of their elected representatives easily and for free. We build politically neutral tools that give people simple but tangible benefits to the political and civic sides of their lives.

We are a virtual organisation which combines some paid developers with volunteer developers, moderators and designers, spread across the UK. We are seeking an intern to do non-technical publicity and partnership building work over this summer, mainly on two projects. The first is
the recently launched PledgeBank.com, and the second a service aiming to change the way in which people hear from their MPs (www.mySociety.org/ycml).

As a virtual organisation we have no fixed offices, and mostly work from our own homes. As such, being an intern with us would be a bit different – instead of a daily commute, we’d meet occasionally, often
in London and sometimes in Cambridge. You need to have your own internet access and decent day-to-day computer skills, as we work and communicate almost exclusively over the Internet. There may also be opportunities to do work with our sister project TheyWorkForYou.com ,
the hugely popular annotated Hansard project.

We’re a flexible, easygoing organisation that doesn’t get out of bed very early. If you want to be involved with a different and increasingly important part of politics and the Westminster village,
please get in touch.

CVs and covering emails to Tom Steinberg at interns@mysociety.org

I guess I should post the photos from our week in Wales. An excellent time was had by all, and I haven’t had to eat since. ;) We started testing PledgeBank in Portuguese (many thanks to Fabiana for the translation), worked on YCML and what to do with international pledges, and more, all whilst sharing a 26.4k (if we were lucky) dial-up connection. I also began getting to grips with juggling, something I’ve always wanted to do but never got around to. :-)

At the moment Tom is hiking across Welsh mountains for three days towards Llanwdra. Chris is in Devon, and is heading towards the same farm by train tomorrow afternoon. Matthew is quite nearby in Cheltenham, and will meet Chris at Newport station. I’m leaving early tomorrow from Cambridge, and they’ll get on the same train as me to Swansea.

We’re all converging on Ed’s farm in the south of Wales, where we’re spending the week working together in one office for a change. Now I just have to think of lots of thorny problems that need long rambling brain-storming conversations, so that office can be in the hills as much as possible…

Meanwhile PledgeBank’s largest ever pledge has nearly succeeded.

Today is the sort of muggy, hot day, which makes you believe that old story. The one about the British having an empire because of the bad weather. They could actually concentrate, plot and do under the clouds, rather than just dozing in a field, lazing in the sun. How true it is I’m not sure, but it definitely seems quiet everywhere today. I’ve got fewer emails than ever, and I’m still amazed I’ve managed to think all day at all.

I’m leaving PledgeBank behind for a bit, as most of the obvious bugs are fixed, and features added. Earlier this week, and at the end of last, I put in local email alerts. Now if you sign up to local alerts you’ll get mailed at most once a day when a new pledge is made within 25km of you. PledgeBank really is having a proper beta test, with about 100,000 visitors since launch a month ago. This has been fantastic, the feedback has made it a very stable and hopefully more usable site. Good software is software which is used lots, with a virtuous feedback loop from users into making it better.

So this morning I went back to WriteToThem. We still haven’t updated county councils after the elections. So I’ve been writing some three way merging code, to import changed council data from GovEval. It is merging, rather than just loading, because we’ve also altered the data. This was to make the ward names match up consistently with those from our mapping data.

Things are working pretty well for councils where neither GovEval or us have added or deleted any representatives. There are unique identifiers, and very few clashes. Only one throughout all the data which has been caused by one of us editing the ward name, the other editing the councillor name, and so the repersentative really being neither.

The hard case is additions and deletions. Obviously, I’d like to keep our work of mySociety additions. But this is no use, as eventually one day they’ll be wrong. The councillor we thought should be there no longer will be. So how do I detect when?

One thing that would be nice to do with PledgeBank is allow people to put funds into escrow, and then when a money related pledge succeeded or failed the money would be passed on, or back to the donor. Does anyone have any ideas about how to start doing this? We know you can using paypal, but it seems to ‘Nethead’ to be mainstream, and has some major drawbacks. Does anyone know of a way we could do it using simple credit card details?

For pretty well the last week now I’ve been adding various new messages and alerts to PledgeBank. First up were a set of emails which go to creators. Two of these are “chivvying” ones, which remind the creator to do publicity, and give them some advice.

Quite a few people make pledges and don’t realise that they have to publicise them themselves. Ambient traffic to PledgeBank will get you some sign ups, especially at the start of last week when we had lots of media attention, but pledge creator’s also have to do some of their own work. So we chivvy the creator once after a few days if they don’t have many sign ups.

Secondly, even if you’ve got some publicity and some signups, sometimes you just haven’t got a high enough rate to reach your target on time. I’ve added some code which checks signup rate on average over the last week (for pledges at least a week old). If things were to carry on at that rate until the deadline, and the pledge still wouldn’t succeed, it sends an email with ideas about publicity/marketing of your pledge. Now we just have to hope this is useful rather than annoying…

Another thing we noticed is that lots of pledge creators hadn’t sent a message to their signers after their pledge succeeded. Admittedly this was partly because of a bug in our announce message code (there was no action= in one of the forms, which caused problems on some browsers). So I also added a successful pledge reminder email which is sent a few days after success.

All the stuff above was done using the nice “message” tables which Chris made a while ago. They make it easy to send messages which go out only once, and to creators, signers and late signers as appropriate. However, for the next set of emails, some new code was needed. These are optional “alerts”. For example, to email you when new comments are added to a pledge, and when there is a new local pledge in your area. So, the last day or two I’ve been coding that.

I’m just filling out my own questionnaire about the direction in which mySociety and UKCOD should head from now on. Interesting times, and I’m looking forward to seeing what everyone else’s answers are like. If you’ve any thoughts on overall strategy, please post them here!

So, I left regular readers on a geographical cliffhanger last week in my search for a decent gazetteer of the whole world which we can use to let pledge creators tell us where their pledges apply to (and to let people to search for pledges near them). No doubt you expect me to now say that I’ve done this and that this marvellous new feature is now up and running on PledgeBank.

Sadly not.

The best data I’ve been able to find is the GEOnet Names Server dumps from the US Department Of Knowing What Places Are Called (or “National Geospatial Intelligence Agency”, as they call themselves). They maintain a big database of all the places in the world (except for the United States, which task falls to the US Geological Survey), mostly, as I understand it, for military applications. Presumably the idea here is that if some US soldier finds himself sharing Hicksville, Iraq, with something he wants to blow up, he can whip out his satellite-telephone, dial 1-800-US-AIR-FORCE (“You Call: We Bomb”) and, once he’s outwitted the phone menu and call-center staff, can arrange an air-strike without having to know anything tedious like his coordinates (“I’m sorry, could you repeat that, please. Do you mean Hicksville, Iraq, or Hicksville, Alabama?”).

Now, when I last looked at this data, it was full of random and quite significant errors (~5km, for the locations of villages in England — much larger than we’d expected from the coordinate transforms from WGS84 to OSGB36 and the National Grid). For its intended application I suppose this just comes down to a question of how big a bomb you’re prepared to use; for PledgeBank, this is irritating, but not fatal, since all we want is approximate location data which is good enough to let users look for things in the same general area as them.

(There are alternative gazetteers but those I’ve looked at are either proprietary, derived from the GNS data, much smaller than GNS, share its problems while adding new ones of their own, or several of the above. That said, I remain open to alternative suggestions.)

So, the plan is simple: grab all the GNS data (717MB of it), import it into a big database, then let people select their country and type in their (nearest) town, and look up coordinates from that. What could be simpler? It turns out that they’ve even abandoned their rather quaint practice of inventing their own characters sets for everything, and now use UTF-8 for (most of) the fields in the database.

Unfortunately, at this point there’s a more serious problem. If somebody in the UK, say, types “Cambridge” as their location, then probably they’re talking about Cambridge, Cambridgeshire; but there’s a small chance that they might be talking about Cambridge, Gloucestershire (population ~1,700), and we’d look like total muppets if we confused the two. Generally, place names have a habit of nonuniqueness; for instance in the GNS data for the UK we have,

Now, ideally we’d disambiguate these by asking which one of those they meant, using the name of the enclosing administrative region or some other piece of information the user might be expected to know as a qualifier. Sadly, though GNS nominally has this kind of hierarchical structure (see the ADM1 and 2 fields in this list), in practice many placenames are coded without any information on enclosing geographical region, but with ADM1 set to, for instance, “00 — United Kingdom (general)”.

(As an aside, we don’t actually need this stuff for the UK particularly, because we can ask users for their postcodes and do coordinate lookups from that. But the reason I’m starting by looking at the UK part of the gazetteer is that I know the UK’s geography better than that of, say, Congo or France or somewhere, so it’s easier to see what processing steps are required. Plus, privacy-conscious users may prefer to name a town rather than give their actual postcode, since this makes it much harder for us to lock on to them with our orbiting mind-control LASER satellites.)

So, my current job is to invent some plausible heuristic for annotating nonunique place-names, either by trying to guess the administrative region in which they live (probably not usually practical) or adding other qualifiers such as “near Gloucester” or whatever. I suspect this won’t work all that well, but it only has to be good enough. After all, I’m not going to be using the results to bomb anyone….

In the three days since PledgeBank launched we’ve had lots of people from around the world asking if they could help translate the site into other languages. We’d love for this to happen, but in order to do so the site needs to be adapted to work in different tongues. The system we plan to use is gettext, and we’re looking for a volunteer to take a lead on making this happen. So if you’ve direct experience with gettext, or you think you could bend it to your will, please get in touch.