Matthew’s just updated ScenicOrNot, the little game that we built to provide a ‘Scenicness’ dataset for Mapumental, to include a data dump of the raw data. The dump will update automatically on a weekly basis, but currently it contains averaged scores for 181,188 1*1km grid squares, representing 83% of the Geograph dataset we were using, or 74% of all the grid squares in Great Britain. It is, in other words, really pretty good, and, I think, unprecedented in coverage as a piece of crowd sourced geodata about a whole country.

It’s available under the Creative Commons Attribution Noncommercial 3 Licence, and we greatly look forward to seeing what people do with it.

Stamen: Talented AND gorgeous

mySociety would never have been able to make Mapumental in the way we did if it wasn’t for the help of San Franciso-based geovisualisation gurus Stamen. They came up with the brilliant idea of sliders instead of static contours lines, they built the flash front end, and, crucially, they helped make sure all the contours had just the right degree of splodginess for a satisfyingly splodgy user experience.

Big thanks, therefore go to Michal Migurski, Shawn Allan, Tom Carden and the rest of the Stamen team who helped us get this far – we look forward to working much more with them in the future.

We’ve been hinting for a while about a secret project that we’re working on, and today I’m pleased to be able to take the wraps off Mapumental. It’s currently in Private Beta but invites are starting to flow out.

Built with support from Channel 4’s 4IP programme, Mapumental is the culmination of an ambition mySociety has had for some time – to take the nation’s bus, train, tram, tube and boat timetables and turn them into a service that does vastly more than imagined by traditional journey planners.

In its first iteration it’s specially tuned to help you work out where else you might live if you want an easy commute to work.

Francis Irving, the genius who made it all work, will post on the immense technical challenge overcome, soon. My thanks go massively to him; to Stamen, for their lovely UI, and to Matthew, for being brilliant as always.

Words don’t really do Mapumental justice, so please just watch the video 🙂 Update: Now available here in HD too

Also new: We’ve just set up a TheyWorkForYou Patrons pledge to help support the growth and improvement of that site. I can neither confirm nor deny that pledgees might get invites more quickly than otherwise 😉

According to some, today is the day when the world is going to be saved. Not sure if I agree with that, but recognise that the inauguration of Barack Obama is of some importance :-)). And, naturally, so is the work carried out by mysociety.org. I therefore thought today would be a good time to point to some of its achievements in 2008.

So where to begin? Well, personally I think the work with the Commuting Time Maps is worth mentioning. Developed in collaboration with the Department of Transport it enables users to work out commuting distances from one point to another. This is arguably very useful information if you are house hunting, looking for a new office or if you are an estate agent wanting to provide clients with that extra information.

Or how about picking up an award for FixMyStreet at the SustainIT eWell-Being Awards. The judges said it was “[a]n excellent example of an independent website which empowers the general public in their dealings with their local council. It is a relatively simple application, yet highly effective and replicable.” Very well done indeed.

I know I have mentioned it before, but an obvious achievement is for the charity to have stayed alive and kicking for five years. The main man behind mySociety.org, Tom Steinberg, was around the time of the anniversary featured in an article in the Guardian. Check it out for some more in-depth information about Tom and the rise of mySociety.org!

Full details of all the achievements and general ongoings in 2008 can be found on the blog, especially the successes and 2008 pages.

This project became Mapumental. Please visit that site for details of our travel-time maps services.
The work was funded and supported by the Department for Transport.

See also: the main travel-time maps report.

——————

Our newly released travel time maps are currently shooting round the internet. It was great fun making them, and you might like to have a go too – there are plenty of public datasets you could overlay on the same base maps, using the same flash app (source code). There are a few notes about how we made them on the page itself, and the associated real time page. For a far more interesting view of the development process, read Tom Carden from Stamen’s account.

The most interesting blog post I’ve seen to come from this is Whitehall staff have no life by Simon Dickson, who was inspired by the maps to think about the destruction of social capital caused by commuting. “Whitehall staff on all but the highest salaries can’t expect to live anywhere near their work, and hence can’t expect to have any kind of a social (capital) life.”

This project became Mapumental. Please visit that site for details of our travel-time maps services.
The work was funded and supported by the Department for Transport.

——————

You may remember that back in 2006 mySociety published some maps showing how long it took to commute places via public transport.

We’ve just made some more which have some lovely new features we reckon you’ll probably like a lot.

If you’d like to see more maps like this in your area, please ask your local transport authority to get in touch with us, or nudge these people 🙂

PS As always, Francis Irving remains a genius.

This project became Mapumental. Please visit that site for details of our travel-time maps services.
The work was funded and supported by the Department for Transport.

See also: the main travel-time maps report.

——————

Introduction

After seeing our travel time maps (original report from 2006
and the current report), the next thing that everyone asks is “can I have one for my journey to work?”. We investigated in detail two ways of providing real time versions of these maps, and we suggest a third way
which requires new data.

Better screen scraping

Suppose we want to create a time contour map of journey time to work by 9am for any selected postcode in the UK. How long this takes depends on where in the country the map is centred. In particular, it depends on the density of bus stops and railway stations. First, some background explanation.

Basic screen scraping

“Basic screen scraping” is the technique which we used to generate the example travel time maps which you’ve seen. This was done by “screen scraping” the Transport Direct website. That is, writing a computer program which operated the normal web interface of the site to calculate journey times between lots of points and the destination postcode.

The obvious thing to do would be to fetch the journey between every single point on the final diagram and the destination postcode. However, this would involve far too many queries. Instead we only queried for the journey between every public transport stop and the destination postcode. We then used our own replication of part of the Transport Direct algorithm to calculate the journey time from a specific point to the destination postcode. This is done by adding the walking time to the appropriately nearest/fastest public transport stop.

So, we have the journey time from every point to the destination postcode. This takes a long time to generate, as the query to the website for each journey takes 8 seconds. For example, there are about 1,900 public transport stops in a 20km square around Cardiff central railway station. That makes in total about 4 hours and 15 minutes for Cardiff. From that point on, the contour maps are fairly quick to produce.

This table shows the calculation time:

City	Public transport stops in 20km square (approx)	Basic screen scraping Generation time (estimate)
Cardiff	1900	4 hours 15 mins
Edinburgh	2700	6 hours
Birmingham	6600	14 hours 45 mins
London	10100	22 hours 30 mins

Parallel screen scraping

The slow generation time was not a problem for making a few maps. However, we would very much like to be able to offer everyone who wants it their own map. Things can be sped up a bit by running parallel screen scraping processes, to make fuller use of the Transport Direct servers. We ran brief experiments at 3 o’clock in the morning to measure this without affecting other users of the service:

Number of parallel screen scrapers	Wall clock time Transport Direct takes to deal with each query(approx)	Average time between query responses from any process (approx)
1 process	8 seconds	8.00 seconds
2 processes	11 seconds	5.50 seconds
4 processes	14 seconds	3.50 seconds
8 processes	17 seconds	2.13 seconds
16 processes	25 seconds	1.56 seconds
32 processes	40 – 50 seconds	1.40 seconds
64 processes	110 -120 seconds	1.79 seconds

As you can see, if we use too few processes, we are not making full use of the servers. If we use too many processes, we overload them, and things get slightly slower again. The optimum is somewhere near 32 processes. Roughly, it takes about 1.5 seconds per query. This is over 5 times quicker than using basic screen scraping:

City	Public transport stops in 20km square (approx)	Parallel screen scraping Generation time (estimate)
Cardiff	1900	47 mins
Edinburgh	2700	1 hour 8 mins
Birmingham	6600	2 hours 45 mins
London	10100	4 hours 12 minutes

Smarter algorithm

The times above are still not quick enough, even though they would fully load the Transport Direct servers.

We can make our algorithm slightly smarter.

It takes only slightly longer when requesting a journey to also get the route, including which bus stops and train stations you change at, and how long each leg of the journey takes. This will give you the destination time at intermediate points, which you then do not have to separately query.

We estimate that this will speed things up again by a factor of at best 5 times, and probably about 3 times. This means Cardiff will take about 15 minutes to generate.

Clearly, a faster source of data is needed than the Transport Direct website.

Using RailPlanner

One way to speed things up is to run an application on the user’s desktop, which has all the data available. We created such an application, based upon the Rail Planner software for Microsoft Windows. Rail Planner can very quickly return a train journey given start, destination and arrival or departure time.

Here is a screenshot of our application, which runs on top of Rail Planner.

© Crown copyright. All rights reserved. Department for Transport 100020237 2007

After you enter a postcode, it starts immediately filling in the map. It does this by running a copy of Rail Planner separately in the background, and programmatically querying it for journey times. As the data arrives, the contours on the map become more complete. It takes about 10 minutes to generate a complete map of the whole country.

This shows that on a computer with all the routing information for a simpler subset (rail only), maps can be generated reasonably quickly.

Interactive Google maps

All of the above ideas rely on existing route finding. Much better can be done with access to the underlying transport model. The way it calculates routes can then be optimised.

For example, Google Maps has a new interactive feature. It lets you drag end or via points of a route in real time. It is optimised to do this, and uses Google’s cluster of servers, so can calculate 10 or 15 routes per second.

This is maybe 20 times quicker than the above, so it could generate a map of Cardiff in perhaps 2 minutes.

Future

There are two clear options for providing custom generation of transport maps.

Local application

Create a desktop application, such as the rail one above, only including bus and car routes. This would require availability of the underlying timetable data. With optimisation, running on modern computers, it would be able to generate a map in 5 or 10 minutes, with useful intermediate information much quicker than that.

Customised servers

Instead of querying the existing Transport Direct site, create a special service just for generating these maps. It would have access to the same underlying data as Transport Direct, but use it in a more optimal manner for the task.

Main page explaining these maps

This project became Mapumental. Please visit www.mapumental.com for details of our transit-time maps and the services we can offer.

The work was funded and supported by the Department for Transport.

—

In 2006, our late friend and colleague Chris Lightfoot produced a series of time travel contour maps, after the Department for Transport approached mySociety about experimenting with novel ways of re-using public sector data.

This mapping work was very important because it provides a potentially revolutionary new way of working out the best place for people to live and work.

Following widespread interest across the net and a major feature in the Evening Standard, the Department for Transport asked us to show them how this work could be taken further, and that is what we are showing here today.

(The work described in this page has since led to the creation of Mapumental, a service that provides custom maps that help house hunters and office managers find the best places for quick and easy commutes.)

Improving legibility and clarity

Many of the maps we produced last time were very pretty, but could be somewhat difficult to interpret. We therefore teamed up with Stamen to improve the visual clarity and fun. Our first approach was to improve the base mapping to something more delicate and appropriate, using OpenStreetMap. We then worked on the colours and textures of the contours to make them quicker to interpret. Click on the images for larger versions.

Old map of London

Showing travel times to work at the Department for Transport in Pimlico, arriving at 9am

© Crown copyright. All rights reserved. Department for Transport 100020237 2007

New map of London

Showing travel times to work at the Department for Transport in Pimlico, arriving at 9am

Car vs public transport

Many people these days are looking to move to public transport, due to reasons varying from congestion, to cost, to environmental impact. But where can you live if you want to have the chance of getting to work speedily?

The following map shows which areas it makes more sense to get public transport to work in Edinburgh, and which areas it makes more sense to drive.

Remember, these are not general maps for the whole city, each map is only useful for the specific target place of work or study marked with the black dot. Please don’t make a mistake and use these to pick your own place of residence unless you happen to work at the location these maps are centred on!

Public transport vs cycling

For some people, the dilemma is not between the car and the train, it is between the bicycle and public transport. This map comprehensively shows that if you want to commute to the Department for Transport, and you live anywhere near the centre of London, it’s best to get on a bike if commuting speed is your main concern.

Introducing interactive maps

Whilst working on improving the usability of these maps, we came to realise that the complexity of graphical display could be substantially reduced by replacing multiple contour lines with a single interactive slider.

Next, it is clearly no good to be told that a location is very convenient for your work if you can’t afford to live there. So we have produced some interactive maps that allow users to set both the maximum time they’re willing to commute, and the median house price they’re willing or able to pay.

This page used to contain our old demos of interactive maps, but these have now been superseded. Visit Mapumental for our latest mapping technologies!

How can I get a map?

There are two ways in which you can get a travel map of this sort centred on a location of interest to you.

If you’re a interested user, you can contact your local transport journey planning organisation, for example Transport for London or San Francisco BART, and encourage them to work with us to get a system like this working interactively on their websites.

If you’re a rich user, or company, you can commission us to create bespoke maps – we’re a non-profit after all and all the money will help run our other projects. And if you’re really rich, you can work with us to develop a real-time service of the sort that the transport agencies should be doing. Francis Irving from mySociety has written a technical review on the challenges of developing a real-time map generation system.

If you’re interested in working with us on this, please contact us.

Acknowledgments

The idea was pioneered by the late and sorely missed Chris Lightfoot. All later code developments implemented by Francis Irving. The street maps were generated by ZXV (now Cloudmade) and Artem Pavlenko from OpenStreetMap data using Mapnik. The map graphical improvements, Flash and general sense of design is thanks to Tom Carden at Stamen; Matthew ran around tidying, spell-checking, validating, and outdenting. Tom Steinberg of mySociety herded all the cats together. The work was funded by the Department for Transport.

Technical notes

Journey times to work are for a week day in 2007. They were generated by screen scraping the Transport for London and Transport Direct journey planner websites. All journeys from public transport stops (in the NaPTAN database of such stops) to the destination were calculated using the journey planner. Points not immediately on top of a stop or station were interpolated using a walking speed to get to the nearest public transport stop.

House prices are based on house sales recorded in the Land Registry for a large random sample of London postcodes. For each point the median is calculated from the price of sales in a 1km radius round that point. Sales from all of 2006 are included in calculating this median, but are house price inflation adjusted to be the price as in December 2006.

If you’d like a copy of the custom software we wrote, or if you have any other questions or comments, please email hello@mysociety.org. Our software is available under the terms of the GNU Affero GPL. Some other data, such as NaPTAN, will require permission from their owners.

So, I’ve just had a shower and I’m waiting for Matthew and Tom to turn up. As time goes on, mySociety seems to get more geographically disparate, and I look forward to meeting my coworkers. Then for 1pm we’ll be heading to CB2 for the mySociety developers meeting. Feel free to come along any time afternoon or evening, whatever your skills or interest in mySociety.

I haven’t posted on here for ages, since October. I’ve been away on holiday quite a lot, and when I’ve not been away I’ve been busy, partly with systems administration. We’ve set up lots of servers in the last month for the E Petitions site. When you go from 3 servers to 7 servers, there’s another step change in sorting out systems administration tools. For example, I had to change the monitoring script so every server wouldn’t monitor every other. And I had to work out the quirks and bugs in the system we have for storing config files for different classes of server in CVS. Because we only had one class of server before.

I’ve also had to learn lots about server monitoring and load balancing. Things have slowed down a bit now, to maybe 10 hits per second. But a few weeks ago the road pricing petition was often getting 50 hits per second. I’ve never worked on a site with that level of traffic before. You find all the bugs in your code, all the missing indices in PostgreSQL, all the badly tweaked FastCGI parameters. I’ve been sucking knowledge off Chris like a sponge, so tools like strace and vmstat begin to become instinctive. Seemingly nobody offers a book or a course which teaches this stuff well – every server setup is different, everyone knows different ways to tune and profile. But maybe you can tell me different in the comments.

Louise has been busily working away on lots of things. Amongst that is a major change to WriteToThem, to let you write to all the members in a multi-member constituency in one go. The last day or two, she’s been installing the WriteToThem test code on one of our servers, when it has only run on my laptop before. This will be fantastic – hopefully can get Matthew to be bolder about making changes to WriteToThem, if he has a test script he can easily run (getting Matthew to be bold isn’t normally a problem, but he seems mildly less bold when it comes to the WriteToThem queue daemon).

Tom and I have also been busy on a second travel maps report for the DfT. More on that soon. Lots of running screen scraping jobs on TransportDirect which take days. On the subject of Tom, he seems to have got expert at “stacking meetings” next to each other. In one day last week he had 7 meetings!

We just had our irregular weekly meeting, which we do most Mondays using a conference call. I thought I’d just write up what we’re all up to this week.

• I’m continuing to test the ePetitions site for 10 Downing Street, and developing an interesting branded version of PledgeBank for CAFOD (more when it launches).
• Matthew is going to look at various things that need doing on PledgeBank and WriteTothem. For PledgeBank more chivvying emails, I think something like this ticket but not exactly. For WriteToThem, various bits of code to do with how we handle error cases.
• Chris is making more pretty maps for the Department for Transport.
• Tom is working out in detail how we’re doing to spend the money from DCLG which has finally come through. It’s mentioned in this post, look for “e-Innovations Product and Marketisation strand via Kirklees MBC”. Which means, we’re being paid to do proper marketing and sales of branded version of our services, such as WriteToThem, PledgeBank, and Neighbourhood Fixit. He’s also chasing up some interesting people met at a conference in Eastern Europe (Bratislava, I think?) last week.

Please ask questions in the comments – for example, if you’d like us to post about particular things on this blog.