Donations to MPs are in the news again, and TheyWorkForYou allows users to easily see what any individual MP has received. In fact, the site has carried a copy of the Register of Members’ Financial Interests (in which, as Parliament’s website explains, “MPs must register within 28 days any interest which someone might reasonably consider to influence their actions or words as an MP“) since at least 2005.
This hasn’t always been straightforward, and has recently become slightly trickier.
The official register is published as static HTML or PDF, with a simple list of all MPs. We scrape that HTML, convert it into light XML and import it onto the site – which means you can easily see not only the current entry on an individual MP’s page, but also see a complete history of their register without having to view many different copies of the official register.
The XML contains all the data from the official register, but it only parses out basic information like the category of interest. Providing more detail would be great, but is quite a hard problem to tackle.
Recently, Parliament has started using Cloudflare’s bot-protection technology. We assume this change was made with good reason, but as a side effect it has prevented effective scraping of the website, as Cloudflare don’t distinguish between good and bad bots or scrapers.
We know that Parliament was working on an API at least as far back as 2016, from their now-removed data blog, but if this is still in development, it is yet to see the light of day. What they said at the time still stands: their website is still the only means of accessing this data. We don’t think it’s necessary to protect purely static HTML pages such as the Register in quite such a heavy-handed manner.
We do have ways of continuing to get the Register, and TheyWorkForYou is still up to date, so anyone else who has been scraping the official site and has hit issues because of this is welcome to use our data, either via the XML or our API.
Image: Adeolu Eletu
The Climate Action Plans Explorer (CAPE) is gathering together every Climate Action Plan from every UK council that’s published one. We’re actively adding more functionality on an ongoing basis; most recently, we’ve extracted the ‘headline pledges’ from each plan, like this:
Pledges like this give an idea of the council’s overarching priorities, but often have not been presented in isolation before, even by the councils themselves.
Why we did this
A core aim of our Climate programme is to improve the information ecosystem around local responses to the climate emergency:
“We’re improving the information ecosystem to allow local and national campaigns, policymakers and other stakeholders to undertake better scrutiny and analysis of local climate action, and develop evidence-based policies and solutions.”
We’ve already written about how we’re working with Climate Emergency UK to collect and score Climate Action Plans for every local authority in the UK. Providing people with an easy way into their local authority’s action plan will give them an unprecedented opportunity to gauge their council’s level of ambition in facing the climate emergency, and how they’re planning to turn those ambitions into actions.
But plans can be complex, and time-consuming to read. Another, faster way people can understand their council’s level of ambition is by finding any targets that it might have set itself for decarbonising either the entire area, or just the council’s own operations, by a particular date.
We call these ‘pledges’, and they’re typically not all that easy to find – they can be buried in council meeting minutes, or slipped somewhere into an unassuming page on the council’s website or action plan.
Knowing what date your council is working towards, and what they believe they can achieve by that date, fundamentally sets the scene when it comes to understanding and contributing to the council’s climate actions.
That’s why we decided to collect these pledges and share them on CAPE. Here’s where you’ll find them on each council’s page, setting the scene before you dig into the full action plan:
How we did this
Collecting these pledges from scratch would be a mammoth task. Luckily, we were able to build on two partial datasets that gave us a headstart.
An important thing to note is that we wanted to collect not only the scope (that is to say, whether the plan covers council operations only, or the whole area) and target date, but also the exact wording of the pledge, and the source where it was found.
We found that local authorities often use terms like ‘carbon neutral’ and ‘net zero’ interchangeably, and the scope of pledges can sometimes be ambiguous. The most objective approach, therefore, was to present the entire pledge, as it was originally worded, and leave it up to the viewer to interpret the council’s intent. Collecting and exposing the source of the pledge would allow them to dig deeper and view the pledge in context, if they wanted to.
Our partners, Climate Emergency UK, had already been collecting climate target dates as part of their ongoing monitoring of council responses to the climate and biodiversity emergencies. But since the target dates were just one small part of a much wider database, they hadn’t collected the direct quotes that we wanted to present.
Still, in the Climate Emergency UK dataset, we effectively had a wide but shallow starting point, covering most councils in the UK, from which we could then proceed to fill in the detail by revisiting the sites, scanning them for anything that looked like a pledge, and pasting them into our database.
We were also incredibly grateful to receive a smaller, but much more detailed, dataset of climate commitments from the National Audit Office, which covered 70% of the principal local authorities in England, some of which had made no commitments. They themselves had manually gathered these commitments from public sources—council’s minutes, websites, and action plans—over April to June of 2021.
Combined with the Climate Emergency UK dataset, this data from the National Audit Office got us 75% of the way towards a full dataset of climate pledges from every council in the UK.
With the help of Climate Emergency UK volunteers, we filled in the gaps on this combined dataset, collecting direct quotes for both council-only and whole area climate pledges, for 341 of the 408 councils in the UK.
For the remaining 67 councils, we were unable to find a public climate pledge, or at least one with a concrete target date – but we’re hopeful that we might yet find this information, and the CAPE website includes a link on these councils’ pages through which visitors (or maybe councillors or council officers!) can contribute the data, if they’ve found a source elsewhere.
The data was collected via an online spreadsheet, making it fairly easy to import into CAPE, as part of the website’s existing data processing pipeline. This feeds the pledges through to both the individual council pages, and also the all councils page, where you can now filter the list to show only councils with a target in a given five-year period, or no target at all.
We will soon also be exposing these pledges via the CAPE API, so third parties can programmatically access and reuse the data in their own services. If this sounds like something you might find useful, do get in touch or subscribe to our Climate newsletter where we’ll be sure to share any news.
Image: Romain Dancre
One service we offer on TheyWorkForYou is an email alert: this lets you know when there is new data published on the site that either contains a word/phrase that you’ve subscribed to, or that indicates new activity from your selected Member/s of Parliament.
(Didn’t know this? Go and sign up now!)
We send around 400,000 of these emails a month. For many years, the look has remained exactly as it was when we first developed them: plain text, which has the benefit of being lightweight and unlikely to get scrambled by email clients. The downsides are that it doesn’t exactly make for a compelling email, visually speaking, and that some find it hard to identify which sections are of interest in a uniform block of unformatted text.
We’ve now finally transformed alert emails into a much more polished HTML format, and at the same time we’ve also improved the look and feel of four other vital elements of TWFY: profile images, the API, the sign-up page, and the Contact page.
As usual, before starting work, we did a bit of research into who uses this feature and why, so we could be sure we were answering their needs. You can see more about this in Alex’s post here.
Photos of MPs
Where there is a more recent and higher quality image available, we’ve updated the profile image we use for MPs. In some cases, this has replaced some pretty youthful faces — it’s clearly high time we caught up with this particular ticket!
Higher resolution or larger images also mean that they’ll be more useful to developers using the images (which are all available under an open licence) on other sites and apps.
Clearer access to the API
The API page (where developers and researchers can access TheyWorkForYou data) has been given a slick new design. We’ve updated it with new examples of how the API might be used, and streamlined the language and content to make it easier to understand.
We hope that all of these features will make it easier and more pleasant for you to use TheyWorkForYou, either when you’re checking up on what’s happened in Parliament for yourself, or using our data to make other parliamentary apps and sites.
Image: David Pisnoy
MapIt has had a bit of a refresh to bring the look into line with the rest of the mySociety projects. At the same time, we thought we’d take the opportunity to make it a bit easier for non-technical folk to understand what it offers, and to make the pricing a little less opaque.
You may not be familiar with MapIt, but all the same, if you’ve ever found your MP on TheyWorkForYou, written to your representatives on WriteToThem, or reported an issue through FixMyStreet, you’re a MapIt user!
That’s because MapIt does the heavy lifting in the background when you enter a postcode or location, matching that input to the boundaries it falls within (ward, constituency, borough, etc). It is, if you like, the geographic glue that holds mySociety services together.
Like most of mySociety’s software offerings, MapIt is available for others to use. So for example, the GOV.UK website uses it to put users in touch with the right council for a number of services, and Prostate Cancer UK uses it on their campaign site, using MapIt’s knowledge of CCG (Clinical Commissioning Group) region boundaries.
And you can use MapIt too: if your app or website needs to connect UK locations with areas like constituencies or counties, it will save you a lot of time and effort.
Pricing and payment is a lot slicker now: while it was previously managed manually, you can now purchase what you need online, quickly and without the need for human intervention. It’s also quite simple to see the pricing options laid out.
We hope that this will make it easier for people to make use of the service, and better understand what level of usage they need. But if you need to experiment, there’s a free ‘sandbox’ to play about with!
As ever, we’re happy to provide significant discounts for charity and non-profit projects: see more details on the licensing page.
If you have any questions or comments please do get in touch.
There’s a new piece of data on MapIt, and it wasn’t added by us. It’s tiny but useful, and it’s slightly esoteric, so bear with us and we’ll explain why it’s worth your attention.
Local Authority codes come from the government’s set of canonical registers. They may not look much, but they’re part of a drive to bring consistency across a wide range of data sets. That’s important, and we’ll try to explain why.
One name can refer to more than one thing
If you try to buy a train ticket to Gillingham in the UK, and you are lucky enough to be served by a conscientious member of staff, they will check whether you are going to the Gillingham in Kent (GIL), or the one in Dorset (GLM).
The names of the two towns might be identical, but their three-letter station codes differ, and quite right too — how, otherwise, would the railway systems be able to charge the right fare? And more importantly, how many people would set off confidently to their destination, but end up in the wrong county?
I mention this purely to illustrate the importance of authoritative, consistent data, the principle that is currently driving a government-wide initiative to ensure that there’s a single canonical code for prisons, schools, companies, and all kinds of other categories of places and organisations.
Of particular interest to us at mySociety? Local authorities. That’s because several of our services, from FixMyStreet to WriteToThem, rely on MapIt to connect the user to the correct council, based on their geographical position.
One thing can have more than one name
I live within the boundaries of Brighton and Hove City Council.
That’s its official name, but when talking or writing about my local authority, I’m much more likely to call it ‘Brighton’, ‘Brighton Council’, or at a push, ‘Brighton & Hove Council’. All of which is fine within everyday conversation, but which is an approach which could cause mayhem for the kind of data that digital systems need (“machine readable” data, which is consistent, structured and in a format which can be ‘understood’ by computer programs).
Registers of Open Data
The two examples above go some way towards explaining why the Department for Local Government & Communities, with Government Digital Services (GDS), are in the process of creating absolute standards, not just for councils but for every outpost of their diverse and extensive set of responsibilities, from the Food Standards Agency to the Foreign & Commonwealth Office, the Land Registry and beyond.
Where possible, these registers are published and shared as Open Data that anyone can use. It’s all part of GDS’ push towards ‘government as a platform’, and in keeping with the work being done towards providing Open Data throughout the organisation. Where possible these registers are openly available, and can be used by anyone building apps, websites and systems.
And now we come to those Local Authority codes that you can find on MapIt.
Anyone can contribute to Open Source code
Like most mySociety codebases, MapIt is Open Source.
That means that not only can anyone pick up the code and use it for their own purposes, for free, but that they’re also welcome to submit changes or extensions to the existing code.
And that’s just how GDS’ Sym Roe submitted the addition of the register.
What it all means for you
If you’re a developer, the addition of these codes means that you can use MapIt in your app or web service, and be absolutely sure that it will integrate with any other dataset that’s using the same codes. So, no more guessing whether our ‘Plymouth’ is the same as the ‘Plymouth’ in your database; the three-letter code tells you that it is.
Plus, these register codes identify a local authority as an organisation, or a legal entity, as opposed to setting out the boundary, so that’s an extra layer of information which we are glad to be able to include.
Census data: there’s lots of it. It contains fascinating insights.
But as with many huge datasets, those insights are not always easy to find at first glance — nor is it easy for the untrained observer to see which parts are relevant to their own lives.
Wazimap in South Africa takes the country’s census data and turns it into something the user can explore interactively. Originally conceived as a tool for journalists, it turned out to be so accessible that it’s used by a much wider range of the population, from school children to researchers. It’s a great example of how you can transform dry data into something meaningful online, and it’s all done using free and open source tools.
Our points-to-boundaries mapping software MapIt is part of that mix, putting the data in context and ensuring that visitors can browse the data relevant to specific provinces, municipalities or wards.
We asked Greg Kempe of Code for South Africa, to fill us in on a bit more.
What exactly is Wazimap?
Wazimap helps South Africans understand where they live, through the eyes of the data from our 2011 Census. It’s a research and exploration tool that describes who lives in South Africa, from a country level right down to a ward, including demographics such as age and gender, language and citizenship, level of education, access to basic services, household goods, employment and income.
It has helped people understand not just where they work and live, but also that data can be presented in a way that’s accessible and understandable.
Users can explore the profile of a province, city or ward and compare them side-by-side. They can focus on a particular dataset to view just that data for any place in the country, look for outliers and interesting patterns in the distribution of an indicator, or draw an indicator on a map.
Of course Wazimap can’t do everything, so you can also download data into Excel or Google Earth to run your own analysis.
Wazimap is built on the open source software that powers censusreporter.org, which was built under a Knight News Challenge grant, and is a collaboration between Media Monitoring Africa and Code for South Africa.
Due to demand from other groups, we’ve now made Wazimap a standalone project that anyone can re-use to build their own instance: details are here.
How did it all begin?
Media Monitoring Africa approached Code for South Africa to build a tool to help journalists get factual background data on anywhere in South Africa, to help encourage accurate and informed reporting.
Code for South Africa is a nonprofit that promotes informed decision-making for positive social change, so we were very excited about collaborating on the tool.
Could MapIt be useful for your project? Find out more here
How exactly does MapIt fit into the project?
Mapit powers all the shape boundaries in Wazimap. When we plot a province, municipality or ward boundary on a map in Wazimap, or provide a boundary in a Google Earth or GeoJSON download, MapIt is giving Wazimap that data.
We had originally built a home-grown solution, but when we met mySociety’s Tony Bowden at a Code Camp in Italy, we learned about MapIt. It turned out to offer better functionality.
What level of upkeep is involved?
Wazimap requires only intermittent maintenance. We had municipal elections in August 2016 which has meant a number of municipal boundaries have changed. We’re waiting on Statistics South Africa to provide us with the census data mapped to these new boundaries so that we can update it. Other than that, once the site is up and running it needs very little maintenance.
What’s the impact of Wazimap?
We know that Wazimap is used by a wide range of people, including journalists, high school geography teachers, political party researchers and academics.
Code for South Africa has been approached a number of times, by people asking if they might reuse the Wazimap platform in different contexts with different data. Most recently, youthexplorer.org.za used it to power an interactive web tool providing a range of information on young people, helping policy makers understand youth-critical issues in the Western Cape.
We also know that it’s been used as a research tool for books and numerous news articles.
The success of the South African Wazimap has driven the development of similar projects elsewhere in Africa which will be launching soon, though MapIt won’t be used for those because their geography requirements are simpler.
What does the future hold?
As we’re building out Wazimap for different datasets, we’re seeing a need for taking it beyond just census data. We’re making improvements to how Wazimap works with data to make this possible and make it simpler for others to build on it.
Each new site gives us ideas for improvements to the larger Wazimap product. The great thing is that these improvements roll out and benefit anyone who uses it across every install.
Thanks very much to Greg for talking us through the Wazimap project and its use of MapIt. It’s great to hear how MapIt is contributing to a tool that, in itself, aids so many other users and organisations.
Need to map boundaries? Find out more about MapIt here
If you’ve visited the MapIt site this week, you might have noticed a change: we’ve introduced key-based authentication for API users.
This enables us to be more flexible about how we provide our service, which means you can be more flexible about how you serve your users.
MapIt is both an open source application and, via https://mapit.mysociety.org, a web service. Use of the API is free for low-volume, charitable use, while all other uses require a licence.
For the moment API keys are optional. We’ll always offer a free level of service to support independent developers and charities.
We’ll have more details soon about the increased flexibility this change will bring.
The Universal Credits system is replacing many other welfare benefits… but slowly. Its roll-out won’t be complete until 2022, meaning that many are, understandably, confused about just what applies within their own local area.
Now Lasa, in collaboration with the Low Incomes Tax Reform Group (LITRG), have launched a tool to help with that problem. Just input a postcode, and it displays information about which benefits apply — and, crucially, where to go for advice in your area.
It’s part of a suite of offerings, also available as widgets that can be placed onto any website. All fall within Lasa’s remit to support organisations in the delivery of social welfare law advice to the disadvantaged communities they serve.
We’re always glad to see MapIt used in other people’s projects, especially those that make a complex system easier to understand.
Apparently advice workers are already expressing their gratitude for the fact that they can have this information at their fingertips — so hats off to Lasa.
Scottish Parliamentary proceedings are now back on TheyWorkForYou.
Back in August 2014, the Scottish Parliament changed the way it published the Official Report of its debates.
TheyWorkForYou works by fetching data from various parliamentary sources—and in this case, unfortunately, the change at the Scottish Parliament end meant that our code no longer worked. We replaced our ‘debates’ section with an apologetic note.
Well, thanks to the Scottish Parliament kindly republishing the data in almost the format we used to use, we’ve managed to make some small tweaks and restore that content—including debates from the previously missing period. If you’re subscribed to alerts, you should have received an email digest with links to the backdated content (always supposing there was any that matched your chosen keywords).
And if you’re not subscribed to alerts? Now is a great time to rectify that. We’ll send you an email every time your chosen word or phrase is mentioned in Parliament, or every time your chosen representative speaks.
While we were doing this work, we also modified TheyWorkForYou so that it now pulls in ministerial data from the Scottish Parliament API. This is a welcome time-saver for us: previously we were creating a list manually from the official PDFs, while we can now automatically fetch it and reformat it into Popolo JSON, meaning it’s consistent with all our other data.
Thanks for your patience; we know that many people were awaiting this repair, and for longer than we would have liked. Enjoy!
From today, it’s much easier to buy transit-time maps from Mapumental. We’ve added a self-service shop which allows you to generate your own maps, instantly and easily.
The technical amongst you may like to know that the service queries the Mapumental API; for everyone else, it’s probably enough to say that your maps will just appear, as if by magic.
Mapumental maps are cheaper when you buy in bulk, so we’ve also integrated a credits system. If you know you’ll have an ongoing need for our maps, stock up on credits (also completely self-service) and you’ll soon start benefiting from some substantial discounts. We’ve included a nifty little credits calculator on the page, so you can find the price band that best suits your needs.
Check out the new interface at Mapumental now. All the benefits of a self-service checkout, none of those irritating “unexpected item in the bagging area” announcements.