Over the last few months, we’ve been working with Hackney Council to design and make a Freedom of Information management system, imaginatively named FOI For Councils — and last time we left you with our pre-development thoughts. Well, now it’s up and running.
With this project, we had two main aims:
- first, to make the process really slick and easy to use for citizens;
- second, to reduce the quantity of FOI requests submitted, relieving some pressure on the Information Officers at the receiving end.
The solution we came up with achieves both those aims, and there’s one feature in particular that we’re super-excited about.
Case Management Integration
One of the development decisions taken early on was for the system to be a very lightweight layer, largely powered by the new Infreemation case management system that Hackney were in the process of commissioning.
Infreemation is targeted primarily at Information Officers, so there was no use in reinventing the wheel and building a heavy backend for our own FOI for Councils software.
Instead we built the FOI request process, using our experience in designing for citizens, and submitted the data directly to Infreemation using their API. This means that every request goes straight in to the case management system used by Information Officers, with no need for double entry; a set-up we’re very familiar with from our work integrating council systems with FixMyStreet.
Information Officers respond to the FOI request through Infreemation, and when they publish the response to Infreemation’s disclosure log, FOI for Councils can pull that response into its innovative suggestions engine, which we’ll discuss shortly.
All this means that Information Officers get to use the tools that are designed directly with them in mind, but citizens get the best experience possible for the process at hand, rather than trying to battle the typical generic forms offered by one-size-fits-all solutions.
On the user side of things we managed to reduce the process to a maximum of 6 screens for the entire process.
Throughout the whole user journey we ask for only three details: name; email address and then the actual request for information.
Each screen provides contextual help along the way, maximising the chances that the FOI request will be well-formed by the time it gets submitted.
Making the process intuitive for the people using it is a key factor in building citizens’ trust in an authority. Too often we see complex forms with terrible usability that almost seem designed to put people off exercising their rights.
So far, so good. But for us, the most interesting piece of the process is the suggestions step.
Before the citizen submits their request to the authority, we scan the text for keywords to see if anything matches the pool of already-published information.
If we find any matches, we show the top three to the citizen to hopefully answer their question before they submit it to the authority. This helps the citizen avoid a 20-day wait for information that they might be able to access immediately. If the suggestions don’t answer their question, the citizen can easily continue with their request.
Suggestions also benefit the authority, by reducing workload when requests can be answered by existing public information.
We’ve tried to make this suggestions step as unobtrusive as possible, while still adding value for the citizen and the authority.
The suggestions system is driven by two sources:
- Manually curated links to existing information
- The published answers to previous FOI requests
The curated links can be added to the suggestions pool by Information Officers where they spot patterns in the information most commonly requested, or perhaps in response to current events.
The intelligent part of the system though, is the automated suggestions.
As FOI for Councils integrates with the Infreemation case management system, we can feed the suggestion pool with the anonymised responses to previous requests where the authority has published them to the disclosure log.
By doing this the authority is making each FOI response work a little harder for them. Over time this automatic suggestion pool should help to reduce duplicate FOI requests.
FOI for Councils also analyses the number of times each suggestion is shown, clicked, and even whether the suggestion has prevented any additional FOI requests being made.
This allows Information Officers to see which information is being asked for, but where existing resources aren’t providing the information necessary to the citizen.
We’ll be keeping a keen eye on how this works out for Hackney, and we’ll be sure to report back with any insights.
As you’ll know if you read our first blog post from this project, we did originally envision a platform that would process Subject Access Requests as well as FOI. In the end this proved beyond the resources we had available for this phase of work.
For us, this has been a really instructive piece of work in showing how authorities can commission process-specific services that connect together to give everyone a better user experience.
If you’re responsible for managing FOI requests or data protection in your own public sector body and you’d like to talk about project in more detail, please get in touch at email@example.com.
FixMyStreet sends users’ street reports to councils across the UK.
But if you’re one of the staff that receives these reports, you might sometimes wish for more insights: which issues are most commonly reported in your area? What’s a bigger problem, dog fouling or potholes? How quickly do reports get fixed, and how does this compare with other councils’ performance?
To make it easy to discover the answers to all these questions, we’ve just rolled out a new stats dashboard on FixMyStreet — and it’s free to access if you work for a council.
Council staff can now view and download information that answers questions such as:
- How many FixMyStreet reports have been made in your area across various time periods?
- How many reports have been made in each category?
- What are the average times between reports being made and being marked as fixed, and how does this compare to other councils?
- Which categories of report are most common within your area?
- Which categories of report does FixMyStreet send to your council, and which email addresses does it use?
From this exclusive area, you can gain a more in-depth understanding of how FixMyStreet is being used in your area, while also getting a taste of the fuller functionality available with FixMyStreet Pro.
So, if you work for a council, head over to the dashboard page now, and start exploring.
This is part II in a series of blog posts about our work with Hackney Council to conduct a discovery and prototyping project to improve the public-facing parts of information request processes. Read the first part here.
With our experience of running WhatDoTheyKnow and Alaveteli, we’re big believers in the importance of information transparency laws in our democratic system. But at the same time, we understand the operational challenges of meeting the statutory requirements of these laws for public bodies.
The challenges of dealing with information requests
While many local authorities have dedicated information governance staff whose job it is to manage requests, finding and compiling the information is often done by hard-pressed staff within services, fitting in information-related work around their core responsibilities.
Requesters sometimes have the expectation that information is all carefully organised in a few databases, ready to be aggregated and extracted at the click of a button. In reality, the degree to which information is well-organised varies a lot between services in a council and between different councils, often because of procurement decisions and departmental priorities stretching back many years.
Working with Hackney Council
We’re part way through our project with Hackney Council that Mark wrote about a few weeks ago, helping them redesign the public-facing parts of their FOI and Subject Access Request (SAR) services, so we’ve seen these challenges up close. We’ve also seen how they can be made even trickier by legacy IT systems that are no longer fit for purpose.
— Rob Miller (@RobMiller31) February 28, 2018
At our ‘show and tell’ last week with Hackney, we shared findings from some of the research we’ve done, along with early prototypes of potential new FOI and SAR submission forms, both created collaboratively with Hackney Council staff.
Prototyping with the GDS toolkit
We decided to go straight from sketches to HTML-based prototypes for this project. We thought that moving in-browser with real interactive elements would make it easier to test out some of the more complicated interactions and conditional workflow of the SAR process in particular.
Prototyping is about quickly testing hypotheses, and not getting bogged down in implementation details. So it was a pleasure to use the GDS frontend toolkit to build our prototypes. Not only did the GOV.UK toolkit help us build something relatively rich in just a few days, but it also meant we benefited from GDS’s previous work on design patterns developed for exactly this kind of context.
Using public information to reduce FOI requests
When submitting a request via WhatDoTheyKnow, users are automatically shown previously submitted requests that might answer their question, and we know that almost 10% of requesters have a look at these suggestions in more detail.
As you can see here, we took this pattern from WhatDoTheyKnow, and enhanced it further in our FOI prototype.
Now, as well as prompting with previous FOI responses from a disclosure log that we’re hoping to build, we also want to include relevant links to other topical or frequently requested information, drawn from other data sources within the council.
If we can get this to work well, we think it could have a number of benefits:
- Helping the person submitting the FOI get their answer more quickly
- Reducing the number of requests that would otherwise have been sent to the council
- Encouraging more proactive, structured publishing of data by the council.
It’s this last point which we’re really interested in, longer term. We think using a common sense technology approach to highlight possible answers to FOI requests before they are made will get people answers more quickly, reduce the burden on responders, and reinforce efforts to proactively release commonly requested data leading to real transparency benefits.
We’ve been doing usability testing of these prototypes over the last few days, and will be looking very closely at whether our assumptions here stand up to scrutiny.
Reducing back and forth around SARs
Given our background, we’re inevitably very interested in the FOI component of this project, but the Subject Access Request component is no less important. And while there’s none of the same opportunity for pre-emptively answering people’s requests, there’s plenty of scope for making the submission process a lot smoother.
A valid FOI is just a written request and some contact details, but the information needed for a SAR to be valid is much greater and more complicated. For example, along with the specifics of the request, a solicitor asking for personal information on behalf of a parent and their 16-year old child would have to provide proof of ID and address for both family members, a letter of consent from the child, and a letter of authority from the parent, confirming that the solicitor is acting for them. Even the most straightforward SAR calls for the handling of personal data and confirmation of the requestor’s identity and address.
Given all this complexity, it’s easy for there to be a lot of back and forth at the start of a SAR: clarifying a request, asking for documents, arranging receipt of documents, and so on.
We’re hoping to build something that makes it much clearer to the person making the request what they’ll need to do, thereby taking some of that responsibility off the shoulders of the team managing the requests.
The sketch above was our first crack at something that could handle (most of) this complexity, and we’ve made a number of changes since then as our understanding of it all has increased.
Recruiting representative users to test SAR submission has proved challenging and wasn’t helped by the Beast from the East rearing its snow-covered head. But we changed tack and successfully ran some guerilla testing in Hackney’s service centre last week, and we’re hoping to do further testing later in the project that has more chance of benefiting from contacts cultivated by individual services.
Changing the conversation
The conversation around information requests often focuses on the burden of responding. And although the number of information requests local authorities receive is unlikely to decrease any time soon, we’re hoping that through our collaboration with Hackney, we can make handling them a whole lot easier.
If we do it well, we think we could eliminate a lot of the process issues created by poorly-designed legacy systems, while baking in a fundamentally more open and transparent way of operating that has the potential to benefit both citizen and state alike.
If you’re responsible for managing FOI requests or data protection in your own public sector body and you’d like to follow this project in more detail—or if you’d like to participate in some of the discovery work—then please get in touch at firstname.lastname@example.org.
Image: Sarah Palmer-Edwards
We’re in the process of conducting a discovery and prototyping exercise to understand how Hackney Council currently respond to FOI requests — and also Subject Access Requests (SAR) — ahead of the new General Data Protection Regulation (GDPR), coming into force in May 2018.
The aim is to explore how we can help members of the public find the information they are looking for when attempting to submit an FOI or SAR request and subsequently, when a request is complete, making it easier to publish the non-personal responses to requests through a searchable public disclosure log.
Information should be free
When someone makes a Freedom of Information request to a public body, we like to think that the information provided, often at a not insignificant cost, should be available freely to everyone, in public.
That’s the basis of our Alaveteli software which runs in at least 28 countries, and WhatDoTheyKnow in the UK which has grown to become a vast database approaching half a million FOI requests and responses over the last 10 years, from almost 19,000 public bodies.
From our own research we know that at least 15% of all FOI requests made in the UK pass through WhatDoTheyKnow, and that rises to over 30% of all requests to some central government departments. That still leaves somewhere over 70% of all requests that feasibly could be made available to the public.
What usually happens instead is that these individual requests remain hidden away in private mailboxes and probably won’t ever see the light of day.
Our FOI strategy
In response to this we’re on a mission to expand the share of FOI requests that are catalogued and released in public through WhatDoTheyKnow. This requires us to have an understanding of the nature and source of these other requests.
Broadly speaking around one third of these remaining requests are made by commercial businesses seeking contractual information from councils, NHS trusts and the like. About a fifth are from journalists researching for stories, another slice come from students or academics, and the remainder from individuals who are often making just a single request.
Our overall strategy is pretty simple: expand the scope of WhatDoTheyKnow where we can to capture more requests directly, and create new services to cater to specific user needs.
This thinking is what led to our service for journalists and campaigners, WhatDoTheyKnow Pro. (So far we’ve been reluctant to directly cater to FOI requests made by commercial businesses, although on balance it would be better if these requests did eventually make it into the public domain.)
Working with Hackney
Through WhatDoTheyKnow we’ve been pretty firmly focused on helping citizens make good FOI requests. Some readers may remember our previous forays into this area, with WhatDoTheyKnow for Councils (since retired). We found issues with that: specifically, the assumption of immediate publication in particular placed us in position as both poacher and gamekeeper, creating a conflict of interest we weren’t comfortable with.
However when we consider the full lifecycle of creating a response to an FOI request we still believe that we can use our experience of FOI to help public bodies support better drafting of initial requests and aid the management of responding to these requests.
Which brings us back to our new collaboration with Hackney.
More specifically, we are working with Hackney to explore how we can:
- help users better submit clear and valid requests
- integrate this request form with other sources of information (including with a disclosure log) to try and help users find what they need more quickly and conveniently
- integrate with case management services so that queries are answered quickly and information published openly wherever possible
- use information from previous requests to speed the allocation of a particular request to a specific council service
- support compliance with current legislation, and pre-empt forthcoming changes (GDPR).
We’re just at the beginning of this process but we’ll be blogging and sharing more updates over the next couple of months. We will also be speaking to other public sector bodies, councils in particular, about how they manage the process of responding to FOI requests, the challenges they face, and the opportunities this offers them to proactively release more publicly available data.
Hackney are a great partner to work with. As you might be already aware they have been very active in adopting user-centred, agile methods to develop new services, they are comfortable and vocal talking about their work in public (check out the HackIT blog here) and they are especially focused on bringing their staff along with them as they evolve their approach.
We’ve got a lot to learn from them, and hopefully they can benefit from some of our experience representing the needs of citizens.
If you’re responsible for managing FOI requests or data protection in your own public sector body and you’d like to follow this project in more detail — or if you’d like to participate in some of the discovery work — then please get in touch at email@example.com.
Update: See part 2 of this blog post here.
If you happen to have visited the WhatDoTheyKnowPro page recently, you might have noticed that we’ve changed from offering free beta testers’ accounts to rolling out the payment interface.
Yes, we’ve launched! But not with a bang: having quietly introduced the payment option, we wanted to wait for a while and check that there were no issues before making a public announcement. Consider this that public announcement.
Just a reminder at this stage, in case you haven’t followed along with previous blog posts: WhatDoTheyKnowPro is a Freedom of Information toolkit which provides extra help with sending and organising requests. It’s designed to provide extra functionality for journalists and other people who use FOI in their jobs; if you’re a standard user of WhatDoTheyKnow, there’s nothing to pay and there never will be. We’ll always keep the site, and all its current functionality, free for everyone.
The question of price
Launching WhatDoTheyKnowPro provoked an interesting debate on where to position it, pricewise.
It’s not that we’ve never put a price on any of our services before: mySociety is a social enterprise, and we charge for some use of our APIs; our council clients pay us for FixMyStreet Pro; you can buy transit-time maps on Mapumental. These sources of income are just part of what help us to provide our core citizen-facing services for free.
And we’re hardly trailblazers in that respect: in fact, it was fascinating to read the Knight Foundation’s recent report Scaling Civic Tech and see how common it is in our sector to rely on a variety of revenue streams, from user donations to philanthropic grants, to paid-for services.
But while we may have experience in charging for our services, it’s definitely the first time we’ve had to price up a Freedom of Information toolkit for journalists and professionals!
What’s it worth?
We were effectively in the position of many an enterprise startup: with a market proposition that doesn’t exist in this exact form anywhere else. How do you know how to price something in those circumstances? Set it too low and you could miss out on important revenue; too high and you’ll alienate potential customers, many of whom are freelance journalists paying for their work tools out of their own pockets.
Still, this project has been a process of tackling problems and questions thoughtfully — from deciding which features to include in this initial version, to debating how to encourage journalists to link back to the news stories they’ve created once they’re live. Perhaps we could bring the same approach to pricing.
Ask the experts
Fortunately, with over 100 beta testers, we had a pool of users who knew the service well enough to be able to give an expert opinion on how much value it was bringing them. Sending out a survey brought some very useful responses, not to mention feedback about what our beta users liked and what they’d appreciate in future roll-outs.
That said, we’re well aware of research indicating that people are not always experts on what they are actually willing to pay.
And of course, it makes sense that different people will attach different values to a service, depending not only on their own finances but, in this case all sorts of other factors such as how frequently they use FOI and how accustomed they are to paying for technology.
Faced with responses to the survey that ranged between a suggested price of £1.00 a month to £50.00, that was, in some strange sense, reassuring to know.
To be fair, those were the extremes. There was a good consensus in the middle and that helped us decide on an introductory price of £10 a month. We’ll assess this after a few months to see whether it’s bringing the number of sign-ups we expect.
Open for business
If you’re a journalist or someone who uses FOI in your work, you can now go and give WhatDoTheyKnowPro a go! We hope you’ll let us know how you find it.
Image: Sven Scheuermeier (Unsplash)
mySociety was built on its Democracy practice, a pioneer in providing simple- to-use tools that demystify the democratic process, allow citizens to understand how decisions are being made on their behalf and ensure that their voices are heard by elected representatives.
We’ve been on a long journey, from the early days of FaxYourMP which eventually became WriteToThem, to our pivotal TheyWorkForYou service which has both stretched the ambitions of Parliament in the UK and led us to develop similar services in Kenya, South Africa and beyond.
Amidst all of this has been our ongoing push to better standardise and make accessible more Open Data on politicians around the world; initially through our Poplus and Pombola projects, but more recently – and with more success – through our EveryPolitician service which has blossomed into a remarkable dataset of almost 4 million datapoints on over 72,000 politicians in 233 countries and territories.
Despite these successes I don’t think we’ve yet sufficiently cracked the challenge at scale of enabling more organisations to monitor and report upon the work of more politicians in more countries. We need to do something about that.
One of the principles that has always underpinned mySociety is that we carry our work out in the open, freely available for others to use. But, as is common with many Open Source projects, we do most of the development work ourselves internally. While community contributions are very welcome, practicality has dictated that more often than not, these are more commonly directed to raising tickets rather than making changes to the actual code.
Unchecked, this situation could lead to us being too internally focused; on developing everything ourselves rather than recognising where we can achieve our objectives by supporting other projects.
Fortunately our collaboration with Wikidata, announced earlier this year, suggests what promises to be a clear way forward to scaling up the impact of our work: we recognised that EveryPolitician could only become sustainable at scale as part of a wider community effort if we want our data to be used more widely.
By contributing to what we’ll call the Democratic Commons — a concept of shared code, data and resources where anyone can contribute, and anyone can benefit — we can help build and strengthen core infrastructure, tools and data that allow other democracy organisations and campaigners to hold their own governments to account.
This was notably put into practice for the snap General Election in the UK in June, where rather than build something new ourselves we directly supported the work of Democracy Club in their efforts to source candidate data and ensured that our existing services like MapIt, TheyWorkForYou and WhatDoTheyKnow were easily accessible for other campaigning and democracy organisations to put into use.
More recently we’ve established a commercial partnership with Facebook to provide them with accurate and independent lists of candidates and elected representatives matched to their relevant Facebook profile pages for the UK, French and Kenyan elections.
There’s a wider benefit to this kind of commercial work, beyond its being a useful source of additional revenue for mySociety. More importantly, it will allow us to feed the data that we source back into the Democratic Commons. It can contribute to EveryPolitician and Wikidata, and even improve boundary data internationally through OpenStreetMap, which in turn powers our own Global MapIt service.
Why is this important now?
Well, it’s not just the rather obvious observation that working with other people is a good idea. The reality is that we need to face the fact that our Democratic practice is just not fully funded, and, as with WhatDoTheyKnow.com, at best we’ll need to consider how more of our services in the UK can be run and directly supported by volunteers and the wider community.
At worst it’s quite possible that we’ll be forced to close some of our popular UK services and restrict the further development of our democracy work internationally.
In April next year we come to the end of our six-year grant agreement with the Omidyar Network who have given us tremendous support over that time. This will leave a substantial hole in our core funding and it’s one reason why we’ve been so diligently focused on developing appropriate new commercial services like FixMyStreetPro and WhatDoTheyKnowPro.
Without sufficient unrestricted core funding — that is, funding which can be applied wherever in the organisation it is most needed — we need to rely much more on specific project funding wherever we can find it. In most cases, however, this project funding comes with its own set of tasks to deliver, and there’s a tendency to want new shiny things, rather than supporting the maintenance of our existing projects. This is especially true of our Democracy work which relies more heavily on grant funding than commercial alternatives.
Sensibly directing our own work more towards contributions to external projects is also a hedge, should we need to find new homes for our services or shutter them for the time being.
In the meantime we’ll be speaking to more funders who we hope might recognise the importance of supporting and building the essential infrastructure of the Democratic Commons, but in the event that isn’t forthcoming we’ll do what it takes to ensure our work to date continues to have some value and impact.
As we start to map out a path to a sustainable future for mySociety and its community, I’d appreciate all thoughts on where we go next with this — after all, we can’t do this without your help.
MapIt has had a bit of a refresh to bring the look into line with the rest of the mySociety projects. At the same time, we thought we’d take the opportunity to make it a bit easier for non-technical folk to understand what it offers, and to make the pricing a little less opaque.
You may not be familiar with MapIt, but all the same, if you’ve ever found your MP on TheyWorkForYou, written to your representatives on WriteToThem, or reported an issue through FixMyStreet, you’re a MapIt user!
That’s because MapIt does the heavy lifting in the background when you enter a postcode or location, matching that input to the boundaries it falls within (ward, constituency, borough, etc). It is, if you like, the geographic glue that holds mySociety services together.
Like most of mySociety’s software offerings, MapIt is available for others to use. So for example, the GOV.UK website uses it to put users in touch with the right council for a number of services, and Prostate Cancer UK uses it on their campaign site, using MapIt’s knowledge of CCG (Clinical Commissioning Group) region boundaries.
And you can use MapIt too: if your app or website needs to connect UK locations with areas like constituencies or counties, it will save you a lot of time and effort.
Pricing and payment is a lot slicker now: while it was previously managed manually, you can now purchase what you need online, quickly and without the need for human intervention. It’s also quite simple to see the pricing options laid out.
We hope that this will make it easier for people to make use of the service, and better understand what level of usage they need. But if you need to experiment, there’s a free ‘sandbox’ to play about with!
As ever, we’re happy to provide significant discounts for charity and non-profit projects: see more details on the licensing page.
If you have any questions or comments please do get in touch.
What’s the best way to get your supporters to campaign, when the finer details of what they’re pressing for may vary from place to place? That’s the issue that faced Prostate Cancer UK as they call for better provision for men across the country with erectile dysfunction as a result of prostate cancer.
There are five core treatments for tackling erectile dysfunction, but whether all of them will be offered to you depends on your postcode. In some areas, all are offered as standard, while in others there may be none.
The tool we built for Prostate Cancer UK used several of mySociety’s areas of expertise, from mapping to user testing — we even used Freedom of Information. And putting it all together, we have a powerful campaigning platform that responds to users’ location, while raising awareness and pushing for improvement.
Prostate Cancer UK’s Erectile Dysfunction campaign site informs people about what care should be available to those who experience the condition as a result of prostate cancer treatment, and urges them to write to their local health commissioner if provision is poor in their area.
Educating, campaigning, sharing
The user is first informed: they are shown the five factors which constitute good treatment of erectile dysfunction. After that, they are prompted to input their postcode to see how many of those measures are provided by the NHS body responsible for their region.
If provision is poor, they are encouraged to help campaign: users can opt to write to their Clinical Commissioning Group (CCG), Health Board or Health and Social Care Board to ask them to improve what’s available. They are given the choice between writing a letter from scratch, or using a pre-composed template which also contains a section for the writer to add a paragraph of their own words — a pragmatic balance that avoids an influx of identical form letters, while still addressing fact that when users are faced with a completely blank page, many will drop out of the process.
When you’ve done that, for those in England there’s also an opportunity to contact Jeremy Hunt, Secretary of State for Health to highlight the variation in treatment for erectile dysfunction and establish which organisation is responsible for the national commissioning guidelines.
Finally, the user is invited to share what they’ve learned, via Facebook, Twitter or email. Our user testing revealed that, contrary to our worries, people were happy to do this without embarrassment.
How it works
Like most of mySociety’s own sites, the ‘Better Care’ site uses MapIt to match the user’s postcode with a boundary, in this case the boundaries of the CCGs, Health Boards and Health & Social Care Boards. That’s how we deliver the information about what’s available in their local area.
When you input your postcode to see how your local provisioners are doing, MapIt also delivers information for other areas, including a couple of close neighbouring ones. This allows us to provide a nice comparison, along with the statistic that shows whether your provisioner is better, worse, or within the same range as the average.
But how did we gather the data to tell you how well each CCG, Health Board or Health and Social Care Board is catering for erectile dysfunction patients? Well, fortunately, thanks to our own WhatDoTheyKnow website, it was relatively easy to send a Freedom of Information request to every one in the country — 235 of them in total. The WhatDoTheyKnow volunteer admin team were able to help with this large batch request.
Once we had all the data and a general idea of how the tool would work, we took an early version out to test it with users. The insights we gained from this process were, as always, extremely useful, and led to us altering page layouts and other elements that made the whole process as clear as it could be.
Finally, we incorporated quite a bit of statistics-gathering into the whole tool, so that Prostate Cancer UK would be able to see where their campaign might benefit from further optimising in the future.
All in all, we’re very glad to have been part of this important campaign to help men understand what’s available to them, and where they might need to push for more.
Census data: there’s lots of it. It contains fascinating insights.
But as with many huge datasets, those insights are not always easy to find at first glance — nor is it easy for the untrained observer to see which parts are relevant to their own lives.
Wazimap in South Africa takes the country’s census data and turns it into something the user can explore interactively. Originally conceived as a tool for journalists, it turned out to be so accessible that it’s used by a much wider range of the population, from school children to researchers. It’s a great example of how you can transform dry data into something meaningful online, and it’s all done using free and open source tools.
Our points-to-boundaries mapping software MapIt is part of that mix, putting the data in context and ensuring that visitors can browse the data relevant to specific provinces, municipalities or wards.
We asked Greg Kempe of Code for South Africa, to fill us in on a bit more.
What exactly is Wazimap?
Wazimap helps South Africans understand where they live, through the eyes of the data from our 2011 Census. It’s a research and exploration tool that describes who lives in South Africa, from a country level right down to a ward, including demographics such as age and gender, language and citizenship, level of education, access to basic services, household goods, employment and income.
It has helped people understand not just where they work and live, but also that data can be presented in a way that’s accessible and understandable.
Users can explore the profile of a province, city or ward and compare them side-by-side. They can focus on a particular dataset to view just that data for any place in the country, look for outliers and interesting patterns in the distribution of an indicator, or draw an indicator on a map.
Of course Wazimap can’t do everything, so you can also download data into Excel or Google Earth to run your own analysis.
Wazimap is built on the open source software that powers censusreporter.org, which was built under a Knight News Challenge grant, and is a collaboration between Media Monitoring Africa and Code for South Africa.
Due to demand from other groups, we’ve now made Wazimap a standalone project that anyone can re-use to build their own instance: details are here.
How did it all begin?
Media Monitoring Africa approached Code for South Africa to build a tool to help journalists get factual background data on anywhere in South Africa, to help encourage accurate and informed reporting.
Code for South Africa is a nonprofit that promotes informed decision-making for positive social change, so we were very excited about collaborating on the tool.
Could MapIt be useful for your project? Find out more here
How exactly does MapIt fit into the project?
Mapit powers all the shape boundaries in Wazimap. When we plot a province, municipality or ward boundary on a map in Wazimap, or provide a boundary in a Google Earth or GeoJSON download, MapIt is giving Wazimap that data.
We had originally built a home-grown solution, but when we met mySociety’s Tony Bowden at a Code Camp in Italy, we learned about MapIt. It turned out to offer better functionality.
What level of upkeep is involved?
Wazimap requires only intermittent maintenance. We had municipal elections in August 2016 which has meant a number of municipal boundaries have changed. We’re waiting on Statistics South Africa to provide us with the census data mapped to these new boundaries so that we can update it. Other than that, once the site is up and running it needs very little maintenance.
What’s the impact of Wazimap?
We know that Wazimap is used by a wide range of people, including journalists, high school geography teachers, political party researchers and academics.
Code for South Africa has been approached a number of times, by people asking if they might reuse the Wazimap platform in different contexts with different data. Most recently, youthexplorer.org.za used it to power an interactive web tool providing a range of information on young people, helping policy makers understand youth-critical issues in the Western Cape.
We also know that it’s been used as a research tool for books and numerous news articles.
The success of the South African Wazimap has driven the development of similar projects elsewhere in Africa which will be launching soon, though MapIt won’t be used for those because their geography requirements are simpler.
What does the future hold?
As we’re building out Wazimap for different datasets, we’re seeing a need for taking it beyond just census data. We’re making improvements to how Wazimap works with data to make this possible and make it simpler for others to build on it.
Each new site gives us ideas for improvements to the larger Wazimap product. The great thing is that these improvements roll out and benefit anyone who uses it across every install.
Thanks very much to Greg for talking us through the Wazimap project and its use of MapIt. It’s great to hear how MapIt is contributing to a tool that, in itself, aids so many other users and organisations.
Need to map boundaries? Find out more about MapIt here
If you’ve visited the MapIt site this week, you might have noticed a change: we’ve introduced key-based authentication for API users.
This enables us to be more flexible about how we provide our service, which means you can be more flexible about how you serve your users.
MapIt is both an open source application and, via https://mapit.mysociety.org, a web service. Use of the API is free for low-volume, charitable use, while all other uses require a licence.
For the moment API keys are optional. We’ll always offer a free level of service to support independent developers and charities.
We’ll have more details soon about the increased flexibility this change will bring.