1. How a payslip taught France about their Transparency rights

    Have you ever wondered what the Prime Minister’s payslip looks like? We’re not talking about how much he’s paid — that’s a matter of record — but the actual paper document.

    Over in France,  Xavier Berne from the Alaveteli-based site Ma Dada has just created a bit of a stir by receiving Emmanuel Macron’s fiche de paie in response to an FOI request. This response set off extensive coverage in the French national media on how citizens can use their right to transparency.

    Payslip of President Macron

    Ma Dada were kind enough to talk us through what happened, and how it’s resulted in a better understanding of FOI across France. Developer Laurent Savaete sent us the picture you see at the top of this post, of the front cover of the national paper Libération, which he describes as France’s rough equivalent to The Guardian.

    “Transparency was the front page story”, Laurent says, “which is highly unusual. The next four pages covered transparency, how everyone can use it, and profiles of four power users.” National TV also picked the topic up.

    Just like here in the UK, Macron’s actual salary was already known. The scoop was in receiving the facsimile of the paper document, which came about because Laurent’s colleague Xavier sometimes sends out interesting FOI requests, “for educational purposes, to show that it is possible”.

    Quite by accident, this one actually turned out to be a perfect piece of FOI promotion: because the content of the payslip was nothing new in itself, the media focused on the means by which it was obtained.

    How it happened

    WhatDoTheyKnow dreams of such amazing coverage, and we’re sure that Alaveteli projects around the world feel the same! So, how did it come about, and can other FOI projects replicate these conditions in their own countries?

    Well, we can try, but Laurent reckons that it was broadly down to the planets aligning. As he explains it, “A journalist had been in touch around that time about something else; Xavier happened to mention he’d received the payslip, and she was interested in writing a story. 

    “She is not a long time FOI user herself. To be honest, it was lucky timing — Xavier had just received the response, and the news cycle needed something to break up their coverage of Gaza and Ukraine.”

    If there’s a lesson to come out of this, we think it might be: keep requesting interesting stuff, and keep talking to journalists.

    A spike in new users 

    Because Ma Dada knew the coverage was coming, they had time to put some safeguards in place to make sure the site could handle extra traffic. “We cranked the server up to the max to make sure everything kept running  — which worked across the swell in interest, but we wouldn’t be able to afford keeping it at that level longer term”.

    The peak in visitors may have retreated now, but there’s been an undoubted uptick in usage. “We’ve been seeing a surge in registration for both standard and Pro accounts, and a definite increase on requests being made on Ma Dada for the last two weeks. 

    “We had around 40,000 visitors on the day of story; and 15-20,000 the day after.”

    Reaching the people

    How do you know your media coverage has touched the nation? Well, Laurent was given a nice clear illustration: “The day after the paper came out, I was chatting with the owner of a bookshop in the south of France, and he asked what I do in life, so I said: ‘I work on a project in public transparency’, to which he replied, ‘Oh, they just talked about that on the radio this morning!’. 

    “That radio segment was discussing our publication and mentioned Ma Dada. It was super cool to bump into someone who’d actually heard of us outside of our own dedicated circles.”

    Very cool indeed! And now? Ma Dada are already thinking how they can replicate this great success. “We’re thinking about other documents we might request to get the same amount of publicity again in the future.”

    Best of luck, Ma Dada! Your next story should be received by a more informed general public, thanks to that alignment of the planets.

  2. Access to Information network: data visualisation Show and Tell

    Organisations all around the world run Freedom of Information sites based on our Alaveteli platform. There are also sites running on other codebases. Either way, we’re all working to the same goals, and we believe that each site can learn from the others: for that reason, we’ve set up the Access To Information (ATI) network, a place for people who run FOI sites to meet, chat and share experiences.

    Within this framework, there’s the chance for practitioners to pair up and exchange specific skills under a mentorship arrangement — and last week, we heard from GONG in Croatia who have been learning from Georgia’s ForSet.

    ForSet began life as a data visualisation organisation, and these beginnings have allowed them to communicate data from their FOI site as compelling, visual stories: skills that GONG were keen to get to grips with. Sara, project co-ordinator at GONG, told us about working with Jubo from ForSet on data visualisations — and how her learnings will change the organisation’s work going forward.

    Sara explained that they agreed two main goals for this project: firstly, to improve GONG’s data visualisation skills; and secondly, to use data visualisation to promote their FOI site Imamo pravo znati (IPZ) to journalists and general users. They were successful on both counts, not only did Sara learn how to use new methods and tools; but their outputs also brought approximately 50 new users to IPZ, and two additional Pro users (Pro usage is free on the site, but numbers had been stagnant of late, so this was notable).

    So, how did they go about it? The mentorship comprised four stages: data collection, analysis, storytelling and visualisation, with the last being very interconnected.

    1. Data collection

    This stage began with both sides brainstorming potential topics for FOI requests that would be good candidates for data visualisation. An initial set of 12 topics was whittled down to five: local referendums in Croatia; special advisors (Spads) in the Croatian government; local participatory budgeting projects; local support for youth civic education; and local financing of civil society organisations. 

    GONG then sent 575 requests to local and county authorities, from which they received 525 responses — a pretty good response rate, and higher that expected. They didn’t hit many problems, although several authorities asked for the requester’s ID details, and there was one ministry that cited GDPR as a reason for refusing information on Spads. This case has now been referred to Croatia’s Information Commissioner. 

    2. Data analysis

    Jubo and Sara organised the responses they received into spreadsheets: they were looking for an angle or a story among the data, and tidying it up in this way was helpful for making comparisons. Could they find a story in there that aligned with GONG’s mission or advocacy?

    By organising the data in this way, the pair could easily see which data wasn’t really strong enough to take any further, and which lent itself to storytelling and visualisation. At this stage they rejected some of the angles they’d begun with, narrowing their projects down to local referendums, Spads, and lowering the voting age to 16 for the EU elections (this last project is pending; they’ll be launching a campaign in late Spring).

    3. Storytelling and visualisation

    Two pieces of software were used for the visualisations: Canva and Flourish. Sara was already familiar with Canva, as she uses it to create social media graphics; but Flourish was new to her, and she says she is very happy to have these new skills under her belt.

    Flourish allows you to create many different types of visualisations: you upload your data and it is fairly intuitive to create maps, charts, etc. Once these were in hand, they added a call to action for each story, encouraging people to use their FOI site and especially Pro.

    The visualisations

    Local referendums

    For the story on local referendums, GONG had requested from each local authority the number that had taken place; the topics under discussion; their outcomes; and the number of referendums that were suspended due to not being valid for whatever reason.

    They received more responses than expected, and this was also the first time nationwide data had been collected on the subject.

    Map showing where Croatian referendums were successful or otherwise in reaching quorate

    The first angle that GONG wanted to support with their data and visualisations was ‘Croatia needs a referendum law that recognises the importance of local democracy’. 

    The data showed that out of 47 local referendums that had been held, just 25 met the minimum turnout for the decision to be valid. Jubo and Sara mapped these, and paired their visualisations with the argument that a lower bar for turnout would encourage better participation in local democracy – demonstrated with human figures.

    Turnout quorum for a local referendum
    A local press outlet picked the story up, using the data to make their own story: they were the area that had had the highest number of referendums, so that was their focus. 

    Special Advisors

    The FOI requests returned the names of Special Advisors, the areas they were in charge of, and the fees they were receiving. As Sara explained, in Croatia Spads are not straightforwardly employees of the government, but they have a lot of influence, and in some cases receive huge fees.

    It became clear that there are two different types of advisors, under two laws; while each type has different functions, both are called Spads. First, there are external advisors who may or may not receive compensation; and secondly there is another class of Spads who are employed internally. Neither is subject to Croatia’s legislation on conflict of interest.

    Number of SPADS in each Croatian ministry

    A pie chart was put to service to clearly show how much compensation Spads had received. This varied widely from Spad to Spad, but the criteria dictating who received how much is still unclear: it appears to be at the discretion of the individual minister.

    Pie chart showing SPAD payment in Croatia

    In collecting this data, GONG unexpectedly unearthed a scandal, as they revealed one Spad who was abusing public funds. He was fired, along with the minister concerned; this resulted in nationwide coverage for the data; albeit again with the media’s own preferred focus.

    Lowering the voting age

    Sara says that it was a lot of work to find data to support the argument for lowering the voting age to 16 in Croatia. They wanted to show that, while young people see voting as the most efficient political action, it is denied to a large portion of them.

    Proving the absence of something is always tricky, and in this case they were uncovering that there isn’t any research to show that 16 year olds lack the cognitive abilities to vote responsibly. So they focused on other angles: in some EU countries, 16 year olds can vote, and they demonstrated that those countries are doing well in democratic processes: they score highly in the democracy index and have good voter turnout.

    Data visualisations around the voting age in Croatia

    Like many countries, Croatia’s population is ageing, so the young are in danger of being totally ignored. GONG plan to share their findings on social media in a simplified form with graphics cards, and a call to action to show support for the campaign.

    Questions and answers

    Once Sara had finished her presentation, members of the audience were invited to ask questions.

    Q: How did GONG and ForSet work together?

    A: At the beginning, they had lots of online video calls, and later on when the data had come in, they communicated a lot via comments on the spreadsheets.

    Q: It feels like each of these the topics would be applicable almost everywhere: perhaps it will spark other people’s interest to do something similar for their own country. Any advice if so?

    A: The questions asked in the first two sets of FOI requests were straightforward, which led to straightforward answers. The third topic was less so; Sara and Jubo had to go through lots of reports, and often the data from one source contradicted another. Also, an uncontentious topic is likely to result in more responses: something like referendums is politically neutral, unlike spads where the authorities may have reasons not disclose information.

    Q: When you put the requests together, were you already thinking about the format it would be best to receive the data in?

    A: In that respect, the best question is one with a yes/no answer. The reason for excluding many of the initial topics at a later stage was that the answers varied so widely that it was hard to pull out statistics or a simple throughline: you’d be comparing apples with pears. So, for example, when asking how much of a local authority’s budget is reserved for supporting civic education, and how civic education is delivered, the responses could range from “We are in favour of civic education, but leave it to schools”, to “We provide money for civic education and produce our own textbooks”. Meanwhile, some authorities wrote two pages of waffle in the free text box. 

    Q: Did you narrow down the topics before or after you had submitted the FOI requests?

    A: Both. There were 12 topics at the start; they decided which of them were best suited to FOI, then sent requests for five of them. One the answers had been received, they narrowed it down to three.

    Q: Could one make data visualisation about the other two? It’s hard to find ways to show that there’s no information. Saying that 80% of authorities don’t reply is not a very exciting way of showing things.

    A: While it might not fit in with the initial aim of the project, this sort of thing can be a great way to show how well or badly FOI is working in your country. Journalists often can’t get the information they need, so build stories around the fact that there’s no data about such an important issue.

    Q: We’ve seen how much GONG has benefitted from this mentorship. What, if anything, did ForSet get from this?

    A: Sara was so quick and flexible, she was great to work with. ForSet also learned from the project: for example, that it is better when requesting a large amount of data, that is sorted by the public institution, so it’s easier to work with. You can request it sorted in the way that you need for your story, which might be different from how it is in public.

    Also, Canva is such a great tool for visualisations. They’ve now merged with Flourish, so the have advanced data visualisation features. You just have to make sure you choose the right format: the type of charts or graphs that will show your findings the most clearly. 

    Finally, ForSet didn’t know about the topics that Sara suggested, so there was plenty to learn there, plus it was great to see the ways GONG employ to publish their stories on both social media and mainstream media. 

  3. Creating datasets from FOI data

    Responses obtained from a widespread FOI project can be difficult to analyse, until they are sorted into neat datasets. This allows you to make valid comparisons, pull out accurate statistics and ultimately ensure your findings are meaningful.

    In our third seminar within the Using Freedom of Information for Campaigning and Advocacy series, we heard from two speakers. Maya Esslemont from After Exploitation explained how to prepare for an FOI project to ensure you get the best results possible (and what to do if you don’t); and Kay Achenbach from the Open Data Institute explained the problems with ‘messy’ data, and how to fix them.

    You can watch the video here, or read the detailed report below.

    Preparing for an FOI project

    After Exploitation is a non-profit organisation using varied data sources, including FOI requests, to track the hidden outcomes of modern slavery in the UK.

    Maya explained that they often stitch together data from different sources to uncover new insights on modern slavery. She began with a case study showing some recent work they had done, using WhatDoTheyKnow to help them understand the longer term outcomes after survivors report instances of trafficking. This stood as an excellent example of how much work needs to be done before sending your requests, if you are to be sure to get the results you need.

    In this case, After Exploitation were keen to understand whether there is any truth in widely-held assumptions around why human trafficking cases are dropped before they are resolved: it’s often thought that there are factors such as the survivors themselves not engaging with the police, perhaps because of a nervousness around authorities.

    But what are these assumptions based upon? Actual information was not publicly available, so we wouldn’t know if cases were being dropped because of low police resource, a lack of awareness or more nuanced factors. Until the data could be gathered and analysed, the perceptions would continue, perhaps erroneously.

    Before starting, After Exploitation thought carefully about the audience for their findings and their ultimate aims: in this case the audience would be mostly the media, with the aim of correcting the record if the results flew in face of what was expected; but they knew that the data would also be of use to practitioners. For example, charities could use it to see which areas to target regionally for training and other types of intervention.

    They placed FOI requests with police forces across the country, making sure to ask for data using the crime codes employed by the forces: were cases dropped because of ‘lack of evidence’; did they have a status of ‘reported’ but not gone on to exist as an official crime record?

    The project had a good outcome: while some requests had to go to internal review, ultimately over 80% of the forces responded with quality data. The findings were worthwhile, too: general perceptions did indeed prove to be wrong and there was no indication that ‘no suspect identified’ was a result of the victim’s lack of involvement. The resulting story was able to challenge the general narrative.

    So, how can After Exploitation’s learnings be applied to the work of other organisations or campaigns?

    Maya says:

    • Planning, rather than analysis, is the majority of the work;
    • Identify the need and purpose before you even start to pick which authorities to send requests to;
    • Be clear who the audience for your findings is;
    • Consult with other stakeholders to make sure your parameters are really clear.

    Planning

    Before you even begin, make sure your project isn’t asking for data that has already been collected and is in the public domain — this might seem obvious but it’s easy to overlook. Check other people’s FOI requests (you can do this by searching on WhatDoTheyKnow); look for reports, research, inspectorate/watchdog outputs, and data released as part of parliamentary enquiries.

    That said, even if you do find previous data, there is sometimes value in requesting more up to date or more detailed information with a new set of FOI requests. If you see a national report collating data from every council for example, you could do an FOI project asking every council for a more detailed breakdown of what is happening in their region.

    But before sending a batch of requests to multiple authorities, ask yourself if there is a centralised source for your data. If so, then just one FOI request might be enough: for example, homelessness data is already collected by the Depts for Housing, Levelling Up and Communities, in which case one request to them would save time for both you, and more than 300 public authorities.

    Another question to ask before starting off on your project is “what is the social need?”. Does this need justify the resource you will expend? Mass FOI projects can be a bit of a time commitment, but the utility might not just be for your organisation: perhaps you can also identify a social benefit if the data would be of use to other groups, academics or journalists.

    Define your intended audience: will the data you gather be of interest to them? Do you have a sense of what they want? For example, MPs often like to see localised data that applies to their constituencies. Journalists like big numbers and case studies. If you think your findings are important but might have limited appeal, you could consider including an extra question to provide details that you don’t need for your own purposes, but which could provide a hook.

    Next, will the data that you gather actually be suitable for the analysis you want to perform? To avoid time-consuming mistakes, make sure the data you’ll receive is broken down in the way that you need. As an example, suppose you wanted to ask local authorities for details of programmes offered to children in different age bands: you might receive data from one council who has offerings for children ‘under 18 months’ and another ‘under two years old’ — and where units differ, they are difficult to compare and contrast. Be really precise in your wording so there’s no mismatch, especially if your request is going to a lot of authorities.

    Consider, too, whether you can you get enough responses to make your data meaningful: 2,000 people is the figure believed to be representative of the population as a whole. Decide how many responses you ideally need for your purposes — and, in a scenario where not all authorities respond, the minimum you can work with.

    You might want to contact other groups or organisations who could be interested in the same data, and ask if there are details that would be useful to their work.

    As suggested in Maya’s case study, try to use existing measurements where you can: if you shape your requests to the methodology the authorities themselves use to collect the information, such as KPIs or their own metrics of success, these will be much easier for them to supply.

    If you’re not sure what these metrics are, you can sometimes access internal guidance by googling the name of the authority plus ‘guidance’. Alternatively, submit scoping requests to a handful of authorities to ask how they measure success, etc.

    At this stage it’s also useful to decide what quality of data you will include or exclude. For example, if you ask about training materials and one authority says they offer training, but don’t include the actual materials, do you include it in your figures? The more authorities you ask, the more ambiguities like this you’ll normally encounter.

    Think about where and how you will log the data as it comes in. Maya recommended WhatDoTheyKnow Projects as a good tool for extracting data. Whatever you use, you should consider accessibility: can your platform be accessed by everyone you’re working with, across different communities? Especially if you are working with volunteers, it’s important to remember that not everyone has a laptop.

    Also consider the security of the platform: how much this matters will depend on how sensitive the data is, but recognise that Google sheets and many other platforms store the data in the cloud where it could be more vulnerable to abuse.

    After Exploitation take great pains to ensure that their data is accurate. They recommend that each response is assessed by two different people, making sure that everyone knows the criteria so they’re applied consistently; and doing regular spot checks on a handful of cases to make sure they are all logged in the same way and there’s no duplicate logging.

    This is time-intensive and arduous, but if you have other stakeholders they might be able to help with the data checking: for example, knowing that they would eventually place the story with the BBC, After Exploitation were happy to hand this task over to their inhouse data checkers.

    What if things go wrong?

    If you’ve done all the planning suggested above, it’s less likely that your project will go awry, but even if it does, Maya says that there’s always something you can do.

    No or few responses: ask yourself whether you have the capacity to chase no/late replies, and if you still don’t get a response, to refer them to the ICO. If not, consider prioritising the bodies that are most relevant to your work, eg the biggest authorities or those in areas with the densest populations; but be prepared to defend accusations that not every authority had a fair hearing unless you do them all.

    If you know your requests were well worded, but you’re not getting a lot of responses — perhaps because you’re dealing with a contentious issue, or simply because the authorities cash-strapped — you could shift to measuring the types of responses you get. If authorities aren’t able to answer the question, this can often be just as revealing.

    Responses that don’t tell you what you set out to understand: Consider whether there are any alternative angles in the data you do have: are there any additional themes, particularly in any free text fields? Or try a new round of requests asking for more detailed information.

    Responses don’t cover the whole country: If you can’t get data from everywhere, could you narrow down to just one area and still have useful findings? Even the most basic data can set the scene for other researchers or organisations to build on: you can put it out and outline the limitations.

    Results

    The impact of gathering data through FOI can be massively powerful, as After Exploitation’s work shows. They have revealed the wrongful detention of thousands of potential victims of human trafficking when the government were denying it could happen; opened the debate about locking up vulnerable people; and uncovered the flawed decision making in the Home Office on modern slavery cases. It was only through FOI requests that all this information came into the public domain and was picked up by mainstream media.

    Combining different sources of data to create datasets

    Kay Achenbach is a data trainer on the Open Data Institute’s learning team; the ODI works with government and companies to create a world where data is working for everyone.

    Kay shared a case study from the medical field, in which an algorithm was being designed to quickly assess high numbers of chest x-rays. The aim was to automate the process so that people identified as needing intervention would be sent to specialists right away.

    The developers wanted to make sure that different demographic groups weren’t being biased against, a common issue with algorithms built on existing data which can contain previously undetected biases.

    The test material was a set of x-rays from a diverse population, that had already been examined by specialists. They ran them past the algorithm to see if the diagnoses produced were the same as those made by human doctors.

    The doctors’ assessments came from three different datasets which, combined, comprised data from more than 700,000 real patients. As soon as you combine datasets from different sources, you are likely to come across discrepancies which can make analysis difficult.

    In this case, one dataset had diagnoses of 14 different diseases, and another had 15 — and from these, only eight overlapped. The only aspect that could for sure be compared was the “no finding” label, applied when the patient is healthy. That limitation set what the algorithm was asked to do.

    Other fields were problematic in various ways: only one of the three sources contained data on ethnicity; one source only contained data on the sickest patients; another was from a hospital that only takes patients with diseases that they are studying, meaning there were zero “no finding” labels. Two of the sources contained no socio-economic data. Sex was self-reported in two of the sources, but assigned by clinicians in the other, which could also affect outcomes.

    The advice from all this is that you should look carefully at each dataset before you combine them, to see what the result of combining them would be. In short: does it reflect real life?

    Ultimately the researchers found that the algorithm was reflecting existing biases: it was much more likely to under-diagnose patients from a minority group; more likely to make mistake with female patients, the under 20s, Black people, and those from low socio-economic groups. The bias was compounded for those in more than one of those groups.

    Cleaning up datasets

    Once you’ve obtained your datasets from different FOI requests, you’re highly likely to find mismatches in the data that can make comparisons difficult or even impossible — but cleaning up the data can help.

    For example, in spreadsheets you might discover empty fields, text in a numbers column, rows shifted, dates written in a variety of formats, different wording for the same thing, columns without titles, typos and so on.

    Kay introduced a tool from Google called Refine that will solve many of the issues of messy data, and  pointed out that the ODI has a free tutorial on how to use it, which you can find here.

  4. This is not just any Impact Report…

    …this is mySociety’s 20th anniversary Impact Report!

    Cover of mySociety's 20th anniversary impact reportThis has been a very special year – mySociety’s 20th anniversary.

    So we haven’t just put together our usual review of the past twelve months: this Impact Report is a special edition, covering our entire history since 2003.

    We look back at our beginnings as a small group of determined coders, and trace our history through the changes our services have made, here in the UK and across the world.

    Discover daring acts of (data) piracy, and learn which vandalised phone box sparked the idea for FixMyStreet. Find out how our “cheap and cheerful open web technology” has been instrumental in helping citizens tackle vital issues, from the climate emergency to human trafficking.

    It’s quite the read. Sit back, grab a mince pie if you have one to hand…and enjoy! Access the Anniversary Impact report here (web), or enjoy the print-faithful PDF version, or plain text and epub formats.

    And if you’re interested in our activity on the SocietyWorks side, don’t miss their own, just as engrossing, annual report: you can read that here.

  5. Shortlist announced for mySociety’s 20th anniversary awards

    The ways in which people and organisations have used mySociety’s services through the lifetime of the organisation have been impressive, inspiring and sometimes astonishing.

    So, to celebrate our 20th anniversary, on 15 November we’ll be presenting awards in five categories, showcasing impactful usage of their services through the years.

    • Driving Institutional Change
    • Accelerating Climate Action
    • Exposing Truth
    • Impactful International Reuse
    • Campaigning for Justice

    The shortlist is as follows:

    Driving Institutional Change

    • The Give Them Time campaign used WhatDoTheyKnow to get the law changed over funding for nursery care in Scotland.
    • John Graham-Cumming In 2009, John used the petitions website that mySociety had built for 10 Downing Street, resulting in Gordon Brown apologising on behalf of the British Government for its treatment of the computer scientist Alan Turing.
    • Richard Bennett used WhatDoTheyKnow, coupled with the Equality Act, to make pathways more accessible for wheelchair users, sharing his methods so that others could do the same.
    • Privacy International The ‘Neighbourhood Watched’ project used WhatDoTheyKnow to reveal the unchecked use of surveillance technology by police forces across the UK.

    Accelerating Climate Action

    • Zero Hour Using mySociety’s WriteToThem software, they’ve garnered the backing of over 150 MPs for their draft Climate and Ecology Bill.
    • Sustain used data from CAPE, our Climate Action Plans Explorer, to analyse the degree to which local authorities are including food within their strategies to cut emissions. 
    • Save the Trees of Armada Way Plymouth’s grassroots campaign fought against the removal of much-loved trees in the city centre, using WriteToThem to send emails to the local councillors — apparently, the most emails they had ever received on a single subject. 

     Exposing Truth

    • Jenna Corderoy Jenna is shortlisted for her investigation — using WhatDoTheyKnow — of the Cabinet Office’s controversial Clearing House, a secretive unit that screened  and blocked FOI requests made by journalists and campaigners, often on matters of serious public interest.
    • The Bureau of Investigative Journalism Their Sold From Under You project used crowdsourced and FOI data to reveal how much publicly-owned property was sold off by councils across England, in an attempt to fill funding gaps caused by austerity measures. 
    • Lost in Europe worked with people running FOI sites on our Alaveteli platform, in 12 different countries, to uncover previously unknown statistics around how many children disappear at borders

    Impactful International Reuse 

    • Dostup do Pravda/Access to Truth The Ukrainian Freedom of Information site continues providing access to information even in the difficult circumstances of war.
    • vTaiwan, Public Digital Innovation Space, and the Taiwanese Ministry of Digital Affairs The Taiwanese government uses mySociety’s SayIt software to make deliberations on difficult subjects public and accessible to citizens.
    • DATA Uruguay The organisation has built both FixMyStreet and Freedom of Information sites on mySociety’s codebases, changing the way their governments  communicate with citizens at both local and national levels.

    Campaigning for Justice 

    • Doug Paulley is a lifelong campaigner for rights for disabled people, using FOI to fight against access discrimination, especially around public transport.
    • Eleanor Shaikh has dedicated hours and hundreds of FOI requests to finding out the truth behind the Post Office Horizon scandal, with her findings making front page headlines.
    • After Exploitation use Freedom of Information to uncover the failings of the government’s measures to protect vulnerable detainees.

    Of course, every single user of our services is a winner in our eyes – but watch this space to find out who takes home the award in each category!

    Image: Rene Böhmer

  6. Ever more transparency for Alaveteli

    Here at WhatDoTheyKnow, mySociety’s service which publishes Freedom of Information requests and responses online, we care a lot about how transparent public authorities are. That being so, it’s only right that we try to be transparent, too. Of course, we already strive to be as transparent as we can, but we’ve recently increased how much information we provide about requests that we’ve had to stop publishing.

    It’s really rare that we stop publishing the entirety of a request, but since starting to produce annual transparency reports, we’ve noticed that in those exceptional cases, we weren’t always able to be clear about why we had to do it. Our records were kept in our support mail system, and couldn’t be published on WhatDoTheyKnow due to some technical limitations.

    Since October 2022, if you visit a link to a request that has been removed, where the information exists, you’ll now see a brief message explaining why the request isn’t available. We were already doing something similar for instances where we’ve had to hide individual messages or attachments. While the information necessary to display these more detailed reasons isn’t always available, especially for older requests, we hope that this small improvement is yet another beneficial improvement to how we run the site.

    Screenshot of a request that has been removed from WhatDoTheyKnow with the message: "Request has been removed. This request has been hidden. We consider it is not a valid FOI request as it was correspondence about personal circumstances, and we have therefore hidden it from other users. If you are the requester, then you may sign in to view the request. Please contact us if you have any questions." The text is linked to sign in and to contact us at the appropriate places.

    Alaveteli, the software platform that powers WhatDoTheyKnow and many other Freedom of Information services globally, doesn’t just let us choose between something being visible or not;we can choose between several levels of visibility. These range from completely publicly visible, to visible only to those with a direct link, to visible only to the user who originally made the request, to visible only to the WhatDoTheyKnow administration team, which we only use in some of the most serious cases.

    Screenshot of Alaveteli's admin interface for editing a request's basic details. Two fields are highlighted in the screenshot. The first is "Prominence", which is a drop-down menu set to "normal". Below that, there is the text "(backpage means hidden from internal and external search engines; hidden means completely hidden; super users can see anything)". The second highlighted field is "Reason for prominence", which is a currently blank paragraph input box. Below the input box is the text "This reason is shown in public. If left blank, a generic reason of 'There are various reasons why we might have done this, sorry we can't be more specific here.' will be shown.".

    This feature allows us to minimise the potential negative impact of publication, while retaining as many of the benefits as possible, with varying levels depending on the results of our carefully considered, case-by-case assessments.


    Image: Mehdi MeSSrro (Unsplash License)

  7. AskGov wins Copenhagen Democracy Summit Tech Award

    Great news from the Copenhagen Democracy Summit this week, where AskGov, the Access to Information website for Georgia, was recognised with an award for Democracy Tech.

    The civic tech organisation ForSet runs AskGov, using our Alaveteli platform, and you may remember that we had a valuable exchange of views and experiences with their cofounder Teona Tomashvili in London last year.

    In Copenhagen, fellows of the the Alliance of Democracies’ Democracy Tech Entrepreneurship Program, of which Teona is one, were invited to ‘pitch’ their project in a Dragon’s Den-like set-up. Teona gave an excellent explanation of the website — which would apply equally to any FOI site running on our Alaveteli platform — and you can watch it for yourself in this video:

    Along with the glory of winning came a very useful prize in the shape of a cheque for $10,000 to be put towards the project, as you can see in the image below. This was presented by Anders Fogh Rasmussen, who founded the Alliance of Democracies.

    Massive congratulations to Teona, whose pitching skills and determination were key to AskGov’s success in these awards.

    Teona Tomashvili receiving the Democracy Tech award at the Copenhagen Democracy Summit
  8. New in Alaveteli: hiding individual attachments

    At our Freedom of Information service WhatDoTheyKnow, when faced with requests to remove material, we operate on the principle of removing the minimum amount possible.

    Alaveteli, the codebase which underlies WhatDoTheyKnow and a number of other FOI sites around the world, gives moderators a range of options for removing content – with the ability to surgically remove text ranging from individual words and phrases, to individual messages, or even entire request threads. This is useful when we spot misuse of our service, for example.

    What we’ve been lacking, up until now, was a way to apply these types of removals to attachments.

    Back in the early days of WhatDoTheyKnow, attachments were less common, but now we see many more: there can often be several attachments to one individual message.

    Over the last few years, there have been occasions where we’ve had to remove an entire message, which may contain several useful attachments, just because of a small issue with one of them.

    We’d then go through an annoying manual process to download the publishable ones, upload them to our file server, and then annotate the request with the links – here’s an example.

     An FOI response, above which is the annotation: We have redacted a name from one of the released documents, acting in line with our published policies on how we run our service. We have republished the response and attachments in an annotation below.

    Back in 2013, when the original suggestion for enabling finer grained control was raised, the site contained around 400,000 attachments. There are now more than 3,500,000! We don’t remove content often, but at this scale it’s inevitable that we need to intervene now and then.

    After a little code cleanup we were able to make individual attachment removal a reality. This allows us much more control over how we balance preserving a historic archive of information released under Access to Information laws, and running the site responsibly and meeting our legal obligations under GDPR.

    As an example, let’s imagine that the FOI officer replying to our request inadvertently makes a data breach when releasing some organisation charts in `organisation chart b.pdf`.

    A fake FOI response in which the officer releases an organisational chart.

    Previously we’d have had to have hidden the whole response. Now, we can go into the admin interface and inspect each individual attachment.

    A list of file attachments

    We can then set our usual “prominence” value – offering a few options from fully visible to completely hidden – and include a reason for why the content has been hidden. We always seek to run the site transparently and explain any actions taken.

    Prominance: Hidden.Reason for prominence: attachment contains significant data breach

    On saving the form, you can see that only the problematic attachment has been removed, with the remainder of the response intact. This saves us considerable time when reviewing and handling material with potential data issues, and keeps as much information published as possible while we do so.

    Response with one hidden attachment

    As an extra bonus, since the main body text of emails is also treated as an “attachment” in Alaveteli, we’re now able to hide potentially problematic material there without affecting the attachments we present.

    A list of file attachments

    We’ve already used this feature several times to republish material where we’d previously had to hide the entire message due to the technical limitations at the time.

    Image: Kenny Eliason

  9. Notes: giving our users more information

    Alaveteli is our platform for running Freedom of Information websites — it underpins WhatDoTheyKnow as well as many other sites around the world. It’s made up of many interconnecting elements, but a key part is the database of public bodies.

    On WhatDoTheyKnow, we list over 42,000 public bodies that are subject to FOI and EIR – possibly one of the largest databases of public bodies in the country. Along with their name, we record information like their FOI request email address, publication scheme and disclosure log URLs, and we categorise them so that they’re easier to browse and make requests to.

    We often want to add additional insight to help citizens understand more about the public body. This can vary from body to body, from describing the information they hold, what they don’t hold, guidance around how to challenge their poor FOI-handling practices or why we list them when they’re not currently subject to FOI.

    We do this via what we call “notes” – a free-text field per authority that admins can update with whatever information is useful for the particular body.

    We often find we need to add or update the same note for a group of authorities — for example, we added the same note to all Business Improvement Districts to explain our reasons for listing them. 

    This can be a major challenge. We list nearly 300 BIDs, and updating each manually, one at a time, would be a several-hour ordeal. An alternative is for our developers to write a quick one-off script to update the text, but that comes with a coordination cost, and can be tricky to work around other text that may be present in the free text field that we want to preserve.

    One of the behind-the-scenes ways we manage the categorisation of authorities is through tags. Many of our other records have the ability to add tags too.

    A picture of several tags, including ones like 'Cardiff' and 'Wales'

    This led to the thought that it would be great to be able to apply a note based on a tag, rather than only having one note field per authority. This would allow us to more quickly add useful information to groups of authorities.

    Another issue we wanted to solve was to be able to add notes to content other than public bodies, like FOI requests for specific types of information and our category lists. In particular we wanted to be able to celebrate requests that led to particularly useful public interest information being released or having wider impact, and also to add clarification around requests that may be misinterpreted.

    We’ve received funding from the Joseph Rowntree Charitable Trust to help support marginalised communities to make more well-formed requests, and to more effectively use the information they obtain to engage with and influence authorities in a way that contributes to fairer decision-making. Thanks to this, we were able to add this underlying capability, which we can use to continue to help users more easily understand how to navigate the complexities of public authorities and what information they hold.

    The first step was to extract the existing notes to a new generic Note model that could be attached to any other record. Ruby on Rails makes this easy through its polymorphic associations.

    We then needed an interface to allow admins to list and search tags for each type of record that has them, and browse all records that are tagged with a particular tag.

    A list of authorities as tags, all starting with the letter A

    A list showing which tags have been applied to each authority

    As well as it being generally useful to be able to browse our database through our tags, this gave us a place to add our new functionality for adding a note based on a tag.

    While Rails’ polymorphic associations make creating a link between one record and another really easy, they don’t cover creating the link through a free-text tag. Fortunately, it took nothing more than a few lines of code to link up notes that are directly associated with a single record, and notes that are applied via a tag.

    Now our public bodies can have notes applied to them and only them, and also notes applied to several authorities based on their tag.

    An authority with two tags applied, as described in the text that follows

    In this example, the Environment Agency has some specific notes about it, but since it’s also a local lighthouse authority, we can apply the relevant notes simply by tagging the authority record and the notes for local lighthouse authorities will automatically get applied. Magic!

    What individual messages look like on the Environment Agency page - a box at the top

    We’ve also applied this pattern to information requests. Now, we can add notes to individual requests, like this example that points to another source for obtaining the information…

    ATOS Anne

    …or applied via a tag so that we can point citizens interested in the climate response to the authority’s CAPE page.

    ATOS

    So far we’ve created 46 notes that get applied via a tag. These notes are applicable to 7,998 requests and 15,283 authorities. Using a rough guess of 30 seconds to manually apply a note to a record, it would take 11,640.5 minutes – 8 full days, or 25 working days – to do so for each of these requests and authorities. This just wouldn’t have been possible before.

    This new feature unlocks a whole new avenue for us to support citizens and users that we just wouldn’t have had capacity for otherwise.

    Image: Keila Hötzel

  10. How ForSet are spreading the word about Freedom of Information in Georgia

    We’re big on open source: much of the software we create at mySociety is freely available for anyone to use, and there are sites all over the world underpinned by our code. 

    While we try to ensure that our codebases are as easy as possible to self-serve, in practice installation can be complex enough that those wanting to use them often get in touch with questions. 

    Not so much with AskGov.ge, the Alaveteli-based Freedom of Information site for the republic of Georgia. Yes, there had been an email in the Alaveteli Developers Google Group seeking a developer, but the first we knew that a brand new fully-formed service had successfully launched was when Teona Tomashvili, from NGO ForSet, emailed to say she was planning a trip to the UK and would love to meet up for a chat.

    And so it was that we spent a happy couple of hours in London, finding out more about AskGov and the context in which it is providing an FOI service for the citizens of Georgia, as well as offering as much advice and experience from running the UK site WhatDoTheyKnow as we could cram in without overwhelming our visitor.

    A fragile democracy

    Talking FOI may not be the obvious way of learning more about a country’s history, culture and politics, but it’s a surprisingly effective one. Teona explained that Georgia joined the Open Government Partnership in 2012, and the country is keen to do all it can to improve transparency.

    They’ve actually had an operational FOI Act since 2000, five years earlier than ours came into force in the UK. But as an ex-member of the Soviet Union, she says, the country is not used to democracy and open government.

    “Georgia had never been a fully democratic state”, she explained. “It’s only 30 years since we were part of the Soviet Union and our democracy is still very fragile. There’s a new impetus towards teaching people that open government and open data are important, but citizens are not used to these things. They have never felt like that, ever!”

    With the site relatively newly launched, ForSet have seen their main task as increasing public understanding of FOI and normalising its use. This is something we were interested to talk about: over time we’ve come to the conclusion that while some people here in the UK are ‘super FOI users’ who might put in several requests a month, the majority of the population are unlikely to feel the need to use it more than a couple of times a year, if that. 

    Even so, there’s always room for awareness-raising and we agree that everyone should know they have a right to information, for the times when they do need it. 

    Bureaucracy woes

    Whenever we’ve been able to gather together an international group of people who run FOI sites, we often find that the core challenges they face are very similar — though they may come embellished with some unique local colour.

    When Teona told us of their woes with bureaucracy, it was definitely a story we’d heard before: authorities required not just a name, but personal details such as the address and phone number of the person making a request before they would process an FOI request.

    For an Alaveteli site, the problem with that is, of course, that both the request and the response are made publicly available online, and this information would publish out too.

    While the issue might be familiar, we don’t think we’ve previously come across the particular solution that ForSet put in place: when someone makes a request, they can fill in all their personal details in a form on the website. This is used to create a PDF which is attached to the email that the authority receives; meanwhile the personal data is automatically destroyed at ForSet’s end — desirable both for users’ privacy and to avoid any worries about data retention.

    Teona said that they’d only had this system in place for a couple of weeks, so it’s too early to know if it’s really working. As Gareth pointed out, in Alaveteli we always try to model the law as we believe it should work — for example, when WhatDoTheyKnow started out, some authorities didn’t accept FOI requests by email; eventually, things changed enough that official ICO guidance now states:

    “Requests made through the whatdotheyknow.com website will be valid” 

    and 

    “we consider the @whatdotheyknow.com email address provided to authorities when requests are made through the site to be a valid contact address for the purposes of Section 8(1)(b).”.

    On the other hand, not all countries have an overseer, and even if they do, change may not be quick to come, so we are keeping an eye on Georgia’s method to see if it’s one we might recommend to other sites. 

    Data visualisations

    ForSet is a social enterprise like mySociety: their commercial activities support their charitable ones. They started life as a data visualisation organisation, and that provenance informs much of their activity. This gives them a different angle to come at FOI from: it’s a data collection mechanism, the results of which can feed into infographics and visualisations that inform the public, often with an ‘expert’ in the middle to transform the raw data into something the public can follow at a glance.

    Knowing all this, it’s understandable that when they started thinking about how best to promote their new site, ForSet landed on the idea of competitions, asking entrants to create a data visualisation from one or more FOI responses.

    The regular contests have had an enthusiastic take-up. Topics vary, but, Teona says, “Of course, thanks to current events, there have been lots of stories regarding Russia in the last couple of months: how dependent the market is on Russia; what authorities’ electricity consumption is; lots about defence. 

    “Before the war broke out the topics were more varied: there were visualisations on domestic violence, economics, socially important issues. One nice one that we hadn’t foreseen was on the grape and wine history of Georgia!

    “We found that out of all the authorities, the National Statistics Bureau and the Ministry of Internal Affairs are the most responsive — they always send lots of data.” 

    Because ForSet have such relevant experience, once a contestant has decided which data they’re going to use, they can tap into advice from the organisation’s designers and analysts. So these contests are creating a new generation of data visualisers and journalists who can use FOI in this way – win/win!

    Some examples of the data visualisations: click to see each one at a larger size.
    You can also find an interactive visualisation here.

    And ForSet are not stopping there: they’ve also been thinking of running focus groups for FOI officers and citizens — again not something we’ve done very much of at WhatDoTheyKnow, but further proof that there’s always plenty for FOI site runners to learn from one another.

    Making connections

    And one final thing: why did we hear nary a peep about AskGov until Teona made contact? 

    We would love to have believed her first explanation, that the site documentation is so clear that there was no need to enquire about anything — but there was another factor at play, too, as Teona explained:

    “The first challenge for us in installing the site was that it’s written in Ruby. There aren’t a lot of Ruby developers in Georgia and they are in high demand — they tend to work for private US companies, and we couldn’t afford to hire them.

    “But as an NGO, you never have enough money anyway, so we can always think of ways to get around things. We looked around our neighbouring countries Moldova and Ukraine, and saw that there was an existing Alaveteli site in Ukraine.

    “We sent them an email and introduced ourselves. It turned out that the organisation Internews was giving them tech support, paying for a web developer – and they offered to share that resource with us! They said they’re always looking for partners.

    “We never got stuck at any point because the developer knew what she was doing, and actually we benefitted from the fact that she’d learned from prior mistakes setting up the dostup.pravda site for Ukraine. And Ukraine and Georgia are very similar countries in terms of the legislation etc, so it was simple.”

    So in other words, ForSet had done what we would have encouraged them to be doing if they had got in touch – networking and learning from others in the Alaveteli community!

    Talking of community, we weren’t the only organisation that Teona would chat with while she was in London. We introduced her to several other civic tech and transparency organisations in the UK, so she had a busy few days ahead of her, and no doubt plenty to discuss, all of which, we hope, will feed into the success of AskGov.ge.