How access to information can help us understand AI decision making
AI and automated decision-making technologies are increasingly being used in government, and due to their opaque nature, it’s vital that we bring more transparency to their workings. In this event, three researchers and civil society actors talk about how they have used Freedom of Information to do just that.
You’ll hear from Morgan Currie from the University of Edinburgh; Gabriel Geiger of Lighthouse Reports, and Jake Hurfurt from Big Brother Watch. Learn what concerns them about this new age of automated decision-making; the practical tips and techniques they’ve used to bring hidden algorithms to light; and what needs to change in our laws as a matter of urgency.
—
More information
- Blog post, with links to the video and slides
- Morgan Currie’s research (with Alli Spring): Algorithmic Transparency in the UK
- Lighthouse Reports’ Suspicion Machines, as presented by Gabriel Geiger
- Big Brother Watch’s report on the ‘error-riddled AI tool to be used by the Home Office’.
- Find out more about the Access to Information Network
Transcript
Louise Crow 0:03
Hello, everyone, welcome. I’m Louise Crow, Chief Executive mySociety.
Louise Crow 0:08
Thank you for joining us for this one hour session on how Access to Information can help us understand AI decision making in government.
Louise Crow 0:16
At mySociety, we create and run digital services that break down barriers to democratic participation, equipping people to take action and drive meaningful change.
Louise Crow 0:26
We run the digital Access to Information service WhatDoTheyKnow, which has now been used to make more than a million Access to Information requests. And we’re also members of the ATI network, which is a network of organisations across Europe and around the world who run similar ATI platforms to support and enable people to use their right to Access Information in effective ways.
Louise Crow 0:49
And this event is produced in collaboration with our ATI network partners, AccessInfo Europe and Frag Den Staat. It’s the second in a short series on the value and practical application of Access to Information in key public interest domains.
Louise Crow 1:02
We’re running these events because we believe that knowledge is power across the complex challenges that we’re now faced with. And a key goal for us as organisations running platforms that can enable and support Access to Information is to build collaborations where we can bring that power to key issues that are affecting all of our lives, and in that way, support effective reporting and research projects and campaigns in a time when there’s a new generation of AI tools that are rapidly being rolled out by governments around the world, potentially bringing significant benefits but also creating significant new risks.
Louise Crow 1:39
The topic of the safe and ethical deployment of technology and decision making is a hugely important one. I think there’s enormous public interest in transparency around the ways in which software systems are being used, and also what the impact of that usage is.
Louise Crow 1:54
So I’m delighted to welcome, and really excited to hear from, our three expert speakers today. Between them, they have a wealth of experience in the way that Access to Information rights can be a valuable tool to illuminate the way in which algorithms are being used in government decision making.
Louise Crow 2:10
I hope you’re going to find some practical inspiration from this session. As ever, we really welcome your feedback, and we’re interested in opportunities to build further collaborations and support work on bringing transparency to the use of new technologies in government decision making. If you do want to get in touch for either of those reasons, please do drop us a mail at ati-cop@mysociety.org, so that’s A,T, I, dash, C, O, P @mysociety.org, in order to contact the ATI network.
Louise Crow 2:42
So with all of that out of the way, in terms of our running order today, we’re going to be hearing first from Morgan Currie, who is the Senior Lecturer in Data and Society in the Science Technology and Innovation studies group at the University of Edinburgh. Her research and teaching interests focus on open and administrative data automation in social services, activist data practices, civil society and democracy and social justice and the city. So a huge range of very interesting topics.
Louise Crow 3:12
She’s going to tell us about her recent research that analysed 51 FOI requests to reveal how the UK Department for Work and Pensions uses secretive data analytics to detect welfare fraud and uncover broader learning for civil society actors using FOI to enhance transparency.
Louise Crow 3:31
Then we’re going to hear from Gabriel Geiger, who’s an investigative journalist specialising in surveillance and algorithmic accountability reporting. He’s been an investigative reporter with Lighthouse Reports since 2021 and his work’s appeared in Wired, the Guardian, De Spiegel and Motherboard.
Louise Crow 3:48
He’s going to share behind the scenes information from the Lighthouse Reports’ award-winning Suspicion Machines investigation, which revealed how machine learning algorithms were disproportionately targeting individuals based on ethnicity and gender.
Louise Crow 4:02
Finally, Jake Hurfurt is Head of Research and Investigations at Big Brother Watch. He previously spent three years working as a news reporter for The Daily Mail and the Mail on Sunday, where he was able to bring stories on big tech to a wide audience and hone a cache of investigative skills.
Louise Crow 4:19
He’s currently digging deep into the use of AI algorithms and predictive analytics in welfare and social care. He’s going to tell us about new developments in government, algorithmic decision making that he’s uncovered, and how transparency loses out as a result. So without further ado, I’m going to hand over to our first speaker, Morgan.
Morgan Currie 4:39
Yeah. Thanks so much, Louise. So this talk is based on a report that we just put out this summer called Algorithmic Accountability in the UK: how FOIA sheds light on automated welfare. And I co wrote this with Dr Alli Spring, my colleague at the University of Edinburgh.
Morgan Currie 4:58
The report was really inspired by my longstanding interest in this topic of the Department of Work and Pensions’ use of automation for fraud detection. This is something I wanted to study for a really long time. A couple of years ago, I remember talking to people who know this topic much better than me, saying, “I really want to study this”, and them saying “you can’t. There’s just no information about it.
Morgan Currie 5:26
“There’s very little information online or in the public domain, and the DWP is not going to talk to you for for security reasons.”
Morgan Currie 5:37
So I put it aside, and then came back to this just recently, in part because I realised that I could use WhatDoTheyKnow as a way to get information about the systems that are being used. I’ll get back to that – for now just to say a little bit about the topic.
Morgan Currie 5:56
So the DWP has been using data matching and data analytics, we know since at least 2010, so this is using their own data on claimants and comparing it to other datasets that have information about claimants.
Morgan Currie 6:13
So for instance, who’s living in their household, to see if they match, or to find anomalies. We also know that since 2022 the DWP has moved into adopting predictive machine learning models, so these compare claimants to past claimants who have committed fraud, to see if new claimants share similar profiles. If they do, then they’ll be assigned a high risk score for potential fraud and error.
Morgan Currie 6:45
The DWP has been asked to share more information on these systems. An example of that is in 2023 the Information Commissioner warned the DWP that it risked being in contempt of court for not releasing more details about this, because very little is known about these systems.
Morgan Currie 7:05
So that’s why we can we can say there’s a real lack of public scrutiny about automated decision making systems being adopted by government agencies. I would direct you to a really great article by Scott and Edwards that came out this year about this. It’s a paper called The Inscrutable Code: the deficient scrutiny problem of automated government.
Morgan Currie 7:31
And they argue that those bodies that are supposed to be overseeing government agencies, so in particular, Parliament, in the courts, they really don’t have the adequate tools right now to address this problem.
Morgan Currie 7:46
And that’s where civil society, I think, comes in and is playing a really, really crucial role. I’ve been noting civil society is way ahead of us researchers in writing reports that are bringing these systems to light.
Morgan Currie 8:04
And they do this in several different ways: public campaigns – you’re going to hear from Big Brother Watch’s Jake. They do really fantastic public campaigns to get the word out to the public.
Morgan Currie 8:19
Some are going to court. An example of this is the Child Poverty Action Group taking the DWP to court over their automated payment system and the way it affects working claimants unfairly.
Morgan Currie 8:34
Online registries of these systems. So the Public Law Project has an excellent database where you can just go to one place and get more information about them, writing evidence to public reviews, to Parliament.
Morgan Currie 8:49
And I would say a lot of these are based on Freedom of Information requests, so these organisations are getting more information than what’s in the public domain, using FOIA as a tool.
Morgan Currie 9:04
So that gets back to how I think we can, as researchers leverage archives like that have been amassed by WhatDoTheyKnow and other websites like this to look across all the different Freedom of Information requests that have been put to different agencies about these tools to get a bigger picture, kind of a gestalt.
Morgan Currie 9:36
And we call this the FOIA mosaic method. So this is where you combine FOIA requests that you can get from websites like WhatDoTheyKnow with other public documents to get a bigger picture than you could just by looking at what’s in the public domain.
Morgan Currie 9:53
So to do this, we we searched for keywords that had to do with automation and social security fraud. WhatDoTheyKnow had this great – you can refine the search by asking to only look at requests to certain agencies. So we only wanted to look at those requests going to the DWP.
Morgan Currie 10:16
Doing this, we collected 51 requests, and we analysed these in chronological order, and we also looked at public documents that were being referred to in these requests and news reports that were being referenced.
Morgan Currie 10:35
We found that most of these requests are responding to the DWP’s annual report and accounts that come out every year. This is kind of a starring document, because it will say something usually about the fact that they’re using automation, because it discloses that they are doing something proactively to try to catch rampant fraud. So they’ll mention it, but usually say very little about what these systems entail.
Morgan Currie 11:07
And so these reports come out, and that kind of spurs a flurry of requests by people wanting to know more.
Morgan Currie 11:16
Okay, so what did we find out? I’m just going to give high level findings. The report has more granular detail on all of this, and I’d be happy to answer questions as well if you have any, if you want to get more details.
Morgan Currie 11:36
Okay, so some of the high level findings we were able to find out from these requests: some of the organisations the DWP has been sharing, they have data sharing agreements with for their data matching and why. We’ve been able to find out just a little bit more detail about five machine learning models that they’ve created. These are the predictive models that risk score people.
Morgan Currie 12:06
Not much detail, but we know what they’re called, and in part, what data they’re drawing from.
Morgan Currie 12:15
We’ve gotten a bit more insight into the various teams that do this work, and we also can see how the DWP is monitoring for risk and for fairness and bias. So this is mostly through Data Protection Impact Analyses that they’ve released to some requesters. DPIAs, as they’re called, are required by law if you’re using personal data in a way that could carry some kind of risk.
Morgan Currie 12:49
So they’ve done a lot of these DPIAs, and in our corpus of answers to these requests, we have the primary source, DPIAs that were given to people who requested them, both about the data matching and the machine learning models.
Morgan Currie 13:10
They’re heavily redacted, in some cases, to a point where you really can’t get anything beyond the names of some of these systems, but very little else. Another really key and a primary source document that was released to one requester in the data set is a fairness analysis that was carried out on the first machine learning model.
Morgan Currie 13:32
That first machine learning model, it’s called the Advances model. It risk assesses people applying for a Universal Credit advance or loan. DPIAs, I should say, are carried out before you start designing these systems or using them. The fairness analysis is post hoc, so you’re testing the system once it’s been built for bias and in particular, to understand how it works across different protected characteristics.
Morgan Currie 14:05
This fairness analysis showed that there were some issues with certain people with certain protected characteristics being flagged more than others, so that’s kind of a high level of what the what the requests told us about what the DWP is doing internally.
Morgan Currie 14:27
So shedding some light on these systems, what’s interesting about doing this, this method of FOIA analysis, is you can also learn about how the DWP responds to FOIA. So you can get kind of an overarching view of what to what degree they’re actually answering requests versus refusing them.
Morgan Currie 14:53
So out of the 51, only seven were fully successful. 24 were partially successful, so the majority. So that means they would share some information, but reject sharing all of it, all of the requested information.
Morgan Currie 15:09
You also have 16 refused; and then in four cases, they said they didn’t have the information.
Morgan Currie 15:16
We also see that when they refuse requests they’re mostly relying on section 31 of the Freedom of Information Act, so they have legal protection not to share because there’s arguing that releasing that information could jeopardise law enforcement – so catching fraudsters, essentially.
Morgan Currie 15:40
We also found that in all cases except one, the DWP only released DPIAs after the requester made a complaint to the Information Commissioner’s Office. What that means is that other people making the same request did not get the DPIAs. Did not get those documents.
Morgan Currie 16:03
And what we found, therefore, is that there was disparate treatment across requesters. So those who had the time and the resources and they kind of know how to go through the complaints processes, the process, the ICO, they would get the information, but people who didn’t wouldn’t get that information.
Morgan Currie 16:27
OK, one thing I was not expecting to find, and I’m starting to wrap up here, is that you also can learn a lot about kind of civil society actors and the tactics they’re using and how they use FOIA.
Morgan Currie 16:43
So for instance, often in their requests, they’ll refer to public documents. So you can trace they saw a public document that kind of instigated them to make the request, and you can trace kind of their their dialogue as they go back and forth, in some cases, with the DWP to get this information.
Morgan Currie 17:03
You can also link FOIA requests to the news cycle. So in the case of the fairness analyses of the first model that release of that document was given to the Guardian, which made this issue very public, so you can see how this work makes gives more visibility to these issues around, you know, transparency and the use of automation and these in the use of these automated tools, you can see how some requests build on past ones.
Morgan Currie 17:40
So some people are actually using, you know, they’re using, WhatDoTheyKnow. So they can refer to past requests and say, “Well, I want to know more about this.”
Morgan Currie 17:50
And what all this shows is the role of civil society making something, you know, a line in a public document into a public issue and giving it, you know, making it something that is publicly discussed and of concern.
Morgan Currie 18:07
We have some recommendations at the end of the report. First of all, we think all these documents have been released; they’re public on WhatDoTheyKnow. We think they should be made kind of more systematically public by the DWP, either through their website or by using the the UK government’s algorithmic transparency recording standard hub, we think they should just disclose, even if they’re redacted, all the DPIAs and any fairness analyses that are being carried out for their machine learning models, and this is how you know if these models are biased or not.
Morgan Currie 18:45
And we think they need to have a more consistent approach to FOIA requests, so that some people aren’t getting more information than others. Thanks so much.
Louise Crow 18:54
Thank you so much, Morgan. We’re now going to hear Gabriel’s reflections.
Gabriel Geiger 18:59
Oh yeah, my name is Gabriel Geiger. I’m an investigative journalist at Lighthouse Reports, where I’ve led our work on algorithmic accountability.
Gabriel Geiger 19:08
So investigations that try to hold AI deployed by government agencies, typically, accountable. Suspicion Machines is sort of an internal name for a body of work that we’ve been conducting since 2021 where we’ve investigated the deployment of risk scoring AI in different European welfare systems.
Gabriel Geiger 19:33
This investigation started way back in 2021 looked at our colleagues in the United States and and elsewhere, who had done a lot of work on risk scoring systems and criminal justice systems in the US.
Gabriel Geiger 19:47
So there’s a seminal machine bias piece from ProPublica looking at recidivism software and deployed in the US, and felt that there was a need to tackle these types of systems in a journalistic sense, on a similar scale here in Europe, and settled on welfare systems as a sort of quintessentially European issue.
Gabriel Geiger 20:10
And when we set out on this work, we wanted to not only be able to report on these systems in the more traditional sense, like the human consequences or the companies that are making them. But we also wanted to actually obtain deep access to the code and the model files of these systems so we could test them ourselves for check if they were discriminating or working as they’re promised to.
Gabriel Geiger 20:38
Our original entry into that body of work took us to the city of Rotterdam in the Netherlands, where a risk scoring system there was being used to predict which welfare recipients were committing welfare fraud.
Gabriel Geiger 20:52
After a year and a half, using FOI laws, we managed to obtain deep access to the system, the programming code, and found that it was discriminating against people with migration backgrounds, women, people with lower income, parents and even more so, people sort of at the intersection of these different categories.
Gabriel Geiger 21:17
And we ran our own data driven audit on the materials we published, co-published that with Wired magazine in the US, and since then, I’ve done a string of similar investigations, again, relying quite heavily on FOI laws to obtain technical materials for AI systems in countries ranging from Denmark to Spain, Sweden, France, Serbia.
Gabriel Geiger 21:39
And this has become an ongoing theme that we’ve been working on, doing similar types of stories in different countries, across Europe right now, and more recently globally.
Gabriel Geiger 21:51
I should say, FOI has been at the core of every single Suspicion Machines investigation. So it’s been something we use quite a lot in those investigations.
Gabriel Geiger 22:04
And of course, figuring out the nuances of different FOI laws can be challenging. We always worked with local partners, though, and sometimes they help to fill some of that gap.
Gabriel Geiger 22:16
I mean, generally speaking, for a cross border FOI, we found that, you know, using some sort of system to track each of your FOIs is quite important. So we have, like, a tracker template, partnering with local organisations on the ground, not reinventing the wheel.
Gabriel Geiger 22:37
I mean, I can’t tell you how much time I wasted in the beginning by sending out all my FOIs the same template across different countries, and, you know, one size fits all approach, and just wasted a bunch of time because I didn’t formulate my requests in a way that that was appropriate for different restrictions.
Gabriel Geiger 22:54
So for example, in the UK, you can ask these more open ended questions, I believe, like, “what’s the name of this software?”. Whereas you can’t ask a question like that in the Netherlands, you have to ask for a specific document. You can’t just be asking a question in an FOI.
Gabriel Geiger 23:13
So those sorts of nuances are really good to understand before you invest a lot of time in doing a big cross border FOI effort.
Gabriel Geiger 23:20
First of all, our strategy, generally, is to first ask some press questions to the the authority, like the press officer, and something really basic that isn’t going to alarm them too much. Like, you know, what type of AI softwares are you using? What are their names? What type of data do they use? And, you know, is there a private vendor involved?
Gabriel Geiger 23:42
So four really basic questions, and if you can get an answer to those, those will allow you to at least make your FOI requests a little bit more specific and less likely to be rejected, as overly broad.
Gabriel Geiger 23:54
So try to -as you well know, going on a fishing expedition in FOI can be a bit risky, because there’s a lot of ways for governments to not answer your request or avoid sharing actually interesting information.
Gabriel Geiger 24:08
We oftentimes think about it in three different tiers. So you start by asking, for the most basic standard documents that in principle, should always be there: things like the Data Protection Impact Assessment, Human Rights Impact Assessments, you know, the database codebook – this is like the documentation for the database that whatever AI systems actually drawing from.
Gabriel Geiger 24:34
If you know a private company’s involved, you know, with a tender or contract, or like, a proposal, so these sort of standard documents that should, in theory, accompany any sort of advanced IT project.
Gabriel Geiger 24:48
Based off of the sort of replies you get there you can go, move on to the second tier, which is again more specific, a bit more likely to meet resistance. So things like the manuals that are given to case workers when using the system, the full set of variables.
Gabriel Geiger 25:05
So these are the points of data that an AI system actually uses to make its decision or prediction, and their weights, the performance evaluations of the model. Again, that’s something that’s usually happening with an AI system that the data scientist, whoever is actually running the system is checking, is it actually working?
Gabriel Geiger 25:23
And then the third tier would be the most sensitive or difficult to get things. This would be things like the source code training data of the AI system, the actual machine learning model that’s making predictions.
Gabriel Geiger 25:37
The rationale behind using this tiered approach is that oftentimes, if you ask for all of this at the same time, a government agency will just freak out and just immediately shut things down and not give up anything, and or it’s too much of a fishing expedition.
Gabriel Geiger 25:55
And by using this tiered approach, you’re more likely to get, you know, this initial set of documents that, you know, 1) helps you make more specific FOI requests for, like tier two and tier three, and, you know, sort of, secondly is sort of a backstop in case, okay, you can’t get the really sensitive stuff, like the code, or maybe even, like Performance Reports, you can’t get that, but at least you have the Data Protection Impact Assessment and some sort of basic materials that help you, if you’re a journalist, produce maybe an initial story, or if you’re civil society, you know, maybe like those are already materials that you could write a report about, or even even issue some sort of press release about, or maybe start some sort of campaign.
Gabriel Geiger 26:43
So over time, we’ve felt that this sort of three tiered approach is the most effective way of doing a deeper investigation into an AI system with FOI. There are some considerations there that one might might make.
Gabriel Geiger 26:57
I mean, if you have a really long FOI timeline, like you do in the Netherlands. So in the Netherlands, it’s quite common for to be six months until you get a response. Then maybe doing three tiers might be difficult, because could be 18 months.
Gabriel Geiger 27:12
But if you’re somewhere in the UK, where you generally have what like a one month timeline that can be expanded, extended, then if you have the time, I would try to split it into tiers like we do.
Gabriel Geiger 27:24
We have a guide that we wrote, published by the Pulitzer Center, which is an organisation in the US, that talks about our FOI process for the Suspicion Machines work. That also includes things like an FOI template that we use for our investigations and some various descriptions of more specific documents you could ask for.
Gabriel Geiger 27:46
So that might be a helpful resource for people. And then if people want to follow our work, just to go on to lighthousereports.com and sign up for our newsletter. We also have all our Suspicion Machines investigations there listed with methodologies, explaining how we did them for each one. And me, personally, I don’t use Twitter anymore, but I’m on LinkedIn. You can find my name, and I post updates there.
Louise Crow 28:13
Really interesting to hear the details of a journalistic investigation there, and I love the open source approach. Finally, we’re going to go to Jake.
Jake Hurfurt 28:22
Yeah, I’m Jake from Big Brother Watch; I run our investigations here. So we focus on tech and digital rights, across the board, really. So everything from welfare to facial recognition, the home office encryption stuff that’s been in the news today. So all kind of that.
Jake Hurfurt 28:38
But in terms of our work on automated decision making, that has mostly been in the welfare system. Also it’s been around facial recognition, as we believe that’s an automation decision.
Jake Hurfurt 28:44
And from my experience, I’ve been working on this for five years, there’s been a real retreat in transparency over that time in terms of how much the government is willing to tell, how much it will disclose about automated decisions, both to people like me, as civil society; people like Gabriel as journalists; or Morgan, as academics, all sorts of people impacted, which is really important.
Jake Hurfurt 29:07
So a lot of people impacted aren’t told about that -aren’t told about these decisions. And in the first place, that’s really a huge problem. These are making decisions about your life and you don’t know about it. How can you then seek redress when it goes wrong? Get that accountability?
Jake Hurfurt 29:21
It’s really important in the same decision making. I think we’ve seen a backtrack on transparency, in the very recent Data Use and Access Bill, many people have worked on that, know about that.
Jake Hurfurt 29:30
But previously in British data protection law, we had protections against solely automated decisions. These are automated decisions that have big effects, that had no human involvement. We’ve seen that gutted in the Data Use and Access bill.
Jake Hurfurt 29:44
So we are going to see over the next 12, 18, 24, months, a massive expansion of the British state’s use of automated decision making, and it’s going to be less transparent than ever, because, as the law stood, there was a right, if you had one of these decisions, you saw the decisions made about you, but a right to be told about it.
Jake Hurfurt 30:04
So many of you, if you applied for a credit card or a loan online, that’s been automated, there’s been something in the T&Cs that you may have seen. You have a right to have human look at this, and that right is going to be – has been – watered down hugely in legislation.
Jake Hurfurt 30:16
So already there we’re seeing a huge drawback in transparency around the use of automated decision making, and the Secretary of State has Henry the Eighth powers. The Secretary of State just designates certain categories of decision making as exempt from the few protections that do exist.
Jake Hurfurt 30:31
So we might be in a situation where they decide policing decisions are suddenly exempt from the protections.
Jake Hurfurt 30:38
So, many people have to be told that these decisions are made about them, automated procedures being used. So that really is a problem. In my work over the past five years, I’ve seen time and time again, even the protections that used to exist, those really been exploited.
Jake Hurfurt 30:52
A lot of government departments, and certainly in local government, the basic user threshold is ever so slightly below the legal threshold for telling people that automated decisions were being made.
Jake Hurfurt 31:03
So there’s a system called the risk based verification – it’s mostly been phased out, but was used to effectively risk or triage people applying for housing benefit or Council Tax Support in lot of local authorities.
Jake Hurfurt 31:15
And they were risk scored as high, medium or low, and this determined how much evidence they had to provide to get access to the benefits they’re entitled to. And the human involved in this decision wasn’t ever allowed to downgrade the score, only upgrade the score, and in incredibly rare circumstances.
Jake Hurfurt 31:30
But every single local authority in this country argued that that was a human involvement in the decision, so it wasn’t ‘solely automated’, which meant individuals’ right to be informed, that right to for it to be transparent was removed.
Jake Hurfurt 31:44
In my personal view, and I’m not a lawyer, I think if that was challenged in the courts, if that was challenged about whether it was a solely automated decision, somebody would have won.
Jake Hurfurt 32:05
But that’s the problem: data protection law, when we’re looking around this transparency, is that individuals have to challenge. It’s very hard for organisations such as Big Brother Watch to lead that challenge – it has to be an affected individual.
Jake Hurfurt 32:06
And how does that affected individual launch a challenge and try and push back on that use of these decisions if they’re not aware of it in the first place? So that leaves it to people like myself and Gabriel and Morgan to dig into it and raise that awareness.
Jake Hurfurt 32:19
And we see that the state really won’t tell us either. So you end up in this kind of Catch 22 where the law is weak, so affected individuals are unable to find out how they’re affected.
Jake Hurfurt 32:33
So people like myself end up having to do these FOI fishing expeditions to work out well, what’s going on, and where can we try and gain transparency. So I think, yeah, this gutting of Article 22 and those protections, we’re going to see a further decline in transparency around algorithms, automation in the next 12 to 18 months.
Jake Hurfurt 32:52
It’s not quite got there yet, because the laws only just went to force, but I think it’s something that’s going to be incredibly concerning going forward.
Jake Hurfurt 32:58
But I’m mostly going to talk about a very small bit of transparency that I think is perhaps the most important one, this one often goes unaddressed, which is transparency around procurement of these automated decision making tools.
Jake Hurfurt 33:10
Because quite often, when smaller bodies of state, so local… Central government does have the capacity to often develop these tools, algorithms; but smaller state organisations, the police, local authorities, don’t. They procure them.
Jake Hurfurt 33:24
So we’ll see Procurement Notices going up. We’ve seen them recently for all the new police forces who are acquiring facial recognition as they’re procuring this technology.
Jake Hurfurt 33:34
Essex police procured a different facial recognition technology, the data science firm Xantura, who are kind of a knock-off British Palantir.
Jake Hurfurt 33:34
They’re procured by local councils to provide the services they do, and because it’s provided by a third party who argues that this is commercially sensitive information, that revealing their algorithm would be detrimental to their business, the transparency one would expect from government decision-making is almost nil.
Jake Hurfurt 33:59
So going back to risk based verification, which was this triaging software that was used to risk score people applying for benefits. The biggest supplier of this, was that, was Xantura the slightly Palantir-y company – they were a British company, quite big in the UK, after a long FOI battle, I got hold of… actually using the what Morgan was talking about, the mosaic FOI, because it was used by a few dozen local authorities.
Jake Hurfurt 34:26
I asked a few dozen local authorities for documents, and they all gave me slightly different ones, and between that I was able to really put together quite a detailed look at how they are working.
Jake Hurfurt 34:36
And in these documents, you’d see a list of predicted characteristics that we use to decide how risky somebody was, but they said explicitly in these documents, we are not telling you what full list of predictive characteristics is, because that would potentially reveal our trade secrets, our commercially sensitive information, to local authorities.
Jake Hurfurt 34:57
And they knew they were subject to FOI as well. This stuff might have to get published. So in a situation where people getting branded high risk by the local authority being put into huge demands to provide evidence to their benefits claim, a local authority had no idea why.
Jake Hurfurt 35:10
They didn’t know how each characteristic was rated, how much weight it was given, but also didn’t even know all the ones that they used. They were just assured by Xentura it was all fine.
Jake Hurfurt 35:21
This is a massive problem. The public sector equality duty isn’t something that’s delegable. State bodies aren’t supposed to say, well, our supply tools, it’s fine. The duty is incumbent on the state act to comply with the PSED. And this situation in, again, in my non lawyer’s view, this was not happening.
Jake Hurfurt 35:41
How can we have transparency in decision making when the state act, who is fundamentally making decisions, doesn’t know all the things that go into that algorithm? So I found this really worrying and as a practice going forward, how can we have state decisions that then can be challenged and be held to account if the state can’t explain in the first place?
Jake Hurfurt 35:58
So it really was a massive problem with Xantura. I think there’s also, against smaller organisations, there’s a technical knowledge gap. Xantura often went into organisations and integrated themselves very strongly with local councils. They provide the technical knowledge.
Jake Hurfurt 36:08
So it isn’t just that local authorities and these other bodies weren’t told what was actually being used to make these decisions. It was they didn’t really understand it, because they these are kind of cash strapped, small local bodies. They don’t have people who are trained in computer science and machine learning.
Jake Hurfurt 36:29
So it’s that reliance on external knowledge and these commercial interests’ secret keeping. It is creating real problem with transparency, and as we’re having more and more AI and algorithms, more companies of various repute selling into the state, we’re going to we’re going to see this grow, because these companies need to defend their their local knowledge, because they as a company, would cease to function if they couldn’t sell to more and more people.
Jake Hurfurt 36:52
So they’re going to defend, defend this. We’re going to have people asking, “Well, why is this decision made about me?” and the state won’t be able to explain it.
Jake Hurfurt 37:00
And then for people like myself, trying to get this through FOI, section 43 – as much as I often object to how it is used to kind of keep this secret, it is a legitimate exemption. As much as I want to know these things, making a company go bust by ruining their trade secrets probably isn’t in the balance of public interest.
Jake Hurfurt 37:00
So we need to, I think, in my view, reform the law around this to find a way to make sure that these private companies can’t just hide behind commercial exemptions and not tell the state how it’s going on.
Jake Hurfurt 37:30
There’s another example: Essex police are the only police force in UK to use Corsight, an Israeli facial recognition firm, who are also been used by the IDF, but their equality impact assessment is an absolute mess.
Jake Hurfurt 37:30
It cites an assessment for a complete different algorithm made by a completely different company as evidence that the Corsight algorithm is fine. It cites evidence from marketing materials, from Corsight, from the US Department of Homeland Security.
Jake Hurfurt 37:55
The Department of Homeland Security have gone on public record to say ‘incorrect’, and really questionable interpretations of some other US studies. Essex police have never done any any testing themselves.
Jake Hurfurt 38:12
Apparently, it is ongoing now. They’ve been using facial recognition for almost a year within a quality impact assessment that is either based on marketing material or based on tests on a completely different algorithm, and that’d be like saying, well, Windows does this, so you can do it on a Mac as well. They’re not comparable.
Jake Hurfurt 38:25
But again, because that marketing dazzling, this lack of technical expertise within procuring body, it means the transparency tests aren’t done that we don’t really understand how they don’t really understand how it works. So how can they be transparent about decision making that’s going on?
Jake Hurfurt 38:39
So I think, for me, we need to really look at the rules from procuring AI in the first place, if you want to be transparent about it going forward, we can’t allow commercial interests and those kind of commercial arguments, to shield private companies who are supplying the state from any kind of scrutiny.
Jake Hurfurt 38:56
And I don’t know whether we need to give the ICO powers to go into this and verify. I don’t know what that looks like. I’m an investigator, not a policy maker, but as it stands, the commercial interest exemptions are incredibly strong, and we’re not enforcing the laws we even have around equalities, because the people are procuring it do not understand how these systems work.
Jake Hurfurt 39:15
So that’s that’s a huge problem, but I think the black box will generate, again, this transparency thing. We could have all heard about it before, but I think as we’re becoming more and more AI, we have to think not just transparency for people like ourselves and civil society, but actually the people who are impacting sales really matter here. If we’re having people having huge decisions made about their lives and that state can’t explain it, that becomes a problem.
Jake Hurfurt 39:41
And I think for me, it’s a red line that we shouldn’t be using this stuff anymore. So that’s sort of interesting, but I concentrated most on the procurement because I think that it’s a huge gap when we look at the impacts of these systems. But for me, the problem starts when these systems are bought and supplied to the state.
Louise Crow 39:50
Thanks, Jake. Yeah, hugely interesting to hear about the amount of procurement going on, and that that gap in knowledge as well. Super interesting. And as you say, as we go into different kinds of AI systems, understanding is much more of an issue. Julia, I’m going to pass over to you to wrangle some questions.
Julia Cushion 40:04
Thank you so much. We have a nice question from Phoebe; What are the government departments that frequently push back on providing data? Don’t know if anybody has any experience on that.
Morgan Currie 40:16
I’ll just very, very quickly, I can just say, and I know this based on the Public Law projects database that they keep on these tools, that the the DWP is frequently mentioned, but the one that’s mentioned, I want to say even more is the Home Office. I don’t know, Jake, if I have not looked into that myself, my focus is, has mostly been on the DWP. Jake, I don’t know if you, Big Brother Watch has looked into this with regards to the Home Office, or if you have other answers as well.
Jake Hurfurt 40:46
I think the DWP are particularly bad. In the tribunal court, tribunal hearing at FOIA. I got to stage where they started national security arguments, which were a complete nonsense. I think of the worst as the Home Office, because they do have national security duty that often becomes something they hide behind. They’re not great, and surprisingly, DSIT as well are an absolute nightmare, I think so. They’re the three that are worst. Surprisingly, MOD, to give them compliments, say they’re the best, maybe because they’re soldiers and they’re a little bit more straightforward, but I was surprised. I found the MOD easy to deal with, which I never would have expected.
Julia Cushion 41:24
I also, if people don’t know the brilliant Martin Rosenbaum, he wrote a reasonably interesting blog post recently about 20 years of Freedom of Information, that has some statistics about government responsiveness and how that’s changed over 20 years.
Julia Cushion 41:34
Do we know of any examples of good practice on automated decision making transparency? Are there any countries doing a good job of this? There’s a really interesting point, not just on the algorithmic use within the countries, but the different FOI practice in the different countries. And Gabriel’s work on the Suspicion Machines is a comparative analysis. So I would point people there. But yeah, Jake or Morgan. Did you come across any examples of countries which who are being very transparent in this area?
Jake Hurfurt 42:02
So I’m mostly UK focused, but Canadians have a decent Transparency Register. They’ve been trying for a while. I don’t know how much detail they’re still going into, but that was quite good. The Dutch seem to be reasonably forthcoming as well. The Suspicion Machines work.
Jake Hurfurt 42:17
And the UK is trying its algorithmic transparency reporting standard. But in my view, it’s too weak at the moment, the various government departments are refusing to comply. So on paper the UK system could be good if if government actually followed through and enforced it properly. But at the moment, in my view, they’ve been too weak, letting departments get away without actually complying with what’s supposed to do.
Morgan Currie 42:41
I have two examples in the US. So one is a few years ago, New York state required that any government agencies that were going to use automated tools to register it and make some details about it transparent.
Morgan Currie 42:58
My understanding is that that hasn’t been robustly followed through on, but I haven’t looked into it lately, but I do know you could at least look up that that law that was passed.
Morgan Currie 43:09
Another, I’d say, kind of gold standard is a system that’s also in the US. It’s in Allegheny County. This is a county in Pennsylvania in the US. They use machine learning risk modeling in order to score people for risk of child abuse, and they have made the underlying data available to researchers. So there’s actually a lot of research into those systems, kind of similar to what Lighthouse has done with using some of the underlying data that they’ve gotten access to in the Netherlands, a lot of people have been able to research the system because of the kind of fairly, I’d say, actually, relatively radical approach to transparency that they have.
Morgan Currie 44:02
I would also point you, to understand more about that, to Virginia Eubanks’ book Automating Inequality, which has a whole chapter on that system. So she comes away very critical of it. But in terms of transparency, it’s been kind of like, like a gold standard bearer.
Jake Hurfurt 44:20
I’d also add, in the UK, Bristol City Council are probably the best in terms of their transparency. They’re not flawless, but they are much, much better than basically anybody else. They tend to, even they do tend to disclose not the actual data, but the datasets they use, the algorithms and stuff. So as much as I love how they use them, I’d be more complimentary, they at least are much more open about what they’re doing. A lot of places are a lot less transparent than that. So if anyone UK based, that’s probably good place to start.
Julia Cushion 44:44
And two questions from Cara here: with the heightened demand for energy and water to power AI, plus the excess heat created, how do we balance the usefulness, ethics and reliability of AI versus the climate impact? So maybe, like a question on the on AI in general, there? Have you had any thoughts? And then more specifically about the DWP, looking at that and how can citizens get fair and just support in the face of AI and system failure? If you have any thoughts of an individual level and what people could do?
Morgan Currie 45:13
I can say something to the first question. So I think it would be important to note that, and I’m not an expert into the energy demands of what the DWP is doing by any means, but I do think it’s important to make distinctions.
Morgan Currie 45:30
So a lot of the high energy intensive AI would be generative AI or AI that’s being trained on massive, colossal data sets. Now this is not what the DWP is doing. What they’re using is actually fairly traditional machine learning to data match, or machine learning to predictably risk score claimants, and that, I don’t think would be in the same category of kind of energy intensive AI that that that a lot of us are starting to kind of adopt in our daily lives with generative AI.
Morgan Currie 46:06
So that’d be my answer to that question. That said, we’re seeing – not to go off on a tangent – but we are seeing more and more policy rolling out requiring agencies to start to to adopt generative AI and to and to experiment with it. And so I think that’s going to be an interesting development in this conversation. Not something I know anything about at the moment, because it’s so new. But I think this will be kind of like Jake was hinting at, kind of next steps or next horizons in this discussion.
Jake Hurfurt 46:37
in terms of the DWP and the carers allowance, a lot of the do these tools do, in theory, seek to find all kinds of kind of fraud and error in the system, but their focus is, obviously, on what they see as criminal fraud.
Jake Hurfurt 46:51
So it may be that this could be that this one small kind of silver lining, this kind of general, kind of horrible use of AI, could be some of this stuff gets flagged. But I think it is the general approach ofthe DWP to be fraud and error, treat as one which, which is a problem, which you’re going to get huge amounts of people who’ve made a genuine mistake on a document, someone who’s in a vulnerable situation,because people make errors, because we’re human, and getting treated as if they could be a criminal.
Jake Hurfurt 47:10
And I think the carers’ allowance is a really kind of worrying example of this, that people are getting hammered for being overpaid a tiny amount. Tiny amount. I think we could see with these kind of large models that are just looking for ‘is a claim wrong?’, rather than someone committing criminal fraud, that people are putting through very stressful things for genuine human error. And I think that’s it’s going to cause all sorts of problems going forward.
Julia Cushion 47:30
Absolutely. Thank you both for great answers. And obviously I wouldn’t be doing my job if I didn’t say that WhatDoTheyKnow is a service where anybody can make a Freedom of Information request. And there are loads of examples on there, and I think it does help highlight the scale of these issues, and us to keep asking these things, digging away.
Julia Cushion 47:52
And even if you’ve never made a Freedom of Information request yourself, you can go on and see literally millions of others and follow requests that you’re interested in.
Julia Cushion 48:03
Another question here: why do you think governments are getting less transparent on automated decisions? And what are the reasons behind this? So maybe thinking through some of those trends, perhaps also sorry, to make the question harder, are there any ways that we can counter this trend? But yeah, the reasons behind this trend and what you’re seeing.
Jake Hurfurt 48:17
I think the first reason is probably bad press coverage we’ve seen, you’ve seen stories like, kind of the stuff Lighthouse have done over in Europe. We’ve seen the sham marriages algorithm getting an absolute battering after PLP’s work over in the UK. So I think that kind of bad press coverage does make governments more secretive.
Jake Hurfurt 48:39
Then also, these models get increasingly, complicated, so that to actually put this information out has become much more difficult. It’s not now just a kind of a basic regression model where it’s like, here’s some characteristics, here’s the output. It’s the black box is inherently opaque.
Jake Hurfurt 48:52
So they’re becoming more opaque in turn. I think they always just become more more integrated. Governments are as opaque they’ve always been. It’s just the AI used to be sitting on top and now it’s much more integrated. So as it goes to the heart of government, it was going to get less transparency as it becomes more integrated into the data decision making.
Morgan Currie 49:14
Yeah, I agree with Jake. I think so. In the dataset I looked at, it was very interesting. The requests start in 2018 and at the beginning and throughout that year, in the next couple of years, they the DWP would respond by to request for DPIAs by sending very lightly redacted ones. This would be for the data matching.
Morgan Currie 49:39
And then there was a point at which, when I say it was in 2020 where that just stops, and any DPIA that they give after that is heavily redacted. And I agree, I think it had to do with… in 2018 that Virginia Eubanks book comes out, that Philip Alston writes a scathing report about the UK government and about the DWP as well, with regards to these tools. And I think it’s it. There was a skittishness that started at a certain time period, and ever since then, it’s been harder. But that shows up in the in the data set, actually.,
Jake Hurfurt 50:13
I think that report I put out, with Big Brother Watch, called Suspicion by Design, has a section on the government’s general matching service, data matching service, been running for about a decade, and I asked, there was some documents already in public domain.
Jake Hurfurt 50:15
I asked for very similar documents this year, and the transparency over that time has declined massively. They wouldn’t give me documents that are basically already in public domain from a few years ago. They wouldn’t give any new versions that really illustrates the the transparency backsliding we’re seeing with this. And I think it’s all the reasons Morgan said.
Julia Cushion 50:45
Absolutely, that’s so interesting. And, yeah, really interesting when you mentioned, Morgan, about how all those successful requests have involved going to the ICO to challenge them.
Louise Crow 50:56
Huge thank you, Morgan, Jake and also to Gabriel, you’ve been fantastically interesting speakers on what is obviously a really important public interest matter. Thank you all for joining us. I do hope this session has given you some ideas for the future, and hope to see you at the next event in the series.