FOI Fest – Keynote 2: David Hamilton (Scottish Information Commissioner)
In the second keynote from FOI Fest, David Hamilton, the Scottish Information Commissioner, explained current challenges, including the arrival of the AI-based request, increased levels of anger on both sides, and what it means when FOI is front page news every three days. He also goes through the forthcoming changes that the Reform Bill is set to bring to Freedom of Information in Scotland.
David refers to his slides a few times during this recording. If you would prefer to watch this presentation as a video, head over to YouTube.
Please fill in our short survey to help us understand more about our listeners!
Transcript
0:00 [Gavin Freeguard:] Welcome to FOI Fest, 2026
0:04 [David Hamilton] I’m David Hamilton. I’m the Scottish Information Commissioner.
0:07 I don’t know. I always end up speaking at conferences on FOI in England after Scotland thump England at rugby. So having alienated most of the audience, let me carry on.
0:22 OK, so the Scottish information regime is a totally separate to the UK. The rest of the UK is one, and it’s a bit confusing. So let me just have a quick recap of this.
0:34 As you’ll see, if you look at the lower part of the section, you’ve got the Freedom of Information Act 2000 which you’re all familiar with. And of course, you have the GDPR and the data protection legislation, which is UK-wide.
0:47 The difference is that in Scotland, where, because we have a devolved parliament and devolved powers, which is essentially things like education, health, policing and so on, a lot of these things are controlled by the Scottish Parliament under Scottish legislation, the Scottish public authorities, and therefore the FOI system is administered by the Scottish Information Commissioner, ie me. So that’s that’s how it works.
1:12 We have legislation which followed two years after… or was drafted two years after the legislation the rest of the UK, but they actually started at the same time, or more or less, so in 2005 so that’s when it was designed. It was designed to be “FOI plus”, or “FOIA plus”, which might explain some of these things.
1:34 And what I’ve been asked to do today is just to give you a quick update on that, and actually try and draw across some of the other pieces, which we’ve heard today from some of the other speakers who have been fantastic. I’ve really enjoyed their inputs this morning.
1:49 And actually just show how sometimes actually using legislation internationally as almost with what we’re doing here, you can actually also use that to get information, so, you know, from within the UK. So in terms of the Scotland position 20 years going, we’ve had one and a half million requests in that time, and some good news stories.
2:12 You know, consistently, 87% of the authorities, of which there are about 600 or so, are answering those requests on time. So 87 requests answered on time, and on those 83% some or all of the information is supplied.
2:29 So I mean, generally, that’s a pretty good news story. About 0.7% of these cases do however come to my office, and that’s resulted in 10,000 – nearly ten and a half thousand – appeals.
2:43 Now we’ve issued 4,000, probably a bit more than that now, again, 4,500 decisions.
2:49 And again, a breakdown there – 35% for the public authority, 37% for the requester, 28% for a bit of both. So that’s our kind of breakdown. But I’ve only got 11 investigators, which is a challenge, and we’ve got to try and get through a lot of a lot of good work there.
3:08 Ok, how do I know all this? Well, this is a dashboard which we have on our website. You’ll see at the bottom, foi.scot/statistics. This is a piece of information which has been a bit of a game changer, actually, because when it was introduced, we started asking public authorities to submit data. Who are they dealing with FOI, and what their returns are, and that’s given us a dashboard that gives us a lot of intelligence in terms of what the system’s looking like.
3:38 You can go to the top left, you’ll see the authority. There’s drop down lists. You can pick out sectors, you can compare yourself within that, and you’ll see those little red lines and so on. Individual authorities can compare themselves to others for the sector averages and indeed, other areas. And there’s actually a map which is blank here because it’s a screenshot, but there’s an interactive map there individuals can see what their local public authorities are doing, how they’re performing, and research that is an absolute rabbit hole of information. So if you love data, go in there, and I’ll see you next week.
4:14 You’ll see also we’ve – this is relating back to the last you know, last eight quarters at the bottom, last four quarters at the top. So 109,000 requests were made in Scotland in the last four quarters of data. We’re just about to release the next one. You expect to see that going flying up again.
4:34 But one of the other things I wanted to point out here is for ourselves, is you’ll see in the bottom left corner that there’s a red line, which is has taken a sharp increase. Now in Scotland, we have a process whereby as soon as you make your request, if you’re not happy with that, you can ask for a review. That’s part of the process. It’s not a complaint. It’s a part of the FOI system.
4:59 If you take a review, we won’t even look at it until you’ve had your request, a review, and then it comes to us. So why don’t we see this, these reviews going through the roof? Because many reviews come to us. That was a early heads up to us, the problems coming, and I’ll come on to that in a minute.
5:21 So the differences themselves, as I said, this is meant to be FOI or FOIA plus. And I want to just highlight a few things from that. So first of all, the harm test. Now disclosure would cause a harm to a particular interest, is the kind of definition, looking at it. But under the UK, it’s prejudice, but in Scotland, it’s substantial prejudice. So immediately, what we’re doing here is the harm test for exemptions is a lot higher, and that, again, is something that can it’s a subtle difference, but it’s actually quite significant in terms of how we view the arguments that are being made in terms of exemptions and the substantial pressure we’ve defined as real, actual and of significant substance.
6:07 So again, we’re just putting that level up there. It’s a high bar to use the exemptions in Scotland, or higher bar anyway. Looking at timescales, much more tightly defined, as I said: 20 working days to respond to requests, that includes the public interest test. So whole thing done and dusted in 20 days.
6:28 If you’re not happy with that, you can ask for a review. 20 days for the review, and then after that, you’ve got the opportunity to report it to ourselves. So again, it’s a much, much faster process, much more slightly defined process, in terms of getting requests done as early as possible. Our cost limits, again, are different because all public authorities have to be the £600 threshold.
6:55 It’s not just central government, and again, it’s the how that breaks down. The maximum charge for staff time in Scotland is £15. I think it’s £25 in the UK. So the cost thresholds are different.
7:13 And again, you’ve got to do it at a later point. The Authority has a different level here that’s better for the applicant in terms of getting this and so to be honest, the a lot of the authorities think it’s unfair, because that’s not changed in the 20 years of the legislation. And I’ve got a degree of sympathy with that, because there’s very few people, to be honest, in the workplace, are working at £15 per hour. So that may change, but that’s something that the parliament has chosen not to do yet it may do in the future.
7:43 OK, appeal routes in Scotland: we don’t have tribunals. We just have essentially an appeal straight. If it comes to me, and I say this, if you want to appeal my decision, it goes straight to the Court of Session. Now that’s quite a significant challenge, because you’re talking about a £20,000 bill just to get it there, plus the adding out expenses of the other party. It’s pretty cost prohibitive for a lot of people, but it’s something that it’s the Court of Session, it’s the highest court in Scotland. That’s where it takes us again, the way the system has been designed.
8:19 A few problems, though, the Lord Advocate and crown office are not fully covered, and that’s particularly a problem at the moment, because in Scotland, just now, there is a huge furore about transparency and the issue of the Lord Advocate and how their relationship with the First Minister because the dual hat in terms of powers as a member of the government, but also an independent prosecutor.
8:45 So there’s a bit of a challenge there, not fully covered by by Scottish FOI. If we, as in the UK, of course it is. If somebody wants to appeal my decision, as a corporate decision, then again, they can’t do that to me. They have to go to the court session, which is, again, you might be thinking, why am I asking for that? But I just our organisation thinks the right thing to do that there should be an extra stage in there so that people can challenge it, and they we also can’t require the publication of data sets.
9:18 We’ve got a model publication scheme, which we’ve produced. It’s well, well out of date, and it’s something that we hope we can see change, because there is again, quite a lot of discussion, and I’ll come on to that.
9:29 About the Reform Bill for FOI. OK, so what that leads to is essentially gives us a position in a kind of global context. You’ll see here. This is created by the Canadian Center for Law and Democracy, and it essentially takes legislation and paper exercise and says, how are people doing across the world? And you’ll see there that you know, some of the more established regimes Western Europe and so on are pretty poor in terms of this, whereas some of the more emerging democracies in a better position.
10:05 There’s always a bit of a chortle with this, because the highest and best RTI rating, Right to Information, is achieved by Afghanistan, which, of course, on paper, is the case, but in practice, I would question whether that’s still the case.
10:23 But again, this is about looking at a legislative perspective only, and in that basis, Scotland ended up getting a score of 102 out of 150 which is although the highest in Western Europe is still not the best by any means in terms of the global picture. Whereas the rest of the UK ended up with 92 and again, that’s not surprising, because Scottish system was designed to be more transparent, but there’s still a bit to go.
10:51 FOI Reform Bill. This is a piece of legislation that’s been talked about for quite a while now, and a has now coming to an extraordinary stage this week. So I was thinking about these slides, and thinking, well, there’s not much point spending much time about it, because it’s dead in the water.
11:06 Well, apparently not because, when it went to the Scottish Parliament, despite the recommendation of the lead committee, the Parliament said, “no, we want to have this bill”. And they decided to keep it on the statute books, and it will roll over to the next session of the Scottish Parliament next year, which is, I mean, I’m absolutely delighted because, not least because we’ve done a lot of work on this, but also it’s what it’s doing, is it’s strengthening and enforcing, kind of, maybe servicing a bit of legislation that, although 20 years old and in good nick, it still just needs a little bit loopholes closed and a few more things reflected in it.
11:47 These are the kind of headlines from it, strengthening the presumption in favour of disclosure – self explanatory – expanding designation power so that more people come under FOI law. Now we’ve got again, quite a wide spectrum of that. But there’s not been many designations of different organisations that come under the foi legislation, and that’s because it’s got to be down, come down to government. So there was proposals to look at the parliament being able to take a role in this and to put forward designations and into different sectors and so on.
12:24 We’re very, very keen to see, and people very keen to see, as the landscape changes, where more and more sort of public services getting outsourced, that FOI follows that and follows that public pound as much as possible. And there’s a lot of discussion about, how can we do that? Can we should we have gateway clauses and so on, so that it makes it strong? But the difficulty is that, as we feared would be that as a regulator, we would just spend our time looking at whether something qualified or not. So it’d be better just to have a regular refresh of the organisations that are covered.
13:01 And just a statutory duty for proactive publication. I kind of mentioned that earlier. This is about being proactive, a duty to publish, and we would then establish a code of publication, and that would be for different sectors to provide information. And again, my question earlier was asking about standardisation, because this is one of the unintended benefits, if you like the spinoffs, of it would be, actually: there is no definition of a pothole.
13:26 So if you go to any county council, local council, “how many potholes have you got – two inches, three inches or four inches?” You know, that’s the type of thing that some standardisation, that would actually help everybody, not least in those sectors. Even in a country that’s got a housing emergency, we don’t have a definition of what social housing is.
13:48 And again, these are the types of things that we can see. A standardisation would be very, very helpful in terms of Information Commissioners’ powers. And again, this is just from experience, looking at things. It would be good to be able to interview people and to speak to people, and require that have that power to do things in a way that is not there just now, because we no surprise to you, there are some organisations that are evasive and you need to actually have a bit of clunk and a bit of teeth about you. So that would be helpful.
14:22 We also think that our practice recommendations, which are currently voluntary, should be enforceable legally, and that if we say you need to do this, rather than saying, “Yeah, we might do it might not”, then the practice recommendation becomes something that if they don’t do is contempt of court.
14:41 The requirement to appoint an FOI officer is mirroring the Data Protection Officer sort of role, and it’s something that I think we recognised could be done by the same people, but essentially, what it does, it gives FOI the same status and same attention that the DPO has in an organisation.
15:02 And again, this is something that came from, from the FOI officers themselves. It could be helpful to have that equality of arms.
15:11 Close loopholes and destruction of records and extend offences: I think we’ve got a similar offence to what you have in UK. But essentially in Scotland, we have an offence whereby if you destroy stuff during your request, it’s a criminal offence.
15:24 But we would actually like to see that extended so that it’s if you destroy data to avoid it being FOIable, then that should be something that’s a criminal offence. Now there are big questions regarding about records retention, with that and discourage and sort of great against data protection and data retention things, but these are all, you know, we can work these all things out, and I hope we will do in the next session.
15:46 And then finally, we’ve got, still got First Minister’s veto power for certain exemptions, and we’d like to see that getting used. It’s never been used. We just like, get rid of it. And again, that would be something that would be, again, stronger for democracy.
16:02 OK, I’m showing you this because I need to demonstrate something about the impact of demand. This is our caseload, which we publish on our website. You’ll see the website there. You can be looking at it yourselves at any time. And essentially the main thing to look at here is the red line, which is this rolling three month average of cases coming into our organisation, and you’ll see we talked earlier about April, it just went nuts.
16:29 What was going on there? Not entirely sure what that is. A lot of it seems to be a bit of noise, because economic things are pretty tight and services are getting cut, but you’re also getting feedback that because we’re making an impact with FOI, people are beginning to become more aware of it.
16:47 We’ve just done some awareness polling, which shows public awareness of FOI is through the roof. I’m not allowed to say what it is, because my team will kill me, but it will be released next week, but it’s absolutely stratospheric in terms of increases and awareness of that, and as people see their rights, they’re happier to use those rights.
17:07 So we’re seeing a lot more of that. We’re also seeing the impact of AI and these yellow cases were two cases that were all AI cases. That’s the difference it makes. Just having two cases that were AI, then we knocked them out.
17:24 And that’s different ways, but you can see that suddenly we’re hitting a battle of machines where we’re getting swamped by things. And when you start drilling into that and looking for the impact that that’s going to make in terms of the burden upon our office and our people, it’s absolutely astronomical. So you can see that’s really having an impact on us.
17:51 So in terms of AI – and, you know, full disclosure, I used to be an AI engineer, which is probably not something you should be too proud about. But anyway, that I did this study this long time ago, and actually when I studied it was called Artificial Intelligence and Pattern Recognition.
18:11 I wish they just kept up the ‘pattern recognition’ bit and not the artificial intelligence, because it is not intelligent. It’s just following patterns. And when people understand that, they will understand why they shouldn’t use it in the way they are using it.
18:24 So spell check, Grammarly, all these types of tools, those are artificial intelligence. I’m just going to use that. There’s me falling into the trap already. Essentially, it’s looking at the patterns and it’s sorting it. So something has been around for forever, and it’s actually embedded into so much of many of our systems.
18:46 You can’t do a Google search now without getting any search results returned to you. So it is coming in all the time, but it is riddled with errors and prejudices. And what I want to do is concentrate on what how people are using it to contact me as a regulator. And what we’re looking at here is that it just lacks those vital human elements, you never get a thank you for your response to this.
19:04 It’s angry, it’s aggressive. It’s trying to follow a particular line that has been set fed into it at the beginning, and you see that percolating all the way through. And what we’re seeing is people starting with a case. Often it’s an AI that suggests.
19:19 So “Can you tell me how many – back to potholes – there are in the City of London?” I will say I don’t have that information. Do you want me to do an FOI request? Yes, here’s your FOI request. Email it to somebody. Yes. It goes, you get the answer back. Don’t have the information, and then it’s due to appeal that or complain about it, however, or in Scotland, it was coming back to see what a review goes to review, they get this review back.
19:46 “Do you want me to put it to the commissioner?” Now, there is no human interface within this. It’s automated. It’s a machine who you’re negotiating with. It might have a human at the back of it who’s got a name at the bottom, but it’s not a human you’re dealing with.
20:13 So, so far we’ve we’ve knocked some out for being invalid, because people won’t engage on those 60 FOI appeals to put to my office. They won’t engage with it. Well, give it an aggregate. No, for every single one, I’m not doing that. OK, that’s invalid. We’re getting rid of it.
20:29 Some of them are vexatious. Some of them are frivolous. Another option we’ve got, but it’s a real, real problem for us as a regulator, and not just for myself and for Darren’s team, but for all regulators across the sector, and I meet with them every two three months.
20:48 In fact, there’s a meeting today about this, and we’re trying to find ways that we can deal with this and manage it, because every time we get sooked into one of these mega cases with AI, it means that we’re not able to deal with the other cases of the people who are taking their time making…
21:05 And the other problem, the denial of service attack. This is something, again, my eyes are very much focused on. I’m beginning to suspect there are some bad actors, shall we say, that are out there in terms of doing this, because you can swamp it. We’ve seen it in America, and there’s no reason to say why it wouldn’t come here. So super alert to that. Are people going to try and swamp the system, as they’re doing in Ukraine, where Russians are currently swamping the system with FOI requests to try and slow up the system, the bureaucracy?
21:37 So aware of it, and it’s something we’re monitoring with others other areas. We’ve got extension to the care sectors, or extension of foi provision to the care sector, that will bring, think, about 3,000 care organisations into play under FOI.
21:57 And again, this is the problem where, traditionally, if a local authority had a nursing home, you could ask them for how many nurses were on, on a particular date, and they would give you an answer. When you have a outsourced version, how many were on?
22:11 Says, well, contractually, they should have five on. That’s not what I asked. I’m asking how many did they have on? But they’re not covered by that. So extension to the care sector would mean that private sectors, private providers, would be brought into that umbrella, and you’d have the same rights of information that you would get had it been a local authority home as it used to be.
22:31 Revision of section 60 code, which is our Code of Practice. We’re currently going through that just now, and that’s going again, just going to make some more clarifications and tighten up on a few bits and pieces, but that’s under out for consultation, and finally, on interventions. This is where we’re going upstream.
22:46 We’re looking at the problems using that graph we had at the beginning of our dashboard, seeing where we see individual authorities having problems, and we’re using that as an intelligence system to allow us to say, you’re struggling here. What’s the problem?
22:59 Usually it’s because they’re not supported by senior managers. So that then means I go see the senior managers, the board, the chief executive, say, you need to fix this, otherwise there’s pain coming your way, and they fix it. That’s essentially how the dimensions work.
23:14 And it’s been very, very successful, because everyone hates us coming in to start it and they’re sorry to see us leave, because the journey is usually a very positive one, where FOI teams feel totally supported at the at the end of it, and they have a status in the organisation that is that is much, much better than it was.
23:29 And finally, there’s one question, and I don’t know if this is recognised down here, but why is everyone so angry? Public discourse in Scotland is absolutely.. it’s polarised. I don’t know why that is, but it’s been mentioned, someone mentioned earlier on, in terms of what we see, not just to our own staff, in terms of the kind of discourse, public discourse, but also from people speaking to public authorities and so on, and actually public authorities, as well being pretty rubbish to requesters, and it’s something that I’m always urging the public authorities, at least, to try and bring the temper down, try and be the bigger person, but it applies equal to requesters in terms of keeping it civil, keeping it sensible, and having an understanding of that.
24:19 There’s a lot of pressure out there for people, but I think we do need to sort of try and just reset some of the atmosphere.
24:27 So conclusion, FOI Act, good place, it’s generally but it needs a little service, a little tweak, and that’s what the reform will hopefully do. Transparency is going to be a massive thing in Scotland at this election. We’ve got Scottish Parliament elections in May. The phrase “secret Scotland” is now almost every day on the front pages because people are talking about different authorities, not just government, but NHS, boards, local authorities, all keeping stuff secret.
25:01 Now, whether that’s fair or not, that’s for others to judge. I only look at individual cases and take that on board, but it’s becoming a massive, massive electoral topic, and I suspect heavily in voting intentions. To be honest, I’m optimistic, refunding of, expanding our interventions capability, but having a dedicated resource to that, I think that’s one way that we can help try and cut down our problems in terms of a demand management.
25:27 The expansion to the care sector, probably next year, that will be, again, quite significant. But again, it was mentioned, I think Warren mentioned today about that last time we did an expansion, we brought in registered social landlords into the Scottish regime.
25:43 They said, we can’t do this. It’s going to cost a fortune. It’s going to be huge problems. But we supported them, trained them up on it, got them ready for it, and they’re our best performing sector. And they’ve always surveyed them before, surveyed them after, and said, “This is much better than we thought it was going to be”.
25:59 And that’s actually a kind of poster sector, if you like, for the care sector.
26:05 So yeah, business is booming, and I could do a little bit of a break, but there you go. So that’s me, folks. Thank you very much. So I will take a round of questions from the audience.
26:17 [Audience member] I’m Stewart Dresner from Privacy Laws & Business and a former FOI enthusiast, well, still enthusiast, but it’s a long time. Anyway.
26:17 I was very excited at your point at the very end about Secret Scotland and the fact that open government is now on the agenda for the elections in Scotland. And I think I’m right and say, I can’t remember a single time in the last 50 years where Freedom of Information has been an electoral issue in England or Wales. Obviously, the campaign is much better informed than me. But is that so? And how have you managed to get it on the front pages rather than on page 16 at the bottom?
26:52 [David Hamilton] No personal credit for this. It’s more about circumstance and timing. So at the moment, I would say my office is in the front pages of the Scottish papers every three days. It’s incredible, and it’s we’ve never seen anything like this before, but it’s down to people taking a lot of interest on some particularly high profile issues. And the most high profile one, of course, is the Hamilton inquiry, which was into the conduct of the former First Minister.
27:25 And that’s a lot of attention, media wise. But there’s also been health boards which have failed. And it’s at all levels. It’s not just even national it’s at local level as well.
27:38 And I think the journalistic mood has been saying looking at things and seeing things, and that’s driving the political temperature. So I can’t see, I certainly don’t take any credit for that, and nor would it be appropriate for me to be driving that.
27:55 But I do think that it’s been a set of circumstances that politicians, of course, going into an election will look for anything, and this has just probably turned up at the perfect time for those politicians meeting these points.
28:09 [Audience member] Thanks very much.
28:11 I wondered if you could say something about what you expect might come of the care sector being included. I’m conscious, you know, there’s been a whole shift to privatisation of public services, and real concerns about how there’s now private equity profiteering out of a lot of these provisions. I wondered to what extent you would envisage that FOI will lead to kind of more scrutiny of the costs of provision and how private sector costs are spiraling.
28:38 Shine that light on what’s actually happening here. One of the big challenges is, of course, ‘define the care sector’.
28:45 And one of the bits we’re looking at is, does this includes, for example, childminders? You know, where do you stop with this? And I don’t think we’re in a place where that’s going to get covered, but we’re more likely to … essentially, the main purpose is looking at, you know, the nursing home scenario.
29:04 And I think the covid situation is probably the best argument for this in its entirety. And actually having the ability to find out not what should be, but what has been a situation, is absolutely critical. And, you know, that’s something which I think most people find difficult to argue against, and there’s a huge amount of support for it in terms of the commercial aspects of it. Well, you know, it’s public money. We’re entitled to know how our public money is getting spent, and if the company is making extortionate amounts of money on that, then we should know.
29:40 [Audience member] Thank you.
29:42 [Myf] Hello to our most completist listeners. If you’re still listening, thank you so much for sticking with this right to the end. I’m Myf from mySociety, and it’s my job to put these podcasts together.
29:55 And you know what? It would be really helpful for us to know a bit more about how and why you are listening to them. So if you have a moment, please do take a look at the show notes, where you’ll find a link to a very short survey. It’d help us so much if you could fill that in. Thank you.