1. PSNI data breach

    On the afternoon of Tuesday 8 August 2023, we ​​were contacted by the Police Service of Northern Ireland (PSNI) with a request to remove a response that they had made to a Freedom of Information request submitted through WhatDoTheyKnow.

    As many news stories have since reported, the response contained personal information pertaining to a large number of Police Officers and civilian staff. In line with our policies around serious data breaches, we took rapid action to hide the material from public view and subsequently deleted it. We also submitted cache removal requests to ensure that any copies of the information would be removed from search engines as soon as possible.

    The overwhelming majority of requests and responses that we process through WhatDoTheyKnow are unproblematic. However, we take the responsibilities that come with operating a large platform extremely seriously, especially around the personal data breaches that can occur when authorities’ release processes fail. 

    We recognise the significant impact that serious data breaches like this one have on the people affected and on their families, and we are assessing whether there is more we can do to help authorities avoid making such breaches in future.

    Image: Jason Leung

  2. FOI and the Irish border issue

    At the time of writing, a No Deal Brexit seems ever more likely. What exactly will that mean for the UK?

    Attempts to answer this question have filled many column inches, hours of broadcast and endless tweets. There is certainly no lack of opinions.

    But opinions are best based on facts, and it was in this spirit that WhatDoTheyKnow user Jon Rush set out to request vital information about the key Brexit sticking point, and the main reason that a deal is so hard to agree — the Irish border.

    Brexit and the border

    As Jon explains, “Brexit creates serious problems for the current arrangements between Northern Ireland and the Republic of Ireland under the Good Friday Agreement  because they depend heavily on both the UK and RoI being in the EU”.

    He wanted to gain access to the results of a mapping exercise, referred to in a joint report from the EU and UK negotiators but not available to the public at that time, which assessed the level to which co-operation between the North and the South depends on the EU frameworks currently in place.

    Crucial information, you might think, for the general public who will be so affected at every level by whatever type of Brexit we enter into. Jon certainly thought so — but getting hold of it would set him on a long journey.

    A hard-won result

    Jon’s initial request, to the department for Exiting the EU (DEXEU) was in December 2017. You can follow its long and complicated journey on that page, thanks to Jon’s detailed annotations.

    FOI is one of the few tools that individuals can use to hold government to account and it’s important to use it — otherwise government will never take transparency seriously.

    Simultaneously he was requesting the same information via our partner AccessInfo’s site, AskTheEU.com, which covers EU authorities — and meanwhile, MPs in the UK’s Exiting the EU Select Committee requested the same information on numerous occasions throughout 2018, but were repeatedly rebuffed by government.

    Pursuing his right to information would take Jon via the ICO, the European Ombudsman and to the brink of a tribunal, but in the end, the report was indeed released into the public domain.

    What was revealed

    What did it tell us?

    “It contains a description of each area relevant to North-South cooperation under the Good Friday Agreement together with an assessment of how far it is underpinned by EU legal and policy frameworks.

    “The focus in the media has tended to be on trade/customs arrangements, but if you go through the mapping exercise, you find that many other areas of cooperation are underpinned by the EU membership, including transport links, water, waste management, energy, Irish language broadcasting, mobile roaming, invasive species, disease control and cross-border police cooperation.

    “Overall, 96 out of 142 different areas covered by the mapping exercise were found to be supported by EU legal or policy frameworks (with well over a third being “directly underpinned or linked”, ie EU membership is particularly significant).

    “This shows that any workable solution is likely to involve the UK committing to quite a close relationship with the EU, at least in the areas identified as crucial to North-South cooperation”.

    Approaching the UK border on Newtown Road for J0719 The houses beyond the sheugh which here marks the Border are in Co Armagh. The gate in the foreground are those of disused ROI Border facilities. The road sign is a reminder of the Republic's 80kmph speed limit on back roads.

    Disused ROI border facilities in Co Armagh. Image by Eric Jones (CC by-sa/2.0)

    A lack of transparency

    The release of this information was a positive result — but Jon believes that the government has been far from open during the whole Brexit process.

    “To be properly informed about Brexit, we need access to information which is often available only from government. It would be very difficult for an organisation outside government to produce something like the mapping exercise because it requires input from numerous experts across different areas and in some cases, access to information that only government is likely to have.

    “Government is therefore uniquely well placed to provide this information – but if government refuses to share it, it’s impossible to get the full picture.

    “In my view, the government’s approach to its own documents concerning Brexit has been to release as little as humanly possible, arguing that disclosure would undermine its negotiating position with the EU.

    “I accept that occasionally, information may need to be withheld for this reason. But it is equally if not more important that people can understand what Brexit will mean for them —  and I don’t think the government has paid anywhere near enough attention to that issue”.

    Evading scrutiny

    This was not Jon’s first experience using FOI: in fact, he had recently exercised his rights to information on another Brexit matter.

    I asked DEXEU for details of the scope and timetable of their consultation on leaving the EU. This was after David Davis (who was then Secretary of State for Exiting the EU) had told Parliament in September 2016 that the government would be consulting widely on the options for leaving the EU.

    WhatDoTheyKnow.com has made the process quite easy to initiate and it also means that others who might be interested in the same information can find your request.

    “By late October, nothing had been published, so I made an FOI request through WhatDoTheyKnow.

    “Initially, DEXEU told me it had this information but refused my request, saying that it planned to publish the information at a later date. I didn’t see why the information couldn’t be published sooner and complained to the ICO.

    “Their investigation showed that DEXEU did not have a formal plan or any formal process for the consultation — which explained their somewhat evasive response.

    “DEXEU should probably have told me that it didn’t hold the information I had requested – but to do so would have involved effectively admitting that it didn’t have a plan or any formal process for consultation. You can make up your own mind by reading what the ICO had to say here”.

    Pursuing a refused response

    But back to the Irish border request. When Jon didn’t receive a response from DEXEU, and after requesting a similarly fruitless internal review, he took the next step and referred the matter to the ICO. They ruled against disclosure in a decision that Jon believes was ill-founded:

    “The ICO decision was based on section 35 of the FOI Act, which relates to information produced for the purposes of policy formulation.

    To be properly informed about Brexit, we need access to information which is often available only from government.

    “It is certainly true that the mapping exercise was produced to inform the government’s thinking about Brexit and Northern Ireland. However, it was a summary of the current arrangements, not a discussion of what the future policy options should be; as such, it was essentially background information, which is usually regarded as less sensitive. Section 35(4) makes it clear that there is a particular public interest in the disclosure of background of information – and case law makes it clear that such disclosure can take place before the final policy has been formulated, as I was requesting here.

    “The ICO also argued that disclosure of the mapping exercise would have a “negative effect on discussions” with the EU and “create a distraction to discussions” — but its decision did not explain how this would occur, especially given that the mapping exercise had been shared with the EU.

    “When I put these points to the ICO as part of my appeal to the tribunal, it accepted that the mapping exercise was background information but argued that it should be treated in the same way as discussion of policy options. It was unable or unwilling to provide any further explanation of the supposed negative effects of disclosure and suggested that this was a matter for DEXEU to explain. I was (and remain) very concerned by this because the ICO is supposed to be an independent regulator; it should not simply be taking what government says at face value but should be questioning it and satisfying itself that what government says is actually correct”.

    And so Jon referred the matter to tribunal.

    But in June of this year, two of the key documents he was requesting were finally  released by the government, and he decided to drop his appeal to tribunal, for reasons which you can read in his annotation of the time.

    Motivation

    While many WhatDoTheyKnow users are determined and driven, it’s also true that others would be easily defeated by an initial refusal, not to mention the further rulings. So what gave Jon the will and tenacity to carry on?

    I would encourage people to use FOI … if you are prepared to persevere and be patient, you can get what you want.

    “I knew that appealing to the tribunal would involve quite a lot of time and effort on my part, but I wasn’t prepared to just let this go for two reasons. Firstly, FOI depends on having an effective regulator which is prepared to question government robustly — and if people like me just shrug our shoulders when that doesn’t seem to have happened, then nothing will ever improve.

    “Secondly, Brexit is going to take many years to sort out and there will be many more occasions where people want to use FOI to get information out of government; unless challenged, government will just continue to refuse to disclose information whenever it suits it to do so.

    “Appealing to the tribunal was a new experience for me. I am a lawyer by profession, which probably helped, but I am not an expert in FOI, nor am I a litigator — and I did feel at times that my lack of familiarity with those areas was a handicap. So I have a lot of respect for people who are not lawyers and take cases to the tribunal on their own.

    “I would encourage people to use FOI and I think that what happened with this request shows that, if you are prepared to persevere and be patient, you can get what you want — even in a situation like this where MPs had asked repeatedly for exactly the same information and hadn’t received it.

    “FOI is one of the few tools that individuals can use to hold government to account and it’s important to use it — otherwise government will never take transparency seriously. WhatDoTheyKnow.com has made the process quite easy to initiate and it also means that others who might be interested in the same information can find your request.”

    Jon is also planning to submit a complaint to the ICO about its handling of this case, including the time taken to deal with it:

    “Although it was expedited, it still took over six months, whereas my complaint to the European Ombudsman (which concerned essentially the same material) was dealt with in about half that time.”

    He intends to post a link to the complaint in a further annotation on the FOI request page on WhatDoTheyKnow – so watch this space!

    Many thanks to Jon for taking the time to talk to us about his long and involved pursuit of information, which despite the delays will still help to inform the UK public at this critical time in our country’s history.


    Top image: the Irish border by Andrein (CC by-sa/3.0)

  3. FixMyStreet now covers Northern Ireland

    Since its launch in 2005, WriteToThem has always covered all parts of the United Kingdom, and the Northern Ireland Assembly was the first body added to TheyWorkForYou after the UK Parliament, in late 2006. So whilst we certainly have not ignored Northern Ireland, it had always been an irritant of mine (and a cause of infrequent emails) that FixMyStreet only covered Great Britain.

    This was due to the way it had originally been funded and set up, but those issues were in the past, due to a myriad of changes both internal and external, and it was now more a case of being able to find the resources to implement the necessary work. Late last year, mySociety worked with Channel 4 on the website for their series of programmes on The Great British Property Scandal. This used, in part, code similar to FixMyStreet to let people report empty homes, and it was required to work in all parts of the UK. So as part of that process, code was written or generalised that let aspects of FixMyStreet like the maps and place name lookup work for Northern Ireland locations.

    It’s taken a few months since then to allocate the time, but we’ve now been able to take the code written back then, add various other bits, and incorporate it into FixMyStreet – which now covers the 26 councils of Northern Ireland, and the central Roads Service. Issues such as potholes, graffiti, and broken street lighting can be reported to Antrim or Newry and Mourne as easily as Aberdeen or Wyre Forest, and just as in the rest of the UK you can sign up for alerts based around your location or to your council.

  4. Helping voters in the devolved elections

    As well as council elections and the referendum, the Scottish Parliament, Welsh Assembly, and Northern Ireland Assembly are holding elections this May. In Scotland and Northern Ireland, there are accompanying boundary changes, meaning this year you might be voting in a different constituency from last time.

    To help people, as we’ve again had a few requests, our service from the 2010 general election is back, at http://www.theyworkforyou.com/boundaries/, just for the Scottish Parliament and Northern Ireland Assembly. Our generic lookup service MaPit also provides programmatic access to these results (technical footnote).

    Alongside this service, we have refreshed our Scotland and Northern Ireland front pages, to slightly better display and access the wide array of information TheyWorkForYou holds for those devolved legislatures.

    Sadly the Scottish Parliament changed the format of their Official Report in mid January and we haven’t been able to parse the debates from then until its dissolution this March – hopefully we’ll be able to fix that at some point, and apologies for the inconvenience in the meantime.

    There don’t appear to be any central official lists of candidates in these elections. Amnesty.org.uk has a PDF of all candidates in Northern Ireland; David Boothroyd has a list of Scottish Parliament candidates. CAMRA appears to have lists for both Scotland and Wales. Those were simply found while searching for candidate lists, we obviously hold no position on those organisations 🙂

    Technical footnote: To look up the new Scottish Parliament boundaries using MaPit, provide a URL query parameter of “generation=15” to the postcode lookup call. The Northern Ireland Assembly boundaries are aligning with the Parliamentary boundaries, so you can just perform a normal lookup and use the “WMC” result for the new boundary.

  5. New features on MaPit

    We’ve added a variety of new features to our postcode and point administrative area database, MaPit, in the past month – new data (Super Output Areas and Crown dependency postcodes), new functionality (more geographic functions, council shortcuts, and JSONP callback), and most interestingly for most people, a way of browsing all the data on the site.

    • Firstly, we have some new geographic functions to join touches – overlaps, covered, covers, and coverlaps. These do as you would expect, enabling you to see the areas that overlap, cover, or are covered by a particular area, optionally restricted to particular types of area. ‘coverlaps’ returns the areas either overlapped or covered by a chosen area – this might be useful for questions such as “Tell me all the Parliamentary constituencies fully or partly within the boundary of Manchester City Council” (three of those are entirely covered by the council, and two overlap another council, Salford or Trafford).
    • As you can see from that link, nearly everything on MaPit now has an HTML representation – just stick “.html” on the end of a JSON URI to see it. This makes it very easy to explore the data contained within MaPit, linking areas together and letting you view any area on Google Maps (e.g. Rutland Council on a map). It also means every postcode has a page.
    • From a discussion on our mailing list started by Paul Waring, we discovered that the NSPD – already used by us for Northern Ireland postcodes – also contains Crown dependency postcodes (the Channel Islands and the Isle of Man) – no location information is included, but it does mean that given something that looks like a Crown dependency postcode, we can now at least tell you if it’s a valid postcode or not for those areas.
    • Next, we now have all Lower and Middle Super Output Areas in the system; thanks go to our volunteer Anna for getting the CD and writing the import script. These are provided by ONS for small area statistics after the 2001 census, and it’s great that you can now trivially look up the SOA for a postcode, or see what SOAs are within a particular ward. Two areas are in MaPit for each LSOA and MSOA – one has a less accurate boundary than the other for quicker plotting, and we thought we might as well just load it all in. The licences on the CD (Conditions of supply of SOA boundaries and Ordnance Survey Output Area Licence) talk about a click-use licence, and a not very sraightforward OS licence covering only those SOAs that might share part of a boundary with Boundary-Line (whichever ones those are), but ONS now use the Open Government Licence, Boundary-Line is included in OS OpenData, various councils have published their SOAs as open data (e.g. Warwickshire), and these areas should be publicly available under the same licences.
    • As the UK has a variety of different types of council, depending on where exactly you are, the postcode lookup now includes a shortcuts dictionary in its result, with two keys, “council” and “ward”. In one-tier areas, the values will simply by the IDs of that postcode’s council and ward (whether it’s a Metropolitan district, Unitary authority, London borough, or whatever); in two-tier areas, the values will again be dictionaries with keys “district” and “council”, pointing at the respective IDs. This should hopefully make lookups of councils easier.
    • Lastly, to enable use directly on other sites with JavaScript, MaPit now sends out an “Access-Control-Allow-Origin: *” header, and allows you to specify a JSON callback with a callback parameter (e.g. put “?callback=foo” at the end of your query to have the JSON results wrapped in a call to the foo() function). JSONP calls will always return a 200 response, to enable the JavaScript to access the contents – look for the “error” key to see if something went wrong.

    Phew! I hope you find this a useful resource for getting at administrative geographic data; please do let us know of any uses you make of the site.

  6. Mapping points and postcodes to areas

    I’m very pleased to announce that mySociety’s upgraded point and postcode lookup service, MaPit, is public and available to all. It can tell you about administrative areas, such as councils, Welsh Assembly constituencies, or civil parishes, by various different lookups including name, point, or postcode. It has a number of features not available elsewhere as far as I know, including:

    • Full Northern Ireland coverage – we found a free and open dataset from the Office of National Statistics, called NSPD Open, available for a £200 data supply charge. We’ve paid that and uploaded it to our data mirror under the terms of the licence, so you don’t have to pay – if you feel like contributing to the charity that runs mySociety to cover our costs in this, please donate! 🙂
    • Actual boundaries – for any specific area, you can get the co-ordinates of the boundary in either KML, JSON, or WKT – be warned, some can be rather big!
    • Point lookup – given a point, in any geometry PostGIS knows about, it can tell you about all the areas containing that point, from parish and ward up to European electoral region.
    • History – large scale boundary changes will be stored as new areas; as of now, this means the site contains the Westminster constituency boundaries from both before and after the 2010 general election, queryable just like current areas.

    If you wish to use our service commercially or are considering high-volume usage, please get in touch to discuss options; the data and source code are available under their respective licences from the site. I hope this service may prove useful – we will slowly be migrating our own sites to use this service (FixMyStreet has already been done and already seems a bit nippier), so it should hopefully be quite reliable.

    Thanks must go to the bodies releasing this open data that we can build upon and provide these useful services, and everyone involved in working towards the release of the data. Thanks also to everyone behind GeoDjango and PostGIS, making working with polygons and shapefiles a much nicer experience than it was back in 2004.

  7. Political parties don’t know where the boundaries are

    In my last blog post, I explained the new service TheyWorkForYou offers to show you what constituency you will be in at the next general election. Now I’m going to show you why you shouldn’t use anything else.

    The defintions of the boundaries for the forthcoming constituencies in England were originally published in The Parliamentary Constituencies (England) Order 2007 (SI 2007/1681), based on ward boundaries as they were on 12th April 2005. However, due to some local government changes since that date, The Parliamentary Constituencies (England) (Amendment) Order 2009 (SI 2009/698) was published changing the boundaries for four constituencies – Daventry, South Northamptonshire, Wells, and Somerton & Frome – to be based on the new council wards as they were on 3rd May 2007.

    The forthcoming constituencies in Northern Ireland were defined in The Parliamentary Constituencies (Northern Ireland) Order 2008 (SI 2008/1486). In this, Derryaghy ward was split between two constituencies – Belfast West is given “that part of Derryaghy ward lying to the north of the Derryaghy and Lagmore townland boundary.”

    All of which means that other sites that try to tell you what constituency you will be in at the election invariably get it wrong.

    Both Labour and the Conservatives say that BA6 8NJ is in Wells at the next election, when it will be in Somerton & Frome. Both say that NN12 8NF will be in Daventry, when it will be in South Northamptonshire. I assume that both sites are using boundary data predating the Amendment Order from March 2009. The Conservatives also say that BT17 0XD will be in Lagan Valley when it will be in Belfast West; Labour simply say “Northern Ireland” for any Northern Irish postcode you provide.

    The Liberal Democrats site currently returns no results for any postcode, which I assume is a bug 🙂

    The current boundary between Belfast West and Lagan Valley.

    The /current/ boundary between Belfast West and Lagan Valley. (Image produced from the Ordnance Survey electionmap service. Image reproduced with permission of Ordnance Survey and Land and Property Services)

    The official election-maps.co.uk service (from where TheyWorkForYou gets its boundary maps) returns the correct results for BA6 8NJ and NN12 8NF, but doesn’t have future boundaries for Northern Ireland. It’s not clear that it doesn’t, as searching for Lagan Valley with “future boundaries” selected returns a result, but that result is the current boundary. This can be seen from the picture on the right – as is clear from the quote I gave above, everything within Derryaghy ward north of the Lagmore/Derryaghy boundary will be in Belfast West at the next election. Plus the site doesn’t work without JavaScript.

    TheyWorkForYou’s “constituency at the next election” service gives BA6 8NJ in Somerton & Frome, NN12 8NF in South Northamptonshire, and BT17 0XD in Belfast West. There is enough confusion with the changes to boundaries for everywhere except Scotland, that it is somewhat frustrating to have it compounded by sites giving incorrect information. The lack of any official service also doesn’t help.

  8. Postcode and boundary refresh

    We updated our boundary and postcode database at the start of the week (apart from two wards in Scotland that I misspelled and updated on Tuesday, sorry), so hopefully everyone in the country can contact their representatives at WriteToThem or have their postcode recognised on HearFromYourMP or TheyWorkForYou. This applies especially to a small number of councils, such as Bradford, for which the boundaries had completely changed at their last election and which we were unable to get working until now – apologies for the inconvenience.

    Related to this, and for interest, on 1st April, a number of councils are being abolished as their county councils become unitary authorities. The district councils within Durham, Northumberland, Cornwall, Wiltshire, Shropshire, and Cheshire/Chester all disappear – Cheshire becomes two unitary authorities called Cheshire West and Chester, and Cheshire East. Lastly, Bedford borough council becomes a unitary authority, and Central Bedfordshire council covers the area previously covered by Mid Bedfordshire and South Bedfordshire.

    Parliamentary boundaries in England and Northern Ireland are changing, but these do not take effect until the next general election – until then, your constituency and MP remains the same.

  9. Postcodes on TheyWorkForYou

    If you enter your postcode on TheyWorkForYou and it’s Scottish or Northern Irish, you’re now presented with your MSPs and MLAs as well as your MP, which makes sense given the site covers their Parliament and Assembly respectively. 🙂 You also get an extra tab in the navigation linking through to Your MSPs or MLAs. In order to do this, I needed a quick way of determining if a postcode was Northern Irish or Scottish. Northern Ireland was easy, as all postcodes there begin with BT. I assumed Scotland was also easy, which turned out to be true apart from the TD postcode area that straddled the border like a mail-sorting Niagara Falls. After some very dull investigation, I eventually worked out that e.g. most of TD15 is in England, but (amongst others) TD15 1X* is in Scotland, except for TD15 1XX which is apparently back in England. The final result was the postcode_is_scottish() function in postcode.inc, which (hopefully) correctly determines if a given postcode is Scottish or not – perhaps someone else will find it useful.

  10. Matthew Somerville stuns co-workers

    By doing this.

    So, come on then Scots and Welshpersons: who’s going to take up his challenge?