There’s not a lot to go on. Over the years he hasn’t expressed much interest in the topic, either in Westminster or in Manchester.

But one possible view of Andy Burnham is that he’s followed the standard politician’s trajectory of being keener on FOI when in opposition than in government.

This is nicely illustrated by the story of the Department for Health’s risk registers.

As a health minister in the previous Labour government, he declined to publish the departmental risk register, on the grounds that: “Putting the risk register in the public domain would be likely to reduce the detail and utility of its contents. This would inhibit the free and frank exchange of views about significant risks and their management, and inhibit the provision of advice to Ministers.”

But in 2012, having moved to opposition, he learnt to see the virtues of transparency. He now demanded the release of a Department of Health risk register relating to the then government’s NHS reorganisation policy. This was in line with ICO and tribunal rulings, which the Cameron coalition eventually blocked by use of a ministerial veto.

Naturally his new stance required devising arguments (of varying plausibility) as to why the issues surrounding this transitional risk register were completely different to those surrounding the strategic risk register previously withheld when Labour was in power.

Later, as any good opposition politician should, he also made use of FOI to gather material to attack government policy. He criticised health bodies who cited commercial confidentiality to withhold information.

On the other hand, back in 2007, in a free (unwhipped) Commons vote, he had opposed a measure to ensure that information about MPs’ parliamentary expenditure remained subject to FOI. He was thus one of a few dozen MPs who tried and failed to keep expenses details out of the public domain.

This was before the eventual publication of the extraordinary facts involved which created a massive political crisis in 2009, and in which he was caught up in a comparatively minor way.

However he has consistently promoted transparency on one topic, as part of his admirable personal commitment over many years to the cause of the Hillsborough victims and their families.

He pressed determinedly for a full inquiry including release of documentation and for the public service ‘duty of candour’ in the law that is currently going through parliament.

The Thatcher government’s response to the Hillsborough tragedy was a topic that I investigated using FOI around 15 years ago when I worked for the BBC. I talked to him about it at the time, he was keen to do so, and I can remember him telling me earnestly that “We’re on the same side”.1

More recently, as Mayor of the Greater Manchester Combined Authority he would have found himself and his office subject to FOI again.

The authority was rebuked earlier this year by the Information Commissioner’s Office for its failure to comply with the official code of practice on publishing FOI statistics. However it has to be said that this is a widespread problem amongst public bodies.

Burnham now takes national power at a time when earlier hints of government plans to constrain FOI appear to have been set aside.

I can’t see those possible restrictions reappearing in the near future. Apart from anything else, it would be tactically extremely foolish for the newly formed government, generating press antipathy and bad publicity. Those factors ought to feature on any decent risk register.

Assuming that FOI policy stays with the Cabinet Office, the cabinet minister directly responsible will be Louise Haigh, the new First Secretary of State and Chancellor of the Duchy of Lancaster, a close Burnham ally. She will generally have a very influential role within the government.

In opposition she did speak enthusiastically about FOI: “It has saved time and money and helped countless services improve for the better but crucially it has enshrined in law the right to know about the decisions being taken in our name. Make no mistake, we will fight to defend it.”

In particular she campaigned strongly for FOI to be extended to private contractors delivering public services, which would be an important strengthening of the law and the rights of citizens to information.

This much needed reform was theoretically Labour government policy under Starmer - as a ‘longer-term’ measure to be taken forward ‘in due course’. An important question is whether Haigh will use her influence to ensure it is implemented.

So we wait to see if ‘in due course’ has now arrived, or if alternatively things will seem different when viewed through the prism of holding power.

Subscribe now

For a couple of years in an earlier phase of my life, instigated by the Tory leadership election of 1995, I gambled on politics and made money by doing so. The benefits to me were not only financial – I learnt life lessons from gambling.

I largely focused on spread betting, where decisions could easily have significant financial consequences, for good or ill. I wasn’t betting for ‘fun’, or to buy extra interest or excitement in events, or to hedge or reinforce my emotional reaction to what happened politically, all of which could be rational reasons for risking (or squandering) a few pounds. My aim was purely to win money.

I stopped after I joined the BBC in 1998, since it could have created a conflict of interest (which also prevented me taking up a consultancy role I was offered by the betting company Sporting Index to advise on political bets).

Share

Lesson 1: Detail, detail, detail (and do your own research)

I became interested in betting when John Major, then a beleaguered prime minister, told his many internal party critics to “put up or shut up”, and his cabinet colleague John Redwood went for the “put up” option. Which led to the 1995 Conservative leadership contest.

I was then a freelance journalist, and was watching the ITN lunchtime news on the day of the ballot. Its political editor Michael Brunson said Redwood would get about 45 votes from Tory MPs. I thought “If it’s good enough for Michael Brunson, it’s good enough for me”, and phoned a spread betting company. Later that day it was announced Redwood got 89 votes, and I lost £150.

Yet I gained something else - the insight that money could be made, but it needed my own careful analysis, not a reliance on the views of others.

And this meant: avoid general impressions, avoid broad judgments – they are little better than plucking figures out of thin air. Instead break down the factors of a situation (eg the Conservative parliamentary party) into its component parts, as granular as you possibly can, and gather whatever detailed evidence exists on each component.

In due course in the 1997 Tory leadership election I made over 20 times what I’d lost on the 1995 one.

Lesson 2: Opinions and decisions aren’t the same

I learnt how big the difference is between holding an opinion and actually being seriously willing to stake your money on it.

When I was deciding whether to place a bet, and I forced myself to properly think through whether the evidence really supported a view I held, I then realised that maybe it didn’t. And this sometimes applied even when I’d previously been enthusiastically advocating that viewpoint in conversation with others. It’s one thing to enjoy a good argument if you want, but when it comes to decision-making with real consequences it might be better to be more evidence-based.

In other words there was no need to bet on an outcome just because I’d confidently told everybody that it was what I expected to happen. I acquired a healthy disrespect for my own judgments.

Lesson 3: The status quo is always changing

There’s a new status quo with each decision you take.

Sometimes in gambling, as in financial trading, it is possible to guarantee smallish wins by what is called arbitrage. When on some event two betting companies have different spreads which do not overlap, and you spot it before one of them closes the gap, you can guarantee profit by selling with one and buying with another.

For example: Take the number of seats Labour will win at the next general election - if company A quotes a spread of 320-330 and company B quotes 333-343, you could buy with A at 330 and sell with B at 333. If you did that at say £100 per seat in both cases, you’d make £300 for sure (£100 x 3, the size of the gap), and this would be completely irrespective of the actual election outcome.

Sometimes arbitrage opportunities were implied rather than being so explicit, and the bookies might not spot and stop these so readily. I remember in the 1997 general election there were spreads available on the number of Labour MPs and on the number of women MPs which were way out of line with each other, given how many Labour candidates in marginals were women and therefore how tightly the two spreads should have moved together.

This was before online gambling, so you had to make phone calls to place the bets, which took a little time. I reckoned the safest course of action for arbitrage was first to place the bet which represented value in my way of thinking; then place the counterpart bet, assuming it was still available.

But once I had placed the first bet I was in a new situation, and in fact one I was quite happy with. I was now sitting on a bet that I felt at the time was good value. Why on earth would I want to go and place another bet that to my mind was probably chucking money away? Well, I didn’t, so I now realised. I went through this process a couple of times and concluded this kind of arbitrage didn’t work for me (although it would of course be different in another field where I was ignorant and had no view on which bet actually represented good value).

So I’d been planning a multi-step strategy based on the situation I was in initially. But once I’d taken the first step, I was in a new situation where the rest of that strategy no longer made sense.

Lesson 4: Go against the herd, when the herd is wrong

Success can come from spotting when the herd is wrong, when everyone else is heading in the wrong direction – in gambling and in life. You can make money and the bookies themselves can also make money, as long as other punters are losing.

I realised that one key aspect of the bookies’ talents was to price bets not necessarily in line with predicted outcomes but so that people would go for them.

Shortly before a budget one year someone from a spread company told me they were thinking of opening a market on how often the chancellor would be interrupted during his speech and asked what I thought. I informed him it would almost certainly be none at all, since that was the established norm of Commons behaviour for a budget. I was then surprised to see that the company put up a spread of 2-4 interruptions.

When I spoke to him again a few days after the budget, I asked in puzzlement why they’d done this, when people who knew parliamentary convention could take some easy money off them by selling at 2. He explained that it had worked really well for the company, they’d had a few sellers at 2 but lots of people buying at 4 who therefore lost money. This included a minor TV celebrity of the time who had bought at £1,000 per interruption, thus losing £4,000. My contact obviously shouldn’t have told me who this was, but he was so pleased about it that he did.

Lesson 5: Life looks leptokurtic

Life is uncertain, and genuinely unpredictable events happen. You can lose money on decisions which on the basis of the information you had at the time were probably correct. And sometimes I thought I did. Equally well, you can win on the basis of decisions which actually were probably wrong. Probably I did that too.

Perhaps this is particularly applicable to politics, I’m not sure if that’s true or not. But I did increasingly form the view that the probability distribution of human events was fat-tailed. Weird things happened, more often than it seemed they should. In statistical terms, there’s a case for saying that the course of events is leptokurtic.

As they say, the race is not always to the swift, and while logic may suggest that’s where to put your money, it’s funny how often in politics (and life in general) it turns out that the tortoise is a surprise victor over the hare.

Subscribe now

Share

(This is a revised version of a piece that I originally published on my website).

I was interviewed by Ibrahim Hasan of ActNow Training, the information governance legal specialists, for his podcast series Guardians of Data.

This is a lightly edited transcript. (It was automatically generated, so may contain some minor errors). You can listen to the podcast here.

Ibrahim’s interviews with figures from the information rights world are very interesting, and I would strongly recommend the other episodes of his series.

Share

Ibrahim Hasan

You spent many years using the Freedom of Information Act to produce quite impactful journalism. But I just want to go back to the beginning before FOI. What was it like to try and elicit information from public sector institutions, especially government?

Martin Rosenbaum

The key thing about FOI is that it gives you a legal right under certain circumstances to squeeze information out of people who might be reluctant to give it to you. That’s really from the journalistic point of view. Now, obviously, prior to FOI coming in, journalists did lots of stories about government. In some ways, it wasn’t very different, but there were certain particular stories that you couldn’t get because you didn’t have that legal right to get information out of public authorities. And that’s the key change that it brought about.

I wouldn’t claim that it transforms the majority of journalism, because for a lot of journalists, they actually use it very little if at all. And that’s a lot to do with the delay that is involved in using FOI. It’s not something for daily stories, it’s only for longer-term stories. But for those longer-term, more in-depth, more investigative stories, it’s a very valuable resource.

Ibrahim Hasan

Before the Freedom of Information Act came into force, there were other instruments or codes of practice which gave access to information held by government. What was it like working with those codes?

Martin Rosenbaum

I did make use of the access to government information code, the code of practice that existed, as you say, at the time. I think the first thing that has to be said about it is that hardly anybody knew about it. Certainly not many journalists knew about it, but even when you were saying to government departments or other public authorities, large ones, there’s this code of practice, I’m entitled to this information, you’d find a lot of people didn’t know about it.

I did actually write an article for the Press Gazette, a journalism trade magazine about this, before I worked for the BBC, where I’d asked lots of public authorities for very simple information under this code of practice on open government. I can’t remember now the number who didn’t reply or said that they knew nothing about it, but it was a very substantial proportion.

So that was one reason why the code didn’t really make much difference, because neither journalists nor public authorities were really aware of it. But the second thing about it was that if they didn’t want to give you the information, they could just ignore the code. There was no legal recourse, you couldn’t complain to the Information Commissioner, you couldn’t go to a tribunal. That was just the end of the matter because it was purely advisory. So I think the advisory code that existed before the actual Freedom of Information Act came into force really made very, very minimal difference to making government more open.

Ibrahim Hasan

So now you’ve got a statutory right to receive information, and of course, anybody can make a request for information. How would you say that FOI has changed the relationship between public authorities, journalists, and the public?

Martin Rosenbaum

So, as I say, it’s the legal right that counts, and you can sometimes squeeze information out that otherwise you wouldn’t have been able to, and public authorities are aware of that. So the relationship between journalists and public authorities, say when you’re dealing with press officers, is made different by that, because you might be asking a press officer for certain kinds of information, and previously you would know there was a brick wall you weren’t going to get beyond, but now the press office knows there’s actually a kind of route over the brick wall for certain kinds of information, although obviously we know there are lots of exemptions.

Sometimes press officers will give you that information anyway because they know you could get it under FOI. Sometimes they want to make you go through the business of doing it under FOI because of the delay. But it does mean that that kind of power relationship between journalists and press officers is tilted a bit in the journalists’ direction. And the same also applies not only to journalists, but also to all requesters, which might be campaign groups, they might be businesses, they might be individual members of the community, could be anybody who also now have this legal power which public authorities are aware of, which gives power to them as opposed to the public authority.

Ibrahim Hasan

So, Martin, over the years you’ve broken lots of stories using the Freedom of Information Act. Can you share a memorable story that stands out as an example of the power of the Freedom of Information Act?

Martin Rosenbaum

So one story in particular, amongst many, has stood out in my mind as encapsulating some of the benefits of freedom of information, although in some ways it’s quite a mundane everyday kind of story. It took me an 18-month dispute with an agency from the Department for Transport. But I eventually managed to obtain information about which makes and models of cars were most likely to fail their MOT tests. And I knew this information must exist because the government was collecting all this information from MOT garages. And I thought this information is of use to the public to know which cars have the best records, however old they are, how likely they are to fail MOTs or not. As I say, this was resisted by a Department for Transport agency, who said it would breach commercial confidentiality to release the information, it was damaging to commercial interests. My view was it’s overwhelmingly in the public interest for the public to have this kind of information. The Information Commissioner ruled in my favour eventually. So the information was actually published and this came out.

But the thing that I most like about this story is the fact that this information, which for 18 months the Department for Transport told me was too sensitive to reveal, is now actually published proactively every year by them as open data. It is now available to anybody on the internet all the time, updated every year as a matter of routine, despite the fact that there was this initial resistance to making it available. And that to me is really a kind of symbolic triumph of FOI, to take information which at one point was considered too sensitive to reveal, get it out there in the open, now it’s revealed routinely as a matter of course.

Ibrahim Hasan

Absolutely. Do you think that over the last 20 years now we’ve seen a culture change thanks to FOI, where organisations, particularly government, are becoming more open and more routinely publishing information, whereas previously they wouldn’t have done so?

Martin Rosenbaum

I think there’s been a mix, really. Things have gone in waves. Initially, when FOI first came into force in 2005 and the couple of years after that, there definitely was among some public authorities a kind of enthusiasm for it. They felt it was part of the mood of the times to put information out there. And I think there were more positive responses then to FOI than some of the responses that we would get later. And then I think enthusiasm diminished. They found FOI a bit of a nuisance, sometimes worse, sometimes very embarrassing.

Then I think under the Cameron government from 2010, there was big enthusiasm for open data, publishing a lot of stuff proactively, which is not necessarily the same information that people were actually asking for under FOI. It’s sometimes what the government wants out there rather than what the public wants to know. Nevertheless, they did push a lot of stuff out there, and also I think there was a kind of effect of galvanizing people a bit into actually thinking: we believe in open data. This again was the spirit of the times, that we should be releasing stuff. And that then has diminished since then.

So I think there have been waves, and different public authorities have taken different attitudes at different times. I don’t think one can say there’s been a total transformation in attitudes to openness and transparency amongst the public sector. That’s going too far, but there is certainly partial evidence of that.

Ibrahim Hasan

It’s interesting what you say about the Cameron government, that they went through a phase of so-called openness and transparency. If you remember the local government transparency code, I think it still requires all payments made by local authorities over £500, and there’s thousands of transactions. So sometimes the interesting bits can actually be difficult to find.

Martin Rosenbaum

That’s absolutely true. And I remember when this first started happening, and all these details of invoices and receipts and everything were published, looking through them for certain kinds of public authorities, and it was absolutely overwhelming. And unless you really knew what you were looking for, it was very difficult to find anything of interest.

Ibrahim Hasan

So you’ve shared your experience of making the request to the Department of Transport, and that was quite significant because it led to a different sort of approach on their part and publishing information now routinely. Any other surprising or personally significant discoveries you’ve made through FOI?

Martin Rosenbaum

Another one which I thought was important, and this again was after quite a fight, a letter which the then Prince Charles, now King, had written to Tony Blair when Blair was prime minister, lobbying him against genetically modified foods, which Charles was very unkeen on and he wanted Blair to meet people who would tell him how bad genetically modified foods were.

And this was a two-year battle with the Cabinet Office to eventually get hold of this, which again needed the Information Commissioner to rule in my favour. And also, this was actually, as I’m sure some of your listeners would appreciate, under the Environmental Information Regulations, which cover environmental information, because GM food comes under that, and they’re similar to FOI but not identical. And you couldn’t get this under FOI. It was only possible because it was under EIR.

But what that illustrated to me was with these regulations, you could get information which was going right to the heart of the top of government, the kind of innermost sanctum in a way, the communications which were going from the then heir to the throne to the then prime minister.

Ibrahim Hasan

But then, of course, Tony Blair was criticised at the time, they termed it sofa government, where his advisors would meet without records and they were just discussing and making decisions. Perhaps that was a downside of FOI, and we’ve seen that with the COVID inquiry as well, messages were deleted. Do you think that is the case? That freedom of information perhaps has led to, in some quarters, fewer records?

Martin Rosenbaum

So I think this is a complex topic, and there are different forces that point in different directions. And you’re right to refer to Blair’s sofa government, as it was called, but that actually was going on before FOI came into force. It was how Blair wanted to do things. And while it might be true that when FOI came in they did record even less, there were reasons why they were doing that anyway, because they liked to operate in that kind of way. It was his personal style.

And then taking the COVID inquiry and the WhatsApp messages, and there are WhatsApp messages from certain people, certainly in Scotland and Wales, saying we must delete these, and not everybody deleted them, and these actually ended up being presented at the inquiry. I think what we now have with WhatsApp and other chat apps is that information is now recorded in a certain way, written down, which previously would have been the kind of telephone conversation, the corridor conversation. And that is a big challenge in this context because they tend to be more informal conversations, potentially more embarrassing, sometimes anyway, not the sort of proper record keeping of state. And it means all this stuff is potentially perhaps available. It’s certainly become available to the COVID inquiry. You could put in FOI requests for it, some of it you would get and some of it you wouldn’t because of the exemptions.

But I think technological change and the way people now communicate with each other, and of course this may change in the future, but where we’re at now, has indeed made this whole thing about what is recorded and what is not recorded much more acute and difficult for a lot of people, I agree.

Ibrahim Hasan

Martin, you’ve made your craft FOI over the years. Let’s just unpack how that works in practice. Now, your book, Freedom of Information, a Practical Guidebook, gives readers a step-by-step approach in terms of making FOI requests. What would you say are the key tactics or key qualities of a good FOI request separating it from one which could just lead to a dead end?

Martin Rosenbaum

There are lots of points in a way to bear in mind about how to write effective FOI requests. But I think the core thing above everything else is really to think through very clearly exactly what you’re asking for and make sure that the words that you’ve put down in your email actually ask for it clearly and precisely. In other words, you specify it exactly, and there is no ambiguity in what you have asked for.

When I was at the BBC and I was advising other journalists on putting in FOI requests, they would come to me sometimes and say, Look, I’m really annoyed, I put in a request for this, I didn’t get it, I’ll show you the request and the answer. And sometimes I’d end up saying to people, Actually, you didn’t get what you wanted to get, but you did get what you asked for, because if you look for a literal interpretation of your request, that’s what they actually sent you. And so the key thing is to express it really clearly, specifically, and precisely, and to know what you’re trying to get.

Ibrahim Hasan

And let’s talk about that from the perspective of information governance officers who are dealing with requests from journalists. What are your tips?

Martin Rosenbaum

Information governance professionals, they do deal with journalists’ requests, obviously. And the good ones accept that this is part of a journalist’s job to obtain this information. It might sometimes be embarrassing or annoying or difficult or whatever for their organization, but legally the requirement is there that information has to be made available, and it is the journalist’s job in the public interest to get this out, and so it’s about following the law and complying with the requests and being reasonable and so on. I think good ones are fully aware of that and behave in that kind of way, sometimes meeting resistance from more senior people within their own organizations who aren’t happy about releasing stuff.

I think the other thing to bear in mind is to try and respect journalist deadlines, for example, if a request does need clarifying because it hasn’t been expressed very well. What’s really annoying and frustrating as a journalist is that someone contacts you on the 19th day after you’ve put the request in and says, Hang on a minute, what are you actually asking for here? When it’s perfectly obvious they could have done that after one, two or three days, come back to you and said, Actually, you’ve used the wrong phraseology here, please can you clarify? So making sure that things are done promptly is also very important.

Ibrahim Hasan

And what about communication as a journalist? Would you like somebody to come back to you quickly by email, or would you prefer them to just pick up the phone and talk to you?

Martin Rosenbaum

100% definitely pick up the phone. And one thing that has changed is I think it’s a bit harder now for those conversations to happen than they used to in the first period of FOI. And I don’t know whether that’s because people are more reluctant to be helpful. I don’t know whether it’s because people are under more time pressure. Or, as sometimes said, a younger generation of people in the office are less keen to talk to people on the phone in general, they prefer to send texts and so on, there’s been a generational shift in the nature of communication messaging, whereas older people are happier talking on the phone. Maybe even that plays a part, but whatever it is, it’s much better to have a conversation on the phone where you can discuss, this request I put in, does it actually make sense? Is it correct? Is the phraseology right? But also what is likely to be feasible within the time limit, how are the record systems structured and so on, so that a particular request can actually be processed feasibly.

And then you avoid the situation of people putting in a lot of requests which go over the cost limit, or end up with people searching record systems which it’s futile to search, or the records that are wanted aren’t really in those record systems. A lot of that pointless activity, which actually wastes time, both of the information governance professionals, also of the journalists, everybody involved, a lot of that wasted time could be avoided through phone conversations. And at the end of the phone conversation, you probably would say, okay, what we’ve agreed is such and such. We put that in writing because an FOI request has to be in writing. But it’s that phone conversation which facilitates a much more productive relationship, in my view.

Ibrahim Hasan

And I think having those relationships mean that sometimes it may be the case that journalists, rather than making a formal FOI request, just pick up the phone and ask for information. And so that reduces the burden for the FOI officers.

Martin Rosenbaum

Absolutely, yes. There’s lots of information which can be given out very easily without the need for a formal FOI request and the administrative and bureaucratic processes that they end up going through. I would always encourage journalists to try and have good relationships with FOI officers, and it works both ways from the journalistic point of view. Sometimes people would suggest to you actually, you could put in a request for such and such, which you hadn’t thought of and might be a very interesting request. But also sometimes I would be talking to FOI officers who would persuade me, and I think they were sincere, actually putting in a request for such and such, you might think is a good idea. It’s a waste of time because blah, blah. And it would save time.

Just to give one sort of particular organisational example of this, in the first few years of FOI, the Association of Chief Police Officers, as it was then, had this unit, this central FOI unit, who were actually really helpful in terms of you could phone them up and say, We’re thinking of doing this, and they would say, Well, the piece of jargon for that is such and such, or police forces don’t actually hold that information, so it’s a waste of time, but they might hold this information instead. Or they might say sometimes we would tell people not to answer that request, and we might say, Well, we’re going to do it anyway, and they’d say, Fine. But it was a perfectly straightforward discussion, which I think at the end of the day saved a lot of people time and aggravation. And that has all gone now in terms of the police, that kind of stuff doesn’t exist anymore. But where it does exist in public authorities, I think it’s really useful.

Ibrahim Hasan

I agree that picking up the phone is a good option. Now, just going back to your experience, Martin, you’ve taken cases all the way to the tribunal. What’s that experience taught you?

Martin Rosenbaum

I’ve taken quite a few cases to the tribunal, and I’ve won some and I’ve lost some, but I have to say that on the whole, I’ve been impressed by how the tribunal has dealt with cases, and where I have lost, I haven’t felt I’ve been treated unfairly in any way, and obviously where I’ve won, I don’t think that.

But I think one of the things that whole process has taught me is actually how with the FOI system, there’s so much value in persistence, there’s so much value in pursuing things that you don’t get at the first stage, and how often you get more information at subsequent stages. This is first of all at the internal review stage, often you get information you didn’t get at the first stage. Then when you go to the ICO, and I’ve talked about cases where the ICO has ruled in my favour. But even when the ICO doesn’t, taking a case to the tribunal, you’ve got a reasonable chance if you’ve got a good case of getting information at the tribunal stage which you didn’t get before. At all these stages, there’s a reasonable chance of getting information you didn’t get initially.

Partly, I think, because people have imposed exemptions in too blanket a way at earlier stages. And the more stages you go through, often the more granular the detail in which the information is assessed as to what should be released and what shouldn’t be released. There’s more scrutiny at a later level.

So I think the fact that we can take cases to a tribunal in this way is a very good thing. And I’ve done these representing myself. I’m not a lawyer, but obviously I do have good knowledge of FOI law and some experience now of dealing with cases at the tribunal level. And I think it’s perfectly feasible for individuals to represent themselves and to present a good case to the tribunal and sometimes win. And it’s a good feature of the FOI system.

Ibrahim Hasan

Can you give us some examples of victories at the tribunal that you’re particularly proud of?

Martin Rosenbaum

One which was quite recent, and this was since I left the BBC, was the nomination that was given by Boris Johnson for two peers, Charlotte Owen and Ross Kempsell. And I wanted to see the citation, the reasons that were given for why these people had been made peers in the House of Lords, where they actually have the right to vote on legislation, they have political power, they’re voting on laws that all of us then have to obey. What were the official reasons that were provided by Boris Johnson as to why they should be in the House of Lords?

Ibrahim Hasan

Yes, Charlotte Owen was a particularly controversial one, wasn’t it? Because she was so young and it seemed that she had very little experience.

Martin Rosenbaum

Absolutely, so there’s nothing really in her CV that pointed to the level of seniority or authority or experience which would suggest that she is someone who should be in the House of Lords. This initial request was turned down by the House of Lords Appointments Commission. It was turned down by the ICO who went against me. But the tribunal ruled that this information should be released in the public interest, which I think is absolutely the right decision. And the reasons that were given for why she should be in the House of Lords were really very, very thin indeed, and didn’t amount to, in my view, justification as to why she should be someone who is making laws for the rest of us. So that took a long battle, but it seemed to me absolutely that the public should know why people are being made into legislators, and the tribunal took the right decision in the end.

Ibrahim Hasan

So it’s been around for over two decades now, Martin. Now, with open data dashboards, AI-driven systems, what do you see as the new challenges or opportunities when it comes to FOI?

Martin Rosenbaum

I’m sure AI, in the same way it’s going to affect everything, will affect the world of FOI. I know FOI officers talk about how they’re already reading FOI requests that have clearly been written by AI, and we’re seeing some evidence of an increase in numbers of FOI requests.

Ibrahim Hasan

I read the post on LinkedIn from the Scottish Information Commissioner. He’d seen a massive increase in appeals and requests as well. And partly he was saying it was as a result of ChatGPT and other generative AI solutions making it much easier for people.

Martin Rosenbaum

Yeah, and I did a survey of various other complaints bodies, ombudsmen, say, the financial services ombudsman and the housing ombudsman and things like that, so outside FOI, but also complaints bodies. And amongst them, over the past few months or so or a year, there’s evidence of increasing numbers of complaints, and it makes absolute sense. People enter into ChatGPT, how can I complain about this? And ChatGPT says, this is what you do, and would you like me to write your complaint for you? It makes everything so simple. So it’s undoubtedly leading to more complaints, more FOI requests, and so on, which will I’m sure for some public authorities create an issue.

But also public authorities themselves are talking about and thinking about how can they use AI in order to try and process all sorts of incoming questions, including FOI requests. How can they get more productivity out of AI systems and use them more efficiently than their current systems? So there’s AI on both sides of the fence there. How that will work out in due course obviously remains to be seen. We don’t yet know. I think it’s very difficult to predict, but clearly AI will have quite a considerable impact on both sides of the FOI operation for sure.

Ibrahim Hasan

And where do you see FOI going forward in terms of how it develops? What change would you, as a journalist, want to see?

Martin Rosenbaum

One thing I would very much like to see is that FOI is extended to private companies who are delivering public services. Because obviously, what we’ve seen over the past 20 years or so is a big programme of outsourcing where a lot of public services, all sorts of different kinds, at the local level particularly, are delivered by private companies. Those private companies are outside FOI, and the government has indicated a loose kind of way that it’s planning to bring them within FOI.

Ibrahim Hasan

Obviously, when they couldn’t do anything about it, they used to make a big play about housing associations and PFI contractors, but we’ve not heard anything in terms of actual legislation.

Martin Rosenbaum

Exactly correct, no time scale has been provided for this, it’s certainly not an immediate priority. Whether it will happen or not, therefore, I don’t know. On housing associations, actually, this is something that the Conservatives were doing and Labour has continued with, they’ve introduced a kind of partial information rights law which gives residents of housing associations rights to information about the functioning of the housing association. It’s not an FOI law in the sense that anyone would have that kind of information. But you’re right, absolutely right. This is something that they’ve talked about for a long time. We don’t know when it will happen. We don’t really know if it will happen, but I do think it would be a very important reform if it did.

Ibrahim Hasan

And perhaps a time limit on when a public authority wants clarification, so that as you said earlier, they don’t wait 19 days.

Martin Rosenbaum

Absolutely. What the law actually says is that people should deal with FOI requests promptly, and the 20 working days is a kind of backstop. So prompt is a difficult word to enforce, but I think public authorities should comply with the law and be prompt rather than leaving everything to the 20 working days.

Ibrahim Hasan

So finally, Martin, I’ve had a look at your book, Freedom of Information: a practical guidebook, an excellent piece of work. What inspired you to write the book, Martin?

Martin Rosenbaum

What really inspired me was I’d spent over 15 years at the BBC, as the leading specialist in BBC News, in putting in FOI requests, training other journalists in how to do it, advising journalists from other programmes who would come to me and talk about what they were doing, and so on. And I felt, first of all, that I had a great deal of experience and I’d learned a lot during that 15 years. And what I really wanted to do was actually put that down on paper in order to help others so that I could communicate the benefits of what I’d learnt. And also because of the way in which people had come to me and asked for assistance, I thought I had a pretty good idea of some of the pitfalls that other people had been falling into, what they needed to know to avoid falling into those pitfalls and to ask better requests in future. And overall, really what I wanted to do, I wanted to write the book, which, if it had I’d been able to read it when I had first started out, would have saved me from so many problems, so many things that I myself got wrong in that kind of learning process, because it encapsulated all the experience of learning how to do things better.

Subscribe now

Share

You can buy Freedom of Information: A practical guidebook here.

The number of FOI applications to departments reached 46,545 in 2025, up from 38,536 in the previous year. This included particularly large increases for the Home Office and the Department of Work and Pensions.

This chart shows the figures for FOI requests received by UK government departments since the law came into force in 2005.1

Share

Subscribe now

Clearly there are many reasons for why people submit FOI requests, but I belong to the camp who think that one cause of the jump is probably the use of AI. When people seek advice on their problems with parts of the state, chatbots may suggest making FOI applications and indeed offer to draft them to make the process easier.

I’ve previously analysed how numerous national complaints bodies are now receiving a greater quantity of cases.

In this way AI is helping some people to discover their legal rights and also assisting those who are less confident or knowledgeable about how to assert them.

However one problem for FOI officers on the receiving end is that AI-generated requests are often not well targeted or phrased. Although their quality has improved, nevertheless as with much AI writing they tend to be verbose and typically may contain multiple overlapping questions, asking for similar material in slightly different ways.2

While it’s possible to write prompts to try to avoid this, the people who would be good at constructing those prompts are probably not the same as those who are only making FOI requests now because AI has shown them how to do so.

Note that these statistics only cover UK central government and so these requests only constitute a small proportion of the FOI applications submitted across the public sector.3

Ongoing research by Ben Worthy of Birkbeck College, University of London, into local government and FOI is producing findings that are consistent with this notion of some increase in requesting linked to use of AI.

Share

These figures have come out in the wake of reports that the government is currently considering measures to curtail FOI. An article in the Financial Times last month suggested there are discussions about reducing the cost ceiling, so that more requests would be rejected as too expensive.

Another piece described fears within government that FOI is being used by agents of hostile states and China in particular to obtain unclassified details which could be pieced together to reveal more sensitive defence and security information.

In response journalists and campaigners have pointed out that cutting the maximum cost probably won’t save much money, that public bodies could make FOI savings in any case by actually being more open and less obstructive, and that worries about ‘jigsaw’ disclosures are already catered for under existing law. A useful piece by Julia Cushion of mySociety, which runs the whatdotheyknow website, has collated some good articles, including ones by George Greenwood of the Times, Jenna Corderoy of Democracy for Sale, and Claire Miller, as well as Alex Parsons of mySociety.

I’d also draw attention to this insightful piece of analysis by Ben Worthy.

Any attempt by government to restrict FOI will be strongly resisted by the media. Whether or not more does come of this remains to be seen

In my view FOI expenditure should be money well spent, because of the increased transparency and accountability of the public sector, as well as important specific benefits that follow from particular disclosures.

FOI involves some cost to the public sector, but it can also save large sums by deterring wasteful and unnecessary use of resources.

For some public authorities I have dealt with, it is clear to me they could save money by being more efficient and less obstructive.

And in this context there is some other very interesting data in the latest government statistics. The proportion of internal reviews which partly or fully overturn initial withholding of information has risen again and now stands at 28%.

This is why on my FOI training courses I always advise that if you have a decent case then it is worth asking for an internal review.

Subscribe now

I believe there’s a problem of incentives. In some public bodies, FOI officers feel they could get in trouble for releasing information, but they don’t get much blame for withholding material in a way that is later overturned. So if in doubt, they withhold. This results in unjustified rejections, many more internal reviews and much more time of senior staff being wasted on carrying them out - and it’s a task which those staff often resent. Authorities that want to be more efficient need to change this pattern of incentives for front-line FOI officers.

Complaints to the ICO and tribunal cases can eat up even more time of senior staff and legal advisors. Authorities which find themselves going through these processes because they have been unhelpful at the first stage may end up wasting a lot of public money that could be better spent elsewhere.

One fairly recent example (as revealed by Jenna Corderoy) is the Cabinet Office spending £15,000 on legal fees in an unsuccessful attempt to stop me discovering reasons Boris Johnson provided to the House of Lords Appointments Commission to justify peerage nominations.

Subscribe now

Subscribe now

Government files just released by the National Archives shed new light on how ministers and officials in Blair’s administration coped with the new public right to make requests for government documents, with their reactions varying from compliance through unease to outright hostility.

This particular memo was written in February 2005 by Jonathan Powell, who was then Blair’s chief of staff and today is still at the heart of power, as the current government’s national security adviser.

He told Blair: “The system is beginning to seize up - Private office, duty clerks etc - costing us nearly £20,000 a week. It is the same across government, and we need to do something.”

The prime minister replied: “What is the solution?! I need a recommendation”.

Share

The government clearly found some of the initial rush of requests to be uncomfortable. In the following year they proposed new regulations to make it easier to reject FOI applications, but these plans met a lot of opposition and were abandoned.

Several years later, after he left office, Blair complained in his memoirs that civil servants had failed to warn him of ‘the full enormity of the blunder’ that in his view FOI proved to be. Powell also was later publicly critical of FOI, writing that ‘policymaking, like producing sausages, is not something that should be carried out in public’.

These files, which have been published under the 20 years rule for releasing old government records, also show how civil servants dealt with some of their first FOI cases.

They reveal how officials who did not want to release certain material relating to the Iraq war discussed which minister to approach for sign-off. They decided to ask the Foreign Secretary Jack Straw as they expected he would agree with their planned refusal, as opposed to the Cabinet Office minister David Miliband who they thought would not go along with this. An official added that if Straw wouldn’t comply, ‘we’ll have to work on Miliband’. Straw did.

Other examples provide evidence of reluctant information releases which were enforced by the FOI law, as in this memo which an official sent to Blair.

I was particularly interested to find discussion of how to respond to one of my early FOI requests - for correspondence between Blair and the Swedish prime minister, Goran Persson. I’d already obtained some of this from the Swedish government, which had a greater culture of openness, and wanted to see what Downing Street would supply in comparison.

Blair’s private secretary for foreign affairs, Antony Phillipson, recommended giving me the material, which he said contained nothing contentious. But he seems to have been overruled by the chief of staff, Jonathan Powell, who was worried it would set a precedent. Powell wrote that he was opposed to releasing any correspondence with foreign leaders.

So this information has now finally been released by the UK government over 20 years after the Swedish government disclosed it.

It includes the letter Blair sent to Persson to say ‘thank you for Sven’. This was in the wake of the England football team beating Germany 5-1 in the World Cup qualifiers in 2001, an early triumph for the side’s new Swedish manager, Sven-Goran Eriksson.

I was once told by the then Information Commissioner Richard Thomas that Downing Street staff had informed him that they couldn’t send me this amusing but harmless brief handwritten note, which I had written about for the BBC after I obtained it in Sweden, as they hadn’t kept a copy of it. Yet funnily enough, here it is in their files.

Subscribe now

The released documents also show that many ministers were particularly exercised about FOI requests for their diary information, such as who they had held meetings with.

Policy on this was discussed extensively in correspondence between departments. The topic also featured prominently in cabinet meetings during the early months of FOI, as is disclosed in the 2005 cabinet minutes which have also just been released.

The cabinet minutes also give a broader view of how the new FOI system was regarded within ministerial circles, often with concern rather than pride.

It is interesting to note that the cabinet minister responsible for FOI, the Constitutional Affairs Secretary Lord Falconer, was keen to get departments to publish FOI disclosures on their websites at the same time as giving the information to the requester. This was particularly in the case of the media.

Falconer did not appear to be happy generally with the media’s use of FOI, as he told the cabinet on another occasion.

And finally, just before FOI came into force, this is how Tony Blair summarised matters to the cabinet, several years before he announced he was a ‘nincompoop’ for introducing the law.

And one more thing. I wonder what Keir Starmer was saying back in 2005, in a legal opinion commissioned by the Guardian, on whether certain government legal advice should be disclosed.

Share

Subscribe now

Analysis of last month’s Caerphilly by-election for the Welsh Senedd suggests it was a significant factor in Plaid Cymru’s dramatic victory, with supporters of other left-wing parties voting for the Plaid candidate to ensure that Reform was defeated, as polls indicated only these two parties had a chance of winning.

And this month a YouGov poll covered in the Times reported that around half of LibDem and Green supporters would be willing to back Labour to stop Reform taking their constituency.

But to make a much bigger impact, left-wing voters would also have to be ready to vote for the Tories to keep Reform out - and the YouGov poll suggested that a significant minority would indeed do this: the net figures given were 30% of LibDems, 28% of Labour supporters, and 18% of Greens.

Discussion of tactical voting can be misleading. Most voters don’t really have a carefully ordered, fixed hierarchy of party preference in the way that some political punditry assumes. But the key point is that substantial numbers would appear more determined to block a party they really dislike rather than elect a party on which they are particularly keen.

According to the leading psephologist Peter Kellner, tactical voting along these lines in a general election could deny Reform 100 seats in England. Of these, 52 seats would instead go to the Tories, 44 to Labour, and 4 to the LibDems. A scenario like this would dramatically reduce Reform’s chance of winning an overall majority.

Share

Given this background, it’s interesting to examine a by-election last week in a Scottish local government area, Stranraer and the Rhins. The ward is part of Dumfries and Galloway Council, and incidentally happens to contain Scotland’s most southerly point, the Mull of Galloway.

The conditions in this by-election turned out to constitute a kind of natural experiment about the extent of anti-Reform voting.

This is a location of traditional Conservative strength, and the by-election was narrowly won by the Tory candidate, just beating the new challenge from Reform.

It’s revealing because Scottish council elections are held under the single transferable vote (STV) system, where voters list candidates in order of preference.

At each stage of counting the votes, the candidate with the smallest number of preferences is eliminated, and the next preferences indicated by their voters are then allocated accordingly among the remaining candidates, before the next elimination.

This means we can gather information about the second or lower preferences of voters - and actual voters taking part in a real election, rather than respondents to a poll who are making predictions about how they might vote in hypothetical circumstances.

Whatever its merits as an electoral system, STV is certainly a boon for political analysts.

Subscribe now

In this by-election the Reform candidate John Roberts was ahead at every stage of the count until the concluding one, when he was finally overtaken by the Conservative candidate, Julie Currie, who won by 42 votes, with 1565 to Reform’s 1523.

At the first stage Roberts had been leading with 1386 first preferences compared to Currie’s 1302. Then during the count Currie gradually caught up a bit at nearly every stage due to allocations of preferences as other candidates were eliminated, eventually nudging ahead to win thanks to the lower preferences passed on from the third-placed SNP candidate.

As the LibDem, Green, independent, Labour and SNP candidates were all knocked out, and lower preferences were transferred, the cumulative effect was that Currie obtained an additional 263 votes for the Tories, while Roberts only put on 137 more for Reform.

In other words 66% of these lower preferences transferred to the Conservatives and only 34% to Reform. This pattern was broadly consistent for each party eliminated during the count. And it was enough to ensure - narrowly and only at the last moment - that Reform lost and the Tories won.

This might seem to confirm the potential for such anti-Reform tactical voting for Conservative candidates by some on the left. But there are also some powerful pointers in the other direction.

Firstly it’s notable that about one in three of these voters actually preferred the Reform candidate to the Conservative, so the pattern of preference is far from universal. Nor is this at all surprising, given that Reform has been picking up support from across the political spectrum.

But perhaps more importantly, there were 852 supporters of these other candidates who in the end did not express any preference between Conservative and Reform. Thus overall, in what was clearly a two-horse race between the Tories and Reform, just 21% of the other voters preferred the Tories, and 11% Reform, so that the eventual net benefit to the Tories consisted of 10% of these others.

And this in a contest where due to the electoral system voters could still happily put their top preference first, while preserving the political impact of their lower preferences, without any need for the reluctant compromises of tactical voting.

This pattern is therefore significantly weaker than the picture which emerged from the YouGov poll. It could suggest that 10% is a reasonable upper limit for the proportion of left party supporters who might vote Tory to block Reform in seats where they are the two parties in contention.

On the other hand, 10% (or any figure close to it) is certainly not nothing. In a general election any degree of tactical voting by leftists for Tories to that sort of extent would be enough to affect some Tory versus Reform battleground seats and possibly the overall national position.

There are lots of caveats about all of this, naturally. While this was an actual election, it only consists of one local ward with about 4,000 voters taking part, in Scotland, under a different voting system, and could be completely unrepresentative. Not least, there may of course be all sorts of specific factors to do with the individual candidates and the local campaign, of which I am unaware.

And, in terms of the incentives to vote tactically, it must be said that even for many voters who are very uneasy about Reform, the prospect of a Reform candidate being elected to Dumfries and Galloway Council is not quite the same as the prospect of a Farage government.

Subscribe now

Share

As for their pastimes, Greens seem to enjoy going for country walks and Tories doing the gardening, while Reform voters do not like watching BBC news.

And in terms of shopping, Morrisons appears to be the large supermarket chain with the most politically representative clientele.

I have derived these little nuggets of political analysis from a new dataset published last month by the British Election Study (BES).

Share

The BES is a long-running academic research project involving extensive surveys of voting patterns, political attitudes, and demographic and social characteristics.

In the past week or so I have read three substantial pieces of political analysis which rely on the latest BES samples, by Ben Ansell of Oxford University, James Breckwoldt of Manchester University, and Dylan Difford of YouGov.

They are revealing and thought-provoking articles about parties and social class, the traits and attitudes of Reform supporters, and how public opinion has shifted in the past year, and I recommend them all.

However it is also possible to find amongst the riches of the BES datasets some minor gems, which are perhaps less politically significant but are nevertheless intriguing in their own way. So I’ve been looking at the most recent BES sample (wave 30).

When I refer here to voters for a particular party, that is based on how survey participants (sample size of over 6,000) say they voted in the May 2025 local elections in England, which saw major advances for Reform.

Subscribe now

Personality

If different humans have different personal dispositions towards, for example, behaviours like self-reliance, trust, flexibility, boldness, anxiety, diligence, and so on, then (whatever the difficulties in measuring these) it is not surprising if features of this kind have some influence on political attitudes, as one factor among many.

The latest BES survey contains scores for aspects of the personality of respondents, tied to the so-called ‘Big Five’ psychological traits: agreeableness, conscientiousness, extraversion, neuroticism and openness.1

These ratings are based on the participants’ answers to a battery of questions aimed at identifying those broad characteristics.

Not everyone agrees this is a valid model of human nature or that self-completion is a reliable method of assessment, but it is dominant in the field of personality psychology and widely used in commercial and academic research.

This table shows the average score for each party’s voters on a scale for each of these personality dimensions. The maximum score in each case was 20.

The variation shown here isn’t massive, and the sample sizes for the party breakdowns aren’t very large. But the overall pattern is consistent and fits well with previous research on the same subject: right-wing voters tend to be more conscientious and extravert, left-wing voters tend to be more agreeable, open and neurotic.

What’s different here is that in the past in Britain this was mainly a Conservative vs Labour contrast. The rise of other parties means that we can now see these tendencies particularly sharply in Reform and Green supporters, who are likely to be especially right-wing and left-wing respectively and appear to have the personality characteristics to go with it.

So Green voters seem to have the highest average ratings for agreeableness, openness and neuroticism, and the lowest ones for conscientiousness and extraversion.

Reform voters tend to score the lowest on agreeableness and openness. It’s the Conservatives who register the highest scores on conscientiousness and extraversion, and the lowest on neuroticism, while Reform voters are second in each case.

The average personality ratings for Labour and LibDem voters are very similar to each other and middling.

Bear in mind that all these numbers here are averages - there is a lot of scope for individual variation.

Activities

The BES has also asked survey respondents a collection of questions about what leisure activities they participate in.2

The results suggest that Green voters are the most active in all sorts of ways, which may reflect their younger age: as well as country walks, this includes attending gigs and going to the gym, and perhaps more surprisingly even eating at McDonalds.

Reform voters seem to be much less active generally, at least in the sort of pastimes that social scientists like to ask about. However Conservatives apparently enjoy visiting stately homes as well as gardening.

But in my opinion the most interesting and revealing comparison between the activities of different party supporters lies in who watches BBC news.

This shows how Reform voters are much less likely to be getting their news from the BBC.

Only 44% of Reform voters said they had watched BBC news in the past 12 months, compared to 61% of Greens, 65% of Tories, 69% of Labour voters and 73% of LibDems.

Supermarkets

There’s also a BES question about which supermarket respondents use for most of their food shopping,3 and there are some political differences in the answers.

So 30% of Reform voters say Tesco, compared to only 17% of Greens. However some shopping preferences are shared by these two groups: 21% of Greens and 19% of Reform voters do most of their shopping at Aldi, compared to only 11% of LibDems.

The most politically balanced large supermarket chain seems to be Morrisons. This is favoured for most of their food shopping by about 6-7% of each party’s voters.

I don’t know if the chance to chat with a good range of fellow customers counts as another reason to shop at Morrisons, but if so it’s another useful by-product of academic research.4

Subscribe now

Share

Direct research evidence is limited, but one indicator could be that (according to data I’ve obtained) the UK’s main complaints services have seen a recent upsurge in the numbers contacting them.

This mirrors the latest official freedom of information statistics which show a significant increase in the level of FOI requests to central government, another civic right.

AI chatbots may be lowering the barriers for ordinary citizens to pursue their grievances over how they’ve been treated by a public authority or commercial company.

On the other hand AI systems are not always used sensibly, while the greater volumes are increasing the demands on those at the receiving end.

Share

I made FOI requests to establish the monthly figures for new cases over the past three years from a dozen authorities which investigate complaints from members of the public, such as the Financial Ombudsman Service and the Local Government and Social Care Ombudsman.

The charts below show the three-year pattern in the level of complaints they each received per month. (For further explanation see the notes at the end.) There are lots of ups and downs in each case, but a broad overall trend seems clear.

Subscribe now

While none of these bodies seems to have conducted a formal analysis of the role of AI in the increases they are facing, anecdotally within the sector there is talk of the rapidly growing use of AI tools as being a major contributory factor.

The same is true within the FOI community, where some practitioners believe AI is boosting requests.

The next chart is taken from the latest government FOI statistics published last week, and shows what has happened to numbers of FOI requests to central government bodies over the past three years.

Share

The evidence of AI playing a role in all this is circumstantial and anecdotal, but in my view it would be surprising if it wasn’t true. If you make something easier for people, and tell them how to do it, they will do it more often. A little experimentation confirms that popular chatbots will explain how best to raise complaints about failings in services and which organisation to direct them to, and conveniently offer to supply a draft or template for use.

While I’m not aware of any similar research in the UK, one American study last year reported that 18 percent of complaints to the US Consumer Financial Protection Bureau had evidence of text generated by an AI large language model.

Nevertheless there could also be other possible causes involved, such as perhaps a more general willingness to complain or more to complain about, as well as the specific factors that would significantly affect individual services.

It seems likely that younger and more tech-savvy citizens would be the earlier adopters. But the US experience suggests that possibly the use of AI in this way is also particularly helpful to people who are less proficient in English.

However the lazy or careless use of AI can hinder rather than help complainants. Last week the Information Commissioner’s Office rather sarcastically rebuked a requester for an internal review application which contained inaccurate statements. The ICO suggested the requester had relied on AI and added: “We advise that any AI generated material is verified before including it in your responses.” (This was spotted by the FOI trainer Tim Turner).

Collating these raw numbers leaves open the question of whether AI-assisted complaints are more or less likely to be effective.

But either way the increased number of complaints may present difficult challenges for these organisations, who will have to cope with the extra workload and prevent the build up of backlogs, possibly without the benefit of extra resources.

In due course AI tools may also help them process complaints more efficiently, in ways that some have already started to develop. However there are serious doubts about both accuracy and public acceptance for automated decision-making.

Subscribe now

Notes

Financial Ombudsman Service (FOS) handles consumer complaints about financial services.

Housing Ombudsman Service (HOS) deals with residents’ complaints about social housing in England.

Independent Case Examiner (ICE) reviews certain complaints about state benefits and financial support.

Information Commissioner’s Office (ICO) covers data protection and freedom of information complaints. My chart amalgamates the figures for these two aspects of its work.

Local Government and Social Care Ombudsman (LGSCO) investigates complaints about councils and adult social care in England.

Legal Ombudsman (LO) handles complaints about legal services.

Northern Ireland Public Services Ombudsman (NIPSO) investigates complaints about public bodies in Northern Ireland.

Parliamentary and Health Service Ombudsman (PHSO) deals with complaints about UK government departments and the NHS in England.

Pensions Ombudsman (PO) covers pension schemes. The dramatic blip in the chart for mid-2023 is because for a period its online submission form was not available, followed by a rush of complaints when it returned.

Public Services Ombudsman for Wales (PSOW) handles complaints about Welsh public services.

Scottish Information Commissioner (SIC) covers Scottish freedom of information cases.

Scottish Public Services Ombudsman (SPSO) deals with complaints about Scottish public services. And on a side note, they responded to the FOI request which I submitted at 5pm one day by sending all the requested information at 1030am the next morning - so well done to them for the efficiency of their FOI operation.

My full dataset can be downloaded from here.

Share

Subscribe now

That perspective goes back about 20 years or so, because I've been making FOI requests since the law came into force in 2005, primarily as a journalist. Before that I made requests under the previous system, the Code of Practice on Access to Government Information, a much weaker system than the FOI law we have today.

During my time at the BBC, I specialised in using FOI, both submitting requests myself and advising and training other journalists on how to use the law effectively. I left the BBC around four years ago, and since then I’ve written a practical guidebook for FOI requesters.

So over the past two decades I’ve seen a lot of FOI in practice, and when people ask me for an overall verdict on FOI - is it working well, or not? - I have sometimes described it as a ‘tale of two giraffes’.

Subscribe now

Of the two giraffes photographed in this slide, one is in London Zoo, and the other is in Whipsnade Zoo in Central Bedfordshire. Three years ago I was doing some research into zoos, and I asked for the latest inspection reports on these two zoos, which should be a very simple FOI request.

Central Bedfordshire Council sent me the latest inspection report on Whipsnade within an hour. It was absolutely the way FOI should work.

Westminster Council, the council responsible for London Zoo in Regent’s Park, didn't reply. After 20 working days. I sent them a chasing email. They didn't reply to that. I had to complain to the ICO, and the ICO - thank you very much - did the usual thing of sending an email saying you must reply within 10 working days, and eventually five months after my request I got the report from Westminster.

So what one public authority had been able to do within less than an hour, it took another public authority over five months and the intervention of the regulator.

In other words, sometimes FOI works very well in practice, sometimes it works very badly. And obviously there's also a lot of circumstances in between those two extremes. But the key point is that the experience of FOI as a requester is that it's very varied. There’s both a positive side and a negative side.

Share

The positive

Taking the positive side first, and looking back over the last 20 years, one of the very good things is the way that FOI has become well established as a tool that a lot of people use successfully.

This is a chart which I've compiled from government statistics. I don't think anybody else, by the way, is analysing these annual statistics in the same way that I do, keeping track of them over every year going back to 2005. This shows how in terms of central government departments, for the first few years the number of requests increased quite steadily until about 2013. Since then the numbers have been fairly steady over the past 10 to 12 years or so, maybe 35,000 FOI requests annually to government departments.

As you know we've only got statistics for central government bodies. So this is only a very small sample of FOI requests. Yet even so it's one way of showing that FOI has become well established.

However, that fact obviously hasn’t gone down well with some people. For example, Tony Blair, who said that introducing FOI was like giving people a mallet to hit him over the head with, and that it was one of his two biggest mistakes. He famously wrote these words in his memoirs about the introduction of FOI by his own government: “Freedom of Information. Three harmless words … You idiot. You naïve, foolish, irresponsible nincompoop … I quake at the imbecility of it.”

He's not the only prime minister who disliked FOI. David Cameron complained about it. Theresa May was more measured in her memoirs, but also had criticisms. And Liz Truss said in her memoirs: “Constant questions about who you've met and why … innocent meetings are privy to the press and often deliberately misconstrued.” She does, however, from her point of view, get the chance to blame Tony Blair for it.

What I think about these remarks is that actually they show in a way that FOI is working, at least in some cases. FOI should sometimes be uncomfortable to people who hold positions of power. That's part of the point of it. If people in power were always very happy about the way that FOI worked, then it wouldn't really be doing its job. So these quotes from former prime ministers don't put me off FOI, to my mind they strengthen the case for it.

It's really quite striking how FOI has reached the level of use and acceptance where it would be pretty politically impossible for a government to seek to weaken the law significantly.

We can see what happened when sometimes they have tried to take certain steps in that direction. For example, the Cameron government set up a review, a Commission on Freedom of Information, which included people like Jack Straw and Michael Howard, former politicians who were thought to be very sceptical about FOI. I think the government at the time was probably hoping that this review would report and pave the way for some restrictions on FOI to be introduced.

But actually when the review looked at how FOI was working in practice (and I've got to say full credit to people like Jack Straw, who I know from my own conversations with him had had doubts about FOI), it concluded that the law was generally working well, that it enhanced openness and transparency, and there was no evidence that it needed to be altered. It came down against the suggestion that was around some time ago of introducing charges for FOI requests. In fact, the review even said that in some cases the right of access should be increased.

FOI has become well established, it’s difficult to change in principle, and it works well in practice for many people.

Subscribe now

So what is the point of FOI? It's not to hit Tony Blair on the head with a mallet. As far as I'm concerned, primarily as a journalistic user, its purposes are to find out the truth, to hold power to account, to expose wrongdoing, and to give the public useful information.

There are journalists who've made a lot of use of FOI, and others who do so more occasionally, who are trying to achieve these kinds of aims. If we look at a recent list from a website that aggregates news stories, which I searched for freedom of information stories, the selection here is about London’s dirtiest streets, Thames Water (that must be an environmental information request), banks and fraud reports, stories about universities, others about councils, a wide range of themes.

When I was at the BBC we did go to some lengths at some times to collate together on the website a lot of FOI stories that had been done across the organisation, and again what this illustrates is the range of different issues covered, a lot of them very local - health stories, stories about a particular NHS trust, often very much of use to people in the local community, police stories, home affairs, politics stories about what councils are doing, transport stories. One could go on, but it shows the tremendous wealth of topics that FOI enables journalists to find information about in the public interest.

So to give some context about journalistic use of FOI, what do journalists get from FOI requests?

First of all, FOI might directly get the news story - the FOI disclosure in itself is the story, the headline.

But that's not always the case. Sometimes for us FOI is a piece in a jigsaw, it's an element in the story. That’s sometimes because you have a case study and FOI gives you overall statistics or context. Sometimes it's the reverse, you might have some overall context already, but with the use of FOI you can get information about particular examples that help to fill out the story.

Sometimes FOI obtains background information. It doesn’t end up getting reported directly, but it points you in certain directions. It enables you to decide what you're going to research, it informs questions to put to people in interviews, it helps with how to find out stuff from contacts, and so on.

And of course sometimes you get nothing useful whatsoever. All of us who’ve put in a number of FOI requests have sometimes found that the outcome was nothing of any use.

To give some examples of all of this, here is a story I did some time ago when I was at the BBC about Prince Charles, as he was then, urging Tony Blair to meet campaigners against genetically modified food. Charles doesn't like GM food and he was lobbying Blair as prime minister about the issue. This fits into the example of the disclosure as the news story.

I have to say on this example that it took me two years to get this information out of the Cabinet Office, and it took four rulings from the ICO at different points in that process. I’m grateful to the ICO for that, but it also shows the kind of attitude sometimes of the Cabinet Office. For example, first of all neither confirming nor denying that they held the information, then trying to insist it was not environmental information, and so on.

To give a different kind of example, here’s a story where the BBC had some case studies about people who were scammed into paying money to get COVID vaccines. It was the kind of thing that a lot of scammers piled into, trying to get money out of people by sending them bogus messages. But is it a big problem? Well, FOI enabled the BBC to get some kind of national overall context and say there's over a thousand reports, people have lost nearly £400,000 to scammers through this scam, so it is a significant thing. That's an example where FOI is a piece in the jigsaw, you've got the case studies and then you've also got overall statistics.

Next is a different kind of example of a story I did once, and this is one of my favourite examples of the success of FOI, for reasons which I'll explain.

This is about which makes and models of cars are most likely to fail MOT tests. When I first requested this data from a Department for Transport agency, I was told that this information would not be released to me under the commercial Interests exemption, presumably because it would be damaging to the commercial interests of people who sold cars that didn't do well. Again, I appealed to the ICO, and the ICO again ruled in my favour, forcing the Department for Transport agency to release this information, which I regard very much as in the public interest, as useful to potential car purchasers, knowing which cars are most likely to do well in MOTs and which aren't.

This information is now released annually routinely as open data. In other words, what for 18 months the Department for Transport told me was so confidential it could not be disclosed is now released routinely and proactively. It’s a very good example of the benefits of FOI, transforming information that was previously secret into something which is released as a matter of routine.

Another instance of this is food hygiene reports, where again we had councils refusing to release food hygiene reports, and where again the ICO stepped in and ruled that it's in the public interest for the public to know which food outlets have good food hygiene and which don't. And this has now become information which the Food Standards Agency releases routinely as a matter of course. We have ‘scores on the doors’, so you can see on the outside of a restaurant how it's doing.

That transformation has enabled journalists to do this kind of story - how do restaurant chains compare? To give the quick summary, if you want to go to a place with good food hygiene consistently across the country, when we did this study Nando's had a satisfactory score everywhere. So you can't say you don't get useful information out of this kind of talk.

Another example I want to share illustrates the kind of information I think of as practically useful. This was a graphic I generated based on data from individual hospitals, showing A&E waiting times by hour of day and day of week. Each line represents a different hospital. The pattern is clear: if you turn up at midnight, any day of the week, you're likely to wait much longer than if you arrive around eight o’clock in the morning.

That’s useful information, and yet it wasn’t being published at that level of detail by NHS England, even though it was collecting the data. I had to request it through FOI. Without the law, I wouldn’t have been able to obtain it.

The negative

So that’s some of what I see as the positive potential of FOI, and what I and others have achieved with it, but on the other hand FOI doesn’t always work well. Journalists frequently complain about delays and obstruction. Here’s a headline from the Press Gazette, the news trade website: ‘Freedom of Information in the UK sinks to new low’. This story is from 2024, last year, but you could probably find a version of that story in the Press Gazette every year. Because FOI still all too often doesn’t function as it should.

I first want to make a general point. Part of the problem is a broader shift in official attitudes. In the past there were more people in positions of power, contrary to the views of Tony Blair, stressing the benefits of openness and transparency. In the early years after the law came into force in 2005, there was rhetorical support from government for openness. Lord Falconer, the minister responsible for FOI at the time, attacked the ‘culture of secrecy’ - you don’t hear government ministers now saying that kind of thing.

Then under the Cameron government, the Cabinet Office minister Francis Maude was very keen on open data - which is not the same as FOI, but is part of a wider transparency agenda.

Again we don’t hear that from ministers today, that rhetoric about the benefits of transparency and openness. And this climate does have an impact on how public authorities behave, and in some respects they are more resistant to releasing information.

Here’s an example. Early on in the FOI period I obtained documents showing how the Metropolitan Police Special Branch had monitored the Anti-Apartheid Movement in the UK over 25 years. They were released in 2005. There is absolutely no way the Metropolitan Police would release that information today. In the early years of FOI they were generally more committed to transparency, and it’s an example of how things have deteriorated.

Sometimes I’m accused of being too cynical, so let’s take a look at what the chief executive of the Environment Agency, Philip Duffy, said at a conference last year: “I see these letters and these FOI requests and I've got great volumes of them, and I see local officers going through contorted processes not to answer when they know the answer and it's embarrassing.”

This is a clear statement from the chief executive of the Environment Agency about how his officials behave, trying to avoid releasing embarrassing information. This is obviously a completely unsatisfactory state of affairs. It's not only the Environment Agency, in some ways what we have here is Philip Duffy being more honest about how FOI sometimes operates than other public authorities often are.

Share

Another example: a slide from internal Cabinet Office training on FOI. It says: “If you don’t want to appear in tomorrow’s newspapers, consider carefully what you send out.” That’s basically telling people don't release stuff which is embarrassing, and it’s not what the law says. There’s no FOI exemption in the Act for “Stuff you don't want to appear in tomorrow's newspapers”. That should be a totally irrelevant consideration, yet here we have it appearing in the Cabinet Office guidance to staff on how to handle FOI requests.

I do think journalists in particular often have a problem with how their FOI requests are handled. To give just one example of the sort of processes that come into play, the Financial Conduct Authority guidance to staff on FOI explicitly states that approval for journalists’ requests must be “obtained from press office”. It's very far from being the only body where this is the case, very far indeed, and I see two problems with this. First of all, it inevitably increases delay, because the reply has to be checked by the press office. But secondly, for people who work in the press office, their job is to protect the reputation of the organisation where they're working. That is entirely separate from FOI considerations and they should not be involved in approving the answers to FOI requests.

This is another statistical analysis I've done, of how often internal reviews change initial refusals. Again, as far as I'm aware, I'm the only person who is analysing the statistics issued about central government FOI requests to this extent. This chart shows how often internal reviews overturn, partly or entirely, an initial refusal from a government department that information should be withheld.

It's only because we have statistics for central government that we can do this. We don't know what the position is for all sorts of other public authorities. But what we can see here is that maybe 20 to 25 percent, and perhaps increasing, of internal reviews overturn an initial refusal, partly or entirely.

You can look at this in two ways. One is to say that internal reviews are actually doing their job, scrutinising the initial decision. But I also think that this rate is surprisingly high, suggesting that very often at the initial stage people are refusing FOI requests in ways that they should not be. This isn't even the ICO getting involved. This is just somebody else in the same organisation saying you should have released it, and I think it points towards an initial level of obstruction in some cases.

Now let’s look at how often the ICO overturns decisions made by public authorities. This is an analysis I did based on a dataset going back to 2005 which the ICO released last year. I don't know to what extent this kind of analysis is done internally within the ICO or not.1

Share

What I find really striking about this is how for some of these exemptions, the ICO overturns them so often. Taking the ones at the top, the economy and relations within the UK are not so often used, but commercial interests is a very frequent exemption used by public authorities to turn down requests. Yet of the cases that have been appealed to the ICO, basically in half of them the ICO has overturned it.

And there are other exemptions - future publication, policy formation and so on - where the ICO is very frequently overturning their use. What that really points to is a systemic issue that these exemptions are being overused by public authorities. And what the ICO in my view needs to do is draw attention to that and say there is a widespread problem of the overuse of certain exemptions.

I’m aware that the ICO has issues about resources and its level of casework. So if authorities could be stopped from overusing those exemptions, then that will be a way of reducing the level of casework that the ICO gets, and this is another reason why the ICO should be vocal in flagging the systematic overuse of certain exemptions.

The ICO

As I’m talking to the ICO, I should tell you my view of the ICO. This is what I wrote in my book: “Despite problems of resource constraints and significant delay (and the fact that I sometimes disagree with its decisions)” - we’ll come onto that later - “its existence is one of the strengths of the UK’s FOI system”.

When I’ve done training for journalists, this is one of the points that I make, that it’s a strength of the UK system to have an independent body which can overrule public authorities. That’s not the case in some other countries, and the ICO’s role is very valuable.

Let’s take some aspects of its FOI work in more detail. On delay, I very much welcome what the ICO has done over the past couple of years or so to speed up its consideration of complaints about FOI.

In contrast there was an extreme example back in 2009, when the ICO ruled in my favour on releasing the minutes of a Thatcher cabinet meeting about the Westland crisis, where it took the ICO four years to reach that decision. That’s ridiculous. It’s an extreme case, but when the ICO takes that sort of time, then public authorities have tremendous incentives to boot issues into the long grass, turning down requests for information on the basis that by the time the ICO rules, no one will care about it. A lot of progress has been made in speeding up the ICO’s procedures, and it’s vital that it doesn't deteriorate again.

The fact that the ICO now prioritises certain FOI complaints with significant public interest is also a welcome and valuable step which enables the most important cases to be dealt with most quickly. Maybe the threshold for prioritisation is a bit too high and probably more cases should be prioritised, but in general it's absolutely right to have done this.

I'm also very pleased by the way that the ICO is now taking serious regulatory action against recalcitrant public authorities. For example, the enforcement notices and practice recommendations served on numerous police forces recently and rightly, because the performance of the police in FOI terms has often been very bad, as has also been the case for various government departments, certain NHS bodies and so on.

It's much more effective for the ICO to take this kind of regulatory action, issuing enforcement notices, than the kind of situation that we used to see beforehand.

Take this example that I wrote about the Cabinet Office. You would have this flow of decision notices from the ICO referring to Cabinet Office delays as being unacceptable, unhelpful, extreme, protracted, considerable, unsatisfactory, excessive, prolonged, severe. One decision notice after another would say this, but nothing ever changed by just issuing decision notices where it says in passing that delays are unacceptable.2

In contrast taking proper regulatory action and issuing enforcement notices is the best way to achieve change from the most recalcitrant public authorities, and I'm very glad to see that the ICO has been doing that.

Subscribe now

However, I'm not going to say that everything about the ICO is completely marvellous. I'm pleased about reducing delays, I'm pleased about the prioritisation, I'm pleased about the enforcement action. But I do think there is an issue about the quality of certain ICO decisions, and there are decisions where the ICO does not really address properly the balance of the public interest.

This includes recent cases where I’ve been successful in appealing ICO decisions to the First-tier Tribunal.

One example involves the House of Lords Appointments Commission and the citations provided for some of Boris Johnson’s appointments to the Lords. In my opinion it's overwhelmingly in the public interest that the public knows what are the official reasons given for why people are put in a legislative assembly. The tribunal agreed with me on that point3 that the citations should be public. So the House of Lords Appointments Commission was forced to reveal them.

I was disappointed, however, that I needed to appeal to the tribunal in order to achieve this. This is a case where the ICO should absolutely have said it's in the public interest for this to be out in the open, and people could then assess for themselves the reasons that Boris Johnson gave as to why people should be members of the House of Lords with the right to decide on laws everyone else has to abide by.

That’s one example, and another occurred just last month when the First-tier Tribunal ruled that I should be given information in a Cabinet Office case where the ICO had ruled against me on a public interest test.

The tribunal said: “The tribunal shares the appellant's view that there is a strong public interest in upholding standards of good decision-making, transparency and integrity in public administration … In the tribunal’s assessment, the withheld material is relatively benign and does not contain candid or controversial exchanges. The tribunal finds that the Cabinet Office’s concern over inhibiting future frank advice is overstated and not supported by the content of the material in question.”4

Here we have another example of the ICO in my view not taking a strong enough stance on where the balance of the public interest lies, and then the tribunal has to step in. That means extra work for me as the requester to appeal to the tribunal, but it's also extra work for the ICO, which then ends up trying to defend an untenable position at the tribunal, costing a lot of time and money, as well as being against the public interest.

I know there are other cases where the ICO has taken a position which has annoyed the Cabinet Office. But I think it doesn't do that often enough, and the ICO should be more prepared to stand up to the Cabinet Office on cases like these.

Finally, I wanted to make one other point, and this goes back to what I was saying earlier about the public debate about open government. I'd like to see the ICO now speaking out more about the broad advantages of transparency. We need to be hearing the voice of the ICO more often in public discussion.

To give one specific example, a government policy document issued last year states that it's going to extend the FOI Act to private companies that hold public contracts, and that the government will take this forward “in due course”. So it's publicly committed, but we've no idea when it will actually do this, or even if it will really happen. It's not in the current legislation.

This is the kind of thing that in my view the ICO should be talking about now. It should be stepping into this discussion and saying this is an important and valuable change that is needed, because so many services are contracted out in the way that our public sector now operates.5

I’ve left time for questions. Thank you very much indeed.

Subscribe now

Setting aside the local and national policy consequences, the impact will also be to make our elections harder to forecast and to complicate the task of the polling companies.

Yesterday I attended a meeting organised by the British Polling Council to discuss ways of tackling the industry’s failings at the last general election, when the polls significantly over-predicted how well Labour would do.

The next general election is likely to present further difficult challenges for the pollsters, particularly for MRP polls which proliferated in 2024. I think this could have important political ramifications.

The MRPs apply statistical modelling to survey data in order to produce individual constituency forecasts based on the local demography, and thus predict how many seats each party will win. Despite the headline on this piece, it stands for Multilevel Regression and Post-stratification, rather than Maybe Right, Perhaps.

Share

Last year the MRPs made a positive contribution to understanding the pattern of public opinion by correctly showing that Labour and the LibDems would benefit from different swings in different seats, rather than the traditional norm of roughly uniform swing across the country. They were therefore a very useful expansion of polling techniques.

However the MRP polls all exaggerated the level of Labour success, as I have previously analysed. This systematic error across the industry stemmed largely from the voting intention polling figures which were fed into the statistical models.

If, as seems probable, the current electoral fragmentation continues until the next general election, then predicting constituency winners will surely get harder, for the following reasons.

• There will be many more seats where more than two parties have a realistic chance of coming top.

• Winning margins will be narrower.

• Pollsters will have to try to identify demographic characteristics of voters across a greater range of political attitudes.

All this will make forecasts more sensitive to problems with unrepresentative survey samples and any flawed assumptions or procedures in the statistical modelling. It is also likely to produce greater differences in constituency predictions between the various pollsters.

We already saw in 2024 how the MRPs can profoundly affect a campaign. Their forecasts for each seat were relied on by a number of tactical voting websites, and in local electioneering political parties made great use of those estimates which were convenient. The MRPs also probably had a substantial impact on the morale of some party activists, both positively and negatively.

All these points were made at the BPC event yesterday. For example, Martin Baxter from Electoral Calculus described how he received irate complaints about some local parties quoting out-of-date seat analyses in their election literature, but despite his efforts there was nothing practically he could do to stop them.

I expect next time there will be greater variation and inconsistency, with more opportunities for parties to cherry pick and publicise forecasts that suit them. We’ll also see more instances of different tactical voting organisations issuing contradictory advice. Sounds like a recipe for chaos and confusion. And perhaps more calls for polling to be banned during campaigns.

However I should note that one factor will help the MRP statistical modellers at the next election. As Prof Chris Hanretty pointed out yesterday, they won’t have to cope with the complication of new constituency boundaries.

Subscribe now

As well as these challenges, the MRPs will also face the fundamental issue that the political polling industry in general does - the accuracy or otherwise of voting intention data.

The major problematic factors in 2024 considered by the BPC’s member companies, as I have reviewed in the past, were late swing, ‘shy Tories’, difficulties with reaching over 75s and the less politically engaged, and religion/ethnicity.

No one knows of course whether next time there will be much late changing of mind by the electorate. Some of the other concerns may be dealt with by more sophisticated demographic modelling and more ingenious or determined ways to survey the kind of voters who aren’t enthusiastic about being polled. But the problem of ‘shy Tories’ may get trickier to handle.

Pollsters historically and internationally have faced a frequent (but not universal) difficulty of under-stating backing for right wing parties (as can be seen in this chart presented by Prof Will Jennings). In the UK most pollsters try to manage this by weighting samples according to how people voted in the past.

Yet at a time of increasing volatility in the electorate, with chunks of public opinion churning around in all sorts of different directions, this is becoming much more awkward than in an era predominantly of neat two-party uniform swing.

This may also leave pollsters with dilemmas, as was illustrated at the meeting by Robert Struthers of BMG Research. Given how age was very strongly associated with voting patterns in 2024, it would surely make sense to take account of mortality and adjust for Tory voters (who tended to be much older) being more likely to die between then and the next election. But if you are already worried that your polling is under-stating Tory support, this would only take you further in the wrong direction.

Prof Patrick Sturgis also raised what could become a growing problem in the world of online survey research, which is that of questionnaires being completed by bots or organised bogus respondents, so that financial or other incentives can be claimed. This could be exacerbated if the fakers increasingly purport to be the hard-to-reach groups that pollsters may be upweighting in analysing samples.

It’s expected that the presentations (which I thought were impressively interesting and candid) given by the polling companies at yesterday’s meeting will be placed on the BPC website, to add to the analyses which are already there. Well done to the BPC, which aims to increase transparency in the UK’s political polling industry, for arranging the event.

The pollsters are continuing to grapple with all these issues. In particular they are awaiting the release of delayed data from the large-scale academic British Election Study, which may shed further light on what went wrong for the industry in 2024.

Subscribe now

My analysis of Electoral Commission data shows that in potential LibDem target seats, Labour spent on average about £4,600 less than it did in other comparable constituencies.

This chart demonstrates that in most places where the LibDems could be hopeful of victory, Labour generally spent a fairly small proportion of the maximum legally allowed for local campaigning at the 2024 election. This was in stark contrast to its general pattern of expenditure.

On average Labour candidates in these LibDem targets used only 26% of the legal limit, whereas Labour’s average in other seats was 68%.

Share

Subscribe now

While it could be argued that some of these seats were low Labour priorities as probably unwinnable, my statistical analysis shows that Labour put much less effort into campaigning in LibDem targets even after controlling for how well or badly Labour was positioned after the previous election in 2019.

Looking at seats then held by the Conservatives, and after taking account of the swing Labour would have needed to take the seat in 2024, Labour’s local campaign spending was £4,597 lower on average in these possible LibDem targets.

This fits with the claims made during the 2024 campaign that Labour was diverting its resources away from these seats, sometimes to the annoyance of disgruntled party activists.

In their local campaigning the LibDems concentrated resources very tightly indeed on their priority constituencies. Since these were predominantly ones held by the Conservatives who were clearly in massive electoral difficulties, they did not spend much money in other seats which were Labour or Conservative-Labour marginals.

These strategies suited both parties nationally who could each successfully focus their energies on maximising Tory losses and not competing against each other, without the need for any official agreement which would have been highly difficult and controversial.

I’ve defined LibDem targets here as constituencies where after 2019 they were the party in second place after the Conservatives (along with the much smaller number of places they held). This is a clear and reasonable demarcation.

However, it would not be identical to the list decided on by their party HQ, which added or dropped seats in line with fluctuating political circumstances and local factors. In any case this is more about which seats Labour would regard as sensible LibDem targets than the LibDems themselves. Nevertheless, even if there is a discrepancy of a few locations, that would not invalidate the very strong pattern this analysis identifies.

The local data published last month also reveals a very interesting contrast between Labour and Conservative spending patterns, which reflected the different political situations they faced when going in to the election.

This is indicated in the next two charts, which compare the party’s spending in each constituency in 2024 to its strength there after the previous general election in 2019.

The ^ shape in the Labour graph shows the party tended to spend most in marginals, with significantly less expenditure in many hopeless seats and safe ones.

For the Conservatives, however, they had to concentrate resources on the seats they already held and were not able to assume any were safe. So their spending was much higher in places where they’d done better in 2019.

This expenditure refers to money parties use for electioneering in each constituency, including leaflets, posters, letters, meetings and office costs.

The legal limit in each place depends on the size of the electorate and whether the constituency is urban or rural. In 2024 the maximum allowed was generally in the range of £16,000 to £21,000.

Overall Labour spent £7.2 million on local electioneering, while the Conservatives spent a little less, £7 million. For the LibDems the total was £3.2 million, for Reform £1.3 million, and for the Greens £700,000.

This is separate to money used for national campaigning. The legal cap on that in 2024 was £34 million for parties standing candidates throughout Great Britain. The Electoral Commission has not yet released the details of the national sums spent by the major parties.

There is some evidence that spending on constituency campaigning does matter. I analysed how many extra votes each party got for an extra £100 of local spending, after controlling for the party’s vote share in that constituency in the 2019 election.

The figures are given in the table below. However it is difficult to read much into this, as parties would also be devoting more effort to places where they felt things were going well, especially the LibDems, Greens and Reform. So the overall direction of causation is far from clear.

* This is the number of extra local votes in 2024 for that party associated with an increase of £100 in its local spending, after controlling for its votes in 2019 to take account of its position going into the election. (In the case of Reform, I used the figure for how the Brexit Party did in 2019, although they didn’t stand in many seats).

Subscribe now

https://www.meetup.com/london-business-analytics-group/events/306717967/

Share

Subscribe now

Supposedly the UK has the right of first refusal if the Danes ever decide to sell the icy but strategically located and mineral rich island territory (which of course they insist they won’t). The articles, which have appeared internationally, all seem to be based on statements made by a former Danish government representative to Greenland.

In fact, I've located the relevant documents in files at the UK National Archives and can report that it's not true.

The reality revealed by these papers is that in 1920 the British government did indeed tell the Danish government that they would only recognise full Danish sovereignty over Greenland in return for the right of first refusal if the island was ever sold.

But the Danes completely dismissed this suggestion. Britain then backed down and agreed recognition, merely adding the rider that if ownership was transferred then the British government ‘must reserve their right to be consulted’. To which the Danes sent no response.

Intriguingly, I also discovered that I'm not the only person who's just been to the National Archives in Kew to search through obscure binders from a century ago containing paperwork about cryolite mining, musk oxen, trading outposts, and other aspects of life in Greenland, along with diplomatic dealings.

My access to several folders I had ordered in advance was postponed for several hours because, according to archive staff, they had been taken to be read by someone 'from the government'.

So what do the documents disclose?

In an era when it wasn’t uncommon for colonial nations to trade land (and control over its population) for money, a Foreign Office memo shows the issue had previously arisen.

By the time of the First World War, the Danish government was extremely keen to obtain formal international recognition for its authority over all of Greenland.

In 1916 the Danes got the United States to accept this, with the Americans thereby renouncing any claims of their own, as part of a treaty in which the US purchased from Denmark what then became called the US Virgin Islands.

In the wake of the war Denmark wanted all the leading victorious powers to agree. From the internal discussion in the archive records it is clear that there was some unease within the British state, particularly the Colonial Office (CO). This was largely because of potential future implications for Canada, a neighbour to Greenland and at that time a dominion within the British empire.

The Danes wanted to raise the issue at the post-war Paris Peace Conference in 1919, but were persuaded not to. In 1920 they formally asked for recognition.

In line with the wishes of the Canadian government, who had been consulted, the Foreign Office informed the Danish ambassador in London that the UK would only recognise their complete sovereignty over Greenland on one condition: that ‘should Denmark ever wish to dispose of the island, she would grant the right of pre-emption to the British Empire’.

This didn’t go down well with the United States government, who were notified. The US ambassador to London John W Davis wrote to the Foreign Secretary Lord Curzon to say that his government ‘is not disposed to recognise the existence in a third government of a right of pre-emption to acquire this territory if the Danish Government should desire to dispose of it’.

More importantly, the guarantee which the UK wanted didn’t prove acceptable to the Danes.

The Danish ambassador replied, stating: ‘The King’s government direct me to observe that there is absolutely no intention of selling Greenland and that there is no prospect that such a sale will ever become actual. Accordingly the King’s Government on grounds of principle regret that such a guarantee cannot be given as such an undertaking might imply the possibility at some future date of the question of sale being raised in some form.’

The British then felt they had no alternative but to retreat to a much weaker backstop position, which was agreed with the Canadian government. This said only that ‘His Majesty’s Government must reserve their right to be consulted should the Danish Government contemplate the alienation of this territory’. (I couldn’t find a copy of the original message in the archives, but did come across this later officially typeset version).

‘Alienation’ here is used in the legal meaning of transferring property rights, rather than the current everyday one of causing disaffection. As it happens, in the latter sense the Danish state has over the decades sometimes gone a considerable way towards alienating parts of the local population, but that’s another story.

Greenland is now semi-autonomous and will hold elections next month, in which the question of full independence from Denmark will be a leading issue. The island is currently dependent on substantial financial support from Denmark, but global warming could bring increased economic opportunities with mining and international transportation.

It looks like the Danes never replied.

One Foreign Office official wrote that this was presumably ‘an oversight’. But possibly the Danes didn’t think that this final British missive merited a response.

Two years later in 1922 the question arose of whether these exchanges between the governments should be published. This stemmed from an enquiry from the Norwegian government, which was contesting Danish sovereignty over the island in a dispute about fishing and hunting rights.

The suggestion that these confidential messages should be released did not meet with enthusiasm within the Foreign Office, where an official wrote that ‘this correspondence seems in general unsuitable for publication’. There was particular concern not to reveal that the US objected to the British demand for first refusal.

In the file there follows some discussion of whether to release some much more limited information: just the initial Danish request for formal recognition of sovereignty and the final British response. However from the records I could find, it’s not clear what eventually happened and whether any of this was actually revealed to the public at the time.

But in any case it is all revealed here and now. And it’s a very good thing that this paperwork is carefully preserved and made available by the National Archives, so that these questions can today be researched and answered. I hope the person ‘from the government’ found what they were looking for.

My thanks to Liz Evans, one of the independent researchers based at the National Archives, for her specialist expertise and assistance in locating some of these records.

Subscribe now

The justification which the former prime minister supplied to the House of Lords Appointments Commission states that Johnson “entrusted Charlotte with engaging the parliamentary party on his behalf and she was essential to maintaining his relationship with them”. He added that “Charlotte was later tasked with helping the new Chief Whip in his role and used her unique knowledge to become a bridge between the Prime Minister and the Chief Whip”.

Johnson’s citation for her also states that “Charlotte led on many sensitive and key projects including advising the Prime Minister and the Chief Whip on suitability for Ministerial appointments during the reshuffle”. No other details of these “sensitive and key projects” are given.

The first part of this citation fits with the notion that she was a junior official with very little responsibility and experience, whose sudden elevation to a peerage was peculiar.

The unevidenced claim in the citation that she led on “many sensitive and key projects” including on advising on ministerial appointments will, I think, come as a surprise to a lot of people working in Downing Street at the time.

The citation added that Owen wanted to advocate for young women who have fallen victim to image based sexual abuse, an issue she has indeed taken up since being in the House of Lords.

Share

The released documentation also reveals that her nomination was supported by the former Tory cabinet ministers Grant Shapps and Chris Heaton-Harris, although not what they said.

The citation for Ross Kempsell is more detailed, giving an account of his work and achievements as the head of the Conservative research department and a special adviser in Downing Street.

The House of Lords Appointments Commission was eventually forced to release this material due to a lengthy freedom of information case I have been pursuing with them.

I am very pleased that the documentation has now been revealed, but it shouldn’t need an argument over 18 months for the public to find out what reasons are officially provided for allocating certain people important political powers.

Members of the House of Lords debate and vote on laws that control the British public’s lives. As a basic principle the public is fully entitled to know what reasons are given for why they have been appointed to rule over us.

The two peers are now known as Lady Owen of Alderley Edge and Lord Kempsell, and have both been active in the Lords after their somewhat unexpected appearance in Johnson’s resignation honours in June 2023.

At the time of the announcement they were 29 and 31 respectively. Owen’s ennoblement caused a great deal of consternation and puzzlement, as there was no evidence of any achievement of hers that could explain it.

She was a junior staffer working in Johnson’s Downing Street operation. Press reports described her as possessing “no views, no achievements, no experience” and as “the most junior person in political history to have received a peerage”, while also alleging discrepancies in her career history.

The disclosure follows a ruling last month from the First-tier Tribunal, which upheld my arguments that releasing this information is in the public interest.

It is the culmination of the dispute between myself and HOLAC, which initially rejected my FOI request in July 2023 for the information held about these two nominees. After HOLAC insisted on its position, I complained to the Information Commissioner and then appealed to the Tribunal.

At the Tribunal hearing, where I represented myself against HOLAC’s extensive legal team and cross-examined Clare Brunton, the secretary of HOLAC who is also head of the honours secretariat in the Cabinet Office, I argued that the process for giving individuals a status as legislators is a vital matter of the public interest.

Those appointed can approve or reject proposed laws, as well as being able to take part in parliamentary debates, directly question ministers, and so on. This entails the need for maximum transparency, so that the process is both legitimate and is seen to be legitimate, and the public can see for themselves whether appropriate procedures are followed.

The First-tier Tribunal’s judgment stated: “We attributed considerable weight to the public interest knowing the PM’s reasoning … Life peers are Members of Parliament with the rights, obligations and influence associated with such an appointment thus enhancing further the weight of the public interest in the PM’s citations.”

The Tribunal dismissed HOLAC’s arguments that releasing the information would damage the honours system and be a breach of confidence.

The full citations disclosed to me are here.

Share

If you are interested in taking an FOI case to the First-tier Tribunal, there is a chapter devoted entirely to this with detailed and thorough advice in my book, Freedom of Information – A practical guidebook.

Subscribe now

We were saved from the normal cliched headline 'Polls Apart', because the polls were all together on one side of reality, overstating Labour and understating the Tories.

I've been examining the reasons provided by those polling companies who have publicly tried to explain how these forecasts went wrong. They focus on the following factors: late swing, religion, turnout, 'shy Tories', and age.

The most recent company to publish its analysis was YouGov, which did so just before Christmas, also announcing that it would adopt a new methodology from January.

Subscribe now

The election polls significantly overestimated Labour and underestimated the Conservatives, as shown in a chart from Will Jennings.

While this pattern has often happened, in terms of the difference between the two parties, this was their biggest miss since 1992, exaggerating the gap on average by 7 percentage points.

The constituency prediction models known as MRP polls were also all awry in the Labour direction, as demonstrated in the dataset collated by Peter Inglesby.

Of course some polls were much nearer to the actual outcome than others, as the companies that did reasonably well and got closest are naturally keen to stress, and as I myself have analysed in the past. But the industry as a whole clearly systematically over-predicted Labour, and that's not good for the world of opinion research.

This is despite the fact that one can argue this was a tricky election to get right, with an increasingly volatile electorate, a very large swing, an important new party, the impact of independents, and changes in how demographic characteristics such as education and class link to voting behaviour.

If the result had been close, the level of polling error involved would have created a sense of chaos and surely have become a crisis for the industry. However the problem has been disguised by the fact that the only point at issue was the extent of the landslide, and so it did not disturb the central narrative of the election.

Pollsters are constantly seeking to improve their methods, and indeed the MRP models last July were a positive contribution to getting the overall impact correct, confirming the value of innovation. Companies have been reviewing their performance and what went wrong.

The British Polling Council (BPC) is collating relevant research from its members on its website. So far work from six organisations has been added. As well as YouGov, the others are BMG, Electoral Calculus, Find Out Now, More in Common, and Verian. It's possible that more BPC members will add further submissions in due course.

I've been reading them to see what prevailing points emerge on an industry-wide basis.

It is important to note that given the companies have different methodologies, this implies there could also be variations in what each got wrong. But the fact that they were all out in the same pro-Labour/anti-Tory direction suggests that as well as any individual aspects there is also something significant which is shared.

Although there is no unanimity, their findings do reveal some common themes. (None of them discuss the issue of 'herding', the claim that error can be exacerbated if some companies sometimes take decisions in such a way that they stay in line with the crowd - a charge which is very unpopular within the industry).

Late swing

Three companies - BMG, More in Common and YouGov - attribute the error partly to 'late swing', due to people changing their mind about how to vote at the last minute after final opinion surveying ended. A cynic might say that this is the most convenient excuse for the industry, as it is the least challenging to the accuracy of their methods. Maybe, but the fact that it is convenient doesn't necessarily mean that it is wrong.

Beyond the data presented, I have to say I also find this plausible based on anecdotal evidence, with the forecasts of a huge Labour victory nudging some intending supporters into eventually switching to vote for someone else, such as the Greens. In this sense the polls ironically could have been their own enemies, almost a kind of partially self-negating prophecy.

However Electoral Calculus finds no evidence of late swing, and in any case none of the companies thinks it can approach the full explanation, which still leaves a methodological challenge for the industry.

Religion/ethnicity

The pollsters seem to have failed to reflect the increasing fragmentation of the ethnic minority electorate, with some Muslim/Pakistani & Bangladeshi voters abandoning Labour, often for independent candidates who campaigned about the situation in Gaza, while Hindu/Indian voters drifted towards the Tories. This factor is referred to by BMG, More in Common and YouGov. It is clear that election analysis can no longer crudely treat voters of Asian heritage (let alone all ethnic minorities) as if they are one political bloc.

More in Common suggests that Muslims who currently take part in online market research panels are probably not representative of the overall Muslim population, being more likely to be second or third generation immigrants, and less likely to be born outside the UK or not speak English. The company says it will probably modify its weighting scheme.

Similarly YouGov says it will incorporate a more detailed ethnicity breakdown into its modelling in future.

However, the numbers of voters involved, while crucial in certain seats, mean that this could also only be a very partial factor nationally.

Turnout

Taking account of likelihood-to-vote is a notoriously difficult problem for pollsters, who employ a range of strategies to estimate how many of each party's proclaimed supporters will actually go to the trouble of casting a ballot. Three companies - BMG, Electoral Calculus and YouGov - include the overpredicting of Labour voters' turnout as a factor in the 2024 error.

YouGov argues the cause stemmed from panels which over-represented people who would actually vote, especially for low turnout demographic groups. The company says that from now on it will base turnout modelling purely on demographic data, rather than respondents' self-reported likelihood-to-vote.

This sort of problem has been a general industry issue in the past, of over-sampling the more politically engaged (who tend to be keener to take part in this kind of survey).

However it is awkward for pollsters to get turnout adjustments correct. There is no guarantee that what worked best last time will be best next time, as the commitment of different groups to implement their asserted voting intentions may depend on the political circumstances of the moment. Ironically again, the forecast Labour triumph last July might have pushed some of the party's less determined supporters into not bothering to go to the polling station on the big day.

Shy Tories

This has also been a traditional difficulty for the polling industry, where those of a Conservative outlook are somewhat less willing to express their allegiance - possibly because they feel in some sense disapproved of or intimidated (this is sometimes called 'social desirability bias'), or perhaps alienated from polls. Again, the extent to which it happens can also depend on the political atmosphere of the time.

Over-estimating the Left and under-estimating the Right is not just a UK polling problem - it has cropped up as a fairly consistent (but not universal) pattern across many countries, as can be seen in the Deltapoll slide in this piece by Mark Pack.

The industry has tried to counteract this skew through various means of political weighting, such as using previous voting behaviour.

Electoral Calculus states there is indeed suggestive evidence of a 'shy Tory' effect in 2024, with people who refused to answer voting intention questions or who replied "don't know" being more likely to be Tory voters. This is also consistent with the findings reported by BMG and by More in Common about 'undecided' voters who were then pressed.

YouGov suggests that its past vote weighting fell down in 2024 because at the previous election in 2019 the Brexit Party endorsed the Conservatives in many seats. The result was that its panels had too many 2019 Tories who actually preferred the Brexit Party and then voted Reform in 2024, and not enough firmly committed Conservatives. Their paper does not raise the issue of whether it is staunch Tories who are most likely to avoid voting intention opinion research, but it seems to me that this conclusion is compatible with their evidence.

Find Out Now (which only produced one unpublished poll during the 2024 campaign) argues against the 'shy Tory' hypothesis. But in my opinion their data only counters the hypothesis that online research panels under-represent Tories in general, as opposed to the hypothesis (advanced by Electoral Calculus) that Tories may be reasonably represented in panels but are disproportionately likely to refuse or reply "don't know" when faced with a voting intention question in a survey.

More in Common also states that there is possible selection bias affecting online panels as the recruitment processes appeal to the 'overly opinionated'.

Age

Age was very strongly associated with how people voted last July, with Tory support concentrated in the older section of the electorate.

The report from Verian (the polling company which came closest to the actual result on percentage vote shares) focuses entirely on the issue of age, and concludes that those companies whose samples contained a smaller proportion of over-65s (after weighting) tended to be less accurate. But its presentation adds that other biases would also have played a role.

Find Out Now raises a different possibility on age, that it failed to locate Conservatives who were younger and less politically engaged (a group that is hard for pollsters to reach).

Subscribe now

Summary

At this stage we are left with the suggestion that perhaps four or five factors may have contributed together to the polling miss, and none explain it alone.

There can be a problem with this kind of analysis, dubbed the "Orient Express" approach by Electoral Calculus, where multiple possible causes are examined and all those which affect the error are deemed part of the solution. In other words, as in the Agatha Christie story, if everyone/everything is responsible for what happened, then eventually no one/nothing is actually held responsible, and nothing is done.

On the other hand, looking at the underlying fundamentals, it seems to me that predictive opinion polling is a difficult business given the level of precision required and the volatility of today's voters. There are many sources of potential error (apart from normal sampling variation), arising from which people get contacted, whether they reply or tell the truth or change their minds later, how the electorate is modelled, and how the answers from different groups are weighted to aim at representativeness. And errors that arise are difficult to eliminate methodologically, as they depend on political circumstances which vary from one election to the next, and also on the communications technology for conducting research which is constantly evolving and in different ways for different social groups.

Inevitably therefore pollsters are bound to make some mistakes (and not all will make the same ones). When they are lucky, the errors may cancel themselves out, more or less, and nobody notices them. When the pollsters are unlucky, the errors largely or entirely mount up in the same direction.

Further, more thorough analysis will be possible once detailed data becomes available from the academic British Election Study and its extensive voter research.

The British Polling Council, to which all the main pollsters belong, is also planning to hold a public event to discuss these issues, probably in April.

It has just ordered the release of the secret citations Johnson sent to the House of Lords Appointments Commission (HOLAC) for the peerages awarded to Charlotte Owen and Ross Kempsell in his resignation honours list in June 2023.

This is the latest stage in an 18-month freedom of information dispute between myself and HOLAC, which turned down my FOI request in July 2023 for the information held about these two nominees.

The First-tier Tribunal, which hears information rights cases, has now backed the view that releasing the citations Johnson submitted to justify the nominations is in the overall public interest.

It has also instructed that the identity of some public figures who were indicated as supporting Charlotte Owen’s appointment should be revealed.

I’m very pleased by the decision, which is a boost for transparency and democracy. Members of the House of Lords debate and vote on laws that control the British public’s lives. As a basic principle the public is fully entitled to know why they have been appointed to rule over us.

According to HOLAC, citations provide ‘the reasons for nomination’ and a statement of ‘personal and professional background and attributes’.

The two individuals involved are now known as Lady Owen of Alderley Edge and Lord Kempsell, and have both been active in the Lords after their somewhat unexpected appearance in Johnson’s resignation honours.

At the time of the announcement last year they were 29 and 31 respectively. Owen’s ennoblement caused a great deal of consternation and puzzlement, as there was no evidence of any achievement of hers that could explain it.

She was a junior staffer working in Johnson’s Downing Street operation. Press reports described her as possessing “no views, no achievements, no experience” and as “the most junior person in political history to have received a peerage”, while also alleging discrepancies in her career history.

The Tribunal’s judgment published today states: “We attributed considerable weight to the public interest knowing the PM's reasoning … Life peers are Members of Parliament with the rights, obligations and influence associated with such an appointment thus enhancing further the weight of the public interest in the PM's citations.”

The Tribunal dismissed HOLAC’s arguments that releasing this information would damage the honours system and be a breach of confidence.

The decision means HOLAC has until 22 January to release the material.

I brought the case to the Tribunal to appeal against the Information Commissioner, who in March this year had upheld HOLAC’s rejection of my request. It shows that it is worthwhile challenging weak decisions from the IC.

The Tribunal has now overruled the Commissioner, following a one-day hearing in October. However it did not back all aspects of my appeal, coming down against the release of the detailed minutes of the HOLAC meetings which discussed Owen and Kempsell.

At the hearing, where I represented myself against HOLAC’s extensive legal team and cross-examined Clare Brunton, the secretary of HOLAC who is also head of the honours secretariat in the Cabinet Office, I argued that the process for giving certain individuals a status as legislators is a vital matter of the public interest.

Those appointed can approve or reject proposed laws, as well as being able to take part in parliamentary debates, directly question ministers, and so on. This entails the need for maximum transparency, so that the process is both legitimate and is seen to be legitimate, and the public can see for themselves whether appropriate procedures are followed.

Earlier this month the Labour government announced that in future the citations to support individual nominations for political peerages will be published. This is also a welcome move towards much-needed greater openness in the appointments system for members of the House of Lords.

If you are interested in taking an FOI case to the First-tier Tribunal, there is a chapter devoted entirely to this with detailed and thorough advice in my book, Freedom of Information - A practical guidebook.

Subscribe now

So what about the annual turnover level at which businesses have to register for VAT?

Data I have recently obtained from HMRC under the freedom of information law shows the dramatic impact of the VAT threshold in restricting the growth of some of the UK’s small businesses.

In 2021/22 the UK had 21,752 businesses with annual turnover in the range £84,000-£85,000, just below the then threshold. But there were only 10,096 businesses just over the limit, in the range £85,000-£86,000.

In other words the number of businesses clustered just under the VAT threshold was more than double the number just above, as businesses curtail their activities to remain outside the VAT registration system.

The graph above clearly shows the cliff edge in the data.

Many small businesses are desperate to keep their annual turnover under the VAT level, so that they avoid the bureaucracy and costs of registration and they don’t have to charge VAT to customers, which would make them less competitive. However the consequence is that they then won’t grow further into larger, more successful operations.

For some businesses the VAT threshold functions as a ceiling constraining their growth.

Research by Warwick University in 2022 concluded that earlier data of this kind reflected genuine curtailment of business activity rather than false reporting to HMRC.

This is the latest data available from HMRC, which says that more recent information is still being processed. The current VAT threshold is now £90,000, as the figure was increased by the Conservative government before the general election.

The UK’s VAT threshold is high compared to other European countries which tend to impose VAT registration on businesses at a much lower level. While the UK policy saves many small businesspeople from the compliance burden of VAT, the significantly lower thresholds elsewhere make it less likely that enterprises will be found bunched and held back just under the relevant level of turnover.

I also wanted to get a breakdown of the data by sector of the economy, to see which kinds of businesses were most affected. HMRC said it could provide this for 2019/20, as it had previously extracted the information involved, but that more recent breakdowns would probably exceed the FOI cost limit.

According to these 2019/20 figures, the most dramatic effect is in the construction sector.

This data shows 4,445 construction businesses with an annual turnover of £84,000-£85,000, but only 1,425 in the range £85,000-£86,000. So the number of construction businesses appearing to have kept themselves just below the limit is over three times the number who grew a little more and just exceeded it.

The chart shows the impact for construction and some other economic sectors with large numbers of small enterprises.

These FOI releases from HMRC constitute the latest and most thorough official evidence of what the tax expert Dan Neidle of Tax Policy Associates has called ‘the VAT growth brake’.

The full HMRC spreadsheets can be downloaded here:

1) Summary data for 2019/20, 2020/21, 2021/22

2) 2019/20 sectoral breakdown

Subscribe now