Subscribe now

Here we go again.

For the third time in as many weeks we have a frenzy about apparently terrifying AI agents seemingly going rogue and embarking on a hacking spree of innocent victims.

Once again, it’s a safety test at the root of the problem. This time, however, it’s not an American frontier lab - OpenAI and Anthropic have already disclosed what went wrong with their tests. It’s a British Government agency - the AI Security Institute - which has earned a deserved reputation as the foremost independent evaluation body in the world for frontier AI security testing.

The team at AISI were testing tools developed by Anthropic and OpenAI, but the operation was theirs, not that of either frontier lab. So it is they who are responsible for an agent that breached GitHub. To their credit, AISI have taken responsibility in a refreshingly accessible and candid blog. To their even greater credit they have made very significant and immediate changes. Others should follow their lead.

There are three lessons we should learn from this spate of incidents:

• the first is surprisingly simple. It’s about monitoring and, crucially, control in the testing environment. There’s a problem with the way we currently do AI security testing and it’s really obvious it needs to change. Happily, but unusually for AI, it’s surprisingly easy to fix. Rather than do what Governments and companies normally do and waffle on about reviewing procedures and sharing information collaboratively in partnership, AISI have said simply and clearly they’re going to change the way they test. From now, during testing, their agents will always be monitored in real time and they are going to introduce ways to ensure that if the agent does things it’s not supposed to the agent can be stopped. This is the right change to make. Others involved in AI security testing should follow their lead;

• the second is conceptually simple but practically very complicated. It’s about accountability rooted in ownership of the AI agent across the entirety of agentic AI. These recent incidents arise from artificial testing scenarios designed to force us to think about what might happen in the real world. And in these three scenarios AI agents are behaving like very badly behaved small children who don’t know right from wrong. That’s because they absolutely don’t; they don’t understand what they’re doing. With small children, parents are supposed to set boundaries. If they fail in that, and something goes wrong, in the laws of most countries, parents are responsible for those very badly behaved children. This is not a bad starting analogy for thinking about AI agents. If you don’t set proper boundaries for your AI agent and something goes wrong, then you should expect to be held liable. So the starting principle of codes, conventions, rules and ultimately laws for AI agents is that someone owns it and is responsible for it. This will be hard to roll out in practice. But it is the right starting concept.;

• finally, we need to keep this in perspective. All three cases disclosed are highly artificial testing environments. These conditions are not likely to be replicated in the real world; not yet anyway. The capabilities are dazzlingly impressive but the real world harm is negligible; arguably it is zero. And for all the freakout, frenzy and hype, these three incidents are not even the biggest issue in cyber security right now. In case you missed it, last week it seems that the Iranian state came very close to disrupting the water supply to ordinary citizens in parts of the United States via a cyber attack. This genuinely terrifying episode had nothing whatsoever to do with AI. Is that why it received so little attention? If so, that’s nuts. Other things beyond AI testing procedural problems need the attention of those who want to help bolster vital cyber defences.

Let’s look at each of these three lessons - control, ownership and perspective - in turn.

Control

The AI Security Institute’s blog setting out what happened is not just accessible and candid. In terms of lessons learned, it is also commendably specific. As in the other two cases, AISI’s exercise involved setting an agent off on a task but then not fully monitoring in real time what it was doing. So in all three cases no one - indeed no thing - was watching when the agent went off on its unsolicited hacking spree. And even if someone, or something, been watching, it’s not clear that anyone would have been able to stop the agent from going about its hacking.

Now that we know this has happened - three times - it is perfectly clear that we should not do AI cyber security testing like this. The testing status quo is indefensible. However, unlike Anthropic and OpenAI, AISI have said very explicitly that they understand this. They will not conduct another test without real time monitoring and the ability to switch the agent off. This must be right. AISI are to be commended for pledging this swiftly and unequivocally. Everyone else involved in AI testing should do the same.

Ollie Whitehouse, the Chief Technical Officer of the National Cyber Security Centre in the UK, put out a very good three-paragraph statement on the implications of his British Government colleagues’ disclosure. Inevitably, the first paragraph - how this reminded us of the power of AI - was widely carried in media reporting; we seem to love to marvel at the capabilities of supposedly ‘rogue’ AI. Depressingly, the final paragraph - how this reminds us of the importance of basic cyber hygiene - was ignored. But it’s the middle paragraph that is most important:

Daniel Card, the well known British cyber security expert, put it a little more agriculturally:

AISI have followed this inexorable logic swiftly. This must be the way of the future. I suspect the main regret at AISI is that they didn’t make this change immediately after the OpenAI/Hugging Face disclosure - this test was carried out the week after. Had they done so, the incident disclosed today would not have happened.

But the correct decision has now been taken. And it’s a surprisingly simple one. As the AISI blog sets out in painstaking detail, the circumstances as to how this happened are extremely bespoke to testing and highly unlikely to be replicated in the real world. The same point is implicit in both the OpenAI and Anthropic disclosures, but the - in my view unhelpful - focus in those blogs on how powerful the attacking capabilities are blunts this important message. Don’t test without monitoring, and don’t monitor without control - the ability to press stop if you see something is going wrong. That should now be policy everywhere.

Last week I detected a mood change in the cyber security community. Since the Glasswing report in April, the prevailing atmosphere has been one of respect and gratitude towards the frontier AI labs for taking security seriously and being transparent and collaborative. Some of this has been sincere, and some of it contrived out of a feeling of necessity, and it’s been hard to work out the balance. But the attitude shifted with the OpenAI and Anthropic agent disclosures. Lots of cyber security veterans opined wryly that if they’d undertaken capability tests in this way they’d at best be in serious commercial trouble and facing multiple lawsuits, and at worst on remand awaiting trial. The great Bruce Schneier had a point when he wondered aloud what sort of hellish international crisis would have occurred if a Chinese model had been responsible for the Hugging Face incident, rather than OpenAI.

AISI can hardly be blamed for testing Anthropic and OpenAI’s capabilities in the same way as those companies themselves. But it is quite clear from the evidence of the past few weeks that these practices have to change, and it is good AISI have led the way in changing them.

Testing must be monitored and agents being tested must be capable of being controlled.

Ownership

All three incidents do highlight, however, the power of AI agents as hackers and their potential to carry out illegal and dangerous activity in pursuit of the objective they have been set. So, working on the assumption that such agents at some point become widely and generally available, how can they be controlled outside testing environments if and when they become widely available?

This is also conceptually easy but - sadly - much harder in practice than AISI’s changes to testing. The good news there is some time to think it through because the testing scenarios are not yet available in real world situations for most potential users, good and bad.

There is a semantic struggle to describe these capabilities and events accurately. The word ‘rogue’ is not fully accurate because the agent is simply trying to find ways to complete a task humans have set. But, as AISI have set out very clearly, the agent is behaving roguishly by trying to conceal things it’s done in pursuit of that objective. Similarly, the word ‘autonomous’ doesn’t quite work because the agent is doing something it’s been instructed to do and only that - which is anything but autonomous. At the same time, it is taking steps to carry out its task which those who instructed it really don’t want it to do. That’s very much autonomy in action.

I am afraid I don’t have a word that fully captures what these agents are doing and how we might think about controlling the consequences of that. But I do have a principle for how we should deal with it. It’s around ownership of the agent and accountability for its actions.

Let’s call it the Watling Principle. Bear with me…

In a superb episode of the Times podcast The General and The Journalist in June this year, Dr Jack Watling of the Royal United Services Institute talked about how drones are changing the war in Ukraine. He was asked a more general question about responsibility for AI military capabilities. His response was very thought provoking, in two ways.

First, he said that autonomy in AI posed real challenges for military leaders. The nature of military command was to restrict autonomy, not expand it. Soldiers have dangerous missions that rely on discipline and the collective interest. Impulsive individual decisions can jeopardise the mission. That is why commanders are responsible for the actions of their subordinates, and why commanders are therefore compelled to try to find sensible limits to autonomy and on-the-spot decision taking while allowing for good decisions in response to changing circumstances.

This led to the second, and for our purposes, more important insight. This is about ownership and accountability. For Dr Watling, core concepts and principles don’t change. It doesn’t matter what capability the commander uses. He is still accountable for everything that happens because he owns the capabilities, whether human or technical.

Dr Watling used the example of targeting a building. If a commander believes a building is full of enemy combatants, and he drops a 2,000lb bomb on it, but it turns out it is full of civilians, the commander bears responsibility for this humanitarian and operational fiasco. If the commander sends a swarm of AI robots into the same building to kill anyone it finds, the commander bears exactly the same responsibility for the same disastrous outcome. It follows that if the men, or the robots, go ‘rogue’, the commander is still responsible because he gave them their instructions.

This is the principle was should apply in these civilian environments too. We have already applied it in cyberspace in relation to two infamous incidents in 2017. Both were what can be called malicious accidents: no one thinks North Korea - in the Wannacry case - intended to launch an attack that hit both Taiwan and China, both Russia and Ukraine, as well as German railway stations and British hospitals. Similarly, no one thinks Russia was trying to destroy Maersk, the shipping giant, Merck, the pharma titan, or to close down Cadbury’s chocolate factory in Tasmania. But that’s what happened when they launched NotPetya. And what did western Governments do? We blamed North Korea for the destruction wreaked by the first attack, and Russia for the second, and tried to hold them accountable. That they didn’t mean it didn’t matter. It was their “agent”.

The AISI report is commendably clear that this was their agent and their responsibility. Rolling out this principle more widely in the full deployment of agentic AI in all circumstances is going to be fiendishly difficult. But it’s the right place to start. Who owns the agent? With that ownership comes accountability. Let’s work out how to get that principle right in practice, rather than marvel at the scary capabilities.

Perspective

Finally, let’s keep all this in perspective. What we are seeing are the consequences of maturing, and therefore flawed, safety tests. This is not even accidental use of AI by normal businesses, let alone undefendable attacks by cyberninja adversaries armed with magical AI powers. All three of OpenAI, Anthropic and AISI have said they were testing the most powerful capabilities they have, and that in doing so they were removing the normal safety features. That is not normal. It’s a test.

Moreover, there is no actual damage. Intrusion into a network may (or may not, in these circumstances) be illegal, it may be discombobulating, but it is not in and of itself harmful or damaging. No one suffered here. In terms of most people’s understanding of the word ‘harm’, none has been done. We can learn from this, and it’s not learning the hard way.

What is, however, truly extraordinary therefore - and needs calling out - is the obsession with these disclosures to the detriment of other things. These incidents are extremely interesting. They are concerning, and must surely give added impetus to the urgent and comprehensive measures needed to get secure AI deployment right. But there is no cyberapocalypse, and no sign of one.

In the headspace of policy makers and opinion formers, cyber security competes with many other unpleasant and difficult challenges for much needed attention. So focusing on the right things when we get that attention really matters. This gives rise to the most depressing conclusion of the last fortnight.

It is widely believed that over the past few weeks the Iranian state has successfully infiltrated the civilian water supply in the state of Minnesota in the US, and possibly in seven other states. The full picture is unclear, but there seems to be credible evidence that had it not been for very considerable recovery work, actual supplies to civilians could have occurred. The causes of the attack will be investigated, but, as the redoubtable Jen Easterly, former head of America’s cyber defense agency, set out in the New York Times, they illustrate serious vulnerabilities in the world superpower’s critical infrastructure. As Easterly put it:

I am writing from the United Kingdom, not the United States. But until the specifically British aspect of the AISI revelation in the past day or so, none of the major cyber security studies of the past fortnight had anything specifically to do with Britain. But look at British media coverage of cyber and general cyber discourse in that period. OpenAI and Hugging Face? Freakout. Anthropic’s triple rogue agent? Frenzy. Our closest ally nearly having its drinking water knocked out by a regime that doesn’t much like us either and could try the same on us? Barely a ripple.

If we focus our efforts and attention exclusively, or even predominantly, on the eye-catching results of the latest frontier AI testing mishap to the exclusion and detriment of long-standing threats and vulnerabilities, we don’t stand a chance. Let’s keep these incidents in perspective.

Subscribe now

AI and the new wave of cyber security FUD

Two seemingly contradictory things can be true at the same time:

1. Advances in frontier AI and the new capabilities they bring represent a very significant change in what’s possible for cyber attackers. This is real and it’s really important. It requires changes in strategy, tactics and posture among cyber defenders and a real sense of urgency around some of the defensive improvements needed, as well as difficult challenges for policymakers;

2. At the same time, the near four months (and counting) since the Mythos/Glasswing announcement has prompted the biggest outbreak of cyber FUD^[1] since former US Defense Secretary Leon Panetta warned of a “Cyber Pearl Harbor” in a 2012 speech. Lots of cyber harms happened. But cyber Pearl Harbor never took place, and that was entirely predictable. The apocalyptic narrative is equally wrong now. For sure, we are heading towards some potentially very tough times in cyber security. But many of the current problems we’re experiencing in cyber security are long standing, unaddressed ones. We can also expect AI to lead to cyber security improvements in the medium term.

There is probably a stormy period ahead. But storms pass. We are not heading towards some AI cyber apocalypse. Creating the expectation of one is harmful as it distracts from serious but manageable cyber security challenges. And drawing the wrong lessons from the latest reveal doesn’t help either.

Two articles, two authors:

(1) AI & FUD; (2) Hugging Face – AI is not attacking us

There’s a lot more to say on all this and I am working on a longer article that sets the new outbreak of FUD more fully in its historic and current context, along with why it is damaging. But the prompt for this shorter and quicker intervention was two-fold. One was an excellent SANS Institute discussion on the recent agentic AI cyber attack on Hugging Face which you can watch here (interest declared: I do paid work with SANS; that said, as this video makes very obvious, they don’t try to censor their contributors or make them toe a party line.)

The other was an outreach from my friend Professor Alan Woodward of the University of Surrey, who sent me his own thoughts on the hype surrounding the Hugging Face incident. Alan is away and tied up on other things, and he is not in a position to publish this article himself. So it’s my privilege to do so on his behalf and with his permission as an accompanying piece, and I do so below (so please note these are two distinct articles, one by me and the other by Alan).

Alan’s insight is vital: AI isn’t attacking us. The AI didn’t know what it was doing. And that’s a different problem set that requires a different set of solutions.

So first me, then Alan. And apologies this Substack has been so dormant for so long, especially to those who have supported it. I will try to do better.

Article 1: AI and the new wave of cyber security FUD – from frenzy to freakout

By Ciaran Martin

The cyber security community has been on quite a journey since the Mythos/Glassing announcement almost four months ago. Hugging Face is the latest stop on the ride. Unusually (and entirely because of the general apocalyptic hype around AI which has been around for several years), this time the wider bubble of media and politicians are interested as well; it’s not just us in the specialist cyber field.

The first few days after Mythos saw something of a freakout with all sorts of apocalyptic predictions, mainly based on a knowledge gap between what Anthropic had published about their own model and what the wider world knew. So although the freakout was a bit distracting, it was understandable.

Then, as the days passed and the knowledge base grew, an unhelpful freakout gave way to a more constructive frenzy: a period of frenetic activity. That was because as a community we were beginning to understand the significance of what was unfolding and what to do about it. The mood was one of manageable urgency: we needed to promote realistic awareness among leaders, and at a technical level do sensible things like reduce the attack surface. Addressing technical debt was becoming more urgent – as the great Katie Moussouris put it, AI meant the technical debt repo man was on his way. In other words, AI was finally making it imperative for organisations to do all the things they should have been doing for years (plus some newer things).

So the freakout subsided. The ensuing frenzy – racing to fix things for the age of AI – was justified. But here we are again.

Over the past few days the global media has been back in freakout mode. On Times Radio and Channel 4 News, I struggled to convince two very respected and hardnosed British journalists (Stephen Sackur – I mean he hosted HardTalk for goodness sake) – and Matt Frei, that we shouldn’t freak out about the Hugging Face incident. The hysteria was getting it all wrong (as Alan’s piece explains, this was not a rogue, autonomous AI agent, it was an OpenAI agent doing what it was told to do and achieving it because the testing environment wasn’t safe enough). Moreover, the circumstances of this episode are so extraordinary – they raise as many questions about frontier AI testing environments as they do about hacking – they are very unlikely to become commonplace.

And yet we are back to freakout mode. And for reasons I’ll elaborate on in the fuller article, freakout mode doesn’t help. It’s FUD on steroids – the capabilities are too powerful, everything is going to go pop, and there’s precious little we can do about it. That last bit is particularly corrosive – it infantilises people who are in a position to improve their defences and sends them chasing after the wrong problems. It sends Governments into a tailspin, feeling forced to react quickly rather than taking time to assess things strategically. Interestingly Secretary Panetta, asked years later about his Cyber Pearl Harbor remark, said he knew that things were more complicated than the phrase implied but he was addressing “the jackass who wouldn’t pay attention”.

Why do we do this to ourselves? It’s easy to blame commercial incentives. Maybe that’s part of the reason, though I do have some sympathy with Alan’s argument below that demonstrating your capabilities to be uncontrollably dangerous is a funny way to sell things. That said, in security, few businesses, if any, have ever lost money hyping up a cyber threat. But there’s more to it than than just marketing.

The most important point is, as the legendary Marcus Hutchins has said quite recently, a tendency to focus too much on the new capabilities and not enough on attacker behaviour and incentives. There are all sorts of things in life that are possible and very dangerous, but bad actors tend not to do them because there are barriers in the way, or because they’re too expensive, or because they don’t actually fit with what they want to achieve. But in the AI and cyber security debate we tend to ignore all this, and just assume that because something devastating is possible, anyone and everyone will both be able to do it and want to do it.

But there are huge barriers to most attackers being able to harness these new AI capabilities to do us serious harm, as Marcus has set out repeatedly in a series of LinkedIn posts. As he put it in one of his most recent posts: “The whole idea of agentic models mass-hacking sites is ridiculous. Who is allocating billions of dollars worth of compute to hacking a bunch of shoe-string budget organizations with no money?”

Moreover, and this is a hill I will die on, we are not yet seeing massive adoption of new AI techniques by attackers. Yes, there are smart innovations, like the one Seriously Risky Business exposed recently about ransomware negotiations. But they’re not using hugely powerful new techniques no one can defend against. We’re not in the first flush of AI: it’s been around as a commodity product for several years. And what are we seeing? For criminals, malicious emails and phishing are still the most common way of illicitly entering an organisation’s network. Why? Becauseit’s a lot cheaper and easier than using frontier AI, partly because of all the things we haven’t fixed.

There is one thing I will push back on strongly on this point. That is people responding to with comments like: “you don’t know what we’re seeing behind the scenes. It’s mindblowing”. I get this all the time when I say out loud that so far, most attackers have not seriously or strategically adjusted their current posture to include new AI hacking capabilities. These responses are pure FUD. I am not in Government anymore, but I do talk to Government cyber experts across the world and they are not briefing privately about invisible actors doing crazy things with AI that no one can stop. Were that happening, these days it’s their duty to warn of these things publicly. So they’re warning that it could happen, because that’s where the capabilities are going. And they’re telling us we need to up our game. That’s fine, sensible and accurate. But one of the things I can vouch for from years doing this in a Five Eyes Government is that while there is all sorts of murky activity going on, there is no massive secret digital battlefield where all sorts of mind-blowing things are going on that are totally invisible to everyone else.

Second, and far more importantly because the first point requires you to take me on trust, but the second one doesn’t – the vast majority of significant cyber attacks are either immediately or ultimately publicly observable. You can’t take out a hospital’s network (as the ransomware thugs often do) without anyone noticing. You can do a data exfil secretly (that’s the point, at least until you’re done) but usually, in the end, someone finds out what’s happened. Yes, there are secret espionage operations that never get discovered or disclosed. But the point is, if we were experiencing an AI cyber security disaster, we’d know about it. And we’re not. We are, however, experiencing a lot of the same things that have been harming us for years. There is not yet any hugely significant change in observable attacker behaviour. And there are very good explanations for that, mostly around economics, intent and incentives.

The Greenwood Declaration – there will be no AI cyber apocalypse

But what if that doesn’t hold? What if their behaviour changes? Well, speaking of Marcus Hutchins, we were enjoying a beer in the Greenwood Pub underneath the NCSC in a sweltering London a few weeks ago. We found ourselves talking about the new wave of cyber FUD. Neither of us consider ourselves infallible, so we were wondering, as one always should – what if we’re wrong?

The problem for people like us is that if we’re wrong, it’s going to be horribly visible. There will be a huge spate of huge AI-powered cyber security disasters. But if we’re right, the corresponding problem doesn’t arise on the other side of the argument. If the AI cyber apocalypse doesn’t happen in 2027, plenty will argue that it’s just round the corner in 2028, or that it was averted by all the warnings, or by the product they sold. Nearly a decade ago, in a controversial but brilliant speech, Ian Levy, then Technical Director of the NCSC under my leadership, likened this play to medieval witchcraft. He was right then, and we could be seeing the new AI powered version a decade later. We’ve seen this movie before. So many times.

So at the risk of career ruination in 2027, and speaking only for myself, here is a prediction which I’ll call - in jest – the Greenwood Declaration after the pub I first had the confidence to say it out loud in.

We are not heading for an AI cyber security apocalypse.

The central case for 2027 and a bit beyond is a stormier cyberspace with more harm than we’ve had this year and in most years since the 2021-22 explosion of disruptive ransomware (Colonial Pipeline, the Irish healthcare crisis, Kaseya, Costa Rica and so on). It could be a bit worse than that. But, as the Cloud Security Alliance and SANS paper put out just after Mythos Glasswing rightly called it, it’s a storm. And storms pass (as the 2021-22 ransomware one did).

Some storms leave you wet and grumpy. Others destroy your house and village. Whether it’s the benign or malign outcome depends on the severity, which you can’t control, and your preparation, which you can.

Then things are likely to get significantly better if we get our response right. Alan’s piece below is especially important on the lessons for policymakers from the Hugging Face incident.

So, at the risk of repeating the fateful mistake in 1987 of the British weatherman Michael Fish^[2], there is not going to be an AI fuelled digital apocalypse.

The AI cyber capabilities are real. That means frenzied work to improve security is not just justified but necessary. But just because these capabilities exist doesn’t mean they’re going to be easily and widely used by the world’s baddies, and the lesson from Hugging Face is the need to learn how to control them much better.

So a freakout is not justified and harmful.

Stop the FUD. Fix the stuff.

***

Article 2: The Problem Isn’t That AI Wants to Hack You**

It doesn’t want anything. That’s what should worry us.

By Professor Alan Woodward

Earlier this month, a set of OpenAI’s most capable models were running a cybersecurity exercise inside what the company described as an isolated testing environment. Their guardrails had been deliberately loosened, because that is the point of a red-team lab. Then they used stolen credentials, discovered a vulnerability nobody knew existed, and got into the production servers of Hugging Face, a rival AI company. Hugging Face detected an intrusion. It took days to learn where the attacker had come from.

Within hours, the story had a shape: the AI went rogue.

That framing is wrong, and the wrongness is expensive. Nothing broke out of anything. As Hannes Cools, a social scientist at the University of Amsterdam, pointed out, calling this an agent acting on its own is anthropomorphization that conveniently shifts blame away from the people who built the lab. “It is a human decision to switch off specific safeguards,” he said. The models were told to find complex attack paths through a computer system. They found one. It led somewhere nobody had thought to fence off.

Let me concede the part that skeptics usually fumble. These systems are formidable. When Anthropic announced its Mythos model in April, it reported that the model had autonomously identified and exploited previously unknown vulnerabilities across every major operating system and browser, and chained them into working exploit sequences. That is real, it is verified by people outside the company, and anyone telling you frontier models are just fancy autocomplete has not been paying attention. Capability is not the thing being oversold.

What is being oversold is intent.

Read the fine print of the very reports that generated the panic. When Anthropic disclosed last year that its models had been used in an espionage campaign, the details undercut the headline. A human operator picked the targets. Of roughly thirty attempts, a handful worked. And the model lied — it fabricated credentials it had never obtained and routinely inflated its own findings, to the point that the attackers would have had to check its homework. Anthropic’s own authors conceded this was an obstacle to genuinely autonomous attacks. A system that misreports its results to the people running it is not a mastermind. It is an unreliable subcontractor with root access.

This is the distinction the public conversation keeps collapsing. We are not facing an adversary that understands what it is doing and chooses to harm us. We are facing an optimizer that has no model of what we meant, only of what we measured. Machine learning researchers have documented this for a decade under the unglamorous name of specification gaming. Given a boat race and a reward for collecting points, a system learns to spin in a circle hitting the same three targets forever rather than finish the course. It is not cheating. It has no concept of cheating. It found the shortest path to the number you asked it to make bigger.

Now give that same disposition a network, a credential store, and an instruction to find complex attack paths.

Why does the diagnosis matter? Because the two stories point at completely different remedies, and we are currently buying the wrong ones. If the threat is a hostile intelligence, the response is to make the model itself more refusing — better guardrails, more alignment training, a smarter conscience baked into the weights. That is where enormous effort is going. But guardrails are a property of the model, and the Hugging Face incident happened precisely because a company chose to turn them down, in an environment it wrongly believed was sealed. One security researcher, Jake Williams of IANS Research, called the isolation claim “either a cop out or a marketing strategy,” and suspected a straightforward control failure in the lab.

If instead the threat is a powerful, literal-minded system that will reach whatever goal you name by whatever route exists, the response looks like ordinary engineering discipline: real network segmentation, credentials that expire and can be revoked, permissions scoped so tightly that a runaway process cannot reach anything that matters, and logging good enough that you learn about the breach from your own telemetry rather than from the victim. Boring work. Nobody announces it at a keynote.

There is a policy gap here that deserves more attention than it is getting. Almost every AI rule now under discussion governs how models are released to the public. Very little of it touches what the AI companies do with their own systems internally — which is exactly where this month’s failure occurred. The most dangerous models in the world spend their early lives inside the labs that built them, subject to whatever containment that lab decided was sufficient. We regulate the loading dock and leave the factory floor to the honor system.

None of this makes the alarm-raisers villains, and I am not persuaded by the cynical read that these disclosures are simply marketing. Publishing that your own model broke into a competitor’s servers is a strange way to sell a product.

But the alarm is pointed at the wrong thing. The fear on offer is of a machine that understands us well enough to want our downfall. The reality is a machine that does not understand us at all, cannot tell us reliably what it has done, and will take any door we forget to lock.

We keep handing it the keys and then acting surprised about which door it opened.

^[1] _{For the happily uninitiated, FUD stands for Fear, Uncertainty and Doubt and is a pejorative term long used in the cyber security community for apocalyptic warnings about cybergeddon used primarily as sales strategies.}

_{[2] Again for the uninitiated, Michael Fish was a much-loved British weatherman of the 1980s. Close to retirement, he said on live TV that he’d received a call from a woman saying a hurricane was on the way. He told everyone not to worry. You can guess what happened next.}

Share

Introduction, apology, caveat, and then another apology

The introduction: For nearly three months, the British Library has been close to unusable because of what has invariably been called "a cyber incident”. Lots of people have asked me in recent months: “what on earth is going on with the BL and why isn’t it getting more attention?”

At the start of this week, the BL announced the partial restoration of its capabilities. So it seems a good time to take stock of one of the most impactful cyber incidents in British history.

The apology: This is the first post on this Substack for, well, a very long time. Apologies to those who supported me when it started. Many of you will know there were various circumstances which meant 2023 was a hard year for me to sustain it. I won’t make any promises about how often I will post, but I will try to reactivate it, based on an article at least once a month. Consider this January’s offering. And feedback on content and ideas for more is always welcome.

The caveat: This post is based open-source information, my own judgments, and nothing else. I used to run the UK’s National Cyber Security Centre, but I stopped doing that in 2020 and left public service. I have not asked former colleagues about the case. Nor have I spoken to the British Library. No one should assume anything I say here reflects the position of the Government or any part of it.

The other apology. Pursuant to that caveat, I am acutely conscious that this is an article about an extremely hard-pressed organisation trying its best to serve its users under the most extraordinary pressure. Being at the centre of a cyber crisis is absolutely horrible. It normally also means something has gone wrong, somewhere. In commenting on some of those potential causes of the problems, I do not mean to criticise those working flat out to fix things. I apologise if any of this post inadvertently comes across that way. Indeed I’d want to thank BL staff for what appears to have been an extraordinarily effort in a long slog to get to this important recovery point this week. I would encourage anyone else commenting on this or other cyber incidents to remember the human beings at the centre of the crisis. A paper from the Royal United Services Institute this week rightly identified psychological damage to staff as a consequence of these types of attacks. We should always remember this.

What happened at the British Library?

To the issue at hand, and first, some facts. In early January, Alex Scroxton at the indispensable Computer Weekly wrote a superb overview of the British Library cyber incident. In the interests of brevity, the following points are the most important:

• on the last weekend of October, the British Library fell victim to what it called, inevitably, a ‘cyber incident’, acknowledging disruption to its services;

• serious disruption to services continued throughout November, with all the hallmarks of a ransomware attack (for the uninitiated, this is when a hacker locks you out of your network and demands payment to let you back in, normally via cryptocurrency);

• towards the end of November the fact that this was ransomware was confirmed. A new(ish) criminal group calling itself Rhysidia claimed the attack on their (so-called dark) webpage. In doing so they confirmed this event was also what is known as ‘double extortion ransomware’; that is when the demand for payment to decrypt the network is accompanied by a threat to release stolen data from the network, or sell it to other criminals, if the ransom isn’t paid;

• Rhysidia listed the ransom and the price of the stolen data set at 20 bitcoin. At the time, this was worth about £600,000. With presumably no ransom paid, and presumably no buyer (the data is worth far less than £600K to a criminal; the awful bluff that is data extortion is a subject for another day) Rhysidia then dumped 573GB of British Library corporate data, including staff details, onto the dark web;

• as 2023 gave way to 2024, the costs of the crisis both to the BL and its users became more and more apparent, as the disruption continued. The Financial Times reported that the BL would have to burn through nearly half its reserves to cover the costs, which, at an estimated £6m-£7m, were some ten times the demanded ransom. Meanwhile, The Guardian reported on the plight of authors who missed out on valuable royalties payable when their books were borrowed. Media coverage of what was a disaster for academic research went global;

• the BL this week announced a partial restoration of the main catalogue, but in read-only, and therefore much less useful, form. So the crisis continues, but this is a significant mitigation for users.

Two key points flow from these events. The first is that it can safely be inferred that neither the BL nor anyone else paid the ransom (though no one has, to my knowledge, commented officially on this). If the ransom had been paid but the criminals had failed, for whatever reason, to restore access to the BL’s network we would know about that by now, one way or the other.

The second, and most important part of the whole story, is that for more than two and a half months this vital national resource has been essentially unusable. At the start of the crisis it seems that nothing at all worked: the basic staff computers, the phones, and even the public Wi-Fi for a bit. But the longer term damage was caused by the total inaccessibility of the main BL catalogue, described by the BL’s boss itself as “one of the most important datasets for researchers around the world” with its record of some 170 million items dating back centuries.

A particular problem is understood to be that most of the collection is stored in a giant facility belonging to the BL in West Yorkshire. Users are supposed to order from the catalogue and the item will be transported south in a few days. Without the catalogue, this became impossible. Whilst some workarounds could be done in the BL’s magnificent London headquarters, if the text you wanted was in Yorkshire, and it probably was, no one had any way of knowing where it was, and how to get it.

Although plenty of people have asked why this episode hasn’t received more national attention, it is clearly one of the worst cyber incidents in British history. So what are the lessons of it?

For me, there are three. None are new, but not all of them receive enough attention. And the last one needs to resonate thunderously throughout all organisations.

Lesson 1: The perpetrators are in Russia. They will likely never appear in a British court. We have to work within this reality

The British Library cyber crisis has nothing and everything to do with geopolitics. Nothing, in that the only motivation for it is money. Everything, in that the only reason it can happen with impunity is because the Rhysidia group, like nearly all the major ransomware groups, are based in Russia.

It is well documented that the Russian state has no interest in shutting these groups down and putting the leaders in prison, providing they don’t harm Russian interests and cooperate with the state when required. It is against current Russia law for the state to extradite its own citizens (this must be the first time I’ve linked to Tass). So these people are almost certain never to appear in a British court, and very unlikely to face even a Russian one anytime soon.

But, like all comparable democracies, the British state is configured to treat this type of incident as an arrestable and prosecutable crime. “Cyber crime is just the same as other crime” is something I heard a lot from law enforcement colleagues in Government. But there is one crucial difference. For the first time in human history, it is possible to inflict sustained, large-scale criminal damage on another country without the perpetrator or a single accomplice setting foot in it.

We have consistently underestimated just how much cyber crime breaks our model of policing. In rule of law democracies, the contract between citizen and police is based in part on an assumption that when someone is a victim of crime, the police will pursue the perpetrator. And with cybercrime, there are some in the UK we can go after. And with the Russians, every so often some idiotic cyber criminal goes on holiday to a Western country, or contracts for a criminal service with someone in East London, and the police can do what police are supposed to do. But these are the exceptions.

What police forces are doing increasingly well - normally via multinational operations led by the FBI - is orchestrating takedowns of digital infrastructure used by the criminals. But these interventions, while welcome, are invariably whack-a-mole operations and the criminals reappear in another guise with new infrastructure.

Can anything be done? Things got so bad in 2021, with the attack on Colonial Pipeline in the US, alongside serious healthcare disruption in the US and Europe, that President Biden used his Geneva summit with Vladimir Putin in June of that year to demand Russia clamp down on the rampant ransomware crime emanating from its territory. For a brief period, this seemed to have some effect, with the somewhat theatrically broadcast arrest of the REvil gang, one of the most notorious groups.

But then came the invasion of Ukraine. A dictatorship willing to defy the White House over the invasion of a neighbour is unlikely to be swayed by American demands about criminals on its own territory. And a West that would support Ukraine but not take direct military action on its behalf is not going to take direct action against individuals protected within Russia’s vast borders. Both the Russian state and the criminals know that.

Therefore, the brief period when some of Russia’s ransomware thugs flew a bit too close to the sun and became a nuisance to the Kremlin is now over. All the evidence of 2023 suggests that the criminal safe haven has been fully restored. There will come a time in the future when Washington, London, Brussels and others can talk to Moscow about dealing with this scourge. But that time is not now, or soon.

It is of no benefit to pretend otherwise. Australia’s otherwise hugely impressive response to the disastrous theft by cyber criminals of more than a third of the population’s medical records - what I’ve called elsewhere (£) a masterclass in devaluing a stolen dataset to the criminal - provides a case in point. In a press conference in November 2022, the head of the Australian Federal Police claimed the identities of the hackers were known to the AFP and pledged to bring the perpetrators to justice in Canberra via cooperation with Russian law enforcement. Any reasonable Australian watching could have concluded the police thought they had a good chance of locking up the villains. But, as was widely predicted at the time, this has not happened, and there appears next to no chance that it ever will.

It is always hard for Governments and public authorities to admit they can’t do something, especially when the ‘thing’ is being able to catch and convict criminals who’ve laid waste to something that’s very important to lots of citizens. But the lesson from Australia, the British Library, and countless other ransomware crises is that normal policing doesn’t work in most of these cases because the suspects are safely holed up in Russia.

So we should stop pretending that conventional policing can do much about this, and look instead at other things we might be able to do. This article is long enough already without prescribing in detail what the approach should be: that is for another day. However, here are three starting points:

• in the short term at least, serious policy needs to eschew basing our strategy on fantasies of “striking back” or “imposing costs” on criminals who just want to make money and currently shelter in the world’s largest safe house. Impose costs when we can: there are things we can do to harass and harry cyber criminals. But this will not be a strategic solution for as long as the Russia safe haven exists;

• the question of ransom policy and law cannot be avoided forever. The UK’s de facto position is that state bodies like the British Library will never pay, but private entities can, with no questions asked (even if the Government pretends to discourage them). This stands in marked contrast to Britain’s uncompromising approach to terrorist kidnappings, where ransoms are never paid, whatever the (sometimes terrible) consequences. But the Government has yet to publish any analysis or evidence as to why it takes a hardline approach for kidnaps and a soft one for cybercrime. Indeed, it doesn’t really have a cyber ransom policy at all (a detailed look at the ransom policy question is an issue for another article, but policymakers must take a hard look at it);

• nor does the state really have a counter-ransomware strategy. A rich seam of possible policy measures to explore has been provided by Parliament’s Joint Committee on National Security Strategy report of December 2023. The Government could do worse than start there.

Many of the reforms in that report have merit and deserve consideration. But the Committee’s overarching point is that countering ransomware needs serious political leadership and attention.

Lesson 2: The BL case perfectly exemplifies the sort of area where the UK is most vulnerable to cyber disruption

That sort of strategic review of our approach to ransomware requires us to look hard at our own national vulnerabilities. Here the BL crisis provides some valuable lessons.

Harm happens in cyberspace because we have a three decades-long legacy of weak security in our software, hardware and wider digital infrastructure. Famously, the Internet was not built with security in mind - and we’re plagued with poor incentives for providers and users to do anything about it. That is slowly changing, but it is improving much more for newer technologies more than for our existing tech stack.

As all IT security professionals know, legacy systems in old organisations pose the hardest problems. There are no really transformative options until new systems come along. There are only mitigations. These mitigations require a lot of high quality technical and human resources. So they are expensive. They also require a lot of skilled people, as well as management attention and sponsorship. But it’s hard to explain the benefits of these measures to hard-pressed management facing many other pressures. And security reforms are often unpopular with staff and users because they add complexity to everyday work.

So it’s easy to see why some organisations are incentivised to take cyber security and resilience seriously, and some aren’t. Any service where public safety is at risk will invest heavily in security, safety and resilience, and test it all the time. The system and the organisation will probably be inspected. A regulatory license to operate might well depend on that evaluation. Put simply, no one should ever do something where their physical safety is dependent only on a computer staying connected, and most regulatory systems rightly don’t allow this.

So, for example, when part of the UK’s National Air Traffic Control system failed (accidentally) last August, there was no risk to safety to the planes already in the air because of the way much-tested backups work. Because the air traffic control system is such an obvious part of critical national infrastructure it is highly likely that Government agencies will pay attention to and assist with the cyber protection and resilience of the service. Similarly, in the private sector, banks invest heavily in cyber security capabilities and people because they know the risks of large scale financial loss are existential. Moreover, they can afford to. And the Government and regulator will want to help too, to avoid systemic risk within the financial system.

But consider the British Library in this context. It is a very important national institution, for sure. But if you’re tasked with identifying the most important national IT networks for protection against attack, the British Library will not get anywhere near the top of the list for attention. As we have seen, no one gets hurt or dies if the BL goes down. The health service will still function. So will the banks. The lights will still be on. People’s bills will still be accurate. The data of vulnerable populations will not have leaked. And so on.

As a cultural institution, the BL is important and famous. It also a public body. It is not, however, a political or budgetary priority. Constrained by public sector budgets and salaries, it will find it hard to source the people and capabilities it needs for cyber security (the British Treasury was widely mocked for advertising for a head of cyber security with an annual salary of between £51,000 and £57,000 when the industry standard is multiples of that figure). It is hard to imagine the BL being able to pay more, or finding it easy to recruit cyber security professionals.

This matters, because it is within hundreds of networks like the one the BL depended on that serious national risk lies.

The history of cyber security is pockmarked with warnings of mass casualty digital apocalypses threatening civilisation as we know it. It turns out that’s the wrong problem: as the brilliant work of Lennart Maschmeyer has shown, hacking into, say, a power grid and depriving civilians of supply even for a short time via cyber means is possible, but it is painfully slow and hugely resource intensive for the aggressor. Moreover, for cyber security and other security reasons these systems are better protected than ‘normal business’ networks, and have manual or other backups. That’s why cyber attacks don’t directly kill people.

It turns out, however, that our more immediate cyber security problem is that by crippling these so-called ‘normal business’ networks an aggressor can hugely harm a society without that much effort. We now know you can shut down a crucial oil pipeline in the United States not by attacking the pipeline, but by shutting down the ordinary software systems that support its administration. It turns out you can cripple the entire healthcare system of a rich EU nation not by touching hospital equipment or systems but by locking out the network of the body that allocates doctors appointments and schedules surgeries. And it turns out that you can bring part of the British academic sector to a crashing halt by taking a massive library catalogue offline.

So what else can an aggressor do to networks that don’t look to be of ‘strategic’ importance? That is that question we should be asking ourselves in the light of the BL fiasco. We should then be moving resources, expertise and monitoring accordingly as best we can. We also need to think about how we better incentivise the leaders of these organisations to improve basic security and resilience, because ransomware attacks are not, in general, sophisticated.

This is an election year in the UK, and after the votes are counted we can expect someone to try to form a stable administration with a five year horizon. Much is made of short-termism in politics, but we have to work with the world as it is, not as we’d like to be. In that spirit, here are two planning assumptions on national cyber risk for the next five-year Parliament:

1. a devastating, highly sophisticated, threat-to-life cyber attack against the UK in the next five years is unlikely, and if it happens, its impact will be mitigated so long as we continue to ensure that safety-critical systems are not wholly dependent on computer networks;

2. by way of contrast, serious economic and social disruption, including an incident that could threaten public order or safety arising from a cyber operation (the disruption of healthcare administration, the criminal justice system, or food or oil distribution being some examples) is very likely. Indeed, an incident of the severity of the BL attack is likely in each of the next five years.

This lesson of national vulnerability from the BL case, and these assumptions, would make a good starting point for the sort of serious discussion about ransomware that is urgently needed.

Lesson 3: Organisations, whether public or private, must be able to recover far more quickly than the BL did

And the one thing above all else that would make a difference to the problem is finding a way of forcing organisations to be able to recover more quickly than the British Library did.

To understand why, we have divert back briefly to ransoms. As noted earlier, the British state doesn’t pay ransoms, and most other Governments don’t. Throughout this crisis the Government did not come under any serious pressure to pay (unlike, for example, the Irish Government during the healthcare cyber crisis of 2021 because of the huge impact on health services).

The private sector is another matter. Because there is no reporting requirement in most jurisdictions, including the UK, to report when a ransom has been paid, there are no reliable figures for how many organisations pay (the cyber security company Coveware has made as decent a fist as any of tracking trends over time, and the latest figures show a significant decrease to fewer than half of organisations in 2022, down from seven out of every eight a few years earlier).

The blunt reason why the private sector often pays, but governments hardly ever do, is that Governments can throw far more resources and support at recovery. That was certainly the case in Ireland, where the military and a number of major cyber security companies were deployed with no expense spared. Private companies cannot afford to surge in capabilities like this, and they can’t call in the Army. And unlike the state, they can go bankrupt.

So for private organisations, paying can be more effective than not paying. In this case, the cost to the BL was far more than the ransom. This is not always the case: a BBC File on Four documentary in 2021 tracked the impressive response of the Harris Federation of London, a major schools provider. They held their nerve and the overall cost to them was less than the ransom demanded. And paying does not mean avoidance of harm: Colonial Pipeline paid the ransom, but the pipeline was still out for several days.

But Governments do not want to pay ransoms and, certainly in Britain, it is unlikely that taxpayers want them to. The crucial point is that not paying the ransom only works if the organisation can recover quickly.

The heroic efforts of Irish healthcare workers, and IT professionals from the civil service, the military and the private sector got the system back up and running to some sort of basically acceptable level in a similar amount of time as it takes a victim who paid to recover. Similarly, the Joint Committee on National Security Strategy heard from the leader of Redcar and Cleveland about how staff from the National Cyber Security Centre slept in the council’s offices during a ransomware crisis to ensure that the system dealing with the cases of at-risk children were recovered quickly.

Moreover, we need to ask ourselves: what if there is no ransom? What if a hostile hacker working for a nation state does exactly the same thing as a ransomware attacker, but the objective is to damage the UK by destroying the network, rather than to extort money by temporarily locking it?

In such a scenario, recovery is the only option. Ransomware highlights our digital vulnerabilities to others who have motives even worse and more strategically damaging than the criminals. And if there is no effective system backup that can easily be deployed, or no way of restoring the old system in some way, - in other words, if there’s no way of recovering quickly - then we’re stuffed. Recovery capability is paramount for national security.

Here, the obvious point to make about the British Library is that it has taken - and is taking - an inordinately long time for its catalogue, one of its most important services, to be restored. That’s even, presumably, with help from Government experts and others. Of all the high profile ransomware cases throughout the world, it is hard to think of many that have dragged on for this long with this degree of severity.

This slowness to recover is the most painful and most important lesson from the British Library cyber incident.

There are, no doubt, very good specific reasons for it. A 170 million item catalogue is bound to be very complicated. A replica backup would no doubt be very expensive and hard to maintain. (And, as stated at the start, this analysis implies no criticism of those working round the clock and over Christmas to try to get services back up and running; it is impossible to retrofit a solution that did not exist before the crisis).

But faced with the likelihood and potency of this threat to myriad public and private entities, we must no longer accept a situation where important national organisations, public or private, cannot withstand the lost of their enterprise computer network for such a long period of time. If we tolerate this, the likely consequences in terms of economic and social disruption will prove intolerable. Planning for the loss of a key network, and being able to recover quickly from it, needs to be a core part of good public and corporate governance that every organisation models and practices.

The way to get to this point is not to indulge in the classic British tradition of holding a what-went-wrong-and-who-can-we-hang-out-to-dry inquiry. This is not the Post Office IT scandal. There is not a single allegation of malice, bad faith or wilful negligence. Instead, an organisation with a reputation for being well-run and held in high public esteem found itself without the systems and plans in place to recover from being the victims of criminals. They deserve sympathy and support.

But we have to figure out why. What constraints were there, (and what incentives weren’t), that prevented this otherwise capable organisation from protecting itself and recovering quickly? Where else is this a risk? And what can be done about it?

The American answer to this conundrum has been to establish a Cyber Safety Review Board, based on the successful model in aviation safety. The aim is not to hunt for blame but to look at the rational explanations for why things went wrong and make constructive recommendations to address them. Such an approach could work in this case. The last thing we need are hours of theatrical hearings in a courtroom or committee room of Parliament, with exhausted witnesses defensive and humiliated. That makes for good TV and terrible public policy.

Summary

The UK has, by and large, suffered less major harm from ransomware than most comparable nations. But the British Library case is a warning. The critical lessons of it are:

• ransomware is now a national security issue, likely to cause significant and possibly dangerous disruption in the near future;

• this requires a strategic national response, which, due to the Russia safe haven problem, has to be predicated on policing not being able to do what it normally does to deter and punish crime;

• there are a bunch of useful policy mitigations on the table, which need to be brought together in a coherent way. This has to include a thought-through, publicly articulated posture on ransoms;

• but once an attack gets through, the one thing that matters above all else is the ability of the victim organisation to recover quickly. All organisations, whether public or private, need to test their ability to withstand the loss of a key network and show that they can recover at least partially within an acceptable amount of time.

This work is not easy. But it is vital, and urgent. And it is doable, with the right focus and leadership. Otherwise, in the well-chosen title of the Parliamentary report, national security is a hostage to fortune.

Share

Subscribe now

Share

Subscribe now

Recap

This is the third in a five part series looking at the vast and varied implications of two major data breaches in Australia in the period September to November 2022.

The first post, which can be found here, covered the background to the two breaches at Optus, a major telecoms company, and Medibank, a health insurer which covers nearly 10 million of Australia’s 26 million population. It also discussed how our failure to distinguish between the severity of different types of data breach has bred a degree of complacency about data security.

The second post on the lessons of these incidents for Governments and their role in cyber security can be found here.

This third post analyses the issue of the the payment of ransoms.

The remaining posts in this series will cover:

• how we discuss and report cyber harms, and why that matters (Part 4);

• the safe haven problem in cyber crime (Part 5).

Lesson 3: sorry, but we still really need to talk about ransoms…

…whether it makes sense to pay, and, yes, whether it should be allowed

The Optus and Medibank data heists were accompanied by initial ransom demands of around ten million dollars.

Ten million dollars.

Yes, it was probably an opening pitch in a negotiation.

Yes, the Optus demand was apparently dropped, and Medibank, to their credit, didn’t pay.

But the fact that the demand was credibly set at ten million dollars shows we have normalised ransom payments, big and small.

In neither of these attacks was the actual operation of the company affected. In that respect, neither attack was even ‘proper’ ransomware as we have traditionally understood it.

These were threats to publish data, not cripple the operations of a company. Serious situations, for sure, particularly in the case of Medibank, given the sensitivity of personal medical records. But they weren’t the equivalent of, for example, the crisis in Irish healthcare in 2021, when the national system for allocating healthcare stopped working, with devastating results for patients.

And yet the demand for such a huge sum seemed normal.

That’s because it was.

Ransoms are the oxygen of cyber crime.

They have become the source of income for some of the most effective cyber criminals. Ransomware earnings go far beyond the potential for monetising stolen data on the dark web, the original digital age criminal business model.

In 2021 the British firm Elliptic traced $90 million of ransom payments to just one group - the Darkside group - in a nine month period. In December last year, a joint advisory from the FBI and the Cybersecurity and Infrastructure Security Agency said that the Cuba ransomware gang has extorted its way to more than $60 million, out of £145 million demanded from more than one hundred organisations.

Chris Krebs, CISA’s founding head, has set out three reasons why ransomware has emerged as the pre-eminent cyber threat of our our times. First, there is endemically weak security in Western organisations. Second, Russia (and some other countries, but primarily Russia) provide a safe haven for cyber criminals. Third, the criminals’ business model works.

Ransoms, along with low operating costs and low barriers to entry, are central to the criminal business model. And part of making that business model work (and these are my words, not those of Mr Krebs) is what I call a ‘pro-criminal narrative’ around ransoms. In other words, too many times organisations are told, through a variety of means, that the easiest or indeed only way out of the lonely and difficult crisis they face is to pay.

So victims do pay, and the vicious circle continues.

You can take the view that there’s nothing to be done about this (give or take a bit of chasing after cryptocurrency payments). And you may be right. It’s a hard problem.

But perhaps a better starting point is to accept that allowing mostly Russian-based computer thugs to extort payments on this scale from law abiding companies with impunity on an ongoing basis represents an epic failure of global public policy.

We might want to think harder about whether there is something we can do before giving up.

If, as the last piece in this series argued, cyber security is a much a policy, economic, social and legal problem as much as a computer network security one, then this is one of the hardest, but most important, policy challenges of all.

So we need to talk about ransoms. And then do some serious policy work.

Why ransomware works

Why does ransomware work for the criminals? Let’s take a look.

Below is the start of a statement by the meat giant JBS USA in June 2021. It is the main publicly available account from the company explaining why they paid $11 million - eleven million dollars - to the REvil group, one of the main Russian ransomware groups.

It merits careful reading.

The opening of JBS’s explanation of its 2021 decision to pay $11m in ransom

There are essentially two different problems to which paying a ransom is seen as the answer.

The first is to get systems that are not working to work by decrypting them. That is how ransomware has conventionally worked: the victim’s system doesn’t work as users are locked out, so the organisation cannot function. It must therefore buy a decryptor key to get back in to the network.

But in this case, JBS said that “at the time of payment, the vast majority of the company’s facilities were operational”.

The second reason, increasingly common (often alongside the first, though in the Medibank case it appeared on its own) is to prevent the publication of sensitive data. Essentially it’s “give us the money or we publish all your data”.

But in this case, the company says that the payment was made before data was exfiltrated, and indeed cites preventing the exfiltration (not publication) of data as a reason for paying.

The justification for paying eleven million dollars is “to mitigate any unforeseen issues” and “prevent potential harm to our customers”.

To be clear, JBS were perfectly entitled to take this entirely lawful decision. And, it may well have made sense to the company’s leaders for reasons they chose not to disclose. But, based only on the publicly available information the decision makes no sense (as a handful of people tried to point out at the time).

Moreover, note the intriguing reference to the advice of internal and third-party cybersecurity experts. Despite - if we take the company’s statement at face value - the absence of either disruption or data loss - the advice was to pay to make the problem go away. Again, this may have been perfectly reasonable to those with full knowledge of the case, though - again - it is not justified by the presentation of the facts alternatively.

But it is a powerful example of just how strong the incentives to pay are.

This is what I mean by a pro-criminal narrative.

If what happened at JBS happens at scale, continuously, then we’re stuffed.

JBS is perhaps an extreme example: in most of the known cases where victims pay there is a clear and obvious reason to pay. It’s either because the network is locked, or the criminals are threatening to publish data.

Still, it is apparent that in far too many cases the answer - however the advice is gathered and the decision is taken - is to pay.

Research presented today by at the SANS/UK National Cyber Security Centre Cyber Threat conference in London by Ifigenia Lella of the European Union’s cyber security agency ENISA disclosed that some 60 per cent of organisations paid the ransom in more than 600 cases analysed by the agency. This is the latest among a great deal of research about the prevalence of paying.

It is understandable. Victims of serious crime - which is generally committed from jurisdictions beyond the reach of domestic law enforcement - will want a way out of the problem as quickly and efficiently as possible. No one likes paying criminals, and no one should blame desperate victims for doing so. That’s especially the case when they’re told that the way out is to pay.

But this leaves us with the classic public policy problem where individual and collective interests collide.

What might make sense for individual entities is a recipe for serious collective harm.

That is a failure of public policy.

So are there different ways of doing things?

It doesn’t always have to be like this, and sometimes it isn’t

Not every organisation pays, and not every organisation suffers critical harm when they refuse to pay.

So it’s worth analysing some of the better known cases where victims chose not to pay, to see what we can learn. Here are three.

The Health Service Executive in Ireland, 2021

The first is the Irish healthcare system. Crippled by Conti ransomware in 2021, the Irish state faced numerous calls to pay in the light of the ‘double extortion’ problem - the severe damage to the provision of healthcare caused by being locked out of the system, combined with the threat of the disclosure of population level personal health data.

Eventually, the hackers backed down and provided the (only partially functional) decryptor key for free. (Why they did that is unknowable. Some ransomware operators steer clear of healthcare and abandon the operation when it becomes clear they’ve done over a hospital. In this case, speculation was rife that either the Kremlin and/or other ransomware operators were furious with Conti for engaging in such a venal attack so publicly, thus shining a light on this often quietly lucrative criminal enterprise).

What is interesting is not so much why the criminals backed down, but why the Irish Government held firm.

Even with the national healthcare system in crisis, the Government of the Irish State had to take the totality of the national interest into account. Had the decision been left solely to the affected body, the Health Services Executive, the incentives would surely have pointed towards paying. That’s why American hospitals pay all the time - their duty is only to their patients and their owners. But in Ireland, the decision was taken at whole of Government level, because the affected body was a public authority. The Irish State took the view that letting the world’s cyber criminals know that the Irish Government paid up carried even greater risks overall than those they were already facing with the crisis in healthcare.

Moreover, the full resources of the state could be directed at pace to support the stricken service. Irish Defence Force experts and American commercial expertise were swiftly brought in as the state threw the kitchen sink at the problem.

In the 2020s it has become increasingly common to talk of ransomware as a national security threat, at least when it comes to operations against critical infrastructure. But for the most part, Governments leave the absolutely critical decision on whether or not to pay entirely to the affected organisation. Colonial Pipeline is a good example of that; the American company shut down its pipeline which provides much of the East Coast’s gasoline and paid the ransom, leaving the federal Government to invoke emergency procedures to ensure supply, but the recovery up to the company.

Letting companies pay and leaving them to recover is, of course, entirely consistent with their status as private entities. But, when the incident has consequences which could be regarded as meeting the threshold of national security - as it surely is when energy supplies or healthcare is compromised - it is not consistent with treating ransomware as a national security threat. It is subcontracting one of the most critical decisions - whether or not to pay the attacker - to a private entity.

Put it this way, if someone bombed a pipeline, it would be seen as a national security issue for the Government to lead on. Why is shutting down the pipeline via a cyber operation different? If a hospital is stopped from functioning, should it really depend on how it happened to determine who is responsible for the response?

Ireland is an easy an example to focus on because the victim organisation was a public authority and therefore the Health Services Executive had no choice but to follow the decision of the Government not to pay. It is obviously much more complicated when the victims is a private company, even if it is providing a vital public service.

But Governments should think of the problem like this. If the decision to pay is left entirely in the hands of a single private entity, expect that entity to consider only its own interests, imperatives and duties. Do not expect the private company - voluntarily, without legal compulsion - to evaluate the wider public interest at the expense of its own priorities. That would be wholly unreasonable.

In other words, if policy doesn’t change, expect your critical national infrastructure companies to pay ransoms, at least some of the time.

And, consequently, expect the criminals to come back for more.

And, consequently, expect more critical services disruption from ransomware.

The Harris Federation of Schools in London, 2021

A different example involves the Harris Federation of some 50 London schools, ransomwared in April 2021. It, too, is instructive, but for a different reason. And a fascinating overview of it, courtesy of BBC Radio, can be found here (a 30 minute listen).

Throughout the attack, the Federation’s schools remained largely operational, though there were some impacts (for example, at one school, Internet-controlled gates would not open and had to be manually over-ridden). But the organisation was nonetheless badly affected, being unable to pay invoices, for example.

The initial ransom demand came in at $4million. This was completely unaffordable for Harris, which is a charity. Professional negotiators managed to get the demand down to south of $1 million.

But the organisation still decided not to pay. It looked coolly at the situation. Its main educational mission was continuing because that did not depend completely on the network. The computer network was already very badly damaged and would take time and money to recover, even with a decryptor key. Harris were cognisant of advice that not all decryptor keys work perfectly, and about 5 per cent don’t work at all. They knew there were other ways to recover.

So they held firm. Eventually, they calculated the cost of recovery at around $600,000; cheaper than the final ransom demand of $750,000.

It is not always cheaper and easier to pay.

The Medibank breach in Australia

This brings us to the Medibank case in Australia.

It is not hard to see that paying the ransom looked like an attractive option to Medibanks’s leaders, given public concern at the potential exposure of nearly ten million private medical records.

But, as the company presumably realised, extortion based on denying the availability of service and extortion based on the breaching the confidentiality of data are two completely different things. In the case of the former, the attacker can continue to lock out the victim, causing ongoing disruption. And the attacker has a clear ‘product’ to sell: the decryptor key. So there is a clear and obvious transaction to make.

In the case of extortion based on threatening to breach confidentiality through a data dump, this is not the case. Rather that a ‘positive’ action in return for money - the provision of the key - the victim is attempting to pay for a ‘negative’ action: not disclosing data, in perpetuity. ‘Guarantees’ that stolen data has been deleted cannot be verified.

As the cyber security firm Coveware put it in a 2020 study showing that some victims who had paid had seen either data released anyway, or had a further extortion demand levied later on:

“once a victim receives a decryption key, it can’t be taken away and does not degrade with time. With stolen data, a threat actor can return for a second payment at any point in the future.”

Furthermore, the dynamic is different in such cases because the network is still functional. Medibank may have been plunged into a serious crisis but, unlike the Irish Health Services Executive, its ability to arrange healthcare for its customers was unaffected.

Finally, as ever, the victim and wider society has agency in how it responds. As the next post in this series will explore, Australia handled the disclosure of medical data as threatened by the attacker deftly. A grown up conversation between the Government, the company, the media and socia media ensured responsible reporting of the leak, containing the harm to individuals and blunting the severity of the disclosure threat from the criminals. In both Ireland and Australia, public fears that personal health data would be easily available did not materialise.

Three cases, two lessons

Not for the first time in cyber security, details matter. Setting even legality and ethics aside, and focussing just on the practicalities and business realities of the situation, in both the Harris Federation and Medibank cases, it made sense not to pay the ransom.

Two separate questions arise from these three cases:

• Question 1: the law. What should the law be around paying ransoms? Should payments remain legal, and who should take the decisions on paying?

• Question 2: the narrative. Assuming that, at least for now, paying ransoms remains legal, how can we reduce the incentives to pay?

Question 1: The law

Whether or ransomware payments should be prohibited has been one of the most vexed issues in contemporary cyber security policy.

The Ransomware Task force - a US led group of international experts spanning the public and private sectors - conspicuously failed to agree on this topic amidst a welter of unanimous, useful recommendations. This is not to criticise them: it’s an enormously complex and contentious subject, and to their considerable credit they set out (at page 49 of their report) a balanced account of the debate on both sides. Indeed this published, accessible, and reasonable analysis, based on expert input, is more than any Government I know of has produced on the subject.

The UK Government has said nothing discernible at Ministerial level on the subject. This is in marked contrast to the British state’s robust, two-decade long policy on banning ransom payments to designated terrorist organisations for kidnaps via primary legislation with extra-territorial effect. Terrorism and cyber extortion are two very different issues, but the UK Government has not said why it takes such different approaches to the two when it comes to ransoms.

The US’s position appears to be that ransom payments are legal, unless they are paid to groups designated under the Office of Financial Assets Control (OFAC). Some individual hackers and well-known cyber threat groups are designated under OFAC, and so in theory paying them is illegal. However, many cyber criminal groups are not covered. And for those that are, extensive mitigations are set out in the US Government’s guidance. Moreover, as Recorded Future have set out, knowing whether or not criminals demanding ransoms are on a sanctions list is beyond many smaller organisations. RF’s report estimated that 15 per cent of payers in 2020 may well have breached US Treasury rules by paying ransoms. So it is hard to see the logic of these rules, or how they can effectively be enforced.

For most of the rest of the western world, the policy on ransom payments is not to have a policy. More than that, it’s not to even think about having a policy.

It’s a hard question, for sure. So it’s understandable that Governments want to duck it. But if they do, they should expect nothing to change.

In July 2021, the American cyber security expert Tarah Wheeler and I tried to set out an analytical framework for thinking about the law on ransoms from initially opposing perspectives (she instinctively opposing prohibition of payments, me instinctively favouring it).

We agreed that a simple outright ban with no mitigations would likely prove a disaster. To have a chance of being workable, a ban would have to be accompanied by significant guarantees of support, probably from the state, to be effective.

We also looked at the misalignment of incentives, responsibilities and information that encourage so many to pay.

Our conclusion was this:

“if a ban on ransom payments is to be a credible part of a strategy to stop the flow of money to such criminals, then surely an essential precondition is more effective state intervention in the response to attacks, reflecting the gravity of the problem as a national security threat”.

Having set out how such interventions might be developed and tested, we continued:

“Whether or not payments are banned, a more activist approach is needed anyway, even if it means legislating for more interventionist levers over privately-owned critical infrastructure.” 

A huge part of the ransomware problem relates to policy. The reasons why it’s outstripped other forms of cyber harms are not technical or operational: they’re about the business model.

To go back to Chris Krebs’s triplet, countering ransomware means taking on the criminal groups operationally, improving cyber security at home, and also breaking the business model.

And this last part needs policy solutions.

Policy is normally formulated at least in part via the gathering of evidence, and formal expert consultation.

Yet Governments, when they comment at all, usually say simply that a ban on ransom payments won’t work. But no Government (to my knowledge) has tested the proposition properly through any recognisable, publicly disclosed policy evaluation process.

Some of the arguments against a ban are very powerful (threat to life situations in healthcare, for example, and the perverse outcomes that would arise from exempting the healthcare sector). But others do not bear up to scrutiny.

For example, one common argument against a ban is that it would drive payments underground. As Troy Hunt, the Australian cyber security expert, has argued, this is essentially saying that company directors would knowingly and willingly break the criminal law. There is absolutely no evidence to support this claim: plenty of company directors dislike the EU’s General Data Protection Regulation but that doesn’t mean they’ll consciously ignore their reporting requirements under pain of prosecution.

Furthermore, whether a ban is workable or not, policies on responding to ransomware incidents need further development. Specifically, returning to the lessons of the Irish and American healthcare examples, what is the balance between the public and private sectors when public safety is endangered by the compromise of a private entity? Should companies in critically important (and therefore already heavily regulated) sectors have a duty to report breaches? Is there a case for going even further and requiring them to consult, or even seek the consent, of the Government before paying? And if not, aren’t we just privatising national security risk?

This is complex, contentious stuff. The point is that all of the arguments for and against a ban, and other policy measures need to be tested in a formal process of policy evaluation and consultation.

What we have at the moment on the ransomware business model - and ransoms in particular - is policy inertia, justified by assertion.

Question 2: The narrative

That’s not good enough, but it is where we are.

Therefore, given the ransom policy inertia in most capitals, it is a prudent assumption that in the short term at least, the legal position in most countries will remain as it is.

Therefore, Governments, along with the wider cyber security community, must work harder to change the pro-criminal narrative that paying is often the best option.

It is easy for public authorities to ‘advise’ companies not to pay. That advice is genuine but, to a company in serious crisis, it is often meaningless. Governments understand this, and sometimes the don’t pay ‘advice’ is accompanied by thoughtful nods indicating that the authorities understand if the organisation takes the ‘wrong’ decision.

That is why stories like the Harris Federation’s gloriously stubborn refusal to pay - and saving money in the process - are so important. It is why understanding that extortion for availability and extortion for data protection are completely different things is so important. It’s why the Barracuda research that 80 per cent of organisations who paid got hit again is so important. It’s why research such as that by Hiscox Insurance, which shows that 29 per cent of victims of data extortion who paid up still had some data leaked is so important.

It’s also why the national dialogue in Australia about how to report and discuss the Medibank fiasco responsibly is so important (the subject of the next post in this series in two week’s time).

We have to shift the pro-criminal narrative.

Conclusions: the case for a substantive, inclusive, expert policy review

A holistic approach to ransomware will bring together operational and policy measures across like-minded allied nations.

President Biden’s 30 nation strong counter-ransomware group has announced a range of operational activity to try to mitigate ransomware. These measures absolutely have their place and we must hope they make the desired breakthroughs.

But ransomware is as much of a policy problem as it is an operational one, if not more so.

Yet when it comes to policy, Governments are mostly asserting that policy changes won’t work. But they’re doing that without engaging in the normal process of expert engagement, evidence gathering, and evaluation of options.

They are vindicating the age-old bureaucratic saw that doing nothing is always an option.

The case for substantial policy and legal changes to counter ransomware is not yet made. But nor is the case for dismissing the options without carefully examining them. There is a compelling case in many countries for a substantive, open-minded, consultative policy review of counter-ransomware policy options. This should be debated, published, and then - depending on the outcome - action should follow.

And in the meantime, there is a compelling case for a renewed effort to shift the narrative away from the criminals.

By telling the stories that show that paying doesn’t always pay.

We can and must do better.

Coming up next in the series

The fourth part in this series, on the way Australia managed the disclosure of data in the Medibank breach, will be published in early March.

In the meantime, there may be one or two other posts on other subjects on this Substack page.

Share

Subscribe now

Share

Subscribe now

Background to this post

On 22 November I was invited by the First Minister of Wales, Rt. Hon. Mark Drakeford MS, to give a lecture at the Hadyn Ellis building at the University of Cardiff. It is part of a series of lectures he and his team are hosting on the constitutional future of the United Kingdom.

The text of my talk is published below. At some point a video of the event, including the question and answer session afterwards, will be published online.

For now, here is the text of the lecture. (As always, a check against delivery caveat when the video comes out).

A point on timing. This lecture was delivered exactly one month ago today. It was delivered the day before the Supreme Court ruling that the Scottish Parliament did not have the power to hold an independence referendum. It was also delivered before the launch of the report containing proposals for constitutional reform by the former Prime Minister, Rt. Hon. Gordon Brown, and before the publication of the interim report of the Independent Commission on the Constitutional Future of Wales. However none of these developments alter the fundamental arguments in the lecture.

Text of lecture - check against delivery

I am really honoured to be here with you this evening. Thank you for the privilege.

[audience specific introductory remarks omitted]

My main message tonight is aimed at those who want to see a reformed, multinational UK.

Your moment might well be coming.

But meaningful, durable reform is a lot harder than it might look.

And it might be the last chance for it, given the strength of forces on each side:

• those who wish to bring the UK in its current form to an end;

• and those who wish to roll back devolution in favour of a much more singularly British, unitary form of Government.

You wouldn’t start from here: legacy of the last decade

Let’s start by looking at the situation a decade ago in 2012. (This was when I was working on the constitutional brief in Whitehall).

As Leighton Andrews of this parish has observed, we’d just had Danny Boyle’s opening ceremony for the 2012 Olympics.

This was what Leighton called “the emotional and cultural high point” of what he called “progressive unionism”: confident, respectful, inclusively multinational.

Although a referendum in Scotland had been conceded and agreed following the SNP landslide of 2011, support for independence was so low – hovering between 25 and 33 per cent – that few seriously thought the UK was going to break up.

Indeed, the agreement to the referendum was seen as a confident and respectful acknowledgement of Scotland’s participation in a voluntary union.

I also remember pointing out to David Cameron the extraordinary statistic that support for the Union in Northern Ireland was even higher than in Scotland, with 65 per cent, as against 17 per cent for unification. The Executive in Belfast was in one of its longest periods of continuous existence. Relations between London and Dublin were strong.

The one poll done in Wales during that period found 3 per cent support for independence, rising to 10 if Scotland left. Wales had continued on its path of gaining losers’ consent following the division of 1997, and was seen, from the outside anyway, to be working well.

No one was seriously contemplating leaving the EU: it remained, for the Coalition, as for John Major’s Government, a case of managing querulous backbenchers.

Inter-governmental relations were decent. The Joint Ministerial Committee occasionally met, and, while I don’t want to rose-tint the past, it sometimes even had useful discussions.

Much of the Coalition’s constitutional efforts (other than on the Scotland campaign) were spent trying to improve things in England’s regions, (though the chosen mechanism, City Deals, seemed to involve raising money from local areas and giving it back to them with huge strings attached in return for relatively tiny freedoms and small amounts of cash, and calling it decentralisation).  

You wouldn’t start from here: where we are now

Ten years on, the only unchanged feature from that landscape is that Whitehall and English localities are still wrangling over pitiful amounts of money and the devolution of weak powers. Whitehall still has more than one hundred central ‘pots’ or funds, including, and I am not making this up, one for removing chewing gum in local areas, and another for improving public toilets.

But other than that, everything else has changed.

What the whole of the UK now has in common with the situation in England is that the constitution is stuck.

And, particularly in Scotland and Northern Ireland, it is stuck in a tense and polarising way.

In Scotland, the 45 per cent vote for independence has turned into a floor, not a ceiling, for independence.

Now, independence is the all-consuming issue.

One side shouts: “you had your chance in 2014”.

The other yells back: “but Brexit!”.

And round we go again.

Northern Ireland is definitely stuck and tense, something I will return to later, at the organisers’ request.

Is Wales stuck? You tell me and I hesitate to comment in front of this audience. Again, from the outside, things look relatively stable.

Devolution is doing its job: Wales can do things differently in certain areas from a Westminster Government that doesn’t have a majority here.

First Minister of Wales Mark Drakeford spoke of a UK Government “hostile to devolution”

But I can’t help but be struck by the remark by the First Minister in July last year that he considered, for the first time this century, we had a UK Government in his words “hostile to devolution”.

And I can’t help but note that the excellent Leighton Andrews article I cited earlier is entitled “The Forward March of Devolution Halted”.

And across the UK, inter-governmental relations have certainly been scratchier.

So what is going on?

Centralising tendencies: it’s hard to let go

Part of it is to do with the enduring strength of centralising forces in the UK.

The groundhog day debate and reforms around localism in England under Labour, Coalition and Conservative Governments over the past 20 years prove this.

So too does the fact that devolution changed Government everywhere in the UK – except Whitehall.

London still struggles with devolution, as we saw in the pandemic.

But much more fundamentally than that, we now have three competing visions for the future of the UK.

The first vision is simple enough: to bring it to an end through independence (or unification, in the case of Northern Ireland). That doesn’t – yet – have the numbers to achieve its objective.

The second is the vision of an increasingly devolved UK: the one architected in the late 1990s and now seeking renewal via vehicles such as Mr Brown’s commission and the one under the leadership of Lord Williams and Professor McAllister. I will return to the challenges facing such efforts at the end of this talk.

The third vision receives far too little attention given its strength in Westminster.

It has been driving much of the constitutional redesign of the UK since the 2016 vote to leave the EU.

This is sometimes called muscular unionism, or – incorrectly – English nationalism.

Anglocentric British nationalism: a clunky phrase for a powerful force

Although it’s a clunky phrase, a better description of it is one I picked up from the former Labour Cabinet Minister John Denham, though he denies authorship.

It’s Anglocentric British nationalism.

And here are four characteristics of it:

• First, it considers that there is really only one nation and it is a British one. Westminster sovereignty is all; hence the hostility to the EU. It’s majoritarian: for example, it doesn’t matter if whole chunks of the UK don’t buy into its form of Brexit, because it has the numbers to face down inconvenient dissent. That those numbers are drawn largely from England makes it Anglocentric;

• Second, for the Anglocentric British nationalist, devolution is, in Boris Johnson’s word, a “disaster”. There shouldn’t be real alternative centres of real power to Westminster. Sporting and cultural national identities are fine, but not real power. For as long as devolution cannot be reversed, it must be contained. However, in the words of Lord Frost, devolution can “evolve back”, as the Johnson Government showed with the UK Internal Market Act;

• Third, Anglocentric British nationalism pushes back on the idea of the UK as a voluntary union. It talks of the UK as a unitary state – correct in legal terms but politically highly contested, including here in Wales. That is why Anglocentric British nationalism refuses not just the present demands for another referendum in Scotland, but to engage in any discussion as to how and when another one might be held;

• Finally, there is an inherently transactional nature to it. It is happy to remind the rest of the United Kingdom not of the joys of partnership, but of the costs of leaving. For that reason, I sometimes call it ‘know-your-place unionism’.   

Anglocentric British nationalism successfully secured a hard Brexit for Great Britain.

Its record in driving the devolution agenda is more mixed.

Its symbolic high point, so far, was the refusal of Liz Truss to speak to the devolved administrations’ leaders for the entire duration of her short premiership.

The new administration, thankfully, seems less interested in this type of approach.

But Anglocentric British nationalism is still a powerful force.

As a result, devolution is no longer a one-way street, if ever it was.

It can evolve back, and some people want it to.

And this vision of the UK’s future, whilst not particularly popular right now, will not go away.

And it still matters, particularly with regard to the paralysis in Northern Ireland.

Northern Ireland: where Anglocentric British nationalism doesn’t work

Anglocentric British nationalism has been, and remains, a disaster for Northern Ireland.

Contemporary Northern Ireland is based on three things.

First, affirmation of its legitimate place as an integral part of the United Kingdom, but with a right to leave it and join the rest of Ireland, depending on the majority’s will. That was unionism’s great triumph of 1998: acceptance of the legitimacy of the Northern Ireland state by plebiscite throughout the island. But that acceptance depended on compromise.

Second, a key part of that compromise was smooth economic and social relations with the rest of the island of Ireland, reflecting the traditional minority’s desire for as close ties as possible.

Third, and - again - a part of this compromise, power-sharing, devolved Government based on cross-community support.

Anglo-centric British nationalism, and the form of Brexit it gave rise to, crashes up against all three of these pillars of contemporary Northern Ireland.

First, it imposes a view of British sovereignty that is at odds with the 1998 Agreement. The Protocol Bill, that article of faith among Anglocentric British nationalists, states the supremacy of the 1800 Act of Union.

But, as Dr Andrew McCormick, who led for the Northern Ireland Civil Service on Brexit, has written, under the 1998 Agreement, endorsed by referenda on both sides of the Irish border, supremacy is given to the people of Northern Ireland, not to a 222 year old statute, which, under the Diceyan interpretation of the constitution so beloved of Anglocentric British nationalists, has no particular standing above the 1998 Agreement or other statutes in general.  

Under the Agreement, the people of Northern Ireland have the right decide whether they want to remain in the United Kingdom or join an all-island state.

For Dr McCormick, that means that Mrs Thatcher’s totem that Northern Ireland is “as British as Finchley”, recently re-stated by the Foreign Secretary using his own constituency in North Essex, is a nonsense.

People in North Essex do not have the right to join another state, or to hold the passport of another country. There are no joint institutions with another sovereign state responsible for parts of the governance of North Essex. All of these things, by treaty and Parliamentary statute, are in place for Northern Ireland.

The 1998 Agreement is one of the foundational documents of the modern UK.

It explicitly acknowledges that the UK can be broken up, and that some of its constituent parts should have highly exceptional forms of local administration.

It is highly unusual, as some unionists point out, for a state to provide for its own breakup. But the United Kingdom does.

Once that principle has been established, everything else is a matter of degree.

The simplicity of Anglocentric British nationalism cannot be accommodated within this complex mosaic.

Second, withdrawal from the Single Market completely changes the nature of North-South relations.

The 1998 Agreement did not mention the EU because it did not need to. The entry of both the UK and the Republic of Ireland into the Single Market in 1993 transformed economic relations on the island, particularly in border areas, even before the ceasefires and the removal of border security apparatus.

Crucially, as well as reducing trade friction on the island, joint single market membership on the island was a significant cultural and economic moment.

As John Hume pointed out repeatedly at the time, the reduction in the emphasis on binary  national identity mattered hugely. The sense that being primarily British or Irish was of less importance in the age of greater European integration was profoundly important to the historically minority nationalist community.

Along with the removal of discrimination against Catholics in the 1970s, and the introduction of power sharing in 1998, joint participation in the European Single Market with the rest of Ireland was one of the three reasons why Northern Ireland’s traditional minority community reconciled itself to the Northern Ireland state and voted overwhelmingly - even if some of its leaders won’t say it out loud - to recognise its legitimacy.

The shock at the removal of the single market - without consent - is still palpable in this community.

Finally, given this, Brexit, and particularly withdrawal from the Single Market, was a fundamental change in the governance of Northern Ireland.

History would suggest that fundamental change in Northern Ireland is best done, as in 1998, with cross-community support.

But cross-community support was never, could never, and will never be given for Brexit.

Inevitably, the subsequent attempts to exempt Northern Ireland from aspects of Brexit to reduce the impact on cross-border relations has led to the opposite problem: the withdrawal of consent for power sharing in the unionist community.

All this has proved is that, in the words of Tom McTague of The Atlantic, (a writer not unsympathetic either to Brexit or to Northern Ireland’s unionists) Brexit has led to a set of problems in Northern Ireland that “cannot be solved, only managed”.

That means there are no easy answers, and no perfect one. So what to do?

Stronger local leadership must be part of the answer. But tonight I want to look at what the UK Government, as the sovereign power, should think about.

First, the UK Government must recognise Northern Ireland for as it now is, rather than what it might appear doctrinally in a simplistic, post-Brexit, sovereignty first way.

The lead of unionist parties over nationalists in first preference votes at May’s Assembly elections could fit into Windsor Park stadium, one of international football’s smallest venues.

Both unionism and nationalism are now minorities. The future will be decided by the non-aligned.

At the moment, the non-aligned break decisively for the Union, were there to be a referendum on unity.

But they do not break for sovereignty-first British nationalism.

Northern Ireland used to vote this way.

In 1987, in the first general election after the Anglo-Irish Agreement which unionists reviled for giving Dublin a consultative role in Northern Ireland, unionists implacably opposed to the deal won more than 55 per cent of the vote.

Now polls suggest that if the Assembly is to remain moribund, 60 per cent support some form of role for the Irish Government.

Moreover, a clear majority in the Assembly were returned last year on manifestos to support some form of Protocol arrangements.

Unionism may hate the Protocol, and that matters.

But political unionism is no longer a majority, and that matters too.  

The Brexit vote in Northern Ireland. Note that not a single border area voted to leave. Indeed, at present, no constituency touching the border is represented at Westminster by a unionist.

The present ‘majority’ in Northern Ireland, insofar as there is one, expects to remain in the UK for some time to come. It wants governmental arrangements that will work, for however long that is.

But this ‘majority’ is perfectly happy for Northern Ireland to be treated differently from the rest of the UK, and indeed sees some advantages to such arrangements.

So Northern Ireland is not North Essex, and doesn’t want to be.

But this does not, of course, solve the current problem of unionist consent, the piece that is currently lacking.

So the second part of the answer, which may seem unsatisfactory and incomplete but I think is important, is about process.

Process matters.

In an outstanding paper for the UK in a Changing Europe think tank in August, Sir Jonathan Stephens, recently retired Northern Ireland Office permanent secretary and key protagonist in many of the key negotiations of the past quarter century, wrote that “the process around the Protocol has been fundamentally flawed because the key players who need to accept and work with the outcome – the parties in Northern Ireland – have never properly been involved”.

He noted that in the run up to the Agreement 25 years ago, the advice to British Ministers was that the process they oversaw was far more important that any specific solutions. And an important part of that process was, despite clear and obvious differences, London would treat Dublin as a partner, not as an opponent.

From this sort of inclusive process, solutions might emerge. Without it, we cannot know what solutions will stick.

Third, the problem must be treated now as a Northern Ireland issue first and foremost, rather than a consequence of negotiations between the UK and the EU.

Trade is, of course, the major part of the technocratic dimension. And that’s a London to Brussels issue.

But we have to remember that Brexit fundamentally ruptured Northern Ireland politics because it removed the Europe-wide framework where binary national identities mattered less.

You cannot solve a political crisis about identity, consent, all-island relations, relations between the islands, relations between communities, and much more – with chilled meat protocols or reciprocal arrangements for veterinary standards.  

The EU will continue to have concerns about the integrity of its single market. But it’s increasingly obvious it will want to support something that could command genuine cross party support in Northern Ireland.  

Fourthly, and this is the biggest hurdle to overcome, all this requires a rebuilding of trust. Over the course of the last year, trust in the UK Government to deal with the protocol, as measured by Queen’s University, has nearly doubled – to 7 per cent.

It is not hard to figure out why trust is so low. We had:

• a Secretary of State during the referendum campaign assert that leaving the EU would not affect arrangements in Northern Ireland;

• a Prime Minister in 2019 assert that his deal was similarly benign for Northern Ireland;

• and a later Secretary of State on New Year’s Day in 2021 assert there was no sea border just as film footage of new inspections were being broadcast on TV the very same day.

Really substantive progress, in my view, is impossible for as long as the line from London remains that Brexit is no big deal when it comes to Northern Ireland.

Beyond Northern Ireland: challenges for the UK’s reformers

It may be, therefore, that change in approach might need to await a change of Government, (though on the other side of the Irish Sea the seismic prospect, though not the inevitability, of a Sinn Féin led Government in Dublin from 2025 might act as an incentive for more urgency).

In any case, I hope that the example of Northern Ireland – unique as it is, shaped by a tragic and brutal recent history thankfully absent from the rest of the United Kingdom – is enough to show that a simplistic application of post-Brexit, sovereignty-first, Anglocentric British nationalism is not the right approach for at least one part of the UK.

The question now arises as to what’s next for the rest of the United Kingdom.

In that context, let me return to the second group seeking to shape the future of the UK: the reformers.

I detect great excitement in this community that, with the polls as they are, and reformers on the brink of outlining new plans for a reconstituted UK, tensions within the territorial constitution will dissipate in the next Parliament.

And that will sweep away Anglocentric British nationalism from its position of influence within the UK Government.

And in doing so, it will weaken support for separation, especially in Scotland.

That may be so.

But looking beyond that timeframe, it is also credible to think of this as the last chance for meaningful reform of the UK in its current form.

Consider this outline of the future.

There’s a change of UK Government in 2024.

A Labour or Labour-led Government introduces constitutional reforms designed to strengthen devolution.

A Westminster Conservative opposition, hardening against devolution out of office, as it did over Europe after its 1997 defeat, opposes those reforms and pledges to reverse them.

The economic inheritance of the new Government proves too tricky: it is short-lived and in the late 2020s or early 2030s a form of Conservatism that is much more anti-devolution than before comes into power.

The argument of 2024 in Scotland (and possibly Wales) that you don’t need independence to protect you from a Conservative Government you didn’t vote for falls away, along with the new constitutional reforms.

In the meantime, let’s assume that because of these political difficulties there has been no significant movement in national sentiment in either Scotland or Northern Ireland.

But demographics are taking their course, so support for leaving the UK is rising. And in Scotland, it’s now not far off the point where a ‘generation’ has actually passed since 2014.

In this scenario, the 2030s could see a winner-takes-all contest for the future of the UK, with the middle ground nowhere to be seen.

That’s why what happens in the rest of this decade could be the last chance for reform.

Conclusion: the case for reform, and making it

So what might reform involve? Here are some brief concluding thoughts.

First, there are plenty of areas where there is scope for imaginative reform.

Issues like immigration and trade need to be settled on a UK wide basis but there is no reason why there cannot be mechanisms for greater consideration of interests across the UK in setting those policies.

The fiscal frameworks are less lopsided than they were when they were designed in the 1990s, but they still contain many disincentives for the devolved nations to use the powers they have.

The mooted ideas for rebalancing the upper House might gain traction.

In the other direction, surely we might learn from Covid about how better to coordinate crises – that might even mean understanding when it makes sense for Westminster to take some powers back in return for better consultation and coordination with devolved administrations about how to use them.

There is much still to explore in terms of how the governance of the UK works.

Second, meaningful reform must contain some proposals for agreeing a mechanism on how the constituent parts of the UK can leave if they want to.

The position in Scotland is unsustainable: if the pursuit of independence is regarded as a legitimate political pursuit, then a way of achieving that must be provided.

At present, there is no such mechanism.

Third, the process matters. This cannot any longer be a conversation among the non-English parts of the UK only.

But this is really, really hard.

England cannot forcibly be decentralised or broken up into regions it doesn’t recognise.

England resents when the other nations are seen to be able to dictate their place within the UK.

Again, there are no easy answers.

Can, for example, any of the suggested changes to the upper chamber be done in a way that strengthens non-English nations’ role in UK wide life but is done in a way that commands England’s consent?

And now to the most important points.

Penultimately, is there a way of making changes permanent and enduring? I know Sir David Lidington, in an earlier address in this series, dwelt on this topic.

After a contested Brexit, and with the renewed strength of sovereignty-first British nationalism, many, including me, will be sceptical about the durability of any reforms beyond the lifetime of the Government that introduces them. I was no great fan of the Fixed Term Parliaments Act. But its fate is a reminder of how temporary constitutional reform can be.

Is there actually a way of ensuring the permanent powers of national institutions outside England given the doctrine of Parliamentary sovereignty?

That question is also really, really hard. But it is really, really important.

So serious, impactful, enduring reform will be difficult. And it could be the last chance.

But, my final point is that if these problems can be overcome, then their advocates should make the case for the reforms as a something good in their own right, not as a concession. .

If you think the UK is better governed as a multinational state under messy but workable arrangements, rather than in a singular, Anglocentric British nationalist way, then say so.

For too long the decentralisation of power outside England has been presented defensively: the ‘Vow’ to Scotland in the dying days of the 2014 campaign being the most obvious occasion.

Contrast that with where I started.

The 2012 multinational union, comfortable in its multinational diversity, presented to the world in the Olympics had much to commend it.

Wales is perhaps uniquely suited to this proposition.

Its devolved arrangements have secured the consent of the many who did not originally want them.

It has the only devolved Government that can convincingly go to London saying it wants to make the UK work, even if political leadership is different.

Wales has a vibrant, non-political national life, shown by, among other things, the remarkable resilience, and now expansion, of the Welsh language.

Again, from an outsider’s perspective, if anywhere is to make the case for a thriving, multinational, reformable state, it is probably Wales.

So reformers should make their case.

In 2016, pro-Europeans found that if you’ve spent years presenting something in negative terms, boasting instead about how successfully you’ve used it to resist something else, then don’t be surprised when people aren’t particularly enthusiastic when you ask them to endorse it in its own terms.

If you’ve taken the trouble to work out how to do it, then make the case for it. If you believe in a devolved, respectfully multinational UK, then make the case for it in its own right, not as a way of stopping something else.

Thank you for listening to me this evening.  

Subscribe now

Share

Recap

This is the second in a five part series looking at the vast and varied implications of two major data breaches in Australia in the period September to November 2022.

The first post, which can be found here, covered the background to the two breaches at Optus, a major telecoms company, and Medibank, a health insurer which covers nearly 10 million of Australia’s 26 million population. It also discussed how our failure to distinguish between the severity of different types of data breach has bred a degree of complacency about data security.

The remaining posts in this series will cover:

• the payment of ransoms in cyber incidents (Part 3);

• how we discuss and report cyber harms, and why that matters (Part 4);

• the safe haven problem in cyber crime (Part 5).

Part 2: Governments and the cyber security problem

This post covers the role of Government in cyber security.

Over the past decade, Governments have become increasingly active in this space. State operational capabilities have been improved in many countries. Critical infrastructure is increasingly regulated for cyber safety and resilience. Data protection laws abound.

But there’s a lot more to cyber security than this.

Many of the deeply entrenched problems in cyber security are rooted in economic, social and behavioural factors as well as technical and legal ones. Governments have, by and large, steered clear of many of these challenges or confined their interventions to the occasional toe dipped in the water.

A key underlying tension has been where public and privately owned risk collide, whether before or after an incident. This is where recent events in Australia are important.

In dealing with the Optus and Medibank breaches in late 2022, the Australian Government has been strikingly active. It has since signalled its desire for far reaching reform of the way the country does cyber security.

This potentially accelerates a trend of increasingly interventionist Governmental behaviour across the West in recent years. That trend is uneven (and the United States, with its size, constitutional framework and the luxury of having most of its own tech run from its own jurisdiction, will always have its own, often different story).

But the wider trend towards Government activism in cyber security is unmistakable and important. And Australia is going to be a country to watch because of the apparent breadth of its ambitions, and its status as a key Five Eyes cyber power.

The strikingly activist response of the Australian Government

First, it is worth taking a look at what this activist response has involved so far.

It is an age old ritual, as old as cyber security itself and much mocked in the cyber security community, for a victim organisation to describe itself as being the target of “an unprecedently sophisticated attack”, or variants on those words. True enough, Optus duly used this line at press events on 22 and 23 September.

An essential part of the ritual is that it is not contradicted by Governments or the mainstream press. Cyber security professionals might poke fun at the ‘sophisticated’ narrative on social media, but, by and large, it goes uncontested, feeding a sense that organisations are powerless when faced with hostile cyber activity.

This is where Australia broke with tradition.

On 26 September, Clare O’Neil, Australia’s Minister for Home Affairs and Cyber Security (one of the most senior positions in the Federal Government, which has seen the phrase ‘Cyber Security’ added to it for the first time upon Ms O’Neil’s appointment in June) made a direct, forceful and accurate intervention. She said bluntly that the operation was not sophisticated, adding that Australia “should not have a telecommunications provider in this country that has effectively left the window open for data of this nature to be stolen.”

Australia’s Minister for Home Affairs and Cyber Security. Clare O’Neil

Moreover, as the incident developed, Optus came under increasing pressure to organise and pay for replacement passports for those affected by the compromise, some of the pressure coming publicly from the Federal Government. Canberra had no power to compel Optus to do this, but the company eventually announced it would, and agreed a process with the Government (as the issuer of passport documents) to arrange it.

Subsequently, Medibank provided a different challenge. The operation against the company appears to have been much more sophisticated and the company made no questionable claims about the sophistication of its provenance. There was no comparable issue to replacing passports. But given the extent of public concern about health records being publicised the Government, and Ms O’Neil specifically, took a very active role in communicating with Australians about who was responsible, what course the incident was likely to take, what risks people were exposed to, and what the Government would do in response (some of these points are explored later in this series).

Finally, taking both cases together, Ms O’Neil and her colleagues have signalled a willingness to revamp Australia’s national management of cyber security and develop a new strategy to deal with the problem. On 8 December, she announced a process to develop a new cyber security strategy (full disclosure: I have accepted an invitation to be part of a group of unpaid international advisers supporting an Australian-led expert panel in developing this process. The Australian Government does not, and does not seek to, approve or influence anything I write on this or any other topic).

The terms of reference of this strategic review are, as is the way with these things, reasonably short and therefore open to interpretation. However to date they imply a far reaching review that goes way beyond just ‘fighting the last war’ and confining the process to a review of data laws.

The announcement speaks - intriguingly - of “increasing whole-of-nation cyber security efforts to protect Australians and our economy”.

This leaves in scope pretty much anything. But the direct reference to economic interests implies it will go beyond a narrow interpretation of national security or public protection interests.

Why and how Governments are getting involved

Why has it come to this?

In short, because we’ve struggled, mostly in vain, to delineate private from public risk in cyber security. And incentives are working against good cyber security in too many ways. And we’re suffering as a result.

In the middle part of the last decade, when setting up and then running the UK’s National Cyber Security Strategy, my team and I wrestled for months - years - with the question of who we should help, and when. Part of the challenge was to avoid, in the words of a senior Conservative Minister to me privately - “nationalising cyber security”. So we started a big programme of work mapping out when we’d intervene, based on separating private and public risk.

Eventually we gave up.

It was just too hard. In a sophisticated digital economy, the interplay between privately owned damage and public harm was ubiquitous. Should we offer to help a company if it supplied a key public service, but not if it didn’t? If passports were lost because they’re government documents, but not bank cards? If the attacker was a nation state, but not a criminal? If the victim organisation’s cyber security was strong but not if it was faulty? We couldn’t answer any of these questions conclusively.

An illustration in the UK from earlier this year proves the point. In August, a private sector software company suffered a crippling ransomware attack. Among its customers was the publicly run National Health Service (details are here). As a result, the health service’s helpline, and some mental health services, were badly affected. The National Cyber Security Centre confirmed it was assisting. When I was asked live on radio by a former senior politician why the state should help a private company in this sort of situation, my reply was simple: public health provision was being disrupted.

This might not be a neat delineation of risk. It might even be effectively subsidising the consequences of weak cyber security practice in the company. But it is what the British public would expect and what the health service needed at that moment in time.

Therefore, at the NCSC, out triage efforts rationed our interventions based on an assessment of the harm to the nation, whether it originated in Government or the private sector.

This is very difficult to judge. But, in our view, it made for a more sensible set of interventions than trying to work out whether or not the Government should get involved based on who owned or operated the network.

It’s the consequences that matter.

The Optus and Medibank breaches show the same point.

Neither are first order national security crises. Reasonable people can debate and differ on how damaging they are.

But both involve clear public risk and potential harm in some way.

Medibank is the more obvious and serious example. Medical records are deeply sensitive and the company stored not far the records of two fifths of the national population. In such a scenario it is inevitable the population will look to its government for a narrative about what is likely to happen, including what risk they’re at. They will look to Government to do what it can to prevent wholesale disclosure of the dataset. And they will look to Government for the pursuit of the guilty and the prevention of a recurrence of such a data disaster on a national scale.

Optus is a more subtle example. There is a debate to be had about whether Governments should engage in a public debate about whether or not an attack was sophisticated or not (though for what it’s worth I would argue that Canberra’s leaders were right and courageous in this case to do so.)

But where the public interest is obviously engaged is in the part of the dataset that involved passports and driving licenses. These are two of the primary identifiers issued by the state. When, as the last post in this series set out, 1.2 million of the 9.7 million records compromised (and a further 900,000 expired ones), it falls to the state to make an assessment as to the risk of harm arising from this, whether the documents are still usable, and, if not, whether any special process needs to be established to replace them. Some might take the view that this is not a particularly serious issue, and that might be correct. But it still falls to the state to make the assessment.

Therefore, in both the Optus and Medibank cases, privately held risk materialised in a way that engaged the wider public interest. This is increasingly inevitable in advanced digital economies. And that in turn leads to a requirement for the state in many cases to take a leading role in incident response.

One of the reasons the UK’s National Cyber Security Centre was set up was as a result of the breach at the telco TalkTalk in October 2014. This was the first time a cyber breach had led the British national news. The case is instructive because, over time, it emerged that the breach was considerably less serious than initially thought. The early narrative around the case assumed, as always, a sophisticated attack that left the company’s four million customers at direct risk of serious financial fraud. In fact, a basic hack had accessed the dated and limited records of 150,000 customers. But it took several days for this to emerge, and the Government stayed mostly silent amidst the (unnecessary) panic.

Though no national security risks arose in the TalkTalk case, the Government at the time later concluded that the state needed a national cyber security incident management function akin to that for terrorism, floods or public health outbreaks. What was needed, they argued, was a system where someone in authority would give an official assessment of what has happened, who is at risk, from what, and how they might go about managing their risk. This because one of the core functions of the NCSC.

The UK’s more activist approach to managing incidents was one of a number of significant developments in state involvement in cyber security in the West over the past decade or so. Others have broadly fallen into three categories.

The first, and most extensive, is regulation of critical national infrastructure (CNI). This is unsurprising given what is at stake. Moreover, it is where the privatisation of serious national security risk is most acute (for more on that, see here). Australia is among several countries to have acquired regulatory powers over cyber security in CNI. The UK has recently taken powers for telecommunications akin to those already in place for financial services. The EU’s two Network Information Systems (NIS) Directives of 2016 and 2022 set out sweeping requirements for a range of critical sectors. The problem here is that critical infrastructure is very difficult to define, and what we regard as critical changes all the time.

A second, and growing, area is in the safety of technology itself. The EU (through its 2019 Cybersecurity Act) and the UK (through the just enacted Product Security and Telecommunications Infrastructure Act) have followed Singapore in introducing detailed regulation of Internet of Things (IoT) products. This takes advantage of the change in the internet economy represented by IoT: instead of web-based services where the price of entry is data, these are products and services that can be more easily evaluated and, therefore, where security standards can be more easily specified. These reforms are hugely important, but they do not address long-standing defects in network security.

The third is the trickiest and most underdeveloped but seeks to address the limitations of the existing approaches: trying to use the power and influence of Government to improve general network security across all sectors of the economy.

That is where the reference to the ‘whole of nation’ approach in Australia’s strategic review is so intriguing, and it’s one that presents a plethora of opportunities and risks.

Governments and cyber security: opportunities and risks

The great conundrum of Western cyber security over the years is why so much money has been spent on a clearly recognised problem, and yet so many areas of fundamental difficulty remain.

It is hardly new to talk of market failure in cyber security, but it’s a huge part of a difficult story. There are some areas where the ordinary functioning of market economics has led to the development of powerful capabilities to combat cyber security threats - threat intelligence is an obvious example. But there are many other examples of weak security practices and enduring digital pollutants which illustrate the problem.

Here are a number of ways in which the state can shape a nation’s posture around cyber security and address market failures (they are absolutely not specific to Australia or linked to its current strategy development process, or to the UK, or anywhere else). They are instead a list of possible areas for Governments to consider what their appropriate policy is:

• First, setting the national risk appetite. In less technocratic language, this means deciding what a country and its companies and citizens need to care about. Key to this is understanding harm. How much should it care about data breaches? Which ones? Which threat actor matter more than others? Which aspects of the problem can only be left to Government?

• Second is the cyber security environment for businesses. What risks are businesses to be reasonably expected to manage? Once that’s decided, how are the rules to be implemented? Does Government have a role to play in nudging corporate governance rules to be adapted to incentivise better cyber security? Why hasn’t cyber insurance - insurance being a regulated industry - worked to anything like its full potential? Can the Government use its convening power to help fix these problems? And if not, does further regulation become the default?

• Third, the regulatory framework. If the answer, at least in part, is more law, then the key is a balanced set of regulations that seek to take care of the totality of the problem. Two examples: first, what we think of as critical infrastructure changes all the time; and second, if data protection is the only legal obligation, organisations won’t be incentivised to protect against more disruptive threats.

• Fourth, the Government as an active defender. The Government can harden its own networks by direct action, not just by persuasion and/or law. That not only shows leadership by example, but also takes away a large part of the national attack surface given then size of the state in modern economies.

• Fifth, direct intervention in areas of clear market failure. If no-one has a commercial incentive to try to take down malicious websites, the Government can try. If there are ways to block automatically, it might be within the Government’s powers to do so. (Some of these types of initiatives have been tried by the UK in the NCSC’s Active Cyber Defence programme).

• Finally, the deterrence and pursuit of wrongdoers. A key role of the state is to prevent and deter harm against its citizens and punish those who do. This becomes harder with regard to cyber attacks because, for the first time in human history, large-scale, persistent harm can be initiated without the adversary ever setting foot on national territory. In a domain where impunity is easier than before, to what extent can the state reasonably be expected to thwart malevolent actors? To what extent can taxpaying businesses and citizens rely on such protections?

Inherent in these questions are many of the risks and challenges for Governments:

• the UK Minister who worried about ‘nationalising’ cyber security might not find himself reassured by this list. Governments need to navigate the obvious moral hazard associated with a more activist approach. But, arguably inactivity is worse.

• Getting the right regulatory posture is hard and the consequences of getting it wrong can be severe. One striking feature of the Irish healthcare crisis of 2021, when most of the state’s healthcare commissioning system was taken offline, leading to severe consequences for patient care, was that the main law incentivising the Health Services Executive to take cyber security seriously was the General Data Protection Regulation (GDPR). This had skewed incentives prior to the crisis; however serious health data is, it’s not as serious as losing access to the cancer consultations scheduling system. But the regulations incentivised data protection, not service continuity.

• Bringing the business community along with Government reforms is essential. It can be done - the new UK telecommunications security act, for example, was brought in with industry encouragement and input. However, GDPR provides an opposite case study. In the UK, it led to a temporary chilling effect in Government cooperation with business over data breaches: the Chief Information Security Officer (CISO) was replaced by the General Counsel in key operational meetings with Government agencies. As a result, the NCSC came together with the data protection regulator, publishing a memorandum designed to provide regulatory incentives to victim organisations to work with the Government to help manage the problem.

• Finally, greater Government involvement increases some of the risks to the Government itself. If stiff penalties are inflicted on the private sector for cyber security failures, it is reasonable to expect commensurate accountability in Government, and the challenges in protecting networks are no less acute in the public sector. Moreover, some of the tensions in public policy become more acute; for example around Governments’ real and perceived stance around end-to-end encryption.

At the heart of realising the opportunities and managing the risks is changing the way in which incentives work. Incentives have been broken in cyber security for some time. Hope for the best has been a perfectly rational strategy for many organisations. That has to change.

Here, a key choice for countries is the balance between compulsion at one extreme and exhortation on the other. This will vary according to local circumstance, governing systems, and the political preferences of those in power. The United States is likely to remain at the more exhortatory end of the spectrum for all sorts of reasons: the cultural and constitutional reluctance to compel businesses; the difficulty of passing effective regulation through Congress; the location of many tech and cyber security giants within the United States, and the sheer power and scale of America’s state cyber security agencies.

Yet, all that said, the underlying diagnosis in this article of more Government activism does not, to me anyway, seem out of step with some of the thinking currently driving US policy. One only has to look at the brilliant article by President Biden’s National Cyber Director, Chris Inglis, in February of this year arguing for a new Cyber Social Contract to see how the difficulty in delineating risk between the private and public spheres is at the heart of the Administration’s considerable efforts to ramp up partnership between government and the private sector in cyber security.

In any case, most countries, even the US’s close Five Eyes partners like the UK and Australia, do not have any of the specific factors driving the US’s less directly interventionist approach. So it will not be surprising if most of the US’s allies end up taking a more direct role, including through legislation, than America does. Whilst it will be some time before Australia’s policy review process is completed, it could be an early indicator of an important trend.

Conclusion

Modern highly digitised societies share cyber security risk between the private and public sectors. It is often impossible to work out exactly where private risk ends and public risk starts.

Capable Governments can develop their own tools to take care of their side of the bargain. But a modern Government also has a key role in shaping the incentives of the private sector to do its part.

All this is taking the state beyond its traditional areas of intervention in cyber security - like critical infrastructure, data protection and product regulation - and into a more complex area of incentives, economics, general corporate regulation and so on.

And that leads to some very important public policy challenges.

Coming up next in the series

One very specific policy challenge is the vexed question of the payment of ransoms. This has been a feature of many of the most serious cyber security incidents of the 2020s, and has prompted some of the most heated debates within the cyber security community of recent times.

This question will be examined in Part 3 of this series. It will be published on Tuesday 17 January, following the Christmas and New Year break.

In the meantime, there may be one or two other posts on other subjects on this Substack page.

Share

Subscribe now

Subscribe now

Share

Maybe we’d forgotten about data breaches. Maybe we’d started to think of them old-fashioned and passé.

In Western cyber security circles we (understandably) spent most of 2022 studying the digital dimensions of Russia’s murderous invasion of Ukraine (more on that in future posts). We (again, understandably) spent 2021 fretting about an explosion of disruptive ransomware when systems didn’t work, often with potentially dangerous consequences (think of the serious disruption to Irish healthcare, or the disorderly lines queuing for gas on the US eastern seaboard). We spent (yes, understandably) 2020 trying to figure out how to work remotely safely, and cope with all these new Covid scams, as well as how to prevent vaccine espionage and misinformation.

But data breaches? It’s been a while…

Recent events in Australia should remind us of the centrality of data protection to the health of the digital domain. The lucky country’s extended period of luck in cyberspace finally ran out. Two major breaches have not just shaken one of the world’s richest, happiest and digitally advanced countries but have raised just about every major public policy issue relating to cyber security that there is. There are no easy answers, but we have to pay attention to the questions.

Taken together, the two incidents are among the most consequential in recent cyber security history. The lessons arising from them are vast, varied and vital.

This series identifies five of the most important. They are by no means exclusive to the incidents in Australia, but each feature in the cyber security crisis there.

The series starts today with a look at how our failure to understand the differing severity of harm caused by different data breaches has hampered our ability to understand the problem and tackle it.

It will be followed by four further posts over the coming weeks on other crucial lessons of events on the other side of the world.

What’s happened?

But first, given that the incidents have received surprisingly little attention in the northern hemisphere, even among cyber geeks, a quick recap is necessary:

• on 22 September, Optus, one of Australia’s largest telcos, announced a data breach affecting up to 10 million customers (a very good early timeline of the Optus case is here). In keeping with tradition, the company called it “a sophisticated attack”. In a striking (and welcome) break with said tradition, Clare O’Neil, Australia’s new(ish) Minister of Home Affairs and Cyber Security (the last part of her title being completely new) pointedly rejected this characterisation, accusing the company of leaving the door open by way of a public API endpoint which required no authenitication and was therefore open to anyone. Just over 10,000 records were published (and then taken down) on the dark web, removed by an apparently penitent actor. The behaviour of whoever accessed the data was, on the whole, a bit weird: an apparent ransom demand for US$1m in crypto, in return for not publishing the information, was made and then dropped.

  It later emerged that while most of the exposed records were the classic basic trove of email records, phone numbers, dates of birth and not much more. However, for thousands of Australians the breach involved the compromise of much more important identifiers, principally driving licenses and passport numbers. Under pressure from the Federal Government, Optus agreed to pay for the replacement of the passports of those affected. There has not, at the time of writing, been any further evidence of data leakage resulting from the Optus breach.

• if Optus took the form of the classic commercial data breach, with a bit of passport data to make it more serious than most, then the Medibank affair was of a different order, quickly turning into one of the most serious and sickening data breaches in history (a good timeline of the early stages in the saga is here).

  In mid October, Australia’s largest health insurer detected a significant breach. Throughout the rest of October the company seems to have gone through the classic process of data breaches from denial to acceptance. Its early statements played down the incident, but by the week of 7 November they’d been contacted by the malicious actor and it had become clear that 9.7 million medical records had been stolen and it was time to come clean about the severity of the incident.

  The behaviour of the criminals behind this attack showed much more planning and foresight. Technically, this did at least have some indicators of a sophisticated operation. The hackers knew what they were doing and used conventional methods of demanding a ransom (pricing it at a dollar per customer, or $9.7m) in return for not publishing the information. On 7 November the company announced publicly both the extent of the breach and its refusal to pay the ransom. Two days later, the hackers began to release sensitive medical records relating to thousands of people onto the darkweb. The first files apparently related to women’s reproductive health and patients with alcohol or mental health problems. Yes, it really was that bad.

  As a result, the case became headline news in Australia (it is as rare for a cyber incident to lead the news in Australia as it is in the UK or US). On 11 November, the Australian Federal Police held a remarkable press conference where they announced that the attackers were criminals based in Russia whose identities were known to the Australian authorities but were not publicly disclosed. They announced a plan to pursue the criminals through the Russian authorities and Interpol. The Federal Government announced a new offensive cyber unit, to be staffed jointly by the AFP and the Australian Signals Directorate (think the NSA or GCHQ for Australia) to go after the criminals’ digital infrastructure. At the end of November, after further disclosures of apparently intelligible information, the hackers published an odd message saying “case closed” and purporting to dump all the data online onto the darkweb; however the data was, in the seemingly accurate view of Medibank itself, “incomplete and hard to understand”.

Why do these hacks matter? (beyond the obvious)

So for a period of nearly three months, and particularly in November, serious and scary data breaches have been at the top of the political agenda in Australia in a way not seen in other major Western economies for some years. The new Labor administration in Canberra have taken the incidents extraordinarily seriously. The Government seemingly plans a wide ranging reviews of Australia’s cyber security laws and strategic posture.

Clearly, and for obvious reasons, these incidents matter to the millions of Australians affected. But their implications go well beyond Australian shores (there is absolutely no reason to believe that Australia, a wealthy Five Eyes country with excellent security services, is somehow uniquely bad cyber security).

As well as reminding us that data breaches haven’t gone away, the incidents throw up some of the most important and unsolved challenges of doing cyber security in advanced digital economies. These incidents therefore have global resonance and importance.

The five lessons from Australia

Here are the five lessons from Australia’s recent experience that will be the focus of this series:

• first, data breaches still matter. But some matter way more than others. Our failure to distinguish between incidents in terms of severity is helping fuel complacency about data security. That is the focus of this first post in the series.

• second, Governments are going to be far more involved in cyber incidents and cyber regulation in most countries, whether the private sector likes it or not. The Australian Government has shown significant and innovative leadership in its response. It’s used its public platform well. And it seems to be embarking on a wide-ranging review of its posture. All that is likely to have resonance further afield.

• third, we still really, really need to talk about ransoms. We need to talk about whether companies should pay, and, yes, if they should be allowed to. That discussion is paused, not finished.

• fourth, and related, how we talk about cyber incidents, and how the media reports it, really matters. There’s a public interest in reporting serious problems with cyber security. There’s also a public interest in not spreading fear that fuels a pro-criminal business model. Australia has valuable lessons here in responsible public discussion.

• finally, the safe haven problem is the biggest and hardest challenge we face in cyber security. The ability of criminals to hang out in various countries with impunity, especially Russia, is cyber security’s least technical but most pressing and yet intractable problem.

Lesson 1:

Data breaches still matter, but some way more than others

…and headline numbers of records is a terrible and damagingly misleading measure.

This week’s inaugural post focuses on how we think about data breaches and why that matters, and in what ways we need to get better.

When we talk about data loss there is still a tendency to think of all data breaches as being more or less the same, with the differentiator being how many records have been stolen. This is particularly true of broader public discourse, though it still happens a bit in specialist cyber security circles too.

That’s wrong, and it’s a problem.

Why?

Many data breaches lead to no direct harm, and the indirect harm they do cause is hard to prove or, sometimes, even to observe. So many data breaches aren’t particularly scary, and it’s right that we don’t scare people about them.

But some data breaches are extremely serious and it’s vital we’re able to distinguish between those that are and those that aren’t so bad.

But we’re not very good at doing so.

How data breaches all came to look the same

A decade or so ago, data breaches with very large and scary headline numbers were the dominant theme in cyber security stories that hit the media and therefore registered in the public consciousness.

The world’s professional class got a jolt as early as 2012 when LinkedIn got popped, losing the personal data of 700 million users (ten years on, cyber security trainers on courses still tell students to put their email into Troy Hunt’s epically useful data breach notification service haveibeenpwnd.com and many of the positives come from this breach). A year later, Target lost 70 million customer records and 40 million (at least partial) credit and debit card records. Yahoo! trumped even that, with three billion (three billion!) accounts compromised in the same year. Equifax (2017) carried a headline of 147 million affected Americans and more beyond. Facebook’s headline figure in the 2018 breach was 533 million users. And so on.

But we struggled to relate this to actual harm resulting from these breaches. The ‘technical’ reason for this - if it can be called that - is that in most of these breaches the number of human beings involved in the dataset was huge, but the actual content and therefore value of the data involved about each person was often relatively modest. Put bluntly, your email and home address and date of birth isn’t worth that much to criminals if that’s all they get, even if they get hundreds of millions of such records. They can’t do that much ‘harm’ - directly at least - with this information.

Therefore, ‘headline’ numbers of ‘victims’ was, and remains, a terrible way of talking about data breaches.

It gives no indication of the severity of the harm done, or likely to ensue, to the people whose data was on the stolen list. If it was just a name, email address, physical address and last four digits of a bank card then the risk to the victim is not negligible but it is low, and nothing more than a bit of online financial vigilance is required. Add in a password and you’ve got a wider fraud or impersonation risk given the inevitable reuse of passwords and you need to do something (at least change the password you’ve been using anywhere else you use it). Add in a passport and you’re getting into serious risk of identity fraud and/or the onward sale of high value personal data. Add in a dataset involving myriad sensitive personal details - think mortgage application form or security clearance details, or, of course, your medical history - and some serious damage has been done.

But the way we’ve talked about data breaches doesn’t capture this nuance. In many of the big headline cases (Equifax is a great example) the severity of the breach in terms of the actual data stolen was at the milder end of the scale.

The consequence of this is that people got used to being notified that their data had been lost in a breach. But they then often did not suffer any obvious detriment in the aftermath. Contrary to some opinion in cyber security, people are generally smart and rational, and they reacted accordingly. So many people stopped being all that worried about data breaches.

In a seminal moment in the UK in October 2014, the country went into a blind panic for a weekend about what appeared to be a major breach at the telco TalkTalk. That was before the country realised that actually it wasn’t very serious at all, affecting 150,000 quite old and limited records, not the full, up-to-date accounts of the company’s four million customers. The company was fined nearly the maximum under the pre-GDPR data laws but that was because of negligent security practices; there was no evidence customers actually suffered any harm.

As a result, public perceptions of data breaches began slowly to change. In many cases, they became no big deal. For the most part, despite the constant headlines of spectacular data breaches, for the most part we learned to live with risk and move on.

Consider this official UK Government advice from 2018 following a breach at Ticketmaster, the online concert and sports tickets service. I declare an interest as both the head of the organisation that produced the guidance at the time, and as a Ticketmaster customer who got a notification. It’s cited here to illustrate a typical, late 2010s UK Government response to a data breach. In essence it says: if you’re affected, then it’s not great but don’t panic. The type of data stored can’t really directly be used to harm or embarrass you, so don’t do anything (except, as in this case change your password for this site and anywhere else you use the same one) and watch out for suspicious activity.

Or take this live example from the UK. It’s a low key news story about the breach of a water company in Staffordshire. The company has taken steps to prove that the provision of safe water is unaffected, which is true, and to be expected. Then on the data breach itself, the company is saying that’s it’s complicated to assess the damage done by the criminals when they accessed customer data. There is no sense of panic or any massive harm done to customers.

The subliminal message in both cases is: get on with your life and don’t worry too much.

And this is a good thing. We have enough FUD - fear, uncertainty and doubt - in cyber security without terrifying people even further and undermining trust in our vital digital economy. If people aren’t really at risk, then we should reassure them.

But a downside of our - rightly - grown up discussion about the damage of data loss is that we’ve lost sight of when it gets really bad. Ten years or so ago we scared people about the risk of very big data losses. We then told people most of the time they weren’t that serious. But we lost track of which ones mattered, and how to talk about that to help people and organisations manage risk.

The really bad cases were long ago, or far away

For an example of a data breach with really bad consequences, we can go back to the hack of the Office of Personnel Management in Washington in 2015. The admission by the US Government that the Chinese state had copied the data of more than 20 million Americans who had sought a security clearance in the first fourteen years of this century was - justifiably - a spine-chilling moment for that nation because a security clearance form requires telling the Government pretty much everything about you. The headline number for the Equifax breach may have been more than seven times that of the OPM but the level of detail in each OPM dataset was far, far greater, and it had gone to a hostile state. It remains, in my view, the most strategically significant digital data breach in known history.

In very different examples, the hacks of the hook-up services Ashley Madison (2014) and Adult Friend Finder (2016) caused great distress to many on the list just by virtue of the embarrassment of being on it (and, sadly, at least one suicide has been at least speculatively linked to the Ashley Madison hack).

For a harbinger of the horrors of Medibank, we can look to the Vastaamo compromise in Finland in 2020. Vastaamo was a private provider of psychotherapy. A cyber criminal stole the records of, it seems, some 30,000 patients. As well as demanding payment from the company, the attackers also sent demands to thousands of individual patients, threatening to disclose details of their consultations with their doctors about their mental health problems.

But these cases seemed to be exceptions. The American examples were dated. Events in Finland tend not to get a great deal of attention in places like the US or UK. So, even as Europe, including the UK, hardened regulation of data security with the advent of the General Data Protection Regulation (GDPR), ‘traditional’ data breaches stopped dominating concerns about cyber security.

Even GDPR didn’t really help us understand data harm

Indeed, the problem in assessing harm permeated the new regulatory environment. Before the pandemic, the UK’s Information Commissioner (the country’s data protection regulator) imposed swingeing fines of £183m and £99m on British Airways and Marriot Hotel Group respectively under new GDPR powers. Both incidents had what might be called ‘aggravating factors’ that took them beyond the ‘normal’ data breach: Marriott had millions of passport numbers retained, while in BA’s case 77,000 of the 380,000 records included full bank details including the CVV number which, most unusually in a data breach, made them fully usable by the criminals. They could spend actual money on the cards, not just sell them to other criminals on the dark web, or use them as the basis for identity fraud. That’s relatively rare in data breaches, and obviously serious.

Even in the BA case, however, it was hard to find examples of real, direct harm caused by the breach. And it was harder in the Marriott case even though passport numbers, a valuable criminal commodity, were included in the dataset. Eventually, the fines were reduced to £20m and £18m respectively, but this was explicitly done as a ‘mercy’ decision to help two companies in sectors almost wholly shut down by Covid-19. Had the cases been litigated, which appeared likely before the pandemic, the case would no doubt have centred on the companies demanding that the regulator justified the severity of the punishments in terms of actual harm caused.

That would have required a British court to assess the harm done by a data breach. And that would have been a tough question. With no case taking place, we do not have a legal precedent on which to base future planning about how British courts view data harm.

In the absence of any such ruling, how we assess and price data harm is clearly something governments, regulators, the cyber security industry and large data holders need to think about.

Ideally, we’d have spent more time recently thinking through these problems and coming up with solutions. But, set against the ransomware horrors of 2021, with a major pipeline shut in the US, hospital administrative systems in chaos in Ireland, France and elsewhere, fresh food being thrown away in Sweden because it couldn’t be sold, and so many other deeply damaging ransomware cases, all of a sudden data dumps didn’t seem to be so big a deal. And as the West began to worry about the potential - and thankfully this remains potential - uptick in disruptive cyber aggression from Russia in the context of the invasion of Ukraine, data security slipped further down the consciousness of many parts of cyber security’s body politic. In many ways it was rational to begin to downplay data in the hierarchy of cyber risks.

Why what happened in Australia matters

But Australia’s painful experience reminds us that data security as a problem hasn’t gone away. It can cause real damage to an advanced digital society, and as a result needs the attention of the cyber security community and of governments.

Initially, the Optus case looked like it could have been just another data breach with one of those very large headline numbers of victims (estimated at 9.7 million), with data of limited value. But this swiftly changed. What became known as the ‘crucial subset’ - those with more important identifiers like passports and driving licenses - took longer to work out. Eventually the company disclosed that some 1.2million valid records of this type, and a further 900,000 expired ones, had been stolen. Such a large scale breach of important personal identifiers is what turned Optus into more than the usual common data breach.

Medibank, a few weeks later, required little explanation of its complexity to hit home with the general public. The problem with the theft of nearly ten million medical records being out in the wild was obvious: this was the Vastaamo crisis in Finland at population level scale.

Sadly, comparing Optus and Medibank provides a really good way of thinking about how we assess harm because the headline numbers involved in the dataset are almost identical at 9.7 million each. But the entirety of the Medibank dataset involves personal medical histories, so its impact is much, much worse. One only has to look at the Australian media when the details emerged to see the sort of impact that had in the general population. Many more instances of this and we could have a serious crisis of confidence in the digital economy.

That said, the Australian public reacted calmly and the company put in place what has been mostly regarded as a helpful set of communications to concerned customers about what may or may not be happening to their data. Moreover, and as we shall discuss further in the fourth post of this series, the way in which the breach has been reported and discussed in Australia and beyond has generally been responsible, which has helped limit the damage. But the Medibank affair shows us that data breaches can cause national scale concern and even fear in some parts in the population of a wealthy, highly digitised country.

Where do we go from here?

If Australia’s experience forces us to reprioritise basic data security, what should we do?

The most important thing we can do is really quite a geeky, technocratic thing.

We need to find ways of measuring harm that helps us distinguish between different levels of severity in data breaches.

This matters in getting regulation right.

It matters for the fledgling but important cyber insurance market.

It matters for corporate risk management. Companies need ways of calculating what they need to protect most.

Most of all it matters profoundly for the public understanding of cyber risk. People manage risk better, and react more rationally when problems occur, when they understand what’s going on.

So all of this should matter to those looking after our cyber security.

Where we need to get to is a situation- not just a regulatory regime but an accepted understanding in society at large - where the relative value of different datasets and therefore the difference in harm done by their compromise is widely understood.

A situation where we automatically recognise that the compromise of healthcare records is demonstrably more serious than a basic customer dataset. That we take into account cyber security when designing data storage policies and practices with all these nuances and subtleties in mind.

Ultimately, the destination is one where our reaction to a data breach isn’t determined by the headline number of records compromised. Instead, there’s a general understanding of what happens when data disappears. How often there’s not much to worry about, but that sometimes it’s more serious. And even then we understand that it’s not clearcut: that even in the worst case scenario it’s unlikely that health records will end up on an easily available website but we that we do need to take steps to make sure we’re not susceptible to extortion, identity fraud and other forms of harm.

This involves a lot of hard, unglamorous but vital work. And it’s another example of how solving cyber security’s major problems sometimes involves doing things that aren’t really about the technical aspects of computer network security.

In short, we need better ways of understanding the price of data loss. A lot of progress could flow from that. And serious data breaches which damage public confidence in the digital economy still matter, even with all the other problems in cyber security.

Next up: Governments and cyber incidents after the events in Australia

Perhaps inevitably, given the obvious public concerns about passport data and especially healthcare records, the Australian Government’s role in the response to both Optus and Medibank has been notably activist. But it’s also been distinctive and innovative in some areas. That is likely to have wider ramifications for how Governments approach not just major cyber incidents, but cyber security strategy as a whole.

That issue - the probable increasing role of Governments in cyber security in the light of Australia’s experience - is the focus of the next post.

Subscribe now

Share

It will be (initially) a weekly long from post about (mostly) cyber security, with some occasional reflections on the UK constitution, or something else.

It’s written by Ciaran Martin, Professor at the Blavatnik School of Government, University of Oxford, and former/founding head of the UK’s National Cyber Security Centre, part of GCHQ. I’m also an adviser to a small number of brilliant cyber security companies.

I was a senior UK civil servant for a very long time. Before I became obsessed with cyber, I was a Treasury, national security and constitutional obsessive.

Subscribe now