I spent last week in Shanghai for the World AI Conference and a Track II dialogue on Agentic AI. Throughout the conference (not the Track II), one glaring observation stood out: for multiple days, in a hall full of politicians, UN officials, and the world’s brightest researchers from leading AI labs, almost nobody said the word “military.”

That is not a complaint. It is the most interesting thing about the whole event. In fact, I made the mistake of saying the word ‘military’ while talking with some roboticists and was politely asked not to. I felt like I had broken an unwritten rule of the conference. In the excellent side events to the conference, there was some talk of the military dimension, but it was easy for people to talk past one another.

The WAIC 2026 was a big one. Xi Jinping gave the keynote in person for the first time since the conference began in 2018. Delegations from more than a hundred countries were there, and twenty-nine states signed the agreement establishing the World AI Cooperation Organization — the first intergovernmental body dedicated to AI, headquartered, of course, in Shanghai. The exhibition floor was massive with loads of robots and embodied AI, including a child-sized robot that kicked a soccer ball (football, if you like) into the crowd with concerning power (which it was not supposed to do…) or a dancing robot that nearly knocked out its handlers. There is clearly room for improvement.

But from a policy perspective, if ever there was a signal that China intends to shape AI governance, this was it.

The speech

Xi’s address was structured around four proposals — openness and cooperation, risk awareness, civilizational inclusiveness, and multilateral governance — and it was, as these things go, unusually substantive. Check out this great reference piece by Matt Sheehan.

The line that will be quoted in every policy brief over the next year is his insistence that AI must “always remain under human control,” paired with the pledge that China will keep development and security in equal balance so that, in his metaphor, the fine steed of AI can run fast and steadily. I love it.

But read the second proposal to the end, and you find another target: a call to oppose “overstretching the concept of national security” in AI and to reject any country placing its own security above everyone else’s. That is not a sentence about killer robots. That is a sentence about export controls aimed squarely at Washington. While the safety vocabulary is real and important, it is doing double duty for trade politics. Both are true at once, and if you read it only one way, you will miss something important.

Share

The two words that dominated – and why they belonged together

The safety content was not merely decorative, and human control at both the macro and micro levels (humanity’s control over AI’s trajectory versus individual control over particular systems) was central to most conversations. The Chair’s Statement is worth reading: paragraph 7 commits to keeping AI under human control, while paragraph 8 calls on frontier labs to build guardrails and deploy their systems prudently.

Then comes paragraph 9, on agents. It says that AI agents should operate within clearly defined decision-making authority and behavioral boundaries, supported by mechanisms for tracing behavior and issuing risk alerts.

The exhibition floor made it clear why those requirements are necessary and why they will be difficult to meet. The trend everyone noted was the shift from question-and-answer systems to systems capable of complex execution, including multi-step planning, tool use, real-world action, and embodiment. It is no longer simply about systems that answer you, but about systems that do things.

And once a system can decide how to pursue a goal, the safety question changes. It is no longer enough to ask whether a human authorized the final action. We also have to ask who authorized the route the system took to get there, how its intermediate decisions can be bounded, and where and how human control is supposed to occur throughout the chain.

Of course, those questions are not uniquely military. But they lead directly into a debate that has spent more than a decade trying to capture human control at one particular moment.

Subscribe now

Shanghai & Geneva: two sides of the same world?

Generally speaking, the autonomous weapons debate has been organized for a decade around a single moment: the selection and engagement of a target exclusively by AI; or, without human intervention. That framing was always a bit of a fiction, but it was a useful one. It gave policymakers, lawyers, and engineers a discrete point at which to examine human control, technical capability, and legal responsibility.

An agentic system dissolves that point. Give a planner a goal, a toolset, and a permissive environment, and it will decompose that goal into subtasks no one outlined in advance, across a timeline no one scheduled, with intermediate steps that were never individually authorized by a human. The implications are enormous. One thing that kept coming to mind was how much harder the job just became for lawyers. For years, lawyers have been grappling with how to ensure autonomous systems comply with international humanitarian law, or IHL, and the solution to that problem just got a lot harder.

The GGE reconvenes in Geneva on 31 August, and its final report goes to the CCW Review Conference in November, where the Delegates will be arguing about target selection. They will be careful and serious, but it seems they will be working from a picture of autonomy that the exhibition floor in Shanghai has already, in certain respects, technically left behind. The AI community has already moved on to something harder.

Whatever Geneva’s shortcomings, its delegates are at least directly confronting the defense implications. In Shanghai, those implications were often treated as outside the conversation, even when the technologies on display made them impossible to ignore.

The problem is no longer simply how to preserve human control at the moment a target is selected. It is how to govern systems in which the consequential moment has been distributed across a chain of machine-generated decisions. It became clear that Shanghai is already in that world, and Geneva is still trying to name it. For now, the word connecting them remains the one nobody wanted to say.

Leave a comment

Share

Subscribe now

The Department of War and Anthropic are in a public fight over AI and weapons. If you’ve been following the headlines, you’ve seen the escalating rhetoric — Anthropic’s CEO calling the White House’s actions “retaliatory and punitive,” Secretary Hegseth firing back that Anthropic “places Silicon Valley ideology above American lives.” It’s dramatic. It’s also a case where both sides are getting the substance wrong.

Here’s what happened. The DoW declared that it “will not be woke” and will operate “without ideological constraints that limit lawful military applications.” It accused Anthropic of trying to “seize veto power over the operational decisions of the United States military.” Then it went further: the President directed the federal government to cease all use of Anthropic’s technology, and the DoW designated Anthropic a Supply-Chain Risk to National Security. Any contractor, supplier, or partner doing business with the U.S. military is now barred from commercial activity with Anthropic, with a six-month transition window.

Anthropic’s response pointed out that it was the first frontier AI company to deploy models on classified government networks, the first to deploy at the National Laboratories, and the first to provide custom models for national security customers. Claude is already embedded across the DoW and other agencies for intelligence analysis, modeling and simulation, operational planning, and cyber operations. Anthropic drew two lines: it doesn’t believe today’s frontier AI models are reliable enough for fully autonomous weapons, and it considers mass domestic surveillance of Americans a violation of fundamental rights.

Where Anthropic Gets It Wrong

Anthropic’s position suggests a misunderstanding of how the military operates and what the law actually requires. The DoW’s own regulations already prohibit the employment of “fully” autonomous weapons — meaning weapons that lack meaningful human control, where all decisions governed by international humanitarian law are delegated entirely to a machine. The DoW hasn’t asked for a system that would be unlawful under the Laws of War. And the DoW has no legal authority to conduct surveillance on U.S. citizens; using AI for that purpose would itself be inherently unlawful.

It’s worth getting precise about what “fully autonomous” actually means, because the term is doing a lot of work in this debate. A fully autonomous weapon, per Human Rights Watch and the Stockholm International Peace Research Institute, is one that is completely independent of human will — where IHL decisions are fully delegated to the machine. Nobody in the DoW is asking for that. What the military does use, and wants to continue using, are autonomous and semi-autonomous systems where humans remain either “on the loop” (monitoring and able to intervene) or “in the loop” (actively approving engagements). These systems already exist and operate within established legal frameworks.

The deeper technical issue isn’t really about autonomy at all — it’s about whether a system is deterministic or stochastic. A traditional autonomous weapon follows predictable, rule-based logic. An AI-enabled weapon introduces probabilistic behavior, and the real question is whether it will operate as intended. That’s a legitimate engineering and policy challenge, but it’s a different conversation than the one Anthropic is having publicly.

Where the DoW Gets It Wrong

The DoW is mismanaging what should be a straightforward relationship. Anthropic is a contractor that doesn’t have deep institutional knowledge of military law and regulation — of course they don’t know the intricacies of how the DoW governs autonomous systems. The right move is to explain it in private, not to publicly brand them a national security risk.

Anthropic’s models are widely deployed and generally considered among the best available. Coming down this hard on a company for raising questions — questions that are, frankly, not unreasonable from a civilian perspective — is counterproductive, and robbing the DoW of a system that is, in my experience, probably the best performing of its type. Google went through a similar episode in 2018 over Project Maven, and the concerns about government surveillance and autonomous lethal systems are not fringe positions. They’re widely shared across the tech industry and the general public. We can address these concerns without shooting ourselves in the foot, while literally, there’s a war on…again.

Oh, and btw, Anthropic is right. The models are not ready for reasoning at the level that many senior DoW leaders think is possible. But, pray tell, why do they think that? Because the AI companies have been feeding them a steady diet of hype for years. Of course the SECWAR is going to be eager to use them for things they cannot do. He's been told they can do them. In many ways Anthropic, Meta, OpenAI, xAI, and on down the line have created unreasonable expectations in senior military leadership and those flag officers want what they have been promised. There’s a war on…again.

What the Military Actually Wants AI For

The military’s interest in AI falls into three broad categories. The first is tools for warfighting speed: AI as a critical interface layer (speech-to-code, text-to-code) turning thought into combat code for automation of tasks like drone maneuver, object classification, and automatic target recognition. The second is quality and novelty of thought — enhanced reasoning, novel insight, the kind of lateral thinking exemplified by AlphaGo’s famous Move 37. And the third is cognitive offloading, which I’ve written about before. Freeing human reasoning to do the tasks it’s best at, like applying the laws of war and solving high-context problems, rather than supervising routine processes.

All of this is, or can be, entirely legal. Readers will remember that the Laws of War basically operate on four core principles: military necessity (is the action necessary for the mission?), distinction (did you distinguish between combatants and non-combatants?), proportionality (was the action proportional?), and precaution (did you take reasonable steps to protect non-combatants?). If an AI-enabled weapon system can satisfy those requirements operating in a human defined context, then likely the autonomous systems would be compliant with the Laws of War.

The real questions worth debating — how to ensure AI systems behave predictably in combat, how to maintain meaningful human control over stochastic systems, how to govern machine autonomy with the same rigor we apply to human autonomy through training, orders, supervision, culture, and consequences — those are getting drowned out by the political theater on both sides.

Been awhile but we haven’t been idle.

I just returned from Estonia and the EstMil conference where I was discussing different angles of AI-enabled decision-making. Particular on the dimension of cognitive dependence.

I thought I’d post the written copy of my remarks. Keep in mind that this is a fairly concise summary, so lots to unpack here. But good for discussion. Those who have followed some of this page will see this as a bit of a rehash but it’s a concise version.

Cognitive Dependence: The Quiet Risk in AI-Enabled Decision Support

Bradley L. Boyd

January 2026

Artificial intelligence is increasingly described as a “force multiplier” for human decision-making. In military contexts, that phrase often carries a reassuring implication: that humans remain in control while machines merely accelerate, organize, or clarify information. Yet history suggests that when tools begin to think with us—or for us—the nature of control changes in subtle but profound ways.

The question facing modern militaries is not whether AI-enabled decision support systems will be used. They already are. The more difficult question is whether those systems will quietly reshape human cognition in ways that undermine the very control they were meant to preserve.

This article explores one such risk: cognitive dependence—a condition in which human operators come to rely so heavily on AI-enabled decision tools that they retain nominal authority but lose meaningful cognitive agency. Unlike science-fiction fears of rogue machines or autonomous weapons run amok, cognitive dependence is neither dramatic nor intentional. It emerges gradually, often rationally, and most dangerously in exactly the environments where speed and precision matter most.

Decision-Making Has Always Been Technological

Warfare has never been a purely human cognitive enterprise. Maps externalized spatial memory. Radio compressed time and distance. Fire-control systems automated ballistic calculations long before the digital age. Each of these technologies reshaped how humans decided, not just what they decided.

AI-enabled decision support systems represent a continuation of this trajectory—but also a qualitative shift. For the first time, machines are not merely executing calculations or relaying information. They are performing cognitive tasks once reserved for human judgment: identifying patterns, prioritizing options, predicting outcomes, and recommending actions.

In conflicts such as Ukraine and Gaza, AI-enabled systems have already demonstrated their ability to dramatically accelerate targeting cycles, fuse intelligence at scale, and operate at tempos impossible for unaided humans. The operational appeal is obvious. Under pressure, delegating cognition to a machine can feel not only efficient, but necessary.

The problem is that cognition, unlike logistics or propulsion, is deeply tied to agency, responsibility, and moral judgment. When machines begin to carry the cognitive burden, humans do not simply gain speed—they may lose something less visible but more important: the ability to independently think through decisions in context.

Humans on the Loop, Machines on the Mind

Much of the current policy discussion around AI in warfare focuses on concepts like human-in-the-loop or human-on-the-loop. These frameworks assume that keeping a human formally involved in the decision process is sufficient to maintain control.

But emerging evidence suggests this assumption may be flawed.

In environments where AI systems are fast, accurate, and institutionally encouraged, human operators may begin to treat machine outputs as default truths rather than inputs for deliberation. Approval becomes procedural. Oversight becomes symbolic. The human remains present, but their cognitive role shifts from decision-maker to validator—or worse, rubber stamp.

This is not because operators are negligent or unethical. On the contrary, it often reflects rational adaptation to constraints. When cognitive capacity is degraded by stress, fatigue, time pressure, or information overload, offloading cognition to a machine becomes a sensible survival strategy.

The danger is that this offloading can harden into dependence.

Cognitive Dependence Is Not a Bug—It’s a Feature

Cognitive dependence is often framed as a failure of training or discipline. In reality, it is a predictable human response to high-performance automation.

Psychology and human-machine interaction research offer several mechanisms that help explain why dependence emerges:

· Cognitive offloading allows humans to conserve mental resources by delegating tasks to tools. This is beneficial—until it becomes habitual.

· Automation bias leads operators to favor machine outputs over their own judgment, especially when the system appears reliable.

· Confirmation bias reinforces trust when AI recommendations align with preexisting beliefs.

· Action bias pushes decision-makers toward doing something—anything—under pressure, particularly when a machine offers a clear path to action.

These tendencies are not pathologies. They are adaptations. In wartime contexts that reward speed and volume, AI-enabled decision support systems may amplify these biases rather than mitigate them.

Once dependence sets in, removing the system does not restore human performance. The cognitive skill has already been eroded. The human may still be “in control” in a formal sense—but no longer in a meaningful one.

Control Is Not Binary

One of the central challenges in this debate is that control is often treated as a yes-or-no condition. Either humans are in control, or machines are. Either autonomy is preserved, or it is lost.

Reality is messier.

Control exists on a spectrum and varies by context. A human might retain control over strategic intent while relinquishing control over tactical execution. They might define desired outcomes but allow machines to shape the paths that lead there. Or they might retain veto authority while surrendering the cognitive work that informs whether a veto is warranted.

Crucially, a human can be globally autonomous—legally responsible, morally accountable—while being locally cognitively dependent within a specific system or decision environment.

This distinction matters. It suggests that preserving meaningful human control is less about where the human sits in a workflow and more about what cognitive functions they retain.

The Subtle Shift from Judgment to Compliance

One of the most underappreciated risks of AI-enabled decision support is not that machines will make bad recommendations—but that humans will stop evaluating whether recommendations are appropriate at all.

As systems improve, the cost of disagreement rises. Questioning the machine slows the process. Challenging its output may require justification up the chain of command. Over time, the path of least resistance becomes acceptance.

This dynamic is particularly acute when AI systems are framed as neutral, objective, or statistically superior. Percentages and confidence scores can obscure uncertainty rather than illuminate it. A system that is “90 percent accurate” invites compliance, not reflection—especially when the remaining 10 percent is undefined.

When this happens, human judgment is not replaced outright. It atrophies.

Decision Support as Choice Architecture

AI-enabled systems do more than recommend actions. They shape the choice architecture within which humans decide.

What options are presented? Which are excluded? How are they ordered, framed, or justified? What information is emphasized, and what is hidden behind confidence metrics or default settings?

A human cannot choose what is not visible. As AI systems increasingly curate the decision space, they exert quiet influence over outcomes—even when humans retain final authority.

This influence is not necessarily malign. In fact, narrowing choices can improve performance under pressure. But it also raises uncomfortable questions: At what point does simplifying the decision environment become steering it? And who decides where that line lies?

Measuring What We Are Losing

One of the most challenging aspects of cognitive dependence is that it is difficult to detect in real time. Outcomes may remain acceptable—even optimal—until they suddenly are not.

Traditional metrics of system performance do not capture whether humans are thinking independently or merely complying efficiently. A successful strike does not reveal whether it was the product of human judgment or machine momentum.

This raises a provocative question: Should we be treating human cognitive autonomy as a system variable, rather than a moral assumption?

If autonomy fluctuates with stress, tempo, interface design, and institutional incentives, then it may be possible—at least in principle—to monitor when humans are becoming overly dependent on machines.

Doing so would challenge deeply held beliefs about trust, professionalism, and accountability. But it may be necessary if “meaningful human control” is to be more than a slogan.

The Mirage of Transparency

Calls for transparency, explainability, and predictability in AI systems are well-intentioned and important. But they may not be sufficient.

A system can be technically transparent yet psychologically overwhelming. It can be explainable in theory but opaque under time pressure. It can be predictable in isolation but generate unpredictable human responses.

In high-stakes environments, transparency does not automatically translate into comprehension. Nor does explainability guarantee engagement. In some cases, detailed explanations may actually encourage deference: if the machine seems too complex to question, the safest option is to trust it.

This suggests that human-machine interaction design, not just algorithmic performance, may be the decisive factor in preserving cognitive autonomy.

Open Questions, Not Answers

If cognitive dependence is an emergent property of high-performance decision support, then mitigation cannot rely solely on prohibitions or procedural safeguards. It requires asking harder questions about how systems are designed, deployed, and incentivized.

Some of those questions might include:

· Should AI systems sometimes intentionally slow humans down rather than accelerate them?

· Is there value in designing systems that occasionally force humans to reason without assistance?

· Could decision support tools make uncertainty more felt, not just displayed?

· Should operators be able to dynamically trade speed for autonomy based on context?

· How might training environments simulate cognitive degradation to expose dependence before deployment?

· Could systems surface not just recommendations, but alternative framings that challenge operator assumptions?

More speculative ideas might include:

· Autonomy budgets, where systems visibly track how much cognition has been offloaded.

· Cognitive friction modes that increase human engagement during ethically or strategically sensitive decisions.

· Second-order questioning interfaces that ask operators why they agree with a recommendation, not just whether they do.

· Dependence alarms triggered not by error rates, but by behavioral patterns like rapid approvals or narrowing choice exploration.

None of these ideas are solutions. All introduce tradeoffs. Some may prove impractical or counterproductive. But avoiding the questions altogether risks drifting into a future where humans remain nominally in charge while machines quietly define the battlespace—cognitively, not kinetically.

Why This Matters Now

The future of decision-making in warfare is not predetermined by technology. It is shaped by choices—design choices, policy choices, training choices, and cultural choices.

AI-enabled decision support systems will almost certainly make militaries faster and more capable. The harder challenge is ensuring they also make humans better decision-makers, not merely faster approvers.

Cognitive dependence does not announce itself with malfunction or rebellion. It arrives as convenience, efficiency, and success—until one day it becomes difficult to tell who is really deciding.

That is not a problem for the distant future. It is already here.

Share

Leave a comment

Subscribe now

You may have noticed I’ve been pretty quiet on KRCP lately. Thanks to Brad for doing a lot of the heavy lifting. 2024 was a hectic year for my family as we had multiple trans-Atlantic moves and welcomed a baby girl in early October. So, I have been in the throes of newborn life. But I am ready to shake the intellectual cobwebs off and share a bit about what I’ve been working on lately.

When I’m not helping a baby learn to laugh, I’ve been working on a book chapter about national security governance and military AI. As many of you know, my research mainly concerns human control questions (policy, legal, operational) of AI-enabled autonomous weapons. Don’t worry – I'll discuss that again soon. Still, I was asked to write this chapter on an interesting topic, and I thought trying something new might be fun.

So, I’ve been thinking through issues of AI governance, why national security is so often excluded from global governance initiatives and the limits of international law in the governance discussion. Let’s get into it.

Subscribe now

There is no shortage of calls for AI governance. We have seen initiatives at international and national levels to identify and regulate uncertainties and risks related to AI. These initiatives often point to the lack of accountability and transparency in AI, the unequal distribution of its benefits, and the consequential effects of risks, such as bias, surveillance, and disinformation. For a truly global perspective, check out the UN HLAB on AI’s final report, which lays out a roadmap for AI governance.

However, some experts, particularly at this global level, have noted the absence of military applications in many of these governance initiatives. The UN HLAB report does not substantively address defense or security applications of AI (aside from noting security concerns), despite some of the members’ expertise in military AI. And certainly, global efforts still call for the ban of AWS, with some state support in international forums.

There are obvious reasons for this exclusion of military applications from governance initiatives. National security concerns and applications would impact the scope and likely limit the potential for these regulatory and policy initiatives. Experts calling for national security governance often do not engage with the many (many) reasons that traditional governance pathways cannot adequately govern the national security space.

A few reasons that come to my mind:

No international oversight. States would not want to subject a rapidly developing capability to external oversight and potential limitations. Particularly as AI can offer:

Competitive advantage in combat. AI-enabled systems can offer a qualitatively new capability and offer a strategic advantage over adversaries, particularly near-peer.

Dual-Use Nature. Governance initiatives have struggled to delineate civilian and military applications, as the capability can easily blur the lines. For this reason, efforts focus on concrete applications that typically do not involve many potential military applications.

There is a lack of global consensus. Governance initiatives for defense and security struggle because it is nearly impossible to reach an international consensus. This is because legal obligations (or interpretations of those obligations) differ, ethical red lines and preferences differ, and security priorities and interests differ and rapidly change in an evolving security landscape. In short, more factors feed a lack of consensus than those that help states reach a consensus about military AI.

Geopolitical landscape. Related to the challenges for global consensus, tensions between the US and China (among others) include central concerns about emerging technologies and AI-enabled systems, especially weapon systems. States locked in that kind of competition are unlikely to accept (or seek out) external governance measures.

Enforcement capacity. Ensuring compliance and enforcing security measures is challenging, even if governance frameworks were established. If there is a security interest in breaking these measures, then it is difficult to establish trust in adhering to agreed-upon restrictions.

For these reasons and others, formal governance initiatives for military AI are unlikely to be implemented. However, that is not to say that nothing will impact the development and guidance of AI implementation.

A few years ago, I co-authored a paper with the brilliant Matthijs Maas, arguing that strategic partnerships centered on the development of military AI will be a strong pathway for governance. The article explores the concept of strategic technology partnerships and discusses the uses, practices, and distinct operational requirements involved in military AI strategic partnerships. We examined four cases: (1) the US-led AI Partnership for Defense (PfD); (2) the AUKUS partnership; (3) robust China-Russia cooperation on military AI; and (4) transatlantic (especially US-UK) cooperation. The article discusses the implications of these partnerships for broader AI governance stakeholders, standing military alliances like NATO, and the development and usage of military AI itself.

Others have recently made similar arguments. For example, Brianna Rosen made some great points in a WotR post. Like Rosen, Maas and I argue that international law alone is insufficient to regulate military AI and function as a primary governance mechanism. International law requires vastly different behaviors based on whether actions occur in an armed conflict, as one example of a legal distinction with different obligations. Ultimately, international law cannot regulate across the portfolio of use cases, even within the military realm alone.

Strategic partnerships will offer substantial guidance for states as they develop these capabilities. The importance and utility of interoperability and strategic integration offer substantial incentives for working alongside trusted partners, particularly outside the US. These partnerships support governance by defining and delineating interests and agendas, promoting cooperation among partners toward a shared goal, (potentially) curating industry and academia to support co-development, and establishing principles and frameworks for partner implementation and use.

These partnerships are an essential tool in the governance arsenal. While they don’t solve many of the issues laid out above, they are, in my opinion, the best tool for guiding development and implementation, with more potential and many global policy forums.

What do you think? Did I miss key challenges to military AI governance? Have I put too much promise in strategic partnerships?

Let me know in your comments. Preferably before I finish my book chapter. :)

As I mentioned in the last posting, here is the full version of the article Tiffany Saade and I penned last month in the Republic Journal.

I was recently at a Track II dialog where I briefed some of these ideas and I received some really good feedback and some new ideas to develop. While there, several of my colleagues asked me to hurry up and post the full article, so here it is, all 6000 words of it.

Be careful what you wish for…

Human-centered warfare: Optimizing human cognitive autonomy and avoiding machines on the loop.

Sep 2024

Bradley L. Boyd

Tiffany Saade

Machines are helping humans think on the modern battlefield. In Ukraine and Gaza, AI-enabled decision-support tools are helping humans fight wars at greater speeds and higher tempos. This increase comes from automating parts of human cognition. Instead of hundreds of humans spending thousands of hours reading and analyzing reports, AI-enabled decision support tools sift through large amounts of intelligence data, including intelligence reports, video, and signals information to identify and recommend targets for action. Governments and militaries consider these tools important technology that improves human performance in war while the humans remain firmly in control.

Initial reporting from the war in Gaza suggests that the reality of human control over AI-enabled decision support tools may be more complex than keeping a human on or in the loop of decisions. Human operators may have become so cognitively dependent on these tools that they no longer have meaningful control over what the machines are recommending or the resulting actions. Human behavior when teaming with AI-enabled decision support systems suggests some difficult questions about whether humans can retain their own cognitive autonomy in an environment of sophisticated cognitive automation. In this article we consider some of those questions and provide a framework for how to think about maintaining human control through cognitive autonomy in the context of AI-enabled warfare.

The relationship between human and machine

The idea of machines thinking for humans seemed far-fetched only two decades ago, but the last ten years saw a dramatic improvement in compute power and new methods of machine-learning that allowed machines to perform some cognitive tasks as well or better than humans—and in most cases much faster. Under the catch-all term artificial intelligence (AI), humans are creating tools that give machines more ability to perform what were previously human cognitive functions: AI-enabled machines are designing and testing new drugs to improve human health;[1] they are beginning to do open-ended scientific research;[2] they are making art;[3] and they are starting to help fight wars.[4]

In warfare, AI-enabled systems are identifying targets, looking for patterns in intelligence data, and helping prioritize efforts. The integration of more AI-enabled systems has militaries and governments considering what role machines can and should play in the cognitive functions of war and how to ensure human oversight.

As AI-enabled machines take on more cognitive tasks from humans, should we be worried about becoming cognitively dependent on them? Instead, should our goal be to maintain as much cognitive autonomy as possible in all circumstances? If we accept some cognitive dependence on machines because of real benefits, how can we know when we have become “too” dependent? Is there a right amount of cognitive autonomy? If so, how do we build a machine that improves outcomes but prevents dependence and preserves cognitive autonomy?

We argue that optimizing AI-enabled decision support tools for human cognitive autonomy, within a specific context, would ensure that outputs are aligned with human-defined ends while allowing maximum benefit from machine automation.

Machines on the Loop

In a 2016 article, Paul Scharre suggested future human-machine teaming would take the form of a centaur: human brain enabled by machine power. Humans guide and control the machines that increasingly execute complex warfighting tasks at beyond human speed. In centaur warfare, the human retains the roles of essential operator, moral agent, and failsafe, leaving the machine to help where feasible and appropriate.[5] The metaphor of a centaur is meant to illustrate that the human-machine team is improved by the machine, but the human retains cognitive and moral control of what the team does.

In 2023, Sparrow and Henschke responded that the machines are becoming so good at cognitive tasks that humans will turn cognitive control—and, by default, immediate moral control—over to machines. Instead of a centaur, human-machine teams will be minotaurs.[6] Sparrow and Henscke believe that AI-enabled cognition tools will become so good that armies will have AI Generals as the brains making decisions that the human bodies execute. They believe this is starting to happen now and that AI technology is improving so rapidly that we are likely to be confronted with minotaurs very soon. Instead of humans on the loop, we will have machines on the loop.

There may be pressure someday to intentionally turn cognitive control of warfare over to machines, but that seems unlikely in the near term. Most nations are insistent that humans will retain control over the actions of machines, and that the role of machines is to help the humans perform better. Turning over cognition to machines would seem to violate the principle of meaningful human control gaining consensus in the international legal community.[7] It would also seem to violate the U.S. Department of Defense’s requirement that any employment of autonomous weapon systems display appropriate levels of human judgment.[8] The problem is that the way those systems are designed, and the context in which they operate, increases human cognitive dependence on them—which means humans may be unintentionally giving up meaningful human control of AI-enabled systems.

Cognitive Dependence in war

There are times when cognitive dependence on a machine seems to be all benefit and very little, if any, risk. Consider a calculator. It takes most of us a lot of cognition to do math. Having a calculator do it for us is a great benefit. Plenty of humans would not consider doing even moderately complex math without it. A calculator almost always performs at 100 percent accuracy, and we do not worry about it enslaving us, or worse, hallucinating answers. But the stakes of cognitive dependence can be much higher when machines help solve cognitive problems in complex contexts like war.

Militaries around the world are racing to integrate artificial intelligence into decision-making systems to help humans make better decisions faster in war. These systems are called AI-enabled decision support tools. With these tools, machines sift through massive amounts of intelligence data to identify, select, and recommend targets for action, sometimes to lethal effect. These decision support tools perform tasks that previously took hundreds of human analysts, they do it in a fraction of the time, and they do it well—so well that there is an incentive to create more powerful systems that can perform ever more sophisticated human cognitive tasks. Those future cognitive tasks could include analyzing and recommending options for military forces, directing the actions of human forces in combat, coordinating multiple lethal systems to destroy targets, and optimizing which forces get which assets to do their missions.

AI-enabled decision support tools, like the U.S. military’s Maven Smart System, have been used in Ukraine and in the Middle East to help identify, track, and prioritize targets.[9] Similar systems have been used to more dramatic effect in the Israel Defense Force’s (IDF) operations in Gaza. Reporting on the use of these systems suggests some troubling emerging behavior in human operators intended to oversee these AI-enabled tools.

Use of AI-enabled decision support tools in Gaza suggests that under certain circumstances, human operators can become overly dependent on AI-enabled systems, and that dependency can result in actions that may violate policy and perhaps international humanitarian law. Reports have centered primarily around two IDF systems, Gospel and Lavender. Both systems scrape massive data repositories to identify targets for military action. Gospel predicts which structures are used by Hamas, while Lavender predicts who is a Hamas operative and where they might be located. That information is then used to strike those targets as part of the larger military operation.

Both tools have increased the speed and volume of IDF targeting operations in Gaza. Prior to the introduction of AI-enabled decision support tools, the IDF targeting process required 20 IDF intelligence officers to produce 50-100 targets in 300 days; Gospel produced 200 targets in 10-12 days.[10] The Lavender system identified 37,000 suspected militants in the first few weeks of the conflict and is at least 90 percent accurate.[11]

IDF leadership encouraged operators to maximize use of the machines for multiple reasons: political imperatives demanded striking as many targets as fast as possible; leadership considered the machines statistically accurate enough to remove direct human oversight; and the machines removed human emotional subjectivity when identifying targets.[12] Some operators admitted to spending only 20 seconds on approving targets, essentially verifying that the target was a male, while others acknowledged that they preferred to just follow the machine because it was statistically accurate and “…the machine did it coldly. And that made it easier.”[13]

The battlefield results from use of AI-enabled decision support tools are difficult to isolate. The introduction of Gospel and Lavender in Gaza corresponds with greater speed and volume of offensive kinetic strikes when compared to other recent conflicts. By one report, the war in Gaza has seen more daily casualties than any conflict in the last 24 years.[14] We probably will not know whether the increased speed and volume was a positive result for operational objectives until after the war. However, by comparing IDF press releases to unofficial investigations and reports, we can infer that use of Gospel and Lavender in Gaza may not have complied with IDF policy or the Laws of War[15].

The IDF made it clear that reports about operator use of Gospel and Lavender are not consistent with IDF policy or IDF commitment to the Laws of War.[16] The IDF suggested that the reports of operator misuse are false, however, the IDF does not dispute reports of what the tools are designed to do. If we assume the open-source operator reports are accurate, and that IDF policy described in press releases is accurate, then we can infer that human use of the tools has diverged from what was intended when they were developed.

It is important to understand that the tools themselves did not violate any policies or laws. The existence and design of the tools as reported do not appear to violate any requirements for distinction, precaution, or proportionality in the targeting process. The issue lies with how humans use and interact with the tools. In the case of Gospel and Lavender, operators became dependent on AI-enabled decision support tools for approving targets for strike because they were encouraged to do so by the context they were in..

In the presence of machine speed and accuracy, and under cognitive pressure to do more, the operators of these systems became dependent on the machine. The humans stopped thinking and let the machines think for them. Why?

Dependency from increased cognitive bias and decreased cognitive capacity

When human cognitive capacity is reduced, meaning the ability to think is disrupted by physical or mental stress and exertion, the human mind can be more susceptible to cognitive biases and shortcuts to keep thinking and acting with less expenditure of energy. These biases and shortcuts can be very useful; however, they may lead to cognitive dependency on machines with detrimental downstream effects.

One way humans reduce their cognitive load is through a process called cognitive offloading. This is when physical acts are used to alter the processing requirements of information and can include actions like tilting your head to see a sideways image or using a calculator as in our previous example.[17] In times of mental stress and reduced cognitive capacity, a mental strategy to offload cognition to a tool is a rational choice. Sustained offloading may result in what is essentially a long-term dependency particularly if the results seem to be exactly what is desired.

Humans tend to seek out or overweight information that aligns with their beliefs while avoiding or underweighting evidence that contradicts them.[18] A recent study suggests that when psychologists use an AI-enabled decision support system they tend to trust and accept AI recommendations that align with their initial intuition, and further, those with greater expertise were more skeptical of results when the AI tool suggestion deviated from their professional judgment.[19]

When interacting with machines, particularly machines that perform some level of human cognitive task, humans can also bias towards automation if they perceive it to be particularly performant or if the results seem to be what is expected or desired.[20] Automation bias means that a human favors a machine’s output over their own. Automation bias can be particularly prevalent in times of reduced cognitive capacity.[21]

In particularly stressful contexts there could also be bias for action. When a human prefers action over inaction, even if inaction may mean higher probability of reaching desired outcomes, they are succumbing to action bias. Research on action bias suggests that there is a preference for action under high stress in professional sports, but data is not yet available on human interaction with decision support tools.[22] The pressure reported by IDF intelligence personnel would seem consistent with conditions that encourage action bias: an available tool that enables more action, reduced cognitive capacity in operators, demands and rewards for increased action, and consequences for perceived inaction.[23] It appears likely that IDF operators could not sustain the operational tempo and results demanded by their leadership if they retained more of the cognitive load in the targeting cycle. Under these conditions operators outsourced cognition they were supposed to retain for themselves and became dependent on the machines.

Under conditions of stress and reduced cognitive capacity there could come a point where the human is completely dependent on the machine for a cognitive process. This dependency may not be acceptable in all circumstances, particularly when authorizing lethal force. One way to deal with this would be to work to reduce the stress that is affecting cognitive capacity, but in warfare that may not be a practical goal. Instead, we may consider ways to reduce cognitive dependence on AI-enabled tools in warfare by optimizing human cognitive autonomy.

Imposing meaningful control

In the United Nations Convention on Certain Conventional Weapons (UN CCW) a consensus is forming around the need for humans to retain meaningful control when employing autonomous weapons. While defining meaningful human control (MHC) is difficult, the concept can help steer us towards knowing when humans become overly dependent on AI-enabled systems. A general summation suggests that MHC ensures adherence to the laws of war (LoW) and international humanitarian law (IHL) as well as global moral consensus. These laws and morality focus on the employment of force in a way that is: 1) militarily necessary: 2) proportionate to the context; 3) distinguishes between combatants and noncombatants; and 4) takes proper precautions to avoid unauthorized harm. We could consider a human as cognitively dependent on a decision support tool, and not applying meaningful human control, if the actions derived from the tool’s recommendations are violating LoW and IHL without the human operator intending it. But LoW and IHL violations are not the only circumstances where human cognitive dependence could be problematic. Cognitive dependence is not just about violations of LoW and IHL, but about who or what is in control of the conditions and actions in a particular context.

Control is the capacity to exert influence over internal states, external states, and desired outcomes. There is evidence that control is a biological imperative because it is associated with organisms ensuring their own survival.[24] However, the amount of control we want can depend on the context or situation we are in. For example, a low-stakes situation (i.e. choosing what cereal to eat in the morning, especially if you are late for work), would encourage you to accept less control because most choices available are reasonable, and while there may be a preference that varies from day to day, the risk of accepting a low preference selection is minimal. Giving up control could allow you to eat faster and spend less energy on cognition. If the situation has high risk (i.e. in the context of targeting adversaries in a village full of noncombatants), you will likely prefer more control to mitigate risk, with greater cost in speed and cognition.

We may also consider that control of some parts of a situation may be more necessary than others. Control of defining the desired state may be critical, while control over the process to achieve the desired state may be a place to assume risk to gain speed. We may be willing to make this trade-off by assuming that we have control over validating that the output state matches the desired state. Problems arise when we cannot tell if the process is in service of the desired state, or if the output state resembles the desired state.

The aforementioned examples from Gaza suggest that humans may have given up direct control of significant parts of the targeting process in Gaza for speed and cognitive efficiency, while still maintaining the illusion of control through human on the loop requirements. Despite human intelligence personnel on the loop, significant portions of the targeting outcomes appear to be linked to the outputs coming from the AI-enabled decision support tools rather than human cognition. The operators became cognitively dependent on the machines and then the outcomes became most closely tied to machine rather than human control.

Discussions on how humans can maintain control of automated and autonomous systems are long-standing. CCW meetings on MHC have considered ways that humans can maintain control over machines by ensuring: 1) the technology is predictable; 2) it is reliable; 3) it is transparent; and 4) the user has accurate information on what the tech does and what it is for.[25] These requirements are good steps towards ensuring MHC but they do not take into account human psychology and how humans interact with machines even when these conditions are present.

Human operators must be able to maintain control over the outcomes in war. To do that they must manage their dependence on machines so that they retain sufficient cognitive autonomy to guide the war towards human defined outcomes. We propose to better understand the trade-off between cognitive dependence and cognitive autonomy to better ensure human control of warfare when humans employ AI-enabled decision support tools.

Cognitive autonomy as a means for meaningful control

We define cognitive autonomy as the ability to perform the mental process of acquiring, storing, manipulating, and retrieving information in an independent and authentic way.[26] However we do not consider cognitive autonomy an absolute. Instead, it exists on a spectrum. At either end of the spectrum, we have opposing states that are neither achievable nor desirable. We do not want complete cognitive autonomy where human reason exists only in relation to itself, nor do we want complete dependence where we no longer think at all. We want our cognition to exist somewhere on the spectrum in between, and that is likely dictated by a desire for a general global autonomy while we selectively sacrifice some autonomy for an acceptable benefit.

Global cognitive autonomy is what philosophers consider when they discuss distinct personhood and free will. Autonomy in this context refers to the states of a person, and assumes that most adults who are not suffering from debilitating pathologies or oppression are considered autonomous.[27] We confine use of the word autonomy to consider cognition in a specific narrow context. As an example, philosopher John Christman describes a situation where a person can be an autonomous being with complete personhood but still lack control of their behavior in circumstances like drug addiction.[28] Similarly, we consider the human operator of AI-enabled decision support tools to be cognitively autonomous in a global sense even if in the context of using the tool, they have become cognitively dependent, and thus no longer in control.

Loss of control in a narrow context does not necessarily mean that the situation is deteriorating. The machine could be doing a fine job. Instead, the nature of that control is changing. In a situation where a human and machine are both exerting influence, control of that situation leans toward being either human or machine-defined, depending on which is exerting the greatest influence. If a human operator relies on a machine to automate and amplify cognitive functions, the human becomes more dependent and control of the situation becomes more machine-defined. The opposite is true if human cognition is the primary influence, control of the situation becomes more human-defined. This means that meaningful human control is at least partially a function of cognitive autonomy, and could be achieved by ensuring that human operators of decision support tools maintain sufficient cognitive autonomy. The difficulty is in determining how much is sufficient autonomy in any given context.

Evaluating cognitive autonomy and dependence

Applying the idea of cognitive autonomy to human-machine interaction is fairly novel but the idea has been applied in developmental psychology when studying how humans seek to increase control in their environment as they age into adulthood. Developmental psychologists use a framework called cognitive autonomy and self-evaluation (CASE) to determine when children become cognitively autonomous from their parents and when determining if persons with mental disabilities can function on their own.[29]

The CASE framework includes five categories to determine cognitive autonomy: a) making informed, independent decisions; b) voicing educated and appropriate opinions; c) weighing the influence of others on thinking; d) considering consequences; and e) self-evaluating practices.[30] These categories inform a questionnaire researchers use to determine how much cognitive autonomy a person has. Research showed that CASE is a viable tool for assessing cognitive autonomy in children and young adults.[31] The CASE framework provides a good starting point for creating a method to measure human cognitive autonomy in human-machine teams, but it needs to be tailored for human interaction with AI-enabled decision support tools.

We suggest a modified framework of six categories for evaluating cognitive autonomy and dependence of humans on a given AI-enabled decision support tool: a) cognitive capacity; b) conceiving a state of being; c) applying value; d) forecasting outcomes; e) understanding influence; and f) controlling choice architecture. We refer to it as the Cognitive Autonomy Variable (CAV) framework.

In the diagram below, a human operator’s cognitive autonomy or dependence are functions of six variables. The primary variable is how much cognitive capacity the operator has at any given moment. Greater cognitive capacity enables greater capacity in the other variables. In each of the other variables, the human is either more or less reliant on the machine for that function. The composite relationships of those functions suggest that the human is becoming more dependent or more autonomous from the decision support tool.

Figure 1

Cognitive Autonomy Variables:

Cognitive capacity: Not to be confused with a measure of intelligence, this variable considers the resources available for cognition and indicates whether a human can perform the capabilities of the other variables. Capacity can be affected by caloric energy, stress, sleep deprivation, and many other factors.

The cognitive biases discussed all have some element of reduced cognitive capacity as a factor of their appearance and effect. War requires high levels of cognition coupled with circumstances of significant stress. Cognition limits of humans, as well as susceptibility to stress, will be indicators of human tendency to become more dependent on a decision support tool to offset cognitive capacity limitations.

Conceiving a state of being: This category captures human creativity and the ability to perceive relationships between information in order to imagine and describe a state that differs from that which is currently in place.

The heart of any war is the belief that the current state of being is so undesirable that lethal force on a national scale has become necessary. Every war, so far, is therefore a function of human conceptions that things should be a certain way. This is the heart of control. Whether this conception is machine- or human-defined will indicate how dependent humans have become on the tool. The ability of a human to decide what could be and what could not be is the beginning of control and cognitive autonomy. If a machine is deciding what could be and could not be for the human, the machine could be said to be in control and the human cognitively dependent.

Applying value: Value is where human cognition determines desire and preferences. When conceiving of new states of being, the human applies value to determine which state is more desired than another, what is acceptable and what is not, and it helps to prioritize and sequence events. Applying value allows human cognition to go from what could be to what should be.

Applying value to conception of a desired state is closely associated with why humans do what they do or want what they want. In warfare the why, or the purpose, of the violence must be tightly controlled, well-understood, and closely tied to a larger human-centered purpose. Our ability to hold on to the purpose behind a decision or a choice will help us ensure that a system is on track to accomplish the correct objective function we have defined and reasoned. We might think that preserving cognition only over the end state is enough; we would know why that completed end state is important, and why we decided on that end state in the first place. However, when a deviation in the process leads to a different end state or output, this is precisely where holding on to the purpose becomes critical; it is to be able to deviate from a generated end state we do not desire, to the one we initially wanted.

Forecasting future outcomes: An ability to predict outcomes from decisions and values, forecasting is essential to making choices that match desired states. Without forecasting, there is only random choice.

Where machines are starting to excel is in their ability to consider far larger amounts of information than humans can in order to forecast future outcomes of actions. This makes it useful to employ machines for predictive analytics to down-select options in favor of those that are most likely to yield the desired end state. While humans could retain control over conception of what should happen, cognitive autonomy is degraded if they cannot forecast an outcome to decide which is the best action to reach the desired state. In warfare the temptation will be to rely on the machine to predict how to get to a desired state. In many cases this may be fine but humans are sacrificing some autonomy for that benefit.

Understanding influence: There are external and internal influences on cognition. Understanding the nature and ubiquity of influences in any given context allows humans to impose as much cognitive autonomy as they can by tempering or enabling influence, by accepting some influences over others, and understanding (to the extent possible) how influences are affecting cognition.

In warfare, not only will there be neutral influences from the information environment, there will be influences designed to sway decision-making. An adversary may attempt to deceive by confusing data collection and poisoning collected data, or a decision-support tool may attempt to influence its operator to meet a mis-aligned reward function. Much the way a commercial algorithm might be influenced as much by advertiser funding as your personal movie preferences, a military decision-support tool will be balancing competing priorities from the chain of agents throughout its deployment cycle. The operational recommendations may not always align with operator preferences in a real-time context because it is near impossible to anticipate every context prior to deployment. The human may think decision support tool recommendations are aligned with their operation because it says so on the package. However, it would be hard for the operator to know the entire chain of influences affecting the machine.

Controlling choice architecture: Choice architecture is the array of available choices.[32] It is the structuring of the decision environment to guide, constrain, and influence decisions. A choice cannot be made if it is not available, and the volume and types of choices available can shift behavior toward specific outcomes.[33]

One’s ownership of a system is ultimately reduced as automation takes over reasoning rather than strictly automating the process by which an action is executed. This becomes a concern especially when systems have control over choice architecture. If a system controls the amount and variety of choices presented to us, that would potentially steer us towards choices we might have not made independently—often without our awareness. As we lose control over choice architecture, it becomes increasingly challenging to preserve the reasoning behind a decision we take or a choice we make, because our options are limited. In other words, we infer that there is some sort of a positive correlation between choice architecture and cognitive autonomy, whereby a broader choice architecture requires greater cognition to make a choice.

The inherent irony in high-stakes decision-making situations is that while the pressure often reduces cognitive capacity, the gravity of the decisions actually demands a higher level of cognition. This creates a critical tension: on one hand, there's a need to make decisions quickly to respond effectively to the situation. On the other, there's a competing need to carefully consider each decision, given the potentially severe consequences of any mistakes.

In such scenarios, the choice architecture plays a pivotal role. Simplifying this architecture can potentially speed up decision-making but might also limit the depth of cognitive engagement, potentially leading to oversights or errors. This raises a significant question about the extent to which we should rely on automated systems or AI in such contexts, and about how much we allow the choice architecture of a given situation to be determined by automated systems. Machines can process vast amounts of data rapidly and can be designed to operate under predefined rules, which is advantageous in time-sensitive situations. However, their ability to appreciate the nuances of human values and the broader context in which decisions take place might be limited.

The dilemma then becomes whether to delegate critical, high-stakes decisions to algorithms that can act faster but may lack deep understanding, or to maintain human oversight, accepting slower decision-making processes in exchange for potentially more considered and contextually aware outcomes.

Applying the CAV framework

We propose that decision support systems could be constructed to consider the CAV framework in a way that allows human operators to know how dependent they have become on a particular decision support tool, and how they could adjust the tool to increase cognitive autonomy when desired. We offer three principles to be considered for decision support tool design, as well as some suggestions for what practical implementation might look like; and we propose that the CAV framework could be used in testing and evaluation prior to deployment.

Principles

1. Design of any AI-enabled decision support tool should consider how humans will interact with machines that can replace some human cognition—particularly how cognitive bias will manifest when humans interact with machines in the expected context.

2. The ratio of cognitive autonomy to dependence should be recognizable and measurable: We do not intend this as a binary marker but rather as a gauge to inform operators when their interactions with the tool indicate that reliance on the tool is increasing and may exceed desired thresholds.

3. Cognitive autonomy should be optimizable by human control of the tool. The human operator must be able to modify system functions when cognitive dependence exceeds desired limits and more cognitive autonomy is optimal for the context.

Practically, these principles could be implemented through tracking systems in the machine that turn into awareness for the operator. For example, activity logs and performance metrics that compare present user behavior to historic behavior—as well as to behavior in peers with similar systems—might suggest to an operator that they have started to rely on the tool much more than they have, or much more than their peers. At this point the operator or a supervisor could evaluate whether they have become too dependent. It could also provide a legal record for future use, which may have some benefit but carries the risk of encouraging timidity on the battlefield.

The system could also communicate uncertainty clearly in ways that encourage reflection in the operator. This means something more complex than “90 percent performant”—a metric that does not communicate what the other ten percent is or what exactly performance at 90 percent means. Rather than encourage reflection and cognitive autonomy, that type of communication encourages complacency and automation bias, and thus cognitive dependence. In the case of target identification, uncertainty could be communicated by modality rather than as a composite. For example, full motion video analysis is 80 percent, electromagnetic spectrum is 50 percent, and text source is 60 percent. This at least makes the human fully aware of the limitations in the system and causes reflection on what those numbers mean.

Further, the user interface could be constructed to encourage engagement and cooperation with the tool rather than input and output. If the tool were to communicate in questions rather than statements, that could change how the operator thinks about recommendations. So an AI-enabled decision support tool might recommend a target in the form of a question: “Is target #2 with identification confidence levels x, y, and z suitable for action?” It is a small change but could have important implications for how the operator views their interaction with the tool and their responsibilities.

There are many other ways these tools could be designed to encourage cognitive autonomy when we need it and allow some dependence when acceptable. Developers could start considering concepts now, but further research must be done to develop this CAV framework into a usable tool—particularly if we want to make a useful testing and evaluation regime that can validate meaningful human control and appropriate human judgment in operational systems.

Share

Subscribe now

Leave a comment

References

(2015). "What is Cognition?" Insights https://cambridgecognition.com/what-is-cognition/ 2024.

(2023). DoD Directive 3000.09 Autonomy in Weapons Systems. D. o. Defense.

(2024). Daily death rate in Gaza higher than any other major 21st Century conflict - Oxfam.

(2024). "OpenArt." Retrieved August, 2024, 2024, from https://openart.ai/home?gad_source=1&gclid=Cj0KCQjw28W2BhC7ARIsAPerrcIXMSxZtla5TGboU6QNmS1wCMFlhRyW0CPhxzNSU6Xz9lrm9zr61dcaAvWxEALw_wcB.

(IDF), I. D. F. (2024). The IDF’s Use of Data Technologies in Intelligence Processing. www.idf.il, Israel Defense Force.

Abraham, Y. (2024) ‘Lavender’: The AI machine directing Israel’s bombing spree in Gaza. +972 Magazine

Bashkirova, A. K., Dario; (2024). "Confirmation bias in AI-assisted decision-making: AI triage recommendations congruent with expert judgments increase psychologist trust and recommednation acceptance." Computers in Human Behavior: Artificial Humans 2(1).

Beckert, T. E. (2007). "Cognitive Autonomy and Self-Evaluation in Adolescence: A Conceptual Investigation and Instrument Development." North American Journal of Psychology 9(3): 579-594.

Brumfiel, G. (2023) Israel is Using an AI System to Find Targets in Gaza. Experts Say it’s Just the Start. National Public Radio (NPR)

Christman, J. (2020). Autonomy in Moral and Political Philosophy. The Stanford Encyclopedia of Philosophy. E. N. Zalta.

Goodfriend, S. (2024) Why human agency is still central to Israel’s AI-powered warfare. +972 Magazine

Kaanders, P. S., P.; Folke, T.; Ortoleva, P.; DeMartino, B.; (2022). "Humans Actively Sample Evidence to Support Prior Beliefs." Elife 28(11).

Leotti, L. A. I., Sheena S.; Ochsner, Kevin N.; (2010). "Born to Choose: The Origins and Value of the Need for Control." Trends in Cognitive Science 14(10).

Lu, C. L., Cong; Lange, Robert Tjarko; Foerster, Jakob; Clune, Jeff; Ha, David (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. Arxiv.

Risko, E. G., Sam J.; (2016). "Cognitive Offloading." Trends in Cognitive Science 20(9): 676-688.

Roff, H. M., Richard (2016). Meaningful Human Control, Artificial Intelligence, and Autonomous Weapons. Convention on Certain Conventional Weapons Meeting of Experts on Lethal Autonomous Weapons. Geneva, Switzerland.

Sanger, D. E. (2024) In Ukraine, New American Technology Won The Day. Until It Was Overwhelmed. The New York Times

Scharre, P. (2016). "Centaur Warfighting: The False Choice of Humans vs. Automation." Temple International and Comparative Law Journal 30(1): 151-166.

Schein, M. B.-E. O. H. A. I. R. Y. K.-L. G. (2007). "Action Bias Among Elite Soccer Goalkeepers: The Case of Penalty Kicks." Journal of Economic Psychology 28: 606-621.

Skitka, L. J. M., Kathleen L.; Burdick, Mark (1999). "Does Automation Bias Decision-making?" International Journal of Human-Computer Studies 51(5).

Sparrow, R. J. H., Adam (2023). "Minotaurs, Not Centaurs: The Future of Manned-Unmanned." The U.S. Army War College Quarterly: Parameters 53(1).

Thaler, R. H. S., Cass R. (2008). Nudge: Improving Deciions About Health, Wealth, and Happiness. New Haven, CT, Yale University Press.

Vora, L. K., et al. (2023). "Artificial Intelligence in Pharmaceutical Technology and Drug Delivery Design." Pharmaceutics 15(7).

[1] Vora, L. K., et al. (2023). "Artificial Intelligence in Pharmaceutical Technology and Drug Delivery Design." Pharmaceutics 15(7).

[2] Lu, C. L., Cong; Lange, Robert Tjarko; Foerster, Jakob; Clune, Jeff; Ha, David (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. Arxiv.

[3] (2024). "OpenArt." Retrieved August, 2024, 2024, from https://openart.ai/home?gad_source=1&gclid=Cj0KCQjw28W2BhC7ARIsAPerrcIXMSxZtla5TGboU6QNmS1wCMFlhRyW0CPhxzNSU6Xz9lrm9zr61dcaAvWxEALw_wcB.

[4] Abraham, Y. (2024) ‘Lavender’: The AI machine directing Israel’s bombing spree in Gaza. +972 Magazine

[5] Scharre, P. (2016). "Centaur Warfighting: The False Choice of Humans vs. Automation." Temple International and Comparative Law Journal 30(1): 151-166.

[6] Sparrow, R. J. H., Adam (2023). "Minotaurs, Not Centaurs: The Future of Manned-Unmanned." The U.S. Army War College Quarterly: Parameters 53(1).

[7] Roff, H. M., Richard (2016). Meaningful Human Control, Artificial Intelligence, and Autonomous Weapons. Convention on Certain Conventional Weapons Meeting of Experts on Lethal Autonomous Weapons. Geneva, Switzerland.

[8] (2023). DoD Directive 3000.09 Autonomy in Weapons Systems. D. o. Defense.

[9] Sanger, D. E. (2024) In Ukraine, New American Technology Won The Day. Until It Was Overwhelmed. The New York Times

[10] Brumfiel, G. (2023) Israel is Using an AI System to Find Targets in Gaza. Experts Say it’s Just the Start. National Public Radio (NPR)

[11] Abraham, Y. (2024) ‘Lavender’: The AI machine directing Israel’s bombing spree in Gaza. +972 Magazine

[12] Ibid.

[13] Ibid.

[14] (2024). Daily death rate in Gaza higher than any other major 21st Century conflict - Oxfam.

[15] Goodfriend, S. (2024) Why human agency is still central to Israel’s AI-powered warfare. +972 Magazine

[16] (IDF), I. D. F. (2024). The IDF’s Use of Data Technologies in Intelligence Processing. www.idf.il, Israel Defense Force.

[17] Risko, E. G., Sam J.; (2016). "Cognitive Offloading." Trends in Cognitive Science 20(9): 676-688.

[18] Kaanders, P. S., P.; Folke, T.; Ortoleva, P.; DeMartino, B.; (2022). "Humans Actively Sample Evidence to Support Prior Beliefs." Elife 28(11).

[19] Bashkirova, A. K., Dario; (2024). "Confirmation bias in AI-assisted decision-making: AI triage recommendations congruent with expert judgments increase psychologist trust and recommednation acceptance." Computers in Human Behavior: Artificial Humans 2(1).

[20] Skitka, L. J. M., Kathleen L.; Burdick, Mark (1999). "Does Automation Bias Decision-making?" International Journal of Human-Computer Studies 51(5).

[21] Ibid.

[22] Schein, M. B.-E. O. H. A. I. R. Y. K.-L. G. (2007). "Action Bias Among Elite Soccer Goalkeepers: The Case of Penalty Kicks." Journal of Economic Psychology 28: 606-621.

[23] Abraham, Y. (2024) ‘Lavender’: The AI machine directing Israel’s bombing spree in Gaza. +972 Magazine

[24] Leotti, L. A. I., Sheena S.; Ochsner, Kevin N.; (2010). "Born to Choose: The Origins and Value of the Need for Control." Trends in Cognitive Science 14(10).

[25] Roff, H. M., Richard (2016). Meaningful Human Control, Artificial Intelligence, and Autonomous Weapons. Convention on Certain Conventional Weapons Meeting of Experts on Lethal Autonomous Weapons. Geneva, Switzerland.

[26] (2015). "What is Cognition?" Insights https://cambridgecognition.com/what-is-cognition/ 2024.

[27] Christman, J. (2020). Autonomy in Moral and Political Philosophy. The Stanford Encyclopedia of Philosophy. E. N. Zalta.

[28] Ibid.

[29] Beckert, T. E. (2007). "Cognitive Autonomy and Self-Evaluation in Adolescence: A Conceptual Investigation and Instrument Development." North American Journal of Psychology 9(3): 579-594.

[30] Ibid.

[31] Ibid.

[32] Thaler, R. H. S., Cass R. (2008). Nudge: Improving Deciions About Health, Wealth, and Happiness. New Haven, CT, Yale University Press.

[33] Ibid.

Just a quick note to steer you all over to an article that Tiffany Saade and I put together for the second edition of the new journal The Republic.

This ended up being a trimmed down version of a larger piece on how humans can become too dependent on decision support tools, how we might detect that dependence, and how the tools might be engineered to help operators preserve their cognitive autonomy.

I’ll post the larger piece here eventually because I think this group might have a lot to say on some of the details we left out of the Republic article.

In the meantime please enjoy Human Cognitive Autonomy: Keeping People on the Loop in AI-driven Warfare at The Republic.

Feel free to post comments on that article here if not on the article page itself.

And of course, have a happy and safe Thanksgiving!

Subscribe now

Leave a comment

My colleague Lena and I just attended a really good Track 1.5 dialogue at the University of Edinburgh Law School in Scotland. The event was sponsored and run by the Paul Tsai China Center at Yale Law School. A quick shout out to that team, because they did a fantastic job of planning, organizing, and executing this dialogue. Absolutely top notch in every way.

Because of the nature of the dialogue I won’t be naming names or attributing thoughts or ideas to anyone, but I feel the need to capture my impressions because I realized some things perhaps worth sharing.

Scholars from Europe, the U.S., and China attended to discuss autonomous weapon systems and the role of military legal advisors in the context of International Humanitarian Law. Predictably we ended up all over the place, which is good. When you’re drinking crazy Chinese sorghum liquor in Diagon Alley with a bunch of law nerds, things can get surprisingly sporty.

So let me go over a couple of my “hmmmm” moments.

In my experience in discussions of this kind with this type of audience, the Western reaction to China’s policies on lethal autonomous weapons could be described as politely mystified. Unsurprisingly, I think some of this comes down to language but it is also a factor of how both sides understand and communicate about the technology that enables lethal autonomy.

China is adamant that no one should construct fully autonomous weapons, and there should be a legally binding instrument to prevent that. I’m not sure that China believes the U.S., and the West in general, agree.

Unsurprisingly the U.S. thinks that China is trying to get the U.S. to abandon all development of autonomous weapon systems while China secretly continues to develop them. Elsa Kania calls this “Strategic Ambiguity” and suggests that it is an intentional strategy to constrain the West.

There is probably some truth to this, maybe even a lot of truth, but I am coming around to thinking that perhaps our views on autonomus weapon regulation are pretty similar. We are just misunderstanding each other because we are communicating poorly.

The unofficial U.S. response to China’s position on fully autonomous weapons is often, “of course we wouldn’t build FULLY autonomous weapons, that’s crazy.” But the policy documents are not as clear. If you look at prominent policy documents on both sides, the Chinese definition of a fully autonomous weapon and the U.S. definition of an autonomous weapon are not incompatible.

The unofficial Chinese definition of a fully autonomous weapon:

“LAWS should include but not be limited to the following 5 basic characteristics. The first is lethality, which means sufficient payload and for means to be lethal. The second is autonomy, which means absence of human intervention and control during the entire process of executing a task. Thirdly, impossibility for termination, meaning that once started there is no way to terminate the device. Fourthly, indiscriminate effect, meaning that the device will execute the task of killing and maiming regardless of conditions, scenarios and targets. Fifthly evolution, meaning that through interaction with the environment the device can learn autonomously, expand its functions and capabilities in a way exceeding human expectations”

The U.S. definition of an autonomous weapon (my emphasis in italics):

A weapon system that, once activated, can select and engage targets without further intervention by a human operator. This includes, but is not limited to, operator-supervised autonomous weapon systems that are designed to allow operators to override operation of the weapon system, but can select and engage targets without further operator input after activation.

Givn a definition that broad, you could forgive the Chinese for thinking that the U.S. is being strategically ambiguous to constrain them.

During the dialogue, a question that kept going through my mind is, if the Chinese think a fully autonomus weapon is something that, by definition, humans cannot control, why do they think the U.S. and the West would make them? To answer that I started to think about how we communicate about the underlying technology.

To me, part of this problem comes down to the crazy stuff people say about artificial intelligence. The knowledge gap about artificial intelligence in policy circles, though descreasing, is a major obstacle to effective regulation of lethal autonomy.

While listening to my esteemed collegues from China last week I became reasonably convinced that for them, artificial intelligence in the context of full autonomy is literally a non human intelligence. It is not a probabilistic software model that classifies hot dogs on the internet. This is something that is not only intelligent, it is sentient. It has a subjective experience and a will to pursue its own ends.

In their mind, humans have no business arming a fully autonomous (sentient) machine for any purpose. This machine would be capable of deciding when, where, and to what puprpose lethal force should be used. Unsurprisingly, I think we all think that sounds terrible. But why do the Chinese suspect we might not agree with them?

When communicating policy options about lethal autonomy the West does a terrible job of separating current AI technology like computer vision and natural language processing from aspirational AI like artificial general intelligence (AGI) and artificial superintelligence (ASI). They say things like “the AI decides this, or the AI thinks that.” There is not a real undersatnding of the reality that most autonomous systems are only using the “AI” for object classification or maybe an interface (LLM). The “decisions” are being made by deterministic software: if an object meets criteria A, follow it, etc. But policy people are not dumb, quite the opposite, but they’re fed this kind of language for many reasons.

Policy wonks are often heavily influenced by pundits and tech celebrities that give a warped sense of what the technology can do now and what it is about to do next. The week prior to this dialogue, Geoffrey Hinton received the Nobel Prize in physics for his work on machine learning and neural networks. Hinton is one of the biggest personalities in discussions on lethal autonomous weapons and apocalyptic AI, and is on record earlier this year stating that there is a 50% chance that an AI will try to “take over” in 5-20 years.

Sam Altman of OpenAI has made a few vague predictions from “soon-ish” to “end of the decade” for AGI. And of course, you can’t talk about AI hype without Elon Musk, who seems confident that AGI will arrive and be smarter than the smartest human by 2026.

These expert opinions are ingested by the policy community and repeated in dialogues about autononous weapons, this one was no exception. When the Chinese hear this I suspect they could be thinking “holy crap, can they build that already? Are they putting it in weapons?” To me it makes sense that they would immediately say “No fully autonomous weapons!” And then desire the U.S. to more clearly commit to that in policy documents like DoDD 3000.09.

Ultimately I think the real solution is a legal and policy community that is better educated on what the technology can actually do and what is reasonable to expect in the next decade. This is happening but I’m always surprised how apocalyptic AI predictions make their way into serious discussions. I think that this partly contributes to the fact that there is very little movement in the international community seeking to regulate autonomous weapons. However it is getting better.

There has been a slow turn back to reality happening in the International Humanitarian Law community. When I read the latest by the ICRC, I can’t help but think “welcome back to where the development and operational communty was five years ago.” Both the activist and policy communities are sorting through all the hype and starting to bring their policy recommednations back into reality, meaning using IHL to prevent unnecessary suffering rather than preventing the development of apocalypti weapons that will destroy humanity. Don’t get me wrong, I don’t want those either, I’m just saying they are not imminent so let’s focus our efforts on what’s real.

The other major observation I had last week was seeing the difference between the different perspectives between those who own the mission and those who own the law.

My impression of the corpus of the laws of armed conflict through the ages was that it was drafted by people who had fought wars that decided to codify that you can win wars without destroying everything and everyone. That much of the violence of some wars is capricious and unnecessary. You can still fight to win without being a bunch of murderers. The laws of armed conflict always struck me as practical and reasonable.

As autonomy has arrived on the scene it seems like much of the discussion around the laws of armed conflict are more oriented towards making it harder to wage war. Autonomy seems like it will make waging war easier and cheaper but no less destructive. Probably moreso. So I get the instinct to say we should avoid that. But if history has taught us anything about war, it’s that humans will always find a way to wage war, and that it is always better to win a war than to lose it. Nations will always do what they have to to win.

If law seeks to deny nations the tools they think they need to win, nations will not adopt those laws. This is why the U.S. and China will not forego autonomous weapons. The mission requires them. The owners of the mission will not give them up. The owners of the law will probably have to bend and make progress where they can. Which is a noble pursuit.

Thanks for reading. We’re working on our series about what it takes to actually field autonomous weapons. Hopefully our first guest post will be here soon to talk about object classfication challenges.

Until then…

Leave a comment

Lena already summarized some issues nicely. Are we actually sacrificing cognitive autonomy when we deploy automated systems? Does human-machine teaming compromise cognitive autonomy?

The short answer is yes. We sacrifice cognitive autonomy when we automate something, especially reasoning. This includes human-machine teaming. And you know what? Sometimes that’s good.

Clearly we don’t want to have to think about when to add water and soap when we wash our clothes. Nor do we want to have to plot every updated target location on a map when we receive new reports. If we did those functions ourselves could we better ensure that each action was optimally tied to our intent and goals for our life or mission? Yes, but the energy we expend on that could better be used elsewhere, and the consequences of failure in a clothes washing incident are fairly minor so let’s just let the machine do it.

What about consequences in improper target location? That may be another story. In this case the process of validating a target as properly identified and properly located may cross over into a level of reasoning where the machine’s action fails to align with human intent. The machine’s “thinking” has violated MHC or AHJ.

I don’t want to re-hash examples from previous posts but I wanted to make a quick reference to these scenarios because ultimately cognitive autonomy shouldn’t be considered an objective to reach. We aren’t trying to get to full cognitive autonomy. Nor do I think anyone is trying to push humanity to zero cognitive autonomy.

Why are those bad? Well, let’s start with the easy one. Zero cognitive autonomy would mean being a simple machine. A toaster perhaps. Absolutely no influence on the conditions of your existence. Not a great life, but perhaps you wouldn’t even notice, happily toasting along.

Full cognitive autonomy wouldn’t be much better. In this scenario we’re probably talking a disembodied brain with no external influences that could affect thinking. Completely autonomous. In this scenario you’re probably right back to being a toaster with a better user interface.

So if those are the two poles of cognitive autonomy, where is the optimal place to be between them?

That depends on context.

Let’s visit Lena’s Danish supermarket scenario.

While I think having many cereal options is great (Count Chocula, Fruity Pebbles, Cinnamon Toast Crunch, mmmm the list goes on), clearly the Danes only want Fiber O’s or Colon Flakes. And that’s cool. The key is that their architecture for that choice was knowingly limited, in this case perhaps outsourced to a machine. If the Danes don’t get to enjoy the delights of monster-themed cereal, well that’s on them. No harm done. What we want to avoid is unwittingly allowing a machine to bound or control our pursuit of our goals.

Share

If the Danes were wistfully looking across the Atlantic wondering why Americans seem so joyful at breakfast, they may have no idea that monster-themed cereal is the source of that tranquility. Then we have a problem. Their lives are not what they want them to be due to constraints placed on them by a system not of their own making. Begin the first human-machine war!

Obviously this is absurd, but to sum, I’m trying to suggest that one scenario while not having compete cognitive autonomy, seems to be the right amount. While the second scenario certainly isn’t zero cognitive autonomy, but perhaps it is not quite enough.

So a few key questions emerge to frame this inquiry:

1. What is the right amount of cognitive autonomy in any given situation?

2. How do you know how much you have or how much you have surrendered?

3. How do you adjust your cognitive autonomy?

4. When should we sacrifice our cognitive autonomy?

The answers to these questions are not well understood and yet we are developing advanced decision support systems without considering all the variables. In some cases, researchers are considering that humans cannot decide for themselves how much cognitive autonomy they should have and when it should be reduced.

At the University of New Mexico researchers are working on drone systems that will reduce human control depending on how well the drones think the human’s cognition is working. As the human becomes more stressed and less capable, the system will increase its decision autonomy and reduce human decision control.

This autonomy is done as a means to help the human but it is done in a way that maximizes action at the expense of human understanding of what is happening. Is that the right answer? I’m not sure, though it is a tempting feature when considering the rigors of warfare.

My forthcoming paper will consider the questions above and attempt to provide a framework for thinking about how much cognitive autonomy we might want in any given situation, how we can develop systems that maximize automation and autonomy while retaining alignment to human intent, and how we can ensure that human cognition remains the master of its own destiny. When it’s published I’ll post a link here if you want to marinate in this discussion some more.

In the meantime we’ll get ready to host some guest authors to start a larger series on what it takes to field autonomous weapons on the modern battlefield. Spoiler: it’s a lot harder than people think, evidenced by the lack of any meaningful lethal autonomy in Ukraine and Gaza (that we know of).

Until then…

Share

But we’re back, so let’s jump into this. We left off by discussing cognitive autonomy and the risks of machines “thinking for us” in the targeting cycle.

Subscribe now

I’ve been exercising my own cognitive autonomy and thinking about this sentence from Brad’s last post:

“… cognitive autonomy is Excalibur. Whoever gets to decide what happens, and why, owns the future.”

There is a lot to unpack here…but let me begin by saying, are you sure? Are we overplaying how much cognitive autonomy we really have? Now, it has been many years since I last took a philosophy class (there’s a reason I chose international relations instead of philosophy), but there are so many remaining questions to consider here that come to my mind:

Are we actually sacrificing a certain degree of cognitive autonomy when using AI-enabled autonomous systems?

Haven’t other factors already chipped away at the decision space, or choice architecture, and this idea of cognitive autonomy is a bit of an illusion – or, at the very least, already constrained? In other words, do other factors limit our cognitive autonomy, but we don’t consider these as limiting or problematic because a competent authority was responsible for that constraint?

o   I’m thinking of geopolitical realities, the national political climate, individual leaders, international law, etc.

o   I guess by the time we get to the targeting cycle… isn’t the decision space already so delineated by other factors that have shaped it to employ cognitive autonomy? Does that really mean the machine is “thinking” for us at this point?

o   Who needs to do the thinking? Any human – as long as it’s human?

Are there circumstances where a constrained choice architecture would lead to better outcomes? Should we consider circumstances where sacrificing a degree of cognitive autonomy is appropriate or even necessary?

Does human-machine teaming compromise cognitive autonomy?

Individuals react differently to choices – some treasure having the ability to choose, while others dread the prospect of making a choice. It makes me think of a conversation I had with a former Danish colleague over lunch. I was complaining about Danish supermarkets (IYKYK), and she told me, “I much prefer our supermarkets to American ones. We get two choices of cereal; why do you need more? I would rather choose between two than twenty.” I struggled to understand – what if you don’t want those two options? What if they are both bad options?

I realize this is a superficial example, but it may be worth considering how cognitive autonomy is perceived, utilized, and preserved at an individual level. Perhaps my colleague would have preferred that a machine picked her cereal and sent it to her home so she did not have to make a choice at all. I can think of folks that would be more than fine with that. Not to mention, as my Danish context might demonstrate, the potential for national differences in this discussion. Perhaps preserving cognitive autonomy is not as high of a priority in other parts of the world (not necessarily Denmark!) as it is in the US and the American military.

This discussion also makes me think about how quickly humans succumb to cognitive fatigue. I know this has been a fruitful research area for MIT to develop AI assistants that can sense cognitive fatigue and design AI assistants that would sense the fatigue and offer advice to fix it or step in to replace the human if the fatigue is bad enough. The project—already a few years old now—explores the question: What if the machine is working perfectly, and the human is struggling? Could this be a situation where it would be better for a machine to “think” for us? It is difficult to answer because there are so many “it depends” answers. But one of the researchers explains the project:

"In the area of human-machine teaming, we often think about the technology, for example, how do we monitor it, understand it, make sure it's working right. But teamwork is a two-way street, and these considerations aren't happening both ways. What we're doing is looking at the flipside, where the machine is monitoring and enhancing the other side — the human.”

The system would “suggest interventions, or even take action in dire scenarios, to help the individual recover or to prevent harm.”

I know other research projects are taking more drastic steps, such as withholding information from human operators to keep stress levels low and performance optimal—is that really cognitive autonomy if the human does not have all the information? Perhaps others can share their knowledge of ongoing research in the comments so we can create a record of this really interesting work.

I realize this is not quite what you are talking about with the targeting cycle—this is far from an AI-enabled autonomous system that can ‘choose’ targets. But I guess these questions and some of this research on human-machine teaming remind me to take a pause when we make an argument like “humans need to do the thinking.” I am not sure it comes down to such a simple truth, and I am not sure which humans may be best suited for the job. Yes, I know – training and making sure the right folks are in the right job- there are many steps one will take to ensure the right people are making these incredibly difficult decisions. But still, each person has different levels of cognitive capacity (on any given day), and our threshold for fatigue varies.

I also think it is equally possible to get the right people to design systems that are guided by laws, rules, policies, and values that are developed and authorized by humans that delineate the choice architecture for machines. It is hard, for sure. I am working on a project grappling with these very questions, but I think it is possible. What do you all think?

First, thanks to those who commented. Please do more of that, it’s great to see some debate. And thanks to people like Michael Kanaan for sharing this substack.

I’ve taken a while to respond mostly because I’m teaching in Spring quarter and preparing my two classes has totally absorbed my attention. But that delay has also been good for me to consider how to better describe what I’m talking about, and what my concerns are. When people like Jack Shanahan and Colin Carroll are like, “you’re wrong on this one”, it’s best to take some time to think. Especially when I agree with everything they said. Even in the face of new reporting on Lavender and Gospel, those mistakes appear to be within the bounds we can expect from automating target identification. The failures appear to be the result of automation bias and action bias on the part of humans at the action stage of targeting rather than machines intruding into the nomination process.

So, what exactly am I trying to get at here?

To me the fundamental questions are: What are we willing to automate, or maybe more specifically, how much agency are we willing to give machines and what are the costs? These questions make me think about why we would be willing to automate some things and not others. Why would we want machines to find targets but not invent them?

There are some viewpoints out there that support automating everything so long as performance is better.Tthe idea of optimizing outcomes to preserve safety or prosperity seems a noble pursuit. Using machines also enables humans to focus on things we do well, avoiding things we don’t do well or don’t want to do. And in some areas, machines are just better than us and we may need to get out of the way. Because of this, the roles of machines in our lives are evolving, including within decision-making structures like targeting. As my friend Jerry Kaplan said to my class last week, today we are discussing the need for a human in the loop but in the future we may be discussing the need for a machine in the loop.

Is that ok? Mostly I agree with using machines to optimize outcomes, including when autonomously striking targets with lethal force. Where I start to hedge a little is in circumstances where machine agency begins to infringe on human cognitive autonomy.

Share

Cognitive autonomy means having as much control over our subjective experience as possible. Meaning we get to decide how things should be and why they should be that way. If you want to pull on this thread, try here and here to start. The idea of cognitive autonomy is not the same as free will. Free will is binary. You either have it or you don’t. Philosophers have been arguing about free will forever, and it’s a fun conversation, but I’m not going into it for the moment. Instead, I’m focusing on cognitive autonomy to distance my idea from the free will discussion for something more relevant. So rather than consider cognitive autonomy to be binary like free will, as something to have or have not, I’m considering it a quest. There is no on or off, there is simply the human imperative of trying to maximize cognitive autonomy to control our own subjective experiences.

This is not an easy quest. Clearly, our cognitive autonomy is influenced by many things. There is constant, sometimes malign, sometimes benign, pressure from our environment and other humans trying to shape what we think, what we do, how we do it. This is an old game, which is sometimes beneficial and sometimes not, but it has been a human game for most of our history, however, 500-ish years ago, that changed.

We can go all the way back to the Gutenberg printing press, its effect on the Protestant Reformation, and the start of the 30 years war, for how machines have affected human cognitive autonomy. Here’s a whole book on it. Suffice to say, it was a big deal, and the scale of the effect increased over the centuries. But, things really picked up in the last few decades with the arrival of the internet, cheap memory, increasing compute power, and of course, AI. Herb Lin and I wrote on this a few years ago if you want to dig into that more.

The point is that machines have dramatically gained in their ability to influence human cognition and thus cognitive autonomy. Is this bad? Well, it depends on how you think about several questions. Here are a few.

Is cognitive autonomy essential to being human, and is the freedom to make mistakes part of that? If you’re trying to maximize cognitive autonomy, doesn’t that mean you get to decide to do dangerous or stupid things? Is there even an objective concept of what is dangerous or stupid? Doesn’t cognitive autonomy demand that we get to decide for ourselves what is dangerous or stupid? Does the fact we exist in a world with other beings mean we have an obligation to optimize “good” decisions? If we could make a tool to help us make good decisions, do we have an obligation to build it and use it?

Those are interesting questions and hard to answer. Perhaps an easier, more relevant, and productive, way to start is to pick a point in human decision-making where machines agents are acceptable. So, where in a decision-cycle do we want machine agents to assist us in optimizing outcomes? Is it at the cognitive phase or in the action phase? Do we want machines thinking for us or acting for us? Does it matter? I think it does.

Debating what happens when machines start to think for us is not new of course. One of the best examples is The Matrix. I’m reluctant to make a sci-fi reference in what is intended to be a practical discussion but I’m going to anyways because I think it will help.

When Agent Smith has captured Morpheus, he starts monologuing, and makes an important distinction between what came before and what is now. The short version:

“The matrix was re-designed to this, the peak of your civilization. I say your civilization because as soon as we started thinking for you it really became our civilization, which is, of course, what this is all about.”

I’m not worried about machines taking over and putting us in the matrix, though some people think that might be great (looking at you Zuckerberg), but that is a consequence of what Agent Smith is talking about. However, his big point is that cognitive autonomy is Excalibur. Whoever gets to decide what happens, and why, owns the future.

So, let’s take a breath and ask: If I want to maximize human cognitive autonomy in war, can I do that through the targeting process? I think so. And the answer I gave before on how to do that was to co-opt some terms and say humans nominate, machines identify.

Now, Colin had a good insight on LinkedIn, and I agree. Targeting terms are not terribly well-defined and it’s hard to know what, exactly, we mean by using terms like nominating and identifying. I’m going to continue to use target nomination as a term to delineate the point where I think machine agency may begin infringing on human cognitive autonomy. I’m happy to adjust and would be interested if people want to throw out better terms in the comments.

So, specifically, when I say machines nominating targets could infringe on human cognitive autonomy, how does that work? I think it begins with the idea of choice architecture. What I’m referring to is the concept from Richard Thaler and Cass Sunstein in their book “Nudge”.

Thaler and Sunstein are economists and suggest that human behavior can be dramatically influenced by shaping the architecture of choices, or more simply, controlling which choices you have available for a given decision. You cannot choose something that is not an option. This idea is useful in many areas because it gets to the heart of decision-making. A narrow choice architecture, say of three choices, means that whoever, or whatever, is operating inside that architecture can only choose those available options. Let’s assume that doing nothing is always an option, for you Dead Poet Society people out there.

Constraining choice architecture is a great way to ensure that autonomous behavior occurs only within acceptable boundaries. This can be human autonomy or machine and it is baked into military organizations and processes.

The military by organization and equipment has a very specific choice architecture from the Stratcom Commander down to a rifleman, reflected in the tools we give them to operate. What is a submarine commander going to do with his torpedoes? He’s going to sink stuff at sea. He’s not going to shoot down a satellite or build a school. Which is ok, we have other tools and organizations for that. So, when that sub commander is going through his version of the targeting process, he’s considering targets within his choice architecture. His staff may prioritize enemy combatants over enemy shipping but that’s not the nominating I’m talking about because it is already built into the menu of choices. Both are authorized targets for which the crew is equipped, trained, and expected to engage.

When we develop software to identify targets, we are essentially doing the same thing that happened to the sub commander. We’ve given an architecture of choices whose individual selection may have some debate associated with it (which ship, where, when, etc) but the outcome is within an acceptable parameter for the overall effort. This is why I agree with John and others in the scenarios they describe.

So, where’s the problem? I suspect things will get squirrely when we start to let machines determine the choice architecture because those determinations are so closely tied with who is doing the thinking. This is what I mean by nominating and what I mean by considering nominating the line we draw to maintain human cognitive autonomy.

Determining what are targets and what are not targets is a fundamental component of the reasons behind war and ownership of the war, for example when measuring military necessity to decide what is or is not a target before using force. Machine control of the choice architecture for targeting means that machines are thinking for us about the fundamental character of the war. Does that mean the war becomes their war and not our war? I don’t know, but I think it is a reasonable question to explore.

Today, we have our first guest post. Major Tramazzo is an active-duty Army judge advocate and a military professor at the Stockton Center for International Law in Newport, Rhode Island, where he co-teaches a course on the Law of Armed Conflict.

John previously served as the Regiment Judge Advocate for the Army’s 160th Special Operations Aviation Regiment (Airborne) at Fort Campbell, KY. He has also served as a legal advisor within the Joint Special Operations Command at Fort Bragg, NC and the Army’s 10th Mountain Division at Fort Drum, NY. He has deployed to Afghanistan and to Jordan multiple times and has traveled to the EUCOM and AFRICOM areas of responsibility for temporary duties.

He holds an LL.M. from the Judge Advocate General’s Legal Center and School in Charlottesville, VA; a J.D. from the University of Baltimore School of Law; an M.A. in Defense and Strategic Studies from the U.S. Naval War College; and a B.A. from the University of Richmond.

Without further ado… here is John’s response.

Subscribe now

A few weeks ago, I posed a question to Lena during lunch in Newport. What is the difference between an AI system nominating a target for destruction and a non-Department of Defense analyst (e.g., an Intelligence Community detailee or a foreign analyst) nominating a target for destruction? In other words, does it matter who (or what) nominates a target, so long as the Target Engagement Authority vets, validates, and adjudicates the target in accordance with the law of armed conflict (LOAC), applicable policy, and commander’s intent?

I don’t think so.

But Brad disagrees. On February 19, he published this piece arguing that AI systems should never, ever, nominate a target. In this post, I offer a different perspective. Specifically, I lay out three reasons why I support deliberate, AI-driven target nominations during armed conflict. I conclude with thoughts about automation bias, including a hypothesis that future warfighters will be more skeptical of AI than their predecessors.

Three Points

First, there is no obligation for a military commander to understand the specific methodology that a nominating source (human or machine) relied on to reach a factual conclusion about a potential target. Target nomination is merely a starting point, albeit an important one. What matters is that a military commander understands the logical and causal link between nominated targets and the overall objectives. As Brad pointed out, in the modern era, “there is so much data available for analysts to consider when nominating targets that sifting through it all might take so long that an adversary could act more quickly and decisively.” In my view, commanders need not perfectly understand an AI system’s functionality to rely on its recommendations. In fact, they might even consider using AI to develop a no-strike list (NSL); of course, analysts should vet and validate AI-driven NSLs to ensure no valid military objects make the list.

Second, AI-nominated targets do not present legal issues, per se. The LOAC imposes an obligation on people to discriminate between military objects and unlawful targets (e.g., civilians, civilian objects, protected things and places) and to ensure that attacks are conducted in accordance with the rules governing attacks (e.g., proportionality, precautions). Paragraph 6.5.9.3 of the DoD Law of War Manual sums this concept up nicely. When an AI system like “The Gospel” nominates a target, it does not guarantee that the proposed target is a military objective. The Gospel does not determine whether an attack may be expected to result in incidental harm that is excessive in relation to the concrete and direct military advantage expected to be gained. Only humans can make those conclusions of law. This is a red line, but AI-driven target nominations do not cross it.

Third, I acknowledge that AI systems do not engage in reasoning. In my view, it doesn’t matter. Indeed, algorithms lack operational experience, moral intuition, purpose, compassion, and…humanity. But consider the original question. Suppose that a friendly foreign analyst was to nominate a target for destruction in the context of an armed conflict. The battle staff would receive the information and investigate it on behalf of the commander. The staff may never see or fully understand why the analyst nominated the target. For various reasons (e.g., political sensitivities, protection of sources and methods) the root reasoning may remain secret or opaque. Target nominators from the Intelligence Community or foreign governments can be “black boxes” in the same way AI systems are.

Thus, the staff seeks to answer questions like:

- Does the nominated target meet the higher commanders’ objectives, guidance, and intent?

- Is the nominated target consistent with law of war?

- Is the desired effect on the nominated target consistent with the end state?

- Is the nominated target politically or culturally sensitive?

- What may the effect be on public opinion (enemy, friendly, and neutral)?

- What are the risks and likely consequences of collateral damage?

- Is it feasible to attack this nominated target? What is the risk?

- Is it feasible to attack the nominated target at this time?

- What are the consequences of not attacking the nominated target?

- May attacking the nominated target negatively affect friendly operations due to current or planned friendly exploitation of the target? 

Depending on the answers, the staff may approach the commander with a recommendation to strike. The staff may also determine that the target is operationally unworthy of inclusion on a target list or that it presents legal issues.

My point is that target nomination is merely one factual step in a process loaded with human reasoning and legal judgments. If you replace the words “friendly foreign analyst” above with “AI system,” the battle staff and the commander would behave in the exact same manner. Staffs and commanders will (and must) always vet and validate targets, regardless of the nomination source. In my experience, commanders do not blindly trust anyone or anything. Even the most well-designed AI systems will not replace the contextual inquiries we always make or the precautions we always take.

Automation Bias and Multi-Generational Workplaces

That said, Brad made some compelling points. First, he noted that AI systems will nominate more targets than humans have the capacity to vet and validate. We must resist the temptation to skip the inherently human targeting functions (i.e., vetting, validating, target engagement decision). Commanders who demonstrate laziness in this regard or blind trust in AI shouldn’t be permitted to serve as Target Engagement Authorities.

Second, he discussed automation bias and how humans tend to trust computers, even in the face of evidence that they should not. Excessive trust in computer systems is a real problem. We’ve all read the stories about drivers who follow navigation apps over collapsed bridges or into lakes. Humans over-rely on technology. In the context of Israeli operations in Gaza, he argued that The Gospel system may be leading the Israel Defense Forces into “a strategic hallucination cycle from which they can’t escape until it’s too late.”

However, as more digital natives enter the workforce, I anticipate automation bias will dull. Hear me out. Unlike older generations in the workforce, Generation Z trusts technology, in general (see here), but understands that AI systems essentially exploit their behavioral data. They have only lived in a world in which computers and smart phones constantly surveil them and force-feed them targeted advertisements. They use Siri and Alexa, but they do not trust them as omnipotent virtual goddesses.

Even Generation X understands the limits of powerful AI. Brad, for example, recognizes that his Facebook account feeds him baking videos as if he were a “70-year-old white lady from Alabama,” which he is not, I assure you. In my case (I’m a Millennial), social media has me wired: bourbon, saltwater fishing, CrossFit, military humor, black labs. I am impressed with the algorithmic accuracy, but I don’t do everything my iPhone suggests that I do. I make independent decisions about how to act and what to buy based on my schedule, my budget, and my circumstances.

Younger people will continue to engage with AI systems, of course (do they have a choice?), but they understand that Instagram and TikTok only encourage them to purchase cowboy boots because they recently searched “flights to Nashville” (or whatever). I predict that consumer awareness of, and skepticism about, AI-driven advertisements will translate to healthy skepticism in AI-driven target nominations. Here are some studies that I think support my intuition:

The Rise of the Digital Native: How the Next Generation of Analysts and Technology Are Changing the Intelligence Landscape

The Kaleidoscope: Young People’s Relationships with News

Fake News Reaching Young People on Social Networks: Distrust Challenging Media Literacy

Study on the Perception of Generation Z in Relation to Robotized Selection Processes

Human Trust in Artificial Intelligence: Review of Empirical Research

Rethinking Technological Acceptance in the Age of Emotional AI: Surveying Gen Z (Zoomer) Attitudes Toward Non-Conscious Data Collection

Gen Z, Explained: The Art of Living in a Digital Age


The gist (from page 20 of this report), which I agree with, is that:

“In the future environment, end-users, who have grown up using AI-enabled, immersive, and mobile technologies their entire lives, will inherently trust those applications to accomplish assigned tasks. That said, these same users will very likely be skeptical of information they engage with in the digital world and will seek out both disconfirming and confirming information as they form opinions or make decisions.”

Of course, I could be wrong. But that’s my working hypothesis and my sincere hope.

Finally, I note that Brad concluded, “AI Nominating Targets is Bad.” While I appreciate his judgment and his clear stance on the issue, I think it’s important to recall Melvin Kranzberg’s First Law: “Technology is neither good nor bad; nor is it neutral.” Allowing AI to nominate targets is not inherently illegal, unethical, or immoral. However, it will certainly change the way we fight, and change is uncomfortable.

The thoughts and opinions expressed are those of the author and not necessarily those of the U.S. government, the U.S. Department of the Army, the U.S. Department of the Navy, or the U.S. Naval War College.

Subscribe now

I’ve been percolating on this for a while. There are several new ideas that need some expansion here, but this isn’t a journal article so let’s just get the discussion going. It’s long. Brace yourself. Also, this essay takes no sides in this conflict, it simply analyzes a component of it.

What concerns me about the information in Lena’s piece and other reports about Gospel and its supporting systems (Fire Factory and Alchemist) is that the IDF might be allowing Gospel to nominate targets, not just identify them. If that is true, Gospel is creating a tremendous amount of strategic risk for the IDF, and they need to be more forthcoming about what it’s doing. Frankly there are some things that AI just shouldn’t do, and nominating targets is one of those things.

AI Nominating Targets is Bad.

In my view, if Gospel is just identifying targets in the battlespace, that is pretty much ok. I’ll include tracking and custody in that. There are some engineering challenges, but it is possible to build an AI-enabled target identification system that operates with reasonable performance under specific circumstances. However, nominating targets should be avoided because of the cognitive and ethical hurdles that machines cannot surmount, which will accrue unacceptable risk for everyone involved.

The difference between nominating targets and identifying targets is significant. In every piece of reporting I’ve seen on Gospel, the term “targeting” is used to cover just about every phase of a targeting cycle. I don’t want to be too pedantic but using that term to cover so many functions obscures what the system might be doing. I assume journalists just don’t know any better, but I suspect the IDF is being deliberately imprecise to avoid some tough questions. So let’s unpack this a bit.

Brief Description of a Targeting Process.

To start, a brief description of the targeting process might help. When militaries are trying to figure out how to accomplish their mission, they use a decision process that includes the nomination, approval, identification, actioning, and analysis of targets. This occurs in a continuing cycle that can take anywhere from hours to months to years. The critical function is to ensure that actions are in service of a greater purpose, are aligned with an ethical framework, and will not waste limited resources.

When nominating targets, analysts are recommending that the higher purpose can be reached by affecting some entity in the environment. As you might guess, nominating (and approving) targets requires reasoning through complex concepts like context, values, trade-offs, risk, strategy, and culture. Analysts must consider immediate and long-term implications of their recommendations while balancing immediate pressures. At best a bad nomination will just waste time and resources. At worst, bad nominations can divert military effort from a winning strategy and cause the collapse of further meaningful pursuit of desired goals. The analysts who nominated the U.S. Pacific Fleet in 1941 made an existential mistake.

Once nominated and approved, finding and identifying the target comes next. In many ways this is an engineering problem with many processes, though not completely devoid of reasoning. Identifying mostly consists of sensors (including people) bringing in data for analysts to confirm that an object is indeed an approved target. AI enabled computer vision can be very helpful in culling through data quickly resulting in faster, and hopefully more accurate, engagements but we know misidentifications will happen.

While misidentifying targets can have serious, sometimes lethal, consequences, in most cases the error is realized quickly and corrected. This is not the case in the complex, sometimes nebulous process of nominating targets. Because the effects from engaging targets is not always immediately known, deciding whether a target is the correct one or not takes time and analysis.  And the risk for getting target nominations incorrect is much higher than misidentifying targets, to the point where it seems obvious that humans would want to keep nominating as a human function only. To stay with our Pearl Harbor example, how long did it take the Japanese military to realize that Pearl Harbor was a strategic mistake. Months? Years? However, using AI-enabled systems to nominate targets is not irrational and there are real incentives to do so.

Why would militaries let AI nominate targets?

First, there is so much data available for analysts to consider when nominating targets that sifting through it all might take so long that an adversary could act more quickly and decisively. This is coupled with the assumption that considering all the data will necessarily improve target nomination quality. This quest for speed and quality in nominating the right targets is key to gaining what the U.S. military terms “decision-advantage”. That is an ability to reason and act at a tempo that overwhelms the adversary.

But there is more to it than just tempo, there is also a quest for Move 37. Those familiar with DeepMind’s AlphaGo will recognize Move 37 as the point in AlphaGo’s 2016 match against the human Go champion, Lee Sodol, that suggested the machine saw a strategy that no human would have seen, much less implemented. When AlphaGo made Move 37, Lee Sodol had to leave the room, he was so taken aback. That move, unforeseeable to the human adversary, appeared to be a critical point for AlphaGo to win the match. Militaries of the world took note. The idea of finding a Move 37 in war is tantalizing to militaries leaders because it provides an almost magical tool against a talented resourceful adversary limited by human cognition. What could go wrong?

Recovering from mistakes.

If AI is employed for tasks where, if it makes a mistake, we can see it, understand it, and recover from it, we can probably consider integrating it safely. This is why AI is well suited for helping humans with process-oriented tasks such as target identification. If the model gets the identification wrong, the mistake may be significant, but we will know and we can then work a way to recover and improve. When introducing AI to reasoning processes, the nature of the technology means that we may not recognize that mistakes are being made until we have a catastrophe. I want to stay this side of saying an existential catastrophe, but I wouldn’t rule it out.

Let’s use an example here. If I use AI to identify tanks on the battlefield and it makes a mistake, we will know within a reasonable span of time, and we’ll be able to refine the model or suspend its actions. Could noncombatants be killed? Yes of course, but unfortunately in most cases this is not catastrophic. However, if AI is used to nominate tanks as a target and the organization efficiently and ruthlessly starts destroying tanks, the cycle will continue until enough data and analysis comes in to suggest that there is a problem. How long will that take? How much ordnance will be expended on tanks and how much damage will be done pursuing tanks? If the AI is also doing the post-effects analysis, how will it ever conclude that it is on the wrong path and that its actions have detached from its purpose? How long would it take human oversight to see the problem?

I suspect that it would take a long time for humans to notice that an AI’s target nominations were not supporting the higher purpose. In that time a lot of damage to the mission could occur. What might that look like? It could look like 28,000 Palestinian deaths in Gaza causing the world to turn its back on Israel and costing Israel a strategic victory in favor of rapid “targeting”. The problem is that the nature of AI technology and the cognitive limitations that humans bring to any interaction with machines makes control of an AI target nomination system perilous.

How would AI target nomination detach action from purpose?

Why would AI target nomination spin action so far from purpose and why is it hard for humans to see? First AI technology is not capable of reasoning. There I said it. It isn’t reasoning at any level that humans find worthy of the term. Second, AI tech has an explainability problem. Meaning the whys and the hows of its outputs remain uncertain. Third, adaptive models will adjust to reinforce positive feedback humans love to outsource hard things to machines and they tend to be comfortable letting the machine take the lead. We call this automation bias. Lastly, many times what we see when machines give humans lots of options is that there is a bias to take action. This idea is a little new. We’ll start to explore it here but I think it deserves some serious research on its own. Some colleagues and I are building this study right now. In the meantime, let’s take a quick look at these four things.

Inability to reason.

First, the obvious. AI is not reasoning. There is no system out there that is reasoning, including LLMs. There is also no system that can understand the complexities that inform human purpose, including constitutional AI from Anthropic (though that is a great idea we tried to build at the JAIC in 2020 and were told it was impossible, so hats off to the Anthropic peeps). So even the most advanced AIs are using probability, token matching, etc to predict which outputs might satisfy the user. This is not the same as reasoning, but I’m not going to do a philosophy summary on reasoning here. Just press the “I believe” button for now.

Transparency/Opacity.

Second, we don’t really know why an AI makes the recommendations it makes. This is the explainability/transparency/opacity problem. When a computer vision model identifies something like a tank, we are not completely sure what features the model is using to determine tank or not tank. Is it the turret, the gun, the tracks, the color? To address this opacity, we box in the model’s deployment parameters. We test the model to determine where and when the model can be used, under what conditions, and where it will fail. With solid T&E data we can try to employ the model well within its performance zone, depending on our risk tolerance. With a target identification model we can test how well it works under given conditions and then mandate that it is employed only in those conditions. How would this work with a target nomination model?

Adaptive models giving you more of what you want.

In military circles we have measures of performance and measure of effectiveness. In shorthand we ask two questions: 1) are we doing things right? And 2) are we doing the right things? There AI models, called adaptive models, that can do the first.  By “observing” their environment the models tweak their software, and thus their actions to increase the probability of reaching the desired output. The feedback mechanism for this is effective but crude so I’m skeptical about applying it to the reasoning problem required for question number two. I don’t think it will work and I think experience shows that it constrains reasoning, creativity, and initiative.

Adaptive models are designed to learn during use so that they get better at giving you what you want. This presumes that what you want now is the same as what you wanted before. Much like your Netflix or Amazon accounts. What can happen is an AI pushes you down a reasoning tunnel that prevents you from seeing or thinking anything new, because it is only showing you stuff based on reinforcement from the user. This is why Facebook’s like and dislike buttons are so important to their models. The model will base its recommendations on previous success rather than on fresh reasoning. How good is it? My accounts think I’m a 70 year-old white lady from Alabama. Please someone make it stop showing me baking videos. Imagine an AI trying to please you with target nominations. AI: “you really liked when we blew up that missile truck last week, shall we blow up some more missile trucks?” You: “I guess. Missile trucks seem bad.”

Automation bias.

That’s crazy, you’d obviously stop that nomination and want a more refined analysis, right? Would you though? There are two AI-human biases that suggest you might let it ride. First, several studies suggest that humans tend to favor outputs from automated sources. At its simplest this means that humans will tend to trust machine outputs even if they receive external input that the machine is wrong. Here is a recent study where students continued following a robot in a “burning” building even though the robot was leading them away from the fire exits, which the students could clearly see.

Automation bias seems related to confidence (or trust, if you must) in the machine’s output and lack of confidence in human reasoning. After all, machines consider more data, and aren’t clouded by emotions, prejudices, and other bias. The confidence created by automation bias can result in increased action, and it seems possible that Gospel could be creating that environment for IDF decision-makers. Particularly when we introduce a second human cognitive limitation: action bias.

Action bias.

In this study, professional soccer goalies were shown to prefer to move rather than stay still during penalty kicks, even if staying still was the better strategy according to the data. Reasons for this are not clearly understood but let’s speculate. There could be pressure to act in the face of action by an adversary. In the case of the goalie, the guy kicking the penalty is acting, what is the goalie doing in response? Just sitting there? To many that would seem unacceptable. The goalie needs to do something. In the goalie’s mind, is it then better to appear to be doing something? Goalies usually do not save penalty kicks. Saving one is very, very hard. If you know you will likely fail, is it better to fail while doing something or doing nothing? If you act, at least you tried right? Or perhaps you do block the penalty kick. If you block it by just standing there, are you rewarded with the same accolades as if you were to block it by making an incredible dive? Were you just lucky?

When we start looking at Gospel we know that it is “producing” many more targets than a human staff would. If that means identifying targets, then action bias might have the IDF striking more targets because there are more identified targets to strike. You can imagine the IDF strike authority being asked why they are not striking all these perfectly good targets. And you certainly would not want to be the one who fails to defeat Hamas because you didn’t action all the available targets. And striking those identified targets would still be validated as being within the overall strategy. The cost for inaction may be worse than the cost of action. Gospel’s productivity might be enabling IDF personnel to do more, but it may also be creating pressure to act far more than may be in their interests.

If Gospel is nominating targets, and action bias is in play, the IDF could be running its strategy off a cliff because it doesn’t know why it’s doing what it’s doing. But it’s doing a lot of it. The confidence and action encouraged by an AI allowed to affect human reasoning through the target nomination process could mean that the IDF is entering a strategic hallucination cycle from which they can’t escape until it’s too late.

The hallucination cycle.

When I say hallucination cycle, I mean that an organization that allows AI into its reasoning processes linking action with purpose no longer knows what is and is not in the service of human goals. Your understanding of the situation becomes so tied to what the AI is feeding you, its interpretation of what you want, and its interaction with your own biases that you can no longer tell what reality is. And the worst part is that you will not even know anything is wrong.

I have no special knowledge of Gospel or what is going on in Gaza, but the sheer quantity of strikes the IDF is actioning makes me worried that their actions have become untethered from a coherent strategy. If that is in part a result from integrating AI systems too far into human reasoning and target nomination cycles, the IDF needs to know that so it can recover and get back on track. The rest of the world needs to know that too so we can avoid a similar fate.

Share Killer Robot Cocktail Party

Subscribe now

According to The Guardian in December 2023, the Gospel system "significantly accelerated a lethal production line of targets that officials have compared to a 'factory.'"

Regarding the pace of Gospel decision-making or target recommendation, media outlets reported different numbers. The Guardian claims that, before employing the Gospel, the IDF produced 50 targets a year and now, using the Gospel, can produce 100 targets a day. On the other hand, NPR reported that the system could produce around 200 targets within 10-12 days. NPR's estimates seem more likely.

According to NPR, the Gospel system is new – the earliest mentions date back to 2020, when the system received a top innovation award. And this is not surprising. The IDF has a reputation as a technologically advanced fighting force, so it makes sense that it would employ relatively new systems in real time. AI tools like the Gospel can accelerate the decision space and provide significant data to target analysts. We can see a similar reliance on emerging tech in the conflict in Ukraine-Russia, where some observers acknowledge a race between Russia and Ukraine to ‘out-innovate’ each other.

The Gospel system can identify static and moving targets and, based on data we know nothing about, it produces a recommendation for human intelligence and targeting analysts to review and determine whether to authorize the recommendation for further review along the appropriate channels.

Certainly, anyone who follows the military AI space will have a thousand questions about the inner workings of the Gospel system and its subsequent performance. But much to our dissatisfaction, there is much about this system that we will not know and are not likely to know any time soon. We do not know what data is used or the quality of other AI systems that are informing Gospel target recommendations; we do not know which target strikes originated from the Gospel, and we do not know how accurate the system is.

Subscribe now

There are other concerns typical for any AI system, namely explainability and automation bias, which are particularly acute in this case. The explainability problem stems from uncertainty regarding the data and decision protocols that led to a particular recommendation. The analyst evaluating the final product does not know if the system made errors or if there was biased or inaccurate data leading to an inaccurate target recommendation. The inability to open the box and examine the decision-making pathways is one of the notable concerns of using AI in this way – even more so if the system is autonomous (which the Gospel is not).

There are also legitimate concerns about automation bias. This refers to an over-reliance, sometimes called overtrust, in the system's output. If the Gospel previously produced good targets, there may be more willingness to rely on the output and not hold the information to the highest degree of scrutiny. It is a phenomenon we are all familiar with. But there is also the consideration that the Gospel produces a lot of information for human analysts to review. If we take NPR’s estimation, the system can produce roughly 200 targets in 10 days; that is a lot of information to validate. It is easy to imagine how analysts could feel overwhelmed (especially in the stress of conflict), how a high accuracy in the target recommendations could result in overtrust in the output, and how somebody could make mistakes. It seems imperative to have an information-overwhelm protocol to help handle this workload responsibly and implement solutions to these challenges. It may be that the IDF has implemented those protocols – it is just not public information.

For a while, the news about the Gospel system flooded my social media (the algorithms know me well) with worrying headlines like the IDF is using a "data factory" to commit "mass assassinations." Hyperboles aside, it is a giant leap to blame the AI for the now over 20,000 Palestinian deaths. Perhaps the outrage is warranted – the issue is that we simply do not know. And I guess that is the point. While global discussions are happening within our communities about 'responsible AI' or 'trustworthy AI', we must consider how quickly we jump to conclusions in both directions – assuming a system is perfect or terrible. At the end of the day, humans are responsible for the output of the Gospel system. As new tools enter our toolbox to make data-driven decisions, we must not forget that humans are (for now) responsible, and we make loads of mistakes.

Share

Before I discuss my thoughts on your last post, let me congratulate my party partner for his appearance in Business Insider’s Top 100 People in AI for 2023! Anyone who reads KRCP can understand why Brad has a place on this list – again, huge congrats!

Okay, now for the good stuff. The takeaways from the Economist article are helpful and not much too surprising from this piece. I was intrigued by your first impressions about the willingness of Ukraine or Russia to use lethal autonomous systems. I’ve been thinking through the constraints Ukraine may indeed be facing, particularly in the context of employing lethal autonomy. However, they are facing far greater constraints in domestic politics (in the US and Europe) than with any kind of technological capability.

There have been recent claims that Ukraine is using ML in attacks (we’ll get into that) and it is worth considering what kind of backlash has already come from this. Not even necessarily by the US, but certainly from European partners and supporters. Let’s consider three things – the current technology, operational environments, and political support.

In October, Forbes reported that Ukraine is using Saker Scout drones (made in Ukraine) which use AI to “identify and attack 64 types of Russian ‘military objects’ [including tanks, personnel carriers, and other hardware] on their own, operating in areas where radio jamming blocks communication and prevents other drones from working.” Saker Scout drones can carry three kilos of bombs to a range of roughly 12 km. According to Forbes, the system is “updated on demand when there is a requirement to detect a specific new object or vehicle type.”

The Saker Scout drones have the capability to carry out strikes autonomously, and according to the Forbes report, these types of strikes have already occurred by Ukraine in fully autonomous mode, targeting Russian hardware on a small scale – most likely when radio interference or jamming prevents direct communication with an operator. The baseline capabilities are not necessarily new – but there are a few things to talk through here.

My initial instinct was that the US would have had a stronger response to these types of attacks – and, indeed, perhaps they have behind closed doors. But publicly, there is no response. There could be multiple reasons for that. For one, reporting regarding these attacks is from early October, and we all know that the US attention shifted to hostilities in Gaza. Second, according to Forbes reporting, the drones used in autonomous mode have only been for small-scale attacks. Of course, there is no information beyond that, but perhaps these systems were used in conditions that did not raise any (or many) eyebrows.

In the abstract, it is not difficult to imagine AI-enabled autonomous systems carrying out bigger attacks in the future. From a legal perspective, there are many questions about the ’64 military objects’ that are constantly updated to include more objects. This certainly demonstrates the real-time concerns that many lawyers have about how legal standards or concepts are incorporated into AI-enabled autonomous systems. I would think a big US response would happen if the systems suddenly started conducting indiscriminate attacks after one of these continuous updates. Perhaps more on that in another post to put some of the legal academic thought into concrete examples.

Also, the operating environment matters a lot to the future use of these systems. In nearly every event I have participated in (including the recent event in Newport – more on that later), there are significantly fewer controversies or concerns using these systems in sparsely populated spaces (sea, desert…) than in an urban environment. Since we have minimal information about where the Saker drones were used autonomously, we can’t untangle that web quite yet. But I would assume most here will agree, that the where will matter for future use and create less of a stir from Western sponsors to Ukraine.

Subscribe now

As we have all seen, Ukraine is struggling to generate more support these days, which is far more dependent on domestic politics and heightened attention to hostilities in Gaza. But as Congress sorts out the future of aid to Ukraine, a $52 billion support plan for Ukraine was blocked in the EU by Hungary. Nonetheless, as of mid-November, the DOD spent 97% of funds allocated by Congress for assistance to Ukraine. It may well be that aid will soon end or be extremely limited. Other watchdogs (human rights groups, for example) may be far more concerned with the use of autonomous weapons than anyone else at this point.

Certainly, I am not nearly as much of an insider as you or many of our readers. But given the enormous discourse devoted to AI, lethal autonomous weapons, and the future of warfare, it is striking that many of these capabilities (especially ML) are being used on a battlefield and the world responds with a shrug. Killer robot fatigue? Short attention span? Greater attention on Israel and Hamas? Not enough reporting?

Who knows. But certainly, forums like this are always helpful, perhaps necessary, to continue to track the technological, political, and operational evolution of these capabilities. Perhaps the global comfort levels with lethal autonomous weapons will adjust not with an overnight shift and the sudden reality of these systems, which is somehow what I imagined. Rather, a slow and steady introduction with very little public indignation, and even a general shrug.

Leave a comment

Oftentimes when you have these discussions it’s very theoretical because we haven’t seen real autonomy on the battlefield yet, and certainly not lethal autonomy. At least as far as we can tell, which is a whole other discussion. However, the point of this post today is that I recently saw an article in the Economist which suggests that what most of us accept as inevitable, the introduction of autonomous systems to the war in Ukraine, is getting closer and closer.

Share

Apologies if you can’t get past the Economist pay wall. Here are some of the key points from the article.

            1. Russia’s electronic warfare capability has always been good but now it is experienced and robust.

This is well known. U.S. military development has long been concerned with Russian GPS jamming as part of a larger technological effort to degrade U.S. networks to slow down and disrupt U.S. capability. No surprise that the Russians are focusing that capability on Ukraine. The risk the Russians are taking is exposing that capability to U.S. collection. I’m literally salivating at all the data that the U.S. and NATO must be hoovering up on Russian EW systems. Thanks Vladimir.

            2. Russia’s EW efforts are disrupting Ukraine’s ability to strike targets with U.S. precision guided munitions.

Again, not surprising but it is disheartening to see that Russian stuff actually works. The Economist reports that GMLRS, Excalibur, and JDAMs are missing their targets due to Russian jamming. It’s unclear how much of a problem this is but the Economist says in some areas the “majority” of ordnance is going astray. This is probably true. And the Ukrainians do not have a robust counter EW force that I’m aware of. I suspect that the U.S. and NATO are not keen to loan or sell highly classified counter-EW equipment to Ukraine and are even less interested in openly burning Russian EW systems themselves. So Russian dominance in this space will likely continue.

            3. Russian EW is defeating 2000 drones per week.

“Defeating” appears to mostly mean severing the link with the base station which causes the drone to hover aimlessly until it runs out of batteries but may include scrambling guidance systems on more sophisticated platforms. It’s not clear if the drones are being recovered and reused. I doubt too many Ukrainian soldiers are excited about recovering drones unless there is very little risk. Can the Russians use captured drones? Yes. Not hard. Most likely most of them are lost to both sides. Hard to tell though.

The real question is, can the Ukrainians continue to fight effectively while losing drones at that rate? If they can’t what are they willing and capable of doing about it? The obvious answer is to upgrade to autonomous systems that can operate without contact with their home base or continuous GPS signals. This is one of the main arguments for adopting autonomous systems.

Leave a comment

Here are some of the considerations for upgrading to autonomous systems the Ukrainians are probably working through:

1. Is technology good enough and cheap enough to insert autonomy into their ISR (Intelligence, Surveillance, Reconnaissance) and combat workflows? The answer is probably yes for ISR but not for strikes. ISR autonomy can be as simple as the aircraft flying a programmed route and recording FMV (full motion video). It can also be as complicated as hunting for objects in an assigned area and identifying those objects in near real time, then sending back the information once communications are reestablished. That type of ISR requires larger, more sophisticated, and more expensive platforms than what Ukraine is buying off the shelf.

Shifting from ISR to strike requires much higher levels of confidence in the model that is working autonomously. If an ISR platform sends you an image and says it found a Russian T-72 but analysts see that it is a Ukrainian T-72, that is irritating but not necessarily catastrophic. If that same system strikes that Ukrainian T-72 thinking it is a Russian T-72…well you get the idea. It is unlikely that there is a model available to Ukraine that can operate in an environment that complex. However, there is a tactical solution. Ukraine could employ a lethal system somewhere where complexity is lower. Perhaps against other aerial targets or sea targets. This doesn’t solve the ground problem directly though.

            2. If Ukraine introduces autonomy to the battlefield, will Russia do the same? One could argue that Russia already has to some extent with its Shahed/Geran-2 strikes but this is not really autonomy so much as it is precision strike with a guided munition. Also, I don’t think Ukraine’s use of autonomy influences Russia. If Russia could field autonomous ISR effectively they probably would. Would they field lethal autonomy? If they could get it to work…probably. A bigger factor than Ukraine’s use of autonomy may be that Ukraine cannot degrade the EW space the way Russia can, so Russian autonomy is less of an imperative.

            3. Would the U.S. and NATO become grumpy if Ukraine started using autonomous ISR? I doubt it but maybe. The risks are pretty low. Even if conventional strikes are being used based on locations collected from autonomous ISR systems, the same target vetting process that occurs now will still be in place. This means that the probability of an incident caused by autonomous systems locating targets is low. That risk will change with lethal autonomy because the systems available to Ukraine are not good enough. If the worst were to happen, a lethal autonomous system striking something it shouldn’t, the U.S. and NATO could pull the plug on support that is already precarious. I suspect the Ukrainians will avoid lethal autonomy until the threat from Russia becomes more immediately existential.

Thanks for reading. For upcoming pieces I’m working on intelligence and consciousness in autonomous systems. So get out your philosopher hats, and possibly your pajamas, for next time when I go deep on sentient lethal autonomy.

Share

Leave a comment

Subscribe now

I confess to finding it a little exhausting because in the public consciousness we’re not making great progress towards consensus on where to start an informed and circumspect discussion on AI-enabled military capability. The only consensus seems to be that any time the term “AI” is used in conjunction with anything, we should be afraid, very afraid. Fear can be useful, but it rarely generates good policy outcomes.

In your last post you wondered about geo-political issues raised by AI-enabled autonomous systems. First, I think that everyone should check out the paper by one of our subscribers, Lt Gen (ret) Jack Shanahan, on the security dilemma triggered by AI. That is a great place to start and maybe we can get him to do a quick piece for us if we ask nicely. But before I dig too far into that I want to relate an observation from the past couple weeks.

One of the reasons I didn’t publish this post on time last week is that I’ve been sitting on several panels related to AI in the last couple weeks. The topics included the ethics of developing and employing autonomous weapons, the dangers posed by generative AI, and “AI and Truth”. The funny thing about these panels is I ended up saying a few things over and over again just to try to frame the discussions. In the spirit of trying to frame our discussion on AI, autonomy, and geopolitics I’ll repeat them here. You may be thinking “yeah, yeah, tell me something new,” but a common set of assumptions may help us to pour some water on the fires of doom.

So, here are three assumptions that I think must be made to have a productive conversation about AWS. Maybe we need more but let’s start with these. First, war is bad. It’s killing and destroying for political purposes and that is not something humanity should be doing. Wow that was really insightful! Most rational people know and accept that. But oftentimes discussions of AWS seem to drift into constraining a new weapon because it has increased killing potential, and because killing is bad, we shouldn’t build stuff that makes it easier. If this was our only assumption this might be okay. But it isn’t.

Our second assumption is that, despite war being universally bad, humans will continue to make war on each other, and most human societies face the potential for some kind of war; civil or state on state. Do I need to cite examples?

Let’s ignore Russia/Ukraine and Israel/Palestine for the moment. Let’s look at the list of ongoing conflicts. How about the Myanmar civil war? 11,846 deaths this year. The Ethiopian civil war? Somewhere between 10,000 and 109,000 deaths last year. Sudan? 11,000 deaths this year.

A war between major powers will be much worse. We can imagine what the numbers will be like if the U.S. and China start fighting in the Western Pacific. Estimates in a CSIS wargame suggest tens of thousands killed in the first few weeks. And that’s just the combatants. Unfortunately, a war between the U.S. and China no longer seems far-fetched. It now seems almost inevitable. But the idea that wars will occur in the future is not new or surprising. The likelihood is built into international law.

In the face of reality, international law assumes that war will occur despite being outlawed by Article 2(4) of the United Nations Charter. How do I know? Because of the existence of IHL that governs the conduct of combatants in war, deciding what is legal in war and what is not. There is a whole industry built around governing the conduct of combatants in war. This is done because governments know that war will occur and we should probably try to keep the harm to what only needs to be done to win, that is to restore a just peace.

If we assume that war is likely, possibly inevitable, what is to be done? What past wars have taught us is that losing a war is not good. In the best cases, losing means social upheaval. In the worst cases, genocide. Considering the significance of consequences like social upheaval, genocide, and everything in between, we are pushed to our third assumption: It is better to win a war than to lose it.

Let’s compare post-war Germany and Japan to post-war America and Europe. The societies that existed in Germany and Japan prior to World War II essentially no longer exist while the allies gained 70 years of economic dominance and prosperity. Disclaimer: I’m not bemoaning the loss of Nazi Germany and Imperial Japan here. Good riddance. I’m just noting that the result of losing that war was not the desired outcome for the losers.

Share Killer Robot Cocktail Party

On a smaller scale, look at what happened to the losing sides in Sudan, Rwanda, Myanmar, and Yemen to name a few. Losing is not good, therefore states and societies will do what is necessary within their means to win.

I can distill these three assumptions into a process that is easy to describe but really complex in the details: risk management. When preparing for the possibility of war, states and societies will decide how to act within their means to buy down risk of failure by considering a trade-off between harm and benefit. In simplest terms, states will estimate the probability and severity of harm that an action might generate, and then weigh that estimate against the probability and significance of the benefit that might result from that action. I’ll symbolize this in a formula so I can show my kids that I do actually use math at work. Where H is harm and B is benefit: (H_p) (H_se) : (B_p) (B_si)

If the math comes out significantly in favor of the benefit side of this equation, particularly if the severity of harm estimation is high in any case, most rational actors will pursue that course of action. In this case, development and employment of AWS.

This process is occurring right now. The problem is that we don’t really know when or if the benefits from deploying AWS are going to be good enough to outweigh the potential harms. We think they will increase the speed, tempo, resilience, and flexibility of warfighting functions to the point where winning a conflict is much more likely. We also think that we are close to building systems that will outperform humans both tactically and ethically, or perhaps improve humans tactical and ethical performance. But that capability has not yet been demonstrated in any meaningful way. Including in Ukraine.

So where does that leave us? Probably with a bunch of people wanting to write in and tell me all the things I didn’t mention. Fair enough. But anecdotally, in every instance where I’ve briefed these basic assumptions, the discussion seems more productive. This includes discussions with activists who would prefer a total ban. We often end up finding common ground because the risk calculation above just won’t allow states to agree to a total ban. What states might agree to is minimizing unnecessary suffering through constraints and guardrails, which is really what the point of IHL is all about any ways.

As always, thanks for reading. Lena and I would love for readers to throw ideas out in the comments or submit for a full post if you’ve really got a lot to say! Have a great week.

Leave a comment

Share Killer Robot Cocktail Party

Brad, wow – you’ve got some great points here. I completely agree that alignment is more of a process than a destination. But your point clearly demonstrates why it is critical to take that lifecycle perspective for these systems. There is a ‘many hands’ issue when it comes to alignment because so many people participate. But it begs the question – is human intent consistent throughout that process?

We keep referring to ‘human intent’ – but whose intent? Whenever I read or engage in discussions around intent, it is usually commander intent that is the priority. And this makes sense. But does an AI developer necessarily know what their software will be used for, or begin to envision command intent or how their product would (or could) contribute to commander intent?

On my last trip to the Pentagon to chat with research & engineering, the issue of programming and maintaining commander intent was the core concern and kept repeating itself in our discussions. The process of calibrating intent (or alignment) is an issue at the forefront of research and engineering priorities.  While I agree a process of alignment and risk management is a useful way forward for embedding human interaction and intent into autonomous systems, there are uncertainties to address. One big one may be aligning (see what I did there?) the conversation with MHC folks and altering the discourse on human influence and interaction.

As for responsibility, this is a related issue. But I am not sure I agree with you that lawyers are not asking these questions – I think there is significant disagreement in the legal community (and certainly in legal academia, but I suppose that’s their job). But the general discussion about the ‘responsibility gap’ always confused me because there are so many different forms of responsibility and accountability – and I understand the ‘gap’ to be largely regarding criminal responsibility. I have never heard the argument that lesser, non-legal sanctions were at issue. And since the lawyers love the precision of terms, let’s disentangle this a bit.

Stockton Center Professor James Kraska’s article about this is extremely useful as it delineates commander accountability and command responsibility as it relates to the deployment of AWS. To (over)simplify, “[t]he commander is accountable for battlefield action regardless of whether subordinates made and compounded errors, machines performed unexpectedly, or an incident arises as an unforeseeable consequence of pure happenstance or the fog of war.” This accountability may not necessarily be criminal violations (though it could include this) but also non-judicial or non-legal mechanisms of accountability. This is a distinct concept from command responsibility.

Command responsibility, by contrast, is a concept in international criminal law in which “the commander may face legal jeopardy for failure to exercise control over forces under command that violate LOAC.” (446). To establish command (or superior) responsibility, then the commander needs to have “effective control” in a superior-subordinate relationship. Can a commander have effective control? Is an AWS a subordinate? Some great work has been done on these questions in the legal community recently. [Yes, lawyers reading, I understand this is an oversimplification.]

Clearly, there is disagreement in the legal community about AWS command accountability and AWS applicability. For example, a Human Rights Watch report argues it is “arguably unjust” to hold commanders accountable for machines “over which they could not have sufficient control.” I find this sentiment in private conversations with some legal-military folks. In these conversations, it becomes clear that “those sitting in air-conditioned boardrooms have the time to make sure these systems operate as they should rather than placing that responsibility on those in the field.” This position certainly exists.

To me, the issues of human intent and responsibility are distinct but intertwined. The focus on commander intent at the engineering stages of AWS development makes sense because, ultimately, this is where command and control exist, as well as ultimate accountability. Despite the disagreement in the legal community, I agree with Kraska’s position that accountability falls with the commander – as with any other type of weapon. But lawyers are paid to disagree – so, of course, there are those who push back.

Nevertheless, it is worth acknowledging the counterpoint. Human intent at design/development is reflected in machine performance, and it is not always clear how the alignment of the ‘many hands and intents’ at multiple stages in the lifecycle timeline can problematize traditional concepts of accountability and responsibility. This is also why a few years ago, there was a flood of legal articles exploring whether AI developers could be charged with war crimes (spoiler alert: they could, but it would be REALLY difficult to prove and extremely unlikely).

All of this is to say, whether we talk about intent or responsibility, this falls on the shoulders of commanders who may (but likely will not) have any technical training or experience with autonomous systems. But this is also why a life-cycle perspective is always important to keep in mind because even with the emphasis on command accountability/intent, there are many others involved in the timeline. And not just developers. But acquisition/procurement officials, testing & evaluation, and (in a very different way) other nations and allies that cooperate or collaborate on technology development.

Which brings me to something I have been researching lately. New technologies (especially dual-use emerging and disruptive technologies) are not developed in isolation. They are often created for civilian applications, and innovation is not spearheaded by military development. But also, nations work together through various arrangements to create this technology.

We’ve seen a lot of these partnerships in the news – AUKUS, NATO’s DIANA projects, and a recent War On the Rocks post about Chinese-Russian technological cooperation. The US also spearheads the AI Partnership for Defense and, of course, the ongoing GGE discussions in Geneva. But when it comes to weapons development specifically (and I realize much of the IR/tech literature is trying to move away from the weapons discussion), how will AWS alter or otherwise influence multinational operations?

Instead, we should design a framework that requires continuous alignment of action to human intent throughout a system’s life cycle. This transforms a philosophical discussion into an engineering and risk tolerance trade-off. Instead of considering what we mean by meaningful control and when it can be applied to eliminate risk, we focus on improving alignment performance and the framework for accepting the residual risk of employing an imperfect system within complexity. Which allows us to better accept or reject risk because that calculation is necessary for employment. We can meet International Humanitarian Law knowing there will be failures, confident the alignment process will address those failures continuously within operational imperatives and legal constraints.

Where I think this conversation often breaks down is when the question of responsibility pops up because in the minds of many, MHC is closely tied to closing the responsibility gap. The idea of a responsibility gap with autonomous weapons was first put into popularity by Robert Sparrow in 2007 when he suggested that “as machines become more autonomous a point will be reached where those who order their deployment can no longer be properly held responsible for their actions,” because “Where an agent acts autonomously, then, it is not possible to hold anyone else responsible for its actions.” Really? I’m married to a lawyer and I can confidently tell you that this is absolute nonsense. I am at least 80% responsible for my daughter riding her skateboard down the staircase, believe me.

The responsibility gap argument makes a couple of key mistakes. First that moral responsibility and legal responsibility are the same thing. Second that an artificial agent, in this case an autonomous system, has the same legal status as a human agent. Spoiler, it doesn’t and we have control of that. Third that autonomy in an agent like an autonomous system negates negligence in other agents. It doesn’t.

Let’s deal with moral vs legal responsibility first. The responsibility gap argument is that an artificial agent cannot be held morally responsible for something because it is not a moral agent. Now there is a huge discussion about whether an artificial agent can be a moral agent, and I’m inclined to think that it can if it can act morally. But there are some good arguments that if an artificial agent doesn’t know what it is like to be moral then it cannot experience morality in a way that makes it a moral agent. Big discussion. I’m going to leave it there. But the salient point is that something that is not a moral agent can be held legally responsible. Think about large corporations that are punished for misbehaving.

Second, an artificial agent does not have the same legal status as a human agent. This means we can apply necessary rules to it that suit our purposes without being burdened by conflicting philosophical morality. And this is being articulated with the CCW and GGE declaring that “accountability cannot be transferred to machines.”

Third, the concepts of negligence and responsibility are not tests of autonomy, they are tests of a reasonable standard of care. The requirement for a reasonable standard of care can be spread across many different agents in a legal discussion of any kind of mishap. Even something as simple as a car accident usually finds every participant to be at fault to some degree. And here is the real kicker, sometimes no one failed to take a reasonable standard of care. Even when deaths are involved.

So, let’s apply this to an autonomous weapon that is employed in war and mistakenly strikes a target killing people who didn’t need to be killed. What are we to do? Want to know who never asks this question? Lawyers. My friend Jerry Kaplan once quipped that lawyers never ask him questions about responsibility when a robot harms someone. I have found in my own travels this to be absolutely true. Why? Because the law has mechanisms for determining responsibility, liability, and negligence. These processes may not help with the question of moral responsibility, but I think they close the responsibility gap enough to focus on the actual issue, which is how to build a policy and legal framework that ensures a reasonable standard of care at all points of an autonomous system’s life cycle.

Let me get my stick out so I can beat on this horse again before you run me off: the best course to a routine and acceptable reasonable standard of care is alignment and risk management.

Ultimately, Brad, I like your suggestion to focus on alignment rather than control, but what you suggest is more precisely what MHC folks are trying to advocate and implement. My use of the term MHC is not necessarily because I think the term captures the relationship we are trying to grasp – rather, there is not a great alternative (for now). You are correct that we need to ditch the term, but to ditch the concept might be premature. We are dancing around the same idea – perhaps alignment is the better alternative.

Allow me to put words in many people's mouths (hopefully accurately). The broader community of experts is also ready to ditch MHC. At this point, it is clear the term is meaningless (pun intended) at properly grasping the human-machine relationship we want to capture. What is the threshold for meaningful? Which human is appropriate to 'control' a system? Is it possible to conceptualize or operationalize control of autonomous systems without compromising the inherent advantage gained in any context? Likely not. And, as you suggest, if we are not going to absorb the benefits of AI on the battlefield, then why go down this path? Let's figure out what, exactly, we are talking about.

AI alignment aims to match machine performance to human intent or goals in a particular context. This is certainly a significant engineering problem. However, in my reading, the MHC people take this point for granted. They assume that any deployed system will have the capacity to reflect the commander's intent or the broader value system embedded in system control. Alignment is assumed. And maybe it should not be, as others have noted – the issue of alignment is far from settled or simple.

The issue for MHC people is misaligned systems. The weight behind a concept like MHC is the importance of intervention if a system fails, malfunctions, or is otherwise misaligned with human intent or goals. This is the crux of it. How much risk are we willing to assume?

In your last post, you suggested we need to accept a certain degree of risk in deploying systems – and, of course, you are right. But it is far easier to take this point than to implement it. Risk calculation and willingness vary from person to person, potentially depending on their ultimate relationship with the system. This is why breaking down different life cycle stages can crack open the 'human touchpoints’ where risk is ultimately calculated and by whom. For example, the developer(s) may have a threshold for risk that will vary amongst each other (or even by the company), which could inform designers' definitions of objective functions or feedback for the system. Similarly, commanders have different risk assessments and need to calibrate the operation based on their personal assumption of risk and leadership style. Finally, operators will certainly have different degrees of risk assumption for any operation for any particular context.

To be sure, there is risk in employing humans for these operations, too. Human autonomy, as Grog and Lunk can attest, is not new. However, we are more comfortable with human autonomy and assuming the risks inherent to human decision-making because it is easy to empathize with the difficulties of the decision made in an armed conflict. After all, ‘to err is human’ – making mistakes is a basic understanding of our humanity. But it is also easier to live with human autonomy and inevitable mistakes (or malice) because there is a system for accountability, which brings me to my next point.

The MHC people often emphasize responsibility and the 'accountability' gap of employing autonomous weapons. One of the primary arguments for MHC is to find a way to determine a pathway for accountability in the (inevitable) circumstance of mistakes or misidentifications made on the battlefield.

To me, the issue of accountability and MHC (and certainly alignment) are separate issues. Even embracing the concept of MHC does not outline a pathway of accountability. As my first post argued, there are numerous points where MHC (or choose your preferred term) is embedded into any autonomous system. The parameters of a human-machine relationship that MHC and AI alignment are trying to demonstrate is better characterized by the technical relationship between the many humans involved and their subsequent influence on machine performance – ideally toward greater alignment. But the accountability for unintended outcomes is related -- and deserving of a separate post. But nevertheless, these issues are far from settled.

While I am risking the wrath of MHC advocates by putting words, and maybe ideas, into their mouths, they are largely getting at the same type of relationship as the alignment folks. We all want to ensure responsible systems that do what they are supposed to and do it better than humans can. There are many ways to embody the relationship between humans and machines, and your suggestion of alignment is one of the better options for that embodiment. 

One thing we can agree on — it is time to move on from MHC. Where we go next is up for debate.