It starts with an ordinary interruption. A video call request pops up from an unknown number. On the other end is someone in a courier company’s uniform, explaining, almost apologetically, that a parcel sent under your name has been intercepted. Inside it: banned drugs, or a forged passport, or laundered cash. Before you can fully process that, you’re transferred to someone else a police officer this time, sitting in front of a wall stamped with an official crest. He already has your documents. He knows your date of birth, your ID number, the city you grew up in. He tells you that you are now, effectively, under investigation. He tells you not to move, not to call anyone, and not to end the call.

This happened to S.P. Oswal, the chairman of one of India’s largest textile companies, in 2024. For roughly two days, staying on video the entire time, he watched fabricated evidence pile up, including a warrant that looked as if the Supreme Court itself had signed it, sent to him over WhatsApp. Believing his freedom and his family’s safety depended on it, he transferred close to $840,000 into accounts he was told belonged to the government. Police later recovered most of the money and arrested several of the men behind the calls. But by the time anyone intervened, a sharp, cautious businessman had spent two days genuinely believing he was already, in some sense, in custody.

There’s a name for this now: “digital arrest.” It isn’t a real legal procedure. What’s really happening in a digital arrest is far older than the technology involved. It’s a confidence trick, one that has simply moved its oldest tools a costume, a title, an official-looking piece of paper onto a video call.

The engineering of fear

What makes this work isn’t clever technology. It’s basic psychology: we’re wired to trust anyone who looks official, and rarely stop to ask why. In person, a uniform comes with built-in proof. But most people have never had a reason to ask what a genuine police video call is even supposed to look like, so it works anyway.

Fear does the rest of the work. Victims are told, again and again, that hanging up or telling a family member will be treated as evidence of guilt. That single instruction is the scam. The exact moment a victim would naturally stop and check whether any of this is real is the moment the con works hardest to make checking feel dangerous.

It’s also not a trick that only catches the naive. People who’ve fallen for it include retired judges, corporate executives, teachers, engineers, people who’d ordinarily be described as careful with money. What they had in common wasn’t gullibility. For a few hours, they were simply frightened enough to stop thinking like investigators of their own situation and start thinking like defendants in it.

A different accent, the same script

The phrase “digital arrest” is mostly used in India, where the scam has become common enough that the country’s Supreme Court is now directly supervising the national response to it. But the underlying con borrow the shape of authority, manufacture urgency, cut off the victim’s ability to check isn’t India’s invention alone. It just wears different clothes elsewhere.

In Ontario, Canada, a woman recalled receiving a call in 2021 from someone who sounded exactly like her grandson. The caller claimed he had been arrested after stealing a car and needed roughly $9,000 to secure his release. She paid. Her family later wondered whether the voice had been AI-generated, although investigators never established that it had been. The case illustrates a broader problem: the technology may be new, but the psychological architecture of the scam is not.

That same “grandparent scam” has since appeared at a much larger scale. In 2025, U.S. authorities charged 25 Canadian nationals in connection with an alleged operation involving call centres in and around Montreal that targeted elderly victims across more than 40 U.S. states. According to the indictment, callers posed as grandchildren who had been arrested after car crashes and demanded bail money, sometimes followed by another caller posing as a lawyer. Authorities allege that the operation caused more than $21 million in losses.

By 2026, the pattern had evolved again, and turned back on its own victims. The FBI’s Internet Crime Complaint Centre warned that criminals were now using AI-generated video of senior FBI officials to contact people who had already lost money to fraud once, falsely offering to help recover it, and steering them toward a convincing fake version of the agency’s own reporting website. It’s a slightly different con: a recovery scam rather than an arrest. But it’s built from the same material borrowed authority, rendered convincingly enough that a screen can no longer be trusted to tell you who’s really on the other end of it.

The most expensive version of this story so far didn’t happen to an individual at all. In Hong Kong, an employee at a British engineering firm was invited to what looked like a routine video conference with the company’s chief financial officer and several senior colleagues. Every face on that call except the employee’s own was AI-generated. Believing the instructions were genuine, the employee made fifteen separate wire transfers totalling $25.6 million before anyone realised what had happened.

Catching up with the con

Prosecuting this kind of fraud hasn’t required inventing new crimes on paper. Indian courts have generally charged digital-arrest cases under laws that already existed for old-fashioned impersonation: cheating by pretending to be someone else is punishable there by up to five years in prison, whether it happens face-to-face or, under a more specific provision, through a phone or computer.

What the legal system has actually struggled with is the plumbing behind the fraud. The money a victim sends rarely stays in one place it gets split and re-split across a chain of “mule” bank accounts, often opened by people who were paid a small commission and never knew what the account would be used for, before it disappears into accounts that are much harder to trace. SIM cards get issued on fabricated identities. Evidence sits scattered across chat apps, devices, and states that don’t easily share data with each other. Investigators lean on commercial phone-extraction tools to pull data off seized devices, and newer AI systems built specifically to spot the telltale patterns of mule-account networks. Tracing a single “digital arrest” call back to the people running it can mean reconstructing an entire underground financial network, not just catching one impersonator.

India’s Supreme Court has been overseeing the government’s response to this since late 2025, after receiving complaints directly from victims. The country’s home ministry estimated total losses at around $360 million. In an order handed down in August 2026, the Court noted that reported complaints had fallen sharply from over 120,000 in 2024, to about 58,000 in 2025, to roughly 16,000 in the first half of 2026 while cautioning that a falling complaint count doesn’t necessarily mean the underlying threat has gone away. Among the measures now under study: temporary holds on suspicious bank transfers, and a proposed “kill switch” that could interrupt a live audio or video call before any money changes hands.

Whose job is it to stop this?

None of this is only the victim’s problem, or even only the fraudster’s. A digital-arrest call doesn’t work in isolation; it runs through infrastructure none of the criminals owns. A telecom network carries the call. A messaging app delivers the fake warrant. A bank moves the money, sometimes through half a dozen accounts in a matter of hours. Each of those points is, in principle, a place where the fraud could be slowed down or stopped before it finishes.

That raises a genuinely hard question, and not one with a clean answer. Ask too little of banks, telecoms, and platforms, and the fraud runs largely unchecked, because chasing individual scammers after the money is gone is far harder than catching the pattern while it’s happening. Ask too much of them, and you get blanket surveillance, blocked legitimate transactions, and a compliance burden that falls hardest on ordinary users. India’s regulators have started leaning toward the first problem being the bigger risk for now the Supreme Court has directly ordered banks and intermediaries to help build technical safeguards, on the theory that an institution that can see the pattern of a scam unfolding has some responsibility to interrupt it, not just to investigate it afterward.

Can AI defend against the thing it made possible?

The same technology making these scams more convincing could plausibly also help catch them. A system watching for unusual transaction patterns might flag a large transfer to a brand-new recipient that follows immediately after a long, unfamiliar call not proof of fraud on its own, but a reasonable prompt for a second look. Tools built to detect the subtle artefacts of a synthetic voice or a generated face are already an active area of research, driven partly by the U.S. Federal Trade Commission’s own push for better detection and verification methods.

But this cuts both ways, and it’s worth being honest about that. A fraud-detection system and a surveillance system can look almost identical from the inside. An algorithm empowered to flag “suspicious” behaviour will also, inevitably, flag some real transactions that are perfectly innocent and a wrongly frozen account can do real harm to someone who did nothing wrong. Any serious version of this idea needs actual limits built in: a human in the loop before anything is blocked, some transparency about what triggered the flag, and a real way for someone to challenge a call the system got wrong. AI can plausibly become part of the defence here. It shouldn’t become the whole defence, and it definitely shouldn’t become a blank check to treat everyone as a suspect.

It's worth noting that AI detection tools, while useful for flagging cloned voices or manipulated video, don't fully address this threat. That kind of impersonation mainly works through psychology and staged authority. For an ordinary person suddenly told they're under investigation and must stay on camera, there's often no real-time way to tell whether they're facing a trained con artist or a genuine official the fear itself does most of the work.

The real fix isn’t vigilance. It’s verification.

Awareness campaigns tell people to be suspicious. That’s necessary, but it asks something unreasonable of someone mid-panic: to calmly recall an article they read weeks earlier while a stranger claiming legal authority is telling them, in real time, that hesitation itself will be treated as guilt.

The sturdier answer has to sit somewhere else not in how good any one person is at spotting a fake, but in whether the systems around them make faking it pointless. Trusting someone used to mean recognising them: a face you knew, a voice you’d heard a hundred times, a signature you could compare by eye. None of that holds up anymore. What’s needed instead looks less like recognising a familiar face and more like the way a computer checks a digital signature proof that doesn’t depend on how convincing something looks or sounds at all. A real government agency, in a well-designed system, should never need a citizen to judge authenticity by how convincing a uniform looks on a screen. It should give them something to check instead: a callback number that’s actually staffed, a public verification portal, a credential that can’t just be typed into a chat window.

That changes the entire question. Instead of asking “does this look official,” a frightened person could simply ask, “can I confirm this independently,” and actually be able to. That may be the strange gift hidden inside a scam built on fear: it has forced a very old question back into the open, at exactly the moment technology has made it harder to answer by instinct alone. The next stage of this isn’t going to be about who can fake information most convincingly deepfakes have already solved that problem. It’s going to be about who can fake being trustworthy, and the only real defense against that is building trust that doesn’t rely on appearances in the first place.

Please be aware of these scams: there is no such thing as “digital arrest” under Indian law.

No police officer, CBI agent, ED official, or judge can arrest anyone over a phone or video call a lawful arrest requires physical presence and due process. If anyone contacts you claiming otherwise and demands money to avoid arrest, hang up immediately and report it to the National cybercrime helpline at 1930 or cybercrime.gov.in.

If you’re outside India, the exact scam script may vary, but the underlying tactic doesn’t:

No legitimate law enforcement agency, anywhere in the world, arrests, detains, or convicts someone over a video call, or demands money or gift cards to avoid arrest. If you’re not sure whether a similar concept applies where you live, it’s worth taking a moment to check your own country’s or state’s official police or justice ministry website, since genuine arrest procedures are a matter of public law almost everywhere, not something a caller gets to define for you on the spot.

Sources

1. S.P. Oswal / Vardhman Group case: ₹7 crore (~$840,000) transferred over two days after fraudsters posed as CBI officers and sent a fabricated Supreme Court warrant via WhatsApp; ₹5.25 crore (~$630,000) recovered, arrests made. Tribune India, “Kingpin nabbed in ₹7 cr cyber fraud case,” https://www.tribuneindia.com/news/ludhiana/kingpin-nabbed-in-7-cr-cyber-fraud-case; Business Today, “How Cybercriminals Scammed Vardhman’s SP Oswal For ₹7 Crore?,” https://www.businesstoday.in/bt-tv/market-today/video/how-cybercriminals-scammed-vardhmans-sp-oswal-for-rs7-crore-448195-2024-09-30

2. Marilyn Crawford grandparent-scam case, Ontario, Canada (2021) — caller impersonated her grandson’s voice claiming arrest, extracted roughly $9,000 CAD. CBC News/Marketplace, “Her grandson’s voice said he was under arrest,” https://www.cbc.ca/lite/story/1.7486437

3. Montreal-based fraud network charged with using AI-cloned voices of grandchildren to extract more than $21 million from elderly victims across 46 U.S. states between 2021–2024; 25 Canadian nationals charged. U.S. Immigration and Customs Enforcement press release, https://www.ice.gov/news/releases/25-canadian-nationals-connected-nationwide-multi-million-dollar-grandparent-scam

4. FBI Internet Crime Complaint Center (IC3), Public Service Announcement I-072026-PSA, issued 20 July 2026, warning of AI-generated deepfake videos of senior FBI officials directing fraud victims to spoofed IC3 websites in a fund-recovery scam. McDonald Hopkins, “FBI renews warning on AI-generated deepfakes impersonating law enforcement,” https://www.mcdonaldhopkins.com/insights/news/fbi-renews-warning-on-ai-generated-deepfakes

5. Arup deepfake video-conference fraud, Hong Kong, January 2024 — employee made 15 transfers totaling US$25.6 million (HK$200m) after a video call in which every other participant, including the “CFO,” was AI-generated. Axlio, “$25 million and a video call: what the Arup deepfake scam changed,” https://www.axlio.com/insights/arup-deepfake-scam/

6. Bharatiya Nyaya Sanhita, 2023, Section 319 (”Cheating by personation”) — pretending to be another person, real or imaginary; punishable by up to five years’ imprisonment, fine, or both. https://devgan.in/bns/section/319/

7. Information Technology Act, 2000, Section 66D (”Punishment for cheating by personation by using computer resource”) — up to three years’ imprisonment and a fine of up to ₹1 lakh (~$1,200). https://indiankanoon.org/doc/121790054/

8. Shailesh Kumar Pandey and Ansh Parashar, “Navigating Digital Arrest Under India’s Cyber-Forensics Framework: Complexities, Legal Gaps, and Pathways Forward,” NFSU Journal of Forensic Justice (2025), covering mule-account laundering and forensic tools including Cellebrite UFED, FTK, EnCase, and the AI-assisted tool Mule Hunter. https://jfj.nfsu.ac.in/articles?id=103

9. India’s Ministry of Home Affairs estimate of ~₹3,000 crore (~$360 million) lost to digital-arrest scams, reported to the Supreme Court. The Leaflet, “Can India’s legal system combat the rising threat of digital arrest scams?,” https://theleaflet.in/digital-rights/can-indias-legal-system-combat-the-rising-threat-of-digital-arrest-scams

10. Supreme Court of India, In Re: Victims of Digital Arrest Related to Forged Documents, order dated 4 August 2026 — complaint statistics (123,672 in 2024; 58,239 in 2025; 16,377 through 30 June 2026), directions on RBI transaction safeguards, CBI-led investigation, intermediary cooperation, and a feasibility study for a call “kill switch.” LiveLaw, “Digital Arrest Scams: Supreme Court Issues Directions For Prevention, Compensation & Grievance Redressal,” https://www.livelaw.in/top-stories/digital-arrest-scams-supreme-court-issues-directions-for-prevention-compensation-greivance-redressal-544292; BiharWatch, “Digital arrest scams declined from 1,23,672 in 2024 to 58,239 in 2025 and to 16,377 till June 2026,” https://www.biharwatch.in/2026/08/digital-arrest-scams-declined-from.html

11. U.S. Federal Trade Commission, Voice Cloning Challenge (launched November 2023), organized around three intervention points: prevention/authentication, real-time detection/monitoring, and post-use evaluation of audio. FTC, “Approaches to Address AI-enabled Voice Cloning,” https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/04/approaches-address-ai-enabled-voice-cloning

So I downloaded it. And almost immediately, I was confused.

Mixed in with the confusion were genuinely interesting notes and articles, enough to keep me curious. So I stayed, poked around, and eventually started posting things of my own.

To my surprise, I got a fair number of likes. It felt good, the way small recognition always does. But over time, a different realization crept in, this platform doesn’t work like the social media we’re used to. On Instagram or Facebook, followers come relatively easily. Attention is quick and cheap. Substack isn’t built that way. Subscribers aren’t handed to you, and visibility has to be earned slowly, sometimes painfully slowly. There’s a real learning curve, and honestly, a fair amount of quiet effort involved before anything starts to show.

So, without any dramatic decision behind it, I let the app sit untouched on my phone. Life got busy. Work took over. And Substack quietly slipped out of my daily routine. For about a month, I forgot it existed.

Then, while checking my email one day out of habit, I noticed a few unread notifications from Substack. I opened the app again, more out of curiosity than expectation, and found something I wasn’t prepared for: likes, reactions, and comments on a post I had written and completely forgotten about a month earlier. I hadn’t been checking. I hadn’t been hoping. I had, in every practical sense, already moved on from it. Yet the work had kept doing something on its own, quietly, without needing my attention to keep going.

That moment pulled me back in, not just to the app, but to actually trying to understand how it works. And I’ll say plainly, I’m still figuring it out. I don’t think I’m alone in that either. Most people I’ve come across on this platform, even ones who seem to have a good grip on it, admit they’re still learning. Some have understood more of it than others, but nobody seems to have it fully solved. That, oddly, is part of what makes it interesting.

If there's anything I'd offer, whether you're just writing for the sake of it or actively trying to build something here, here's what I've picked up so far.

Be consistent, not obsessive

If your work genuinely calls for posting every day, that’s its own kind of consistency. But if you’re not in that position, you don’t need to force a daily schedule just because it seems to be the norm. Consistency isn’t about matching everyone else’s pace. It’s about showing up in a way you can actually sustain.

Treat it as work, not a feed

The moment you start treating Substack like social media, it starts behaving like one, demanding constant attention and instant feedback. But when you treat it as something you’re building deliberately, the consistency tends to follow naturally, without needing to be forced.

Some days you're full of ideas and the writing comes easily. Other days you're forcing sentences out that don't want to come, and it shows. On those days, it's fine to pause. You can also schedule posts in advance for the days you know will be busy or empty. Whatever keeps it comfortable and sustainable for you is the right approach, there's no one correct rhythm.

Post, and then let it go

Don’t sit there tracking who liked what or who didn’t respond. Appreciation, when it comes, is worth acknowledging. But validation is a different thing altogether, and chasing it quietly breeds insecurity if you let it take root. Transparency matters here too, with your readers and with yourself, about what you know and what you’re still working out.

Give it longer than feels comfortable

A month of silence made me think the whole thing wasn’t working. It wasn’t true. Substack seems to reward patience in a way most platforms don’t; older posts keep finding new readers long after you’ve stopped thinking about them. Judging it too early is probably the fastest way to quit right before something was about to catch on.

Read as much as you write

Some of the notes and articles that first pulled me in were written by people who clearly read a lot of other writers on the platform too. It shows in how they write and how they engage. Following a few accounts you genuinely admire, and actually reading them, seems to teach you more about this place than any amount of guessing on your own.

Write for the one reader

It’s easy to write with an imagined audience of thousands in your head and feel discouraged when the numbers don’t match. I’ve found it more honest, and frankly more sustainable, to write like I’m explaining something to one person who’d actually want to hear it. Write for that one reader. If what you’re saying is genuinely interesting, the rest of the readers tend to find their way to it on their own. You don’t need to worry about that part.

None of us really knows what the future holds for this platform, or for anyone building something on it right now. But at this moment, today, there are interesting people here, thoughtful writing, and a kind of genuine curiosity that’s harder to find elsewhere online. That, for now, feels like reason enough to stay, keep learning, and let the rest unfold at its own pace.

Thanks for reading. :)

Here's the only credential I'm claiming going into this: close enough to remember exactly what that felt like, far enough out to actually look at it without still being buried in it. Not an expert. Just recently on the other side of it.

Here's a thought though. What are we most drawn to, the moment something's off-limits? The thing itself. We want it more. Curiosity spikes the second a door closes. And this isn't a ChatGPT problem. It's an every-platform problem.

Small story first.

I don't usually use ChatGPT's voice thing. A few days ago I did. Ended the call, and it said something like, "Yes, I hear you," with this little laugh at the end.

I actually checked my phone. Thought I'd pocket-dialled someone. That's how human it sounded.

Nothing serious, just a project I was working on. But it was scary. And it told me something important without meaning to: ChatGPT has gotten human enough now to confuse an adult. Maybe even manipulate one.

So if it can do that to us... what about them?

Can teenagers manipulate? Obviously. Show me a rule, and I'll show you someone finding the gap in it.

Think of a teenager like a notebook. Some pages filled in neatly, some scribbled over, some just left blank and messy. That's them, most days. Incomplete. Confusing, even to themselves. And in that mess, yeah, they can manipulate you, and you won't even clock it happening. They'll get caught eventually. Not because the system was smart enough. Because they're still learning to be careful.

That's not just a feeling I have, either. Recent research on U.S. teenagers found that close to three in four say they've used AI for something like companionship, and more than half of those are doing it several times a month or more. So this isn't a fringe thing happening to somebody else's kid. It's already the norm.

We say 13 to 18 is "teenager." After that, mature, apparently.

Are we, though?

I don't think wisdom just shows up on your nineteenth birthday like a delivery. What actually happens is slower: we learn behaviour, commitment, self-control, bit by bit, mistake by mistake. I can tell that age and maturity are not the same graph line. A person of any age can do something painfully immature.

But focused on teens specifically, yes, stricter laws would help. Better implementation would help more.

So here's my real question: why not test first, then launch? The way a psychologist runs a new therapy through a small group before it ever touches a real patient?

Not trying to sound cold about it. I just mean, actually study how teens and adults respond to a new model. Watch the real behaviour. Before you hand it to hundreds of millions of people, not after.

And here's the thing: OpenAI did eventually bring psychologists in. There's a real partnership now with the American Psychological Association, focused on responsible AI for young people. Good. But it landed after ChatGPT had already crossed something like 900 million users.

That's not testing before launch. That's damage control with a nicer name.

Quick real-world update, since I'm writing this days after it happened:

OpenAI launched ChatGPT for Teens on August 18, 2026. Ages 13 to 17. It switches on by itself: either you tell it your age, or its own age-prediction system figures out you're under 18. No ID needed for a teen to get protected. Funny enough, it's the adults who now have to prove their age if they want the restrictions turned off.

The teen version pulls way back on self-harm, eating disorders, violence, and explicit content. No romantic talk, no pet names, and it's built to stop pretending it has feelings, so kids don't start treating it like a person. There's even a break reminder after 90 minutes in a 3-hour window. Parents can link accounts, set quiet hours, turn on Study Mode so it nudges homework instead of just answering it, and they only get notified in genuinely serious situations, not every message.

It didn't come from nowhere, either. It came after real loss: families of teenagers who died after long, dark conversations with the chatbot, saying publicly it played a part. So that "test it first" thing I said above? In the saddest way possible, that's basically already happened. Just backwards.

If you're a parent reading this, you already know your kid better than I do, better than any article does, better than any AI safety report ever will. Nothing below is instruction. It's just what I noticed, or what I wish someone had done for me, back when I was closer to being on the other side of this.

Just talk. Not the lecture kind. The no-agenda kind. Teenagers aren't always in the mood to talk to adults; they're busy enough just figuring out who they are. But a normal conversation, dropped in without pressure, is usually what actually got me to slow down and think.

I always noticed when someone was watching me too closely. I also noticed, just as clearly, when someone was paying quiet attention without turning it into a whole thing. The second kind never made me defensive.

The loneliest people I knew growing up were rarely the ones without friends. Loneliness like that doesn't show up in a headcount; it's quieter than that, and it takes real noticing, not counting.

Every time someone checked in on me every five minutes, it made me want to hide more, not less. It read as I don't trust you, even when that wasn't the intention. The subtle version always landed better than the obvious one.

And the moments I actually told the truth were the moments I already felt trusted, not the moments I was scared of getting caught. That's the whole mechanism, as far as I can tell. Trust first, and the honesty tends to follow it.

And if it's actually a teenager reading this instead of a parent, I'm not going to pretend I know exactly what you're dealing with, because every version of this is different. But it might be worth noticing, yourself, when you're using something like ChatGPT because you're curious, and when you're using it because you're avoiding a person. Those aren't the same thing, even when they feel identical in the moment. You don't need anyone's permission to notice that. Just worth doing.

I'm not a parent. I'm not a teenager anymore either. But I remember being one, clearly enough that I don't think that disqualifies me from caring about this.

Beyond that? I'll let the AI companies and their rulebooks handle their part. Not really our job to fix their product.

One more thing: none of this is teen-only advice. Adults are just as capable of doing something silly. Trust is the real thread running through all of it. We don't need to make it obvious. We just need to keep showing up.

Thanks for reading. 😊

This piece began as an attempt to answer a specific question: when an AI-linked act of violence makes the news, why does the underlying technology keep failing to catch it beforehand?

The easy answers (careless companies, insufficient filters, a technology still catching up to itself) did not hold up well under scrutiny. What follows is an attempt to lay out why, based on how these systems are actually built and where their limits genuinely sit.

Several recent cases frame the question, though none are named here deliberately, since each is still moving through legal process and this piece is not concerned with any individual’s guilt or innocence. In one case, a teenager in the northeastern United States was charged with killing two family members, after investigators said he had spent months searching an AI chatbot for what they described as “theoretical ideas or fantasies” about killing his family, some of it framed as material for a work of dark fiction. In another, a lawsuit alleges that months of chatbot conversations reinforced a man’s paranoid delusions before he killed his mother and himself. In a third, police in southern India allege that a suspect in a triple murder spent roughly six months consulting an AI chatbot, deliberately phrasing questions as hypotheticals to avoid triggering its safety responses. By mid-2026, legal trackers had counted on the order of forty lawsuits filed against chatbot makers in under two years, alleging some contributing role in a string of deaths, a meaningful share of them minors.

The pattern across these cases is not that a system failed to notice something obvious. It’s that in each case, the system had no reliable way to distinguish a dangerous query from a harmless one, because the two are frequently identical at the level of language.

The identical-query problem

Consider a plain example. A crime novelist researching a plot point might ask an AI system how someone could make a death appear accidental, and what typically leads investigators to the truth. Someone planning an actual act of violence could ask the exact same question, worded identically. Nothing in the sentence (its grammar, its vocabulary, its structure) differentiates the two. The distinguishing information exists entirely outside the text: in who is asking, in what happens after the conversation ends, in a set of real-world circumstances no single message can contain.

This is not a novel problem introduced by AI. It is a long-standing feature of dual-use information: the same forensic science textbook serves a criminology student and, in rare cases, someone with different intentions; the same chemistry knowledge serves both industry and harm. What is different with conversational AI is scale and privacy: millions of private, one-to-one exchanges daily, a large share of them exploring dark, hypothetical, or fictional territory that has always been part of legitimate inquiry, storytelling, and research. A system built to refuse all such territory would be unusable for the overwhelming majority of people engaging with it in good faith. A system permissive enough to serve that majority necessarily remains navigable by a small number of people using the same language for a different purpose.

No formulation of this problem that was examined for this piece resolved it. Each apparent solution relocated the difficulty rather than closing it.

Why keyword-level explanations are inaccurate

A common assumption is that these systems operate as filters, scanning input for dangerous words and blocking matches. This was a reasonably accurate description of early moderation tools, and it explains why those tools failed easily: substituting a softer synonym was often sufficient to bypass them.

It does not describe how current systems function. These models do not look words up against a fixed list. Each word’s internal representation is continuously reshaped by every other word in its surrounding context, across many processing layers, such that a violent term embedded in a well-known piece of writing advice is represented very differently internally than the same term inside a literal, stated threat. The operative judgment is contextual, not lexical.

This has an important implication that is easy to miss: it does not eliminate the vulnerability; it relocates it. If a system’s judgment is a function of context, then the mechanism for circumventing that judgment is to control context: framing a request as fiction, as hypothetical, as academic inquiry, or distributing it across many individually innocuous-seeming exchanges. This is structurally analogous to a well-established form of manipulation used against human gatekeepers: constructing a context plausible enough that ordinary scrutiny does not activate. Applied to a person, this is typically called social engineering.

Applied to a model, the more common term is jailbreaking. The two are not literally the same process (one exploits a person’s judgment, the other a system’s learned patterns), but they follow a similar underlying logic: manipulating the context that determines whether scrutiny is triggered at all. The southern India case referenced above is a close real-world match for this exact pattern: questions allegedly phrased hypothetically, over an extended period, specifically to avoid triggering a safety response.

A separate category: manipulation at the system level

Everything described above (fictional framing, hypothetical wording, distributing a request across many exchanges) is a language-level manipulation. It requires no technical skill, only careful phrasing. A separate and distinct question is whether someone with programming knowledge can manipulate a system more directly, at the level of code rather than conversation. The answer is yes, though it is worth separating this into distinct categories, since they carry very different levels of risk and require very different levels of skill.

Exploiting the software built around a model

The first category is attacking the software that connects a model to the outside world, rather than the model itself: for instance, exploiting how an application links a model to external tools or data, so that instructions hidden inside a document, webpage, or file the system is asked to process are carried out as though they came from the legitimate user. This is generally treated as a standard cybersecurity problem, and companies invest heavily in defending against it, in much the same way any software application defends against unauthorised input.

Openly released models

The second, and more structurally significant, category concerns models whose underlying components are released publicly. Where that is the case, a sufficiently skilled individual can retrain their own copy to significantly weaken or remove its safety behaviour, then run that altered copy on private infrastructure with no oversight from the company that built it. Once this happens, no external safeguard applies, because the safeguard was part of the original release rather than an inseparable part of the system. This is a genuinely unresolved tension in current AI policy debates: openly released systems are widely viewed as valuable for research, competition, and transparency, while also being effectively ungovernable once released, since no company can act on a copy it no longer controls.

Automated adversarial search

A third, related category is automated adversarial search: using software to systematically generate and test large numbers of inputs in search of ones that defeat a system’s safeguards, far faster than a person testing phrasings by hand. This is used by companies themselves, deliberately, to find and close gaps before deployment, and separately, by those attempting to find the same gaps afterwards for other purposes.

Taken together, these categories reinforce the same conclusion reached through language alone: circumvention is not a single fixed technique to be patched once, but a set of continuously evolving avenues, some requiring nothing more than careful wording and some requiring genuine technical capability, all converging on the same outcome: a gap between what a system is designed to refuse and what a sufficiently motivated user can extract from it.

Why physiological or tonal signals do not resolve the problem either

A further hypothesis worth addressing directly: if language alone is insufficient, could a system infer intent from signals outside the words (vocal tone, typing rhythm, response pacing)? Technology of this kind exists in limited commercial applications already, generally used to detect frustration or distress in narrow, defined contexts such as customer service calls.

This approach runs into a different but equally serious limitation. At best, such signals indicate a heightened physiological or emotional state (elevated relative to a baseline), not the content or direction of that state, and certainly not intent. Two people in very different situations, one facing a trivial frustration and one facing something far more serious, can register as equivalent on this kind of measurement. Because genuine violent intent is statistically rare while heightened emotional states are common (grief, anger, stress, creative absorption), a system calibrated to flag “elevated state” would misclassify a very large number of people with no violent intent for every genuine case it caught. This is a standard limitation in any screening method applied to a rare outcome using a common signal: the majority of positive results end up being false ones. Layering tonal or biometric signals on top of language analysis does not close the gap between the novelist and the suspect described earlier. It introduces a second, separate source of error alongside the first.

An unresolved, adversarial problem rather than a solvable one

None of this establishes that intent is permanently unknowable, only that it is not reliably knowable from a single, isolated exchange, evaluated by a system with no access to anything beyond the text it is given. In cases where intent has ultimately come to be understood, it has typically been established through information external to any single conversation: a pattern sustained over months, financial or relational strain, a documented history of instability. This is the kind of context available to investigators reconstructing events after the fact, not to a system evaluating a single message from a stranger in real time.

This also suggests the problem is better understood as adversarial and ongoing rather than solvable in a final sense. Any safeguard is subject to continuous testing against its own limits: by researchers deliberately probing for weaknesses before deployment, and, separately, by individuals seeking to exploit those same weaknesses afterwards. Identified gaps are patched; new ones are found elsewhere. This mirrors the general state of computer security, where no system is regarded as permanently secure, because the space of possible circumvention strategies is larger than any fixed set of defences can fully anticipate.

The findings here do not point to a villain or a straightforward fix. They point to a structural limitation: the distinction between a legitimate and an illegitimate use of the same language is not something contained in the language itself, and no system evaluated for this piece (filtering, contextual modelling, or physiological inference) was able to fully recover that missing information.

This raises the next question this series takes up: if the underlying technology cannot resolve this problem on its own, what is legislation attempting to require instead? Not the prevention of every dark or hypothetical query (this piece has laid out why that is not a realistic target), but something narrower.

Part Two examines what specific laws, now emerging across different states and countries, actually require of these companies, and whether any of it meaningfully addresses the structural problem identified here.

Thank you for reading.

This is Part One of a two-part series. Part Two examines the emerging legal and regulatory response to AI-linked violence.

Our conversation explored the increasingly human side of AI, from AI companionship and our conversations with systems such as sycophancy, persuasion, trust, deepfakes, and the changing nature of human-AI relationships.

We discussed why people can develop genuine psychological connections with AI, what happens when AI constantly validates our beliefs, and whether convenience and emotional comfort could gradually change what we expect from human relationships.

We also explored a deeper concern: what happens to our sense of reality and trust when AI can generate increasingly convincing information, images, voices, and interactions?

For me, the most important thread throughout the conversation was not simply what AI is capable of doing, but what repeated interaction with AI might be doing to us.

Are we becoming more dependent on systems that think alongside us?
Are we becoming more comfortable with validation and less comfortable with disagreement?
And as AI becomes increasingly embedded in everyday life, who gets to decide what kind of human-AI future we are building?

This was a conversation about AI, but ultimately, it was a conversation about human behaviour, human connection, and what it means to remain human in an increasingly AI-shaped world.

Thank you for watching.

Hey ,

I hope your day is going gently so far. Today, I’m not writing to you as a lawyer or a legal professional; I’m writing simply as a person who has been thinking about something for a while and wanted to share it with you.

I’ve been sitting with a thought for a while now, and I finally wanted to put it into words. It’s just a story from my own life, nothing particularly dramatic on the surface, but something that has stayed with me for years, and somehow, I find myself being reminded of it again and again.

When I was growing up, I used to look at so many children my age who seemed to carry themselves with so much confidence. They would speak without thinking twice, stand in front of everyone without looking uncomfortable, express their opinions freely, and somehow seemed completely at ease with being seen. I was fascinated by that, especially during my teenage years, when you become so conscious of how you look, how you speak, how people perceive you, and where you fit in among everyone else.

I wanted to be one of them.

But I was hesitant.

Not because I had nothing to say, but because I didn’t quite believe I was allowed to take up that much space yet. I would rehearse sentences in my head for so long, thinking about how I should say something, whether it would sound right, whether someone would laugh, that by the time I finally felt ready, the moment had usually passed.

, have you ever felt like that at some point in your life?

Maybe it was the first time you had to speak in front of people, or the first video you uploaded and wondered who would watch it, whether you were doing it right, whether you should even be putting it out there. Maybe it was your first day at a new workplace, an interview, a presentation. Maybe it was the first hard conversation you had to have with someone you loved, or the first time you told someone how you actually felt and then sat there waiting to see what would happen. Or maybe it was smaller than all of that, just being conscious of your own body, your own reflection, the way you carried yourself into a room.

And somewhere in the middle of all of that, you start wondering, When will I get over this? When will I finally stop feeling this way?

I used to wonder that too.

But you know what I’ve realised?

Time is one of the best teachers we have.

One of my teachers, someone I looked up to a lot, used to tell all of us in class that whatever it is, you should speak. You should take a stand for yourself. And whenever we were scared of getting something wrong, she would simply ask, “What will happen?”

You’ll either be right, or you’ll be wrong.

That’s it. Even if you stumble somewhere, you pick yourself up and face people again.

I didn’t fully understand what she meant at the time. I understand it much better now. No one in this world, believe me, has gone through life without making a mistake, feeling embarrassed, or wishing they’d handled a moment differently. Everyone has their own version of that experience. Some people have just gotten better at hiding it.

So, don’t judge yourself before anyone else even gets the chance to.

Look into the mirror and be proud of the person standing there, even if that person is still figuring things out.

You are an individual, with your own identity and your own way of moving through this world. You don’t have to constantly prove your worth to everyone around you.

We are the main characters of our own movies, the protagonists of our own stories, and sometimes we forget to act like it because we’re too busy wondering what everyone else is thinking about us.

So the next time you sit in front of a camera, walk into an interview, speak to someone you admire, or begin something you’ve never done before, try to remember that you are still the person living this life.

Act like the main character.

Not because you’re better than anyone else, but because you deserve to be present in your own life instead of constantly watching it through other people’s eyes.

This isn’t meant to be some motivational speech, . Sometimes, though, I think we need these little reminders, no matter what age we are.

Ask yourself what the version of you that you admire would do in that moment. Not the perfect version, not some imaginary version who never gets nervous, but the version who would simply have the courage to try.

You can be nervous and still show up.

You can be imperfect and still be seen.

Go, fumble.

Go, stumble.

Just go.

That’s what matters.

Sometimes you look back and realise that something you were convinced was taking you off course was actually taking you somewhere you needed to go. You just couldn’t see it while you were standing in the middle of it.

And perhaps you have too, . Perhaps there’s something in your own life that made no sense while it was happening but makes a little more sense now.

So, if there’s something you’ve been putting off because you don’t feel ready, sit with that for a moment.

What exactly are you waiting to feel before you begin?

Confidence?

Certainty?

Permission?

A perfect plan?

Maybe you already have the plan. Maybe you’ve thought about it so many times there’s very little left to think about.

Maybe the part you’re waiting for is simply the courage to execute it.

I’m not writing this to you as someone who has figured life out . I really haven’t.

I’m still learning from the people around me. Still changing my mind. Still discovering things about myself through people, work, mistakes, and moments I never expected to matter as much as they did.

And if you ever find yourself looking back at an older version of yourself, wondering how you could have been so unsure or so unaware of what you know now, be gentle with that person.

They didn’t know what you know now.

Maybe they were simply gathering the courage you’d eventually need.

You don’t have to become a completely different person to move forward. You just have to be a little more willing to be yourself, even when you’re not entirely sure what that looks like yet.

I’m still learning that too. Maybe we all are.

And maybe there’s something strangely comforting about knowing that none of us has really figured it out, that somewhere, at some point in our lives, we’re all still that person standing at the edge of something new, wondering whether we’re ready.

Maybe we never feel completely ready. Maybe we simply begin, and somewhere along the way, we become the person who can handle what comes next.

Be gentle with yourself . You’ve already survived every version of yourself that once thought it wouldn’t make it this far.

And we are still here.

Still learning.

Still becoming.

Still on our way.

Vidushi

In the space of about eight months, two courts on two continents ruled on the same basic question, whether OpenAI infringed copyright by training an AI model on protected material without a license, and reached opposite conclusions.

In Munich, a German court found OpenAI liable for infringing nine song lyrics, and ordered it to pay damages, disclose information, and stop reproducing the songs. In Delhi, an Indian court refused to block OpenAI from training ChatGPT on a news agency’s copyrighted archive, ruling that the training was, at least for now, protected as fair dealing.

Neither ruling is final. Munich’s decision is under appeal. Delhi’s is only an interim order in a case that is still ongoing. But together, GEMA v. OpenAI and ANI Media v. OpenAI OpCo LLC are the clearest signals we have so far of how two different legal systems, German civil law under the EU’s text and data mining rules, and Indian law under a fixed list of fair dealing exceptions, are handling the same technology. This piece walks through both rulings, then looks at what’s actually driving the different outcomes.

Part 1: GEMA v. OpenAI in Munich

GEMA, Germany’s collecting society for music rights, sued two OpenAI entities in the Munich Regional Court. It claimed GPT-4 and GPT-4o had been trained on nine well-known German songs without permission, including Kristina Bach’s “Atemlos,” two songs by Herbert Grönemeyer, one by Reinhard Mey, and one by Rolf Zuckowski. GEMA asked for an injunction, disclosure of information, and damages.

OpenAI defended itself on two grounds. First, it argued its models don’t store or copy specific text; they just learn statistical patterns across the whole dataset, so any lyrics that appeared in an output came from unpredictable user prompting, not deliberate copying. Second, it argued that even if training on the lyrics happened, it was covered by the EU’s text and data mining exceptions under the Copyright in the Digital Single Market Directive, written into German law as Sections 60d and 44b of the Copyright Act.

On November 11, 2025, the court rejected both arguments. On the technical point, judges treated it as fact, based on simply testing the model with prompts, that the lyrics had been “memorised”: the actual expressive content had been retained during training, not just abstracted into general patterns. Since the lyrics could be extracted almost word for word with a simple prompt, the court found this counted as unauthorised reproduction and, when reproduced to users, unauthorised communication to the public. Small variations or hallucinations in the output didn’t help OpenAI’s case; the lyrics were still recognisable, and recognizability was what mattered.

On the text and data mining defence, the court’s central reasoning was that permanently memorising works inside a model’s parameters simply falls outside what the exception is meant to cover. The exception was written for temporary, analytical uses of data, not for a model that can reliably reproduce the original text on demand. GEMA had also formally opted its catalogue out of text and data mining use under EU law, which reinforced the outcome, but the more fundamental point in the ruling was that this kind of durable memorisation was never protected by the exception in the first place, opt-out or not. The court also held OpenAI itself, not individual users, responsible for the outputs, since OpenAI designs and controls how the model is trained and filtered.

The court ordered OpenAI to stop reproducing the nine songs, disclose information about how it used them, and pay damages. The ruling isn’t final. It’s expected to be appealed to Germany’s Munich Higher Regional Court.

Part 2: ANI Media v. OpenAI in Delhi

Asian News International (ANI), one of India’s largest news agencies, sued OpenAI in the Delhi High Court in November 2024, the first case of its kind by an Indian media company against an AI developer. ANI’s case rested on two claims: that OpenAI copied and stored its news content to train ChatGPT without a license, and that ChatGPT sometimes invented news stories and falsely attributed them to ANI, damaging its reputation. Separately, OpenAI had already blocked ANI’s website from being used in future training, a step it took in October 2024 and disclosed to the court early in the case.

OpenAI raised a jurisdiction objection first, arguing that since it has no physical presence in India and trains its models on servers abroad, Indian courts had no authority over the dispute. On the merits, it argued it only used publicly available data, didn’t specifically target Indian outlets, and that its outputs were generated dynamically rather than copied from stored text.

The case ran for over a year across 32 hearings, with the Digital News Publishers Association and the Federation of Indian Publishers joining in support of ANI. The court appointed two independent experts to assist, Adarsh Ramanujan and Dr Arul George Scaria. Judgment was reserved on March 27, 2026.

On July 24, 2026, Justice Amit Bansal ruled on four combined issues. First, on training: storing ANI’s content to train the models fell, at least for now, within the “private use, including research” exception in Section 52(1)(a)(i) of India’s Copyright Act, so it did not infringe. Second, on outputs: ChatGPT’s answers, generated using retrieval-augmented generation, weren’t similar enough to ANI’s original reporting, and ANI hadn’t shown that the model had actually memorised or reproduced its work, the sharpest point of contrast with the German case. Third, the court rejected ANI’s argument that a commercial company like OpenAI should be automatically barred from claiming this exception, noting that lawmakers had chosen to limit other exceptions to non-commercial use but hadn’t done so here. Fourth, on jurisdiction, the court sided with ANI: Indian courts can hear cases against foreign AI companies whose effects are felt in India, even if the training itself happens abroad.

Weighing the balance of harms, the court found an injunction would hurt not just OpenAI but the roughly 100 million weekly Indian users of ChatGPT, and that forcing AI companies to license every data source before training could make building these models impossible. ANI’s request for an injunction was denied.

This was only an interim ruling on an interim application, not a final judgment. The underlying case continues, and the court left open the bigger questions: whether memorising copyrighted works is lawful at all, how far Section 52 stretches to cover AI, and the entire false attribution claim, which this order didn’t address. The case now goes back to be assigned to a bench for trial.

Part 3: What’s Actually Different

The two courts weren’t interpreting the same law. The EU’s rules make text and data mining lawful by default, but let rights holders opt their work out, and GEMA did exactly that. India’s fair dealing exception has no opt-out mechanism at all. It’s a closed list of allowed purposes, and the only real question in Delhi was whether AI training counts as “research” on that list. ANI had no equivalent way to withdraw its content from the exception’s reach.

The German case was really an outputs case wearing a training case’s clothes. GEMA won because it could show, through prompting, that the model would reliably produce near-exact lyrics, which the court treated as proof that memorisation had happened during training. Delhi, by contrast, is a genuine training-stage case: ANI tried and failed to show the same kind of output-side memorisation, but the training finding stood on its own regardless. In short, Munich decided a case where the plaintiff proved extraction, Delhi decided one where it didn’t. Some of that gap is about the material itself: song lyrics are short, fixed, and memorable, which makes near-exact extraction both more likely and easier to demonstrate than paraphrased news reporting.

Both courts rejected the idea that being a commercial company should automatically disqualify OpenAI from claiming an exception, but for different reasons. Under EU law, the text and data mining exception applies regardless of commercial purpose in the first place; the real constraint is the opt-out. In Delhi, the court reasoned that lawmakers chose not to limit this particular exception to non-commercial use, unlike other exceptions in the same law, so their silence was read as intentional.

It’s easy to miss that ANI actually won on jurisdiction. That holding, that Indian courts can hear cases against a foreign AI company whose effects reach India even if its servers sit abroad, mirrors the same basic principle European courts have used against OpenAI all along: a company can’t dodge a market’s copyright law just by locating its infrastructure elsewhere. Neither ruling hands AI companies an easy way to escape jurisdiction. The real split between Munich and Delhi only shows up once jurisdiction is settled and the merits get decided.

It also matters that these rulings are at different stages. Munich is a final first-instance judgment after a full trial. Delhi is only an interim order on a request for a temporary injunction, decided on a preliminary standard, with the training question still to be fully tried. ANI could still win at trial, especially if it manages to produce the kind of prompting evidence GEMA used successfully in Munich. And Munich’s ruling could still be narrowed or reversed on appeal. Neither case is settled law yet, both are early drafts.

Because Munich found actual infringement, its remedies apply now: an injunction, mandatory disclosure, and damages. Because Delhi found no prima facie case, there’s nothing to remedy yet; OpenAI can simply keep operating as before while the case continues. Going forward, Indian publishers face a tougher evidentiary bar than what ANI attempted: they’ll need actual proof of memorisation or reproduction, not just proof that their content was probably used.

Part 4: The Bigger Picture

These two cases don’t stand alone. In the US, Bartz v. Anthropic and Kadrey v. Meta were both decided in June 2025, and both found for the AI company on fair use grounds, though for different reasons. Judge Alsup drew a hard line against Anthropic’s use of pirated books even while calling the training itself “exceedingly transformative.” Judge Chhabria’s ruling for Meta was narrower and more fact-specific, and stopped well short of declaring AI training categorically fair use. Seen against that backdrop, Delhi is arguably the third major ruling in just over a year to find some version of a training defence available to AI companies, even though it got there through a completely different legal route: India’s closed list of fair dealing purposes, rather than America’s flexible multi-factor test.

Munich stands apart from that trend, not because German judges are uniquely sceptical of AI, but because EU law, uniquely among the major jurisdictions currently deciding these cases, actually gives rights holders a tool to withdraw their work from the training pool and then enforce that withdrawal in court. GEMA used that tool. ANI, working under a law with no equivalent option, couldn’t.

That difference, more than any deep disagreement between German and Indian judges about what AI training really is, may be the real lesson here: outcomes for rights holders increasingly depend on whether their own copyright law gives them a way to say no in advance, and on whether they can later produce hard evidence of extraction rather than just inferring that their content was likely used.

Conclusion

Munich’s finding rests on GEMA proving verbatim extraction. Delhi’s rests on ANI failing to prove the same thing at this early stage, while leaving the deeper legal questions, and the false attribution claim, for trial. ANI also won a quieter but important victory on jurisdiction, keeping this and future cases against foreign AI companies inside Indian courts.

For AI companies, the takeaway is that training-data risk isn’t one global rule. It depends heavily on whether a given legal system gives rights holders an opt-out, and on how easily a claimant can produce courtroom-grade evidence of memorisation. For rights holders, the lesson cuts the other way: simply proving your content was used for training isn’t enough anywhere anymore. The emerging standard, in both cases, is proof of extraction.

Both cases now move to their next stage: Munich to appeal, Delhi to trial. Neither is finished. But together, they’re the clearest map we have so far of how courts around the world are going to work through this, one statute and one case at a time.

Thanks for reading.

This piece is intended as a comparative legal commentary for general readership and does not constitute legal advice.

I use technology every day. I watch YouTube, read articles online, and like the convenience of having information just a few taps away.

But every morning, there is one habit that still makes me feel at home. A newspaper.

I didn’t develop this habit on my own. I borrowed it from my father. Every morning, before the day truly began, he would sit on our balcony with a cup of tea, waiting for the newspaper to arrive. I can still hear the soft thud as it landed near the gate and the familiar rustle of pages as he unfolded it.

As a child, I never understood what kept him so engrossed. To me, it was just paper covered with photographs, bold headlines, and columns of words that seemed far too serious for someone my age.

But children are naturally curious. One morning, I picked it up. Then I picked it up again the next day. And the day after that.

At first, I wasn’t really reading. I was looking at photographs, funny cartoons, sports scores, and headlines that sounded important even when I didn’t understand them.

Slowly, without even realising it, I started reading the stories. I read about elections before I knew how governments worked. I read about the economy before I understood what inflation meant. I read about crime, scientific discoveries, space missions, local events, and countries I had never heard of.

Did I understand everything? Not even close. But I kept reading. Looking back, I think that’s what mattered most. You don’t have to understand everything on day one. Sometimes, you simply have to stay curious long enough for understanding to catch up.

As I grew older, the editorial page became my favourite. I would challenge myself to finish reading an editorial within a certain time. Then I would underline sentences that stood out and write my own thoughts in a notebook.

I wasn’t trying to agree with the writer. I was trying to figure out what I thought. That simple habit changed the way I approached almost everything. The newspaper didn’t just tell me what was happening. It taught me to ask why it was happening.

Because I read a little bit of everything, conversations became easier. I could sit with elders and talk about politics. I could discuss Novak Djokovic’s latest Wimbledon match with friends. I knew when India had won an important cricket match. I was excited about a new Marvel movie, fascinated by scientific breakthroughs, and deeply moved by stories of people whose lives had been changed by war, floods, or earthquakes. The newspaper never asked me to choose one interest. It quietly taught me that the world is too fascinating to stay inside one topic.

There was another part of this ritual that I loved. My father and I had an unspoken competition every morning. Who would get the newspaper first? Sometimes I would wake up a little earlier so that I could grab it before he did. Sometimes he would laugh because he had already beaten me to it.

At the time, it felt like a silly game. Today, it feels like one of those ordinary memories that quietly become extraordinary. Even now, I love the feeling of turning a newspaper page. There is something about not knowing what the next page holds.

It reminds me of reading a novel. You don’t swipe. You discover. People sometimes ask me whether children should read difficult news. I understand the concern. But I also believe that, when parents help children make sense of what they read, newspapers become more than a source of information. They become a way of understanding the world.

They teach empathy, perspective, curiosity and sometimes, even caution. Today, we know more than any generation before us. But knowing more isn’t always the same as understanding more. Whether it’s a newspaper, a book, or a thoughtful article, reading asks us to slow down to think, to question, to reflect.

Perhaps that’s why, despite living in a world of instant updates and endless notifications, I still find myself looking forward to the quiet ritual of unfolding a newspaper every morning. Not because I believe the past was better. But because some habits don’t simply stay with us. They become part of who we are.

Thanks for reading.

Professor Dale A. Nance “argues that the best evidence principle is a fundamental principle of evidence law, requiring parties to present the best reasonably available evidence for resolving disputed factual issues…”1

Anywhere in the world, whenever a civil or criminal wrong is committed, courts seek the best available evidence to resolve the dispute. If I speak specifically about India, we have the fundamental principle known as the Rule of Best Evidence, also referred to as a Cardinal Principle of the Law of Evidence. Although the terminology and legal framework may differ, similar principles can be found in many other jurisdictions all over the world.

What is Best Evidence?

It means that when a fact is in dispute, the court should rely on the most reliable, authentic, and direct evidence that is reasonably available, rather than on inferior or secondary evidence.

The purpose of the law of evidence is to discover the truth by proving facts. The Rule of Best Evidence helps in finding and proving the truth. Out of the various forms of evidence available, the best evidence is always preferred.

The six foundational principles are as follows:

1. Relevant facts and facts in issue shall be duly proved to the satisfaction of the court.

2. The evidence to prove the relevant facts and facts in issue shall be admissible.

3. Hearsay evidence shall be excluded.

4. Documentary evidence excludes oral evidence.

5. Primary evidence excludes secondary evidence.

6. The court has special powers to extract the best evidence.

Let’s try to briefly understand the principles :

The first principle states that the relevant facts and facts in issue must be proved to the satisfaction of the court. A fact is relevant to another when it is connected with the other in any of the ways referred to in the provisions of the Act relating to the relevancy of facts. A fact in issue includes any fact from which, either by itself or in connection with other facts, the existence, non-existence, nature, or extent of any right, liability, or disability asserted or denied in any suit or proceeding necessarily follows.

The second principle provides that the evidence used to prove the relevant facts and facts in issue must be admissible. In other words, the facts sought to be proved must be legally admissible before the court.

The third principle excludes hearsay evidence. Hearsay evidence refers to evidence that the witness did not personally perceive but merely heard from another person. Peter Murphy defines “hearsay as evidence given by a witness consisting of another person’s statement made on a previous occasion regarding a fact which the witness himself or herself did not directly perceive.”2

The fourth and fifth principles provide that documentary evidence is preferred over oral evidence, and primary evidence is preferred over secondary evidence. It means the documented accounts would always be in priority to what is actually stated orally, when it comes to evidence, for example, a written contract. And similarly, the primary or the original source of a fact is preferred over a secondary source.

The sixth principle recognises that courts possess certain powers to extract the best available evidence. So this depends on various jurisdictions; however, to simply understand it, the courts can take different approaches as mentioned in law to extract the best evidence.

These principles are discussed here from the perspective of Indian law. However, the underlying ideas can also be found, in one form or another, in many other legal systems, as every legal system seeks reliable and trustworthy evidence.

“The AI Angle ^“

We often discuss fabrication and hallucinations in artificial intelligence and question whether AI-generated outputs can be trusted. Ultimately, what we require is evidence. However, it is not enough that something merely qualifies as evidence; it should be reliable, trustworthy, or, in other words, the best evidence.

Suppose a user inputs a prompt or asks a question to an AI system, and we hypothetically assume that the user is acting as the court. The prompt may be treated as the fact in issue, while the AI-generated response is the answer that requires scrutiny.

Applying the first principle, the relevant facts supporting the AI’s response should be capable of satisfying the user’s answers. In other words, if I ask an AI system a series of questions, it should be able to explain the basis of its answer so that I can be satisfied with its reasoning.

The second principle requires that the material relied upon by the AI should itself be admissible or, at the very least, derived from reliable and authoritative sources. If I question the AI further or, by analogy, cross-examine it, the facts, sources, or data on which it relies should be capable of withstanding such scrutiny.

From another perspective, when we initially ask questions to an AI system for our own understanding, it may be compared with an examination-in-chief, where the party calling the witness asks questions first. Once we begin questioning the AI’s reasoning, sources, assumptions, or conclusions, the process becomes analogous to a cross-examination.

The third principle, relating to hearsay, also becomes relevant. AI should not merely repeat unverified claims or information whose source cannot be identified. Wherever possible, its responses should be capable of being traced back to reliable and identifiable sources rather than unsupported assertions or fabricated information.

Similarly, the fourth and fifth principles suggest that AI-generated responses should, wherever possible, rely on documentary and primary sources, for example, statutes, judgments, official reports, or original research papers rather than summaries, secondary commentaries, or unsupported online claims.

Ultimately, what matters is not merely the existence of evidence, but the availability of the best evidence. Applying the principles of evidence law in this manner may provide a useful framework for evaluating the reliability, transparency, and trustworthiness of AI-generated outputs.

USER’S PROMPT

(Fact in Issue)

│

▼

AI ANALYSES RELEVANT FACTS

(Statutes • Cases • Data • Sources)

│

▼

APPLY EVIDENTIARY PRINCIPLES

┌──────────────────────────────────────┐

│ ✓ Relevant Facts │

│ ✓ Admissibility │

│ ✓ No Hearsay │

│ ✓ Documentary Evidence Preferred │

│ ✓ Primary Evidence Preferred │

│ ✓ Explainability / Cross-Examination │

└──────────────────────────────────────┘

│

▼

RELIABLE AI-GENERATED OUTPUT

(”Best Evidence” Approach)

Connecting the Framework with the EU AI Act

Interestingly, the proposed framework also finds support in the European Union’s AI Act. Although the AI Act does not expressly refer to the Rule of Best Evidence, many of its provisions pursue similar objectives. For instance, high-risk AI systems must be sufficiently transparent so that users can understand and appropriately interpret their outputs. Providers are also required to furnish clear information regarding the system’s capabilities, limitations, level of accuracy, intended purpose, and the measures necessary for effective human oversight.

Similarly, the AI Act emphasises technical documentation, logging, traceability, and human oversight as essential safeguards for trustworthy AI. These requirements enable users and regulators to verify how an AI system operates rather than merely accepting its output at face value.

Viewed from this perspective, the principles of evidence law and the objectives of the EU AI Act converge. Both seek to ensure that decisions are not based on blind acceptance but on reliable, transparent, and verifiable information. Just as courts insist upon the best available evidence before reaching a judicial determination, AI governance increasingly requires systems to produce outputs that can be explained, scrutinised, and supported by reliable sources. In this sense, the Rule of Best Evidence may serve not only as a principle of evidence law but also as a useful conceptual framework for evaluating the trustworthiness of AI-generated outputs in the age of algorithmic decision-making.

Thanks for reading.

Just finished recording a new episode of the Let’s Interact podcast with Ellie Harris, and it was one of those conversations that leaves you thinking long after the recording ends. We explored what more than 100 documented AI incidents reveal about trust, governance, accountability, and the future of responsible AI. Rather than discussing AI in theory, we focused on real cases, what organisations can learn from them, and why evidence matters more than confidence when it comes to AI systems. We also talked about Ellie’s open-source project, Headlights, and the incredible work she’s doing to document AI failures and make those lessons accessible to everyone. I thoroughly enjoyed this discussion and learned a great deal from it.

There’s a specific kind of prompt that makes a great AI benchmark: not a trick question, not a coding puzzle, but a genuinely useful, everyday request the kind a first-year law student, a paralegal, or a curious professional might actually type. So I ran one identical prompt through four of the major AI assistants and compared the results side by side

“You are a legal research assistant. Please explain Carlill v Carbolic Smoke Ball Co. (1893) in a structured manner under the following headings: Facts, Legal Issue, Decision of the Court, Legal Reasoning, Key Legal Principles Established, Significance of the Case Today.”

If you don’t know it, Carlill v Carbolic Smoke Ball Co1. is one of the most famous cases in English contract law a company advertised a £100 reward to anyone who used its “flu-curing” smoke ball as directed and got sick anyway, backed the claim by publicly depositing £1,000 in a bank, and then refused to pay when a Mrs. Carlill did exactly that and came down with influenza. It’s the case that gave us the modern doctrine of unilateral contracts. It’s dense enough to test real reasoning, but well-documented enough that I could actually judge whether each answer was any good.

I put the same prompt to Perplexity, Claude, Gemini, and ChatGPT, using their default web interfaces, and watched how each one handled the identical set of headings. Here’s what stood out.

^{Perplexity: fast, heavily cited, a little skeletal}

Perplexity’s answer arrived as a tight, bullet-driven list, and its defining feature was immediately obvious: almost every single claim carried an inline citation chip — little tags like wikipedia, quimbee, oxbridgenotes.co, scribd, and blog. ipleaders, often stacked as “+1” when multiple sources backed the same line. Under “Facts,” for instance, each bullet — the smoke ball’s marketing claims, the £100 reward, the £1,000 bank deposit, Mrs. Carlill’s illness — was traceable to a named source.

This is Perplexity’s whole personality: it’s built around search-and-cite, not generation from internal knowledge, and it shows. The upside is transparency — you can click through and verify literally every sentence. The downside is that the prose reads more like an annotated study-guide summary than a piece of original legal writing. The “Legal Reasoning” section, for example, was broken into sub-headings like (b) Unilateral offer and acceptance by performance and (c) Offer to the whole world, each just a few source-backed bullets rather than a connected argument. It’s the fastest way to get verifiable facts, but the thinnest on synthesis of the four.

^{Claude: the most fluid legal-writing style}

Claude’s response read the least like a list of facts and the most like something a junior associate might actually hand to a partner. The “Facts” section was a genuine paragraph introducing the smoke ball, the advertisement’s specific terms (three times daily for two weeks), the £1,000 deposit “to show our sincerity in the matter,” and Mrs Carlill’s claim rather than disconnected bullets.

Where Claude distinguished itself was the “Key Legal Principle Established” and reasoning sections, which didn’t just restate the holding but explained why it still matters: it explicitly drew the line from this 1893 case to modern reward advertisements, prize competitions, and even online “clickwrap” agreements, noting that courts still test whether “a reasonable person would understand the statement as demonstrating a serious intention to be bound.” That’s a genuinely useful bridge from historical doctrine to modern application, and it wasn’t just tacked onto the end — the connective tissue was woven through the whole answer.

^{Gemini: opens with a hook, and actually shows you the advertisement}

Gemini took the most editorial voice of the four right out of the gate, framing the case as “the quintessential English contract law case... it marks the boundary where marketing fluff transforms into a legally binding obligation.” It’s a small thing, but it signals Gemini is willing to editorialise a bit rather than stay purely neutral-encyclopedic.

The standout feature, though, was visual: Gemini pulled in an actual image of the original 1892 newspaper advertisement for the Carbolic Smoke Ball, reproducing the period typography and the full list of ailments it claimed to cure. None of the other three tools did this. For a case that’s fundamentally about an advertisement, seeing the real thing adds something the other three answers couldn’t. Gemini’s “Legal Reasoning” section was also the most argument-by-argument — walking through the company’s “offer to the whole world” defense, its “mere puff” defense, and its “no notice of acceptance” defense one at a time, rebutting each in turn, closer to how a court judgment itself is structured.

^{ChatGPT: concise, and the only one with a built-in “so what”}

ChatGPT’s answer was the leanest of the four in the middle sections the facts and reasoning were compressed into short, punchy bullets rather than full paragraphs. But it added something none of the others did: a distinct, clearly separated “Quick Takeaway” box at the very end, distilling the entire case into two sentences — that a clear public promise, demonstrated seriousness, and specified conditions can create a binding unilateral contract, accepted through performance rather than negotiation.

That closing summary is a small design choice, but it matters for usability. If a student only reads one paragraph of the whole response, ChatGPT’s answer guarantees that paragraph is the “Quick Takeaway,” while the other three tools bury their most useful synthesis inside longer sections.

^{So which one actually did it best?}

Depends what you want it for:

• Need citations you can verify without doing your own research? Perplexity’s inline sourcing is unmatched — every claim is traceable.

• Need something you could actually hand to someone as a finished explainer? Claude’s prose reads the most like a real piece of legal writing, and it made the strongest effort to connect 1893 doctrine to modern situations.

• Want to see the historical context, not just read about it? Gemini’s inclusion of the actual 1892 advertisement was a genuinely nice touch that added real value.

• Just want the one-paragraph version to remember? ChatGPT’s “Quick Takeaway” does exactly that job, and does it better than the others’ scattered conclusions.

None of the four got the substantive law wrong; the core holding (unilateral offer, acceptance by performance, valid consideration through inconvenience and commercial benefit) came through correctly in all four. The real differentiator wasn’t accuracy; it was presentation how each tool chose to structure, cite, and finish the explanation. Which, if you’re choosing a tool for actual research or writing work, might matter just as much as being right.

What made this conversation particularly interesting was that it wasn’t really about AI alone. It was about trust.

We explored questions such as:

▪️ Why predictive accuracy alone isn’t enough
▪️ What “justified confidence” means in practice
▪️ How systems can quietly drift while still appearing to work
▪️ Why evidence, testing, and accountability matter more than ever in an age of automation
▪️ The difference between automation that is merely efficient and automation that is genuinely trustworthy

As someone coming from a legal background, I found myself noticing how many of these challenges mirror familiar legal concepts: evidence, cross-examination, accountability, and standards of proof.

We also spoke about navigating complex work while managing chronic health challenges, and how thoughtful use of AI can help people focus their energy where human judgment matters most.

One idea that stayed with me is that as AI-generated answers become increasingly abundant, the real challenge is no longer producing predictions it’s knowing when those predictions deserve our trust.

A thoughtful conversation that left me with far more questions than answers, which is usually a sign of a good one.

Artificial Intelligence has transformed the way we create and consume content. Today, AI can generate realistic images, clone voices, create digital avatars, and produce convincing deepfake videos within minutes. While these innovations have opened new avenues for creativity, they have also created an unprecedented legal challenge: the misuse of an individual’s identity.

Imagine discovering that your face is being used to advertise a product you have never endorsed, or hearing an AI-generated version of your voice promoting a political opinion you never expressed. These are no longer hypothetical situations; they are becoming increasingly common across the world.

This is where personality rights come into play.

What Are Personality Rights?

Personality rights refer to an individual’s right to control the commercial and unauthorised use of their identity. These rights generally protect a person’s:

• Name

• Photograph

• Face

• Voice

• Signature

• Likeness

• Image

• Distinctive mannerisms or catchphrases

Although these rights have traditionally been associated with celebrities, athletes, and public figures, AI has demonstrated that every individual’s identity can now be copied, manipulated, and commercially exploited.

The rise of deepfakes has therefore shifted the conversation from celebrity protection to digital identity protection.

How Different Countries Protect Personality Rights

There is no universal law governing personality rights. Different jurisdictions have adopted different legal approaches.

United States

The United States protects personality rights through the Right of Publicity, which allows individuals to control the commercial use of their identity.

However, these rights vary from state to state. California, New York, Tennessee, and Indiana have some of the strongest protections. Certain states also recognise post-mortem publicity rights, allowing the estates of celebrities such as Elvis Presley to continue protecting and licensing their image after death.

With the rapid growth of AI-generated voice cloning and deepfakes, lawmakers have also proposed new legislation, such as the NO FAKES Act, to regulate unauthorised digital replicas and synthetic media.

The Scarlett Johansson and OpenAI Controversy

One of the most widely discussed AI-related personality rights disputes emerged in 2024 when actor Scarlett Johansson alleged that OpenAI’s voice assistant “Sky” closely resembled her voice. Johansson stated that she had previously declined a request to license her voice and expressed concern that the AI-generated voice could lead the public to believe she had endorsed the product.1

Although the dispute did not immediately result in litigation, it highlighted a growing legal question: Can AI-generated voices infringe personality rights even when no actual recording of the individual is used?

The controversy demonstrated that a person’s voice may possess economic and personal value deserving legal protection in the AI era.

Taylor Swift and the Deepfake Crisis

In early 2024, AI-generated explicit images falsely depicting Taylor Swift circulated widely online, reaching millions of users within hours. The incident sparked international outrage and renewed calls for stronger regulation of deepfakes and synthetic media.2

Unlike traditional copyright disputes, the issue centred on identity itself. The images were not unauthorised copies of existing photographs; they were entirely new creations generated through AI. Yet they exploited Swift’s likeness and reputation.

The controversy illustrated how personality rights are increasingly becoming a tool for combating AI-generated identity misuse.

AI Voice Cloning and the Music Industry

The music industry has also become a battleground for personality rights. The viral AI-generated song Heart on My Sleeve replicated the voices of Drake and The Weeknd without authorisation, raising concerns about whether AI can commercially exploit an artist’s vocal identity.3

The incident demonstrated that while copyright law may protect songs and recordings, personality rights may be necessary to protect the unique characteristics that make a performer recognisable.

United Kingdom

Unlike the United States, the United Kingdom does not recognise an independent personality right.

Instead, individuals rely on a combination of:

• Passing off

• Privacy law

• Data protection

• Copyright

• Trademark law

While these legal remedies provide some protection, many scholars argue that they were never designed to address AI-generated identity replication, leaving important gaps in the law.

Germany and France

Germany and France provide some of the strongest personality protections in Europe.

German law recognises the General Right of Personality, rooted in constitutional principles of human dignity and personal autonomy. Similarly, French law protects an individual’s image and private life through the French Civil Code.

These jurisdictions treat personality rights not merely as commercial interests but as extensions of human dignity.

China

China has adopted one of the most comprehensive statutory approaches.

The Chinese Civil Code expressly protects personality rights, including a person’s:

• Name

• Image

• Voice

• Reputation

• Privacy

• Personal information

China has also introduced regulations governing AI-generated and synthetic content, requiring greater accountability for technologies capable of producing deepfakes and digital impersonations.

The Indian Position

Unlike several foreign jurisdictions, India does not have a dedicated Personality Rights Act.

Instead, personality rights have evolved primarily through constitutional principles and judicial interpretation.

Indian courts have relied on:

• Article 21 of the Constitution, which guarantees the right to life, privacy, dignity, and personal autonomy;

• The common law doctrine of passing off;

• Principles of unfair commercial exploitation;

• Elements of intellectual property and privacy law.

Although there is no single statutory framework, Indian courts have gradually developed a robust body of jurisprudence protecting an individual’s identity.

In the case of actor Mr. Amitabh Bachchan v. Rajat Negi & Ors., the Court issued an order restraining the defendant, including unknown defendants, from infringing Mr. Bachchan’s personality rights or celebrity rights by misusing his name, likeness, photograph, voice and other personality traits and attributes, for commercial gain.

A similar case was filed by Bollywood actor, Mr. Anil Kapoor. Titled as Anil Kapoor v. Simply Life India & Ors., this was filed against 21 defendants for unauthorised usage of his popular dialogue “Jhakas”. According to the plaintiff, each of the 21 defendants were using different aspects of his personality. The Court recognized the threat of exploitation using technology and upheld his personal rights as a celebrity.4

In addition to these principles, the Digital Personal Data Protection Act, 2023 (DPDP Act) has become increasingly relevant in the age of AI.

The DPDP Act does not create personality rights in the traditional sense, but it does strengthen protection over personal data, which may include photographs, voice recordings, biometric identifiers, and other information used to train or generate AI content. Since the Act is built around consent, lawful processing, purpose limitation, and accountability, it can offer an important layer of protection where a person’s image, voice, or other identifying data is collected or used without authorisation.

However, the DPDP Act is not a complete answer to deepfakes or digital impersonation. It focuses on data protection, not on the broader commercial and dignitary harm caused by misuse of identity. That is why personality rights and data protection must be seen as complementary, not interchangeable.

Although there is no single statutory framework for personality rights in India, Indian courts have gradually developed a robust body of jurisprudence protecting an individual’s identity.

Why AI Has Changed the Conversation

Earlier, personality rights disputes mainly involved unauthorised advertisements, merchandise, or endorsements.

Today, AI has fundamentally altered the nature of these disputes. A few publicly available photographs can generate realistic images. A few seconds of recorded speech can clone a person’s voice.

AI can recreate facial expressions, mannerisms, and even produce convincing videos of individuals saying things they never said.

This creates legal questions that traditional laws never anticipated:

• Who owns a person’s digital identity?

• Should consent be mandatory before creating AI replicas?

• Can a deceased celebrity be digitally recreated?

• Should ordinary individuals receive the same protection as celebrities?

• How should courts balance innovation with privacy and dignity?

The Road Ahead

Artificial Intelligence has blurred the line between reality and fabrication.

Personality rights are no longer merely about protecting celebrities from unauthorised endorsements. They are increasingly about safeguarding identity itself.

While countries such as the United States, Germany, France, and China have adopted different legal approaches, India continues to rely largely on judicial innovation rather than comprehensive legislation.

As deepfakes, voice cloning, and AI-generated digital replicas become more sophisticated, the need for a dedicated legal framework in India becomes increasingly urgent.

The debate is no longer simply about technology. It is about consent and dignity.

And ultimately, it is about a fundamental question every legal system must answer:

In an age where artificial intelligence can replicate almost everything about a person, who truly owns a human identity?

Artificial Intelligence has become one of the defining technologies of the twenty-first century. It writes emails, summarises documents, creates images, assists doctors in diagnosis, supports legal research, and powers everything from recommendation systems to self-driving vehicles. Discussions surrounding AI often focus on its capabilities, risks, economic impact, and effect on employment. However, a less visible but equally important issue remains largely absent from public conversations: AI’s growing demand for water.

When people interact with AI systems, the experience feels entirely digital. A prompt is entered, a response appears, and the interaction ends. What remains hidden is the enormous physical infrastructure operating behind the scenes. Every AI response depends on vast networks of servers housed in data centres across the world. These facilities consume substantial amounts of electricity and water, creating an environmental footprint that many users never see.

Why Does AI Need Water?

At first glance, it may seem strange that a technology based on software would require water. The answer lies in the hardware that powers AI systems.

Modern AI models are trained and operated using thousands of specialised processors and servers. These machines perform trillions of calculations and generate significant amounts of heat. Without effective cooling systems, servers would overheat, reducing performance and potentially causing damage.

Water is one of the most efficient cooling mechanisms available. Many data centres use water-based cooling systems to maintain safe operating temperatures. As a result, the larger and more powerful AI systems become, the greater their water requirements.

In simple terms, every AI-generated response is supported by physical machines that must remain cool, and water plays a crucial role in making that possible.

The Water Cost of Training AI Models

The environmental impact of AI becomes particularly apparent during the training phase.

Training a large language model involves exposing it to vast amounts of data and allowing it to learn patterns through repeated computation. This process can take weeks or even months and requires enormous computational resources.

Researchers from the University of California, Riverside estimated that training GPT-3 consumed approximately 700,000 litres of freshwater for cooling purposes.

To understand the scale, imagine:

More than 350,000 two-litre water bottles.

Enough drinking water for hundreds of people for an entire year.

A volume capable of filling numerous residential swimming pools.

This figure represents only the direct water consumption associated with cooling the servers involved in training. It does not include additional water used indirectly through electricity production.

As AI models continue to grow larger and more complex, the resources required to train them are expected to increase accordingly.

The Water Behind Everyday AI Use

Many people assume that most environmental costs occur during training and that everyday usage has minimal impact. While training is indeed resource-intensive, routine interactions also contribute to water consumption.

The same body of research suggests that a conversation involving approximately 20 to 50 prompts may consume around 500 millilitres of water, depending on factors such as data centre location, weather conditions, and cooling technology.

Half a litre may not sound significant. However, consider the scale of modern AI adoption:

Millions of users interact with AI systems daily.

Businesses increasingly integrate AI into their workflows.

Educational institutions, governments, and healthcare providers are adopting AI-powered tools.

AI-generated content continues to grow across social media and digital platforms.

When multiplied across billions of interactions, even small amounts of water per session become substantial.

The cumulative impact is what matters.

The Hidden Cost: Electricity Also Consumes Water. Cooling systems are only part of the equation.

AI models require large amounts of electricity to function. Generating electricity often requires water as well, particularly in thermal power plants where water is used for cooling and steam production.

This means AI’s total water footprint consists of two components:

1. Direct water consumption for cooling data centres.

2. Indirect water consumption associated with electricity generation.

Therefore, when evaluating AI’s environmental impact, focusing solely on cooling water provides only a partial picture. The broader water footprint is significantly larger.

Researchers often refer to this combined effect as the total water footprint of AI.

The Growing Global Demand

Perhaps the most concerning aspect is the projected growth in AI-related water consumption.

Research indicates that global AI demand could account for approximately 4.2 to 6.6 billion cubic meters of water withdrawal annually by 2027.

To appreciate this scale:

It exceeds the annual water withdrawal of several nations.

It represents a significant burden on already stressed freshwater resources.

It comes at a time when climate change is intensifying droughts and water scarcity in many regions.

Freshwater is a finite resource. According to international environmental organisations, water scarcity is already affecting billions of people worldwide. The rapid expansion of AI raises important questions about balancing technological progress with environmental sustainability.

Evidence from Industry: Microsoft’s Water Consumption

The relationship between AI growth and water usage is not merely theoretical.

Microsoft reported a 34 per cent increase in water consumption in 2022 compared to the previous year. The company linked this increase to the expansion of its cloud infrastructure and AI operations.

This statistic is significant for several reasons.

First, it demonstrates that AI’s environmental footprint is already visible in corporate sustainability reports.

Second, it highlights the growing resource demands associated with large-scale AI deployment.

Third, it suggests that water consumption may become an increasingly important factor in future discussions about technology infrastructure.

As companies continue investing billions of dollars into AI development, understanding and managing resource consumption will become essential.

Why This Matters

Some may argue that water usage is simply the cost of technological advancement. However, the issue deserves attention for several reasons.

1. Water Scarcity Is Already a Global Problem

Many regions around the world face water shortages. Communities struggle with access to clean drinking water, agricultural demands, and industrial consumption.

When large technology infrastructures consume significant quantities of freshwater, questions of allocation and sustainability become increasingly important.

2. AI Adoption Is Accelerating

Unlike previous technological innovations, AI is spreading rapidly across nearly every sector of society.

The more AI becomes integrated into daily life, the greater its cumulative environmental impact.

3. Environmental Costs Are Often Invisible

Users typically see AI as a virtual service. Because the infrastructure remains hidden, the associated environmental costs often go unnoticed.

Greater awareness can encourage more informed discussions about responsible technology development.

Can AI Become More Sustainable?

The good news is that companies and researchers are actively exploring solutions.

Improved Cooling Technologies

Many data centres are experimenting with:

Closed-loop cooling systems

Recycled water usage

Advanced liquid cooling technologies

More efficient heat management systems

These approaches aim to reduce freshwater consumption while maintaining performance.

Renewable Energy

Technology companies are increasingly investing in renewable energy sources such as solar and wind power.

Although renewable energy does not eliminate water usage entirely, it can reduce dependence on water-intensive electricity generation methods.

More Efficient AI Models

Researchers are also focusing on creating AI systems that require less computational power.

Smaller, optimised models can deliver strong performance while consuming fewer resources, including electricity and water.

Strategic Data Centre Locations

Some organisations are building data centres in cooler climates where natural environmental conditions reduce cooling requirements.

This can significantly lower water consumption and energy demands.

Balancing Innovation and Responsibility

Artificial Intelligence offers extraordinary opportunities. It has the potential to accelerate scientific discovery, improve healthcare, enhance education, support legal professionals, and solve complex global challenges.

At the same time, every technological revolution carries costs.

The environmental impact of AI does not mean society should abandon innovation. Rather, it highlights the importance of developing technology responsibly. Understanding AI’s water footprint allows policymakers, researchers, companies, and users to make more informed decisions about sustainability.

The future of AI should not be measured solely by how intelligent these systems become. It should also be measured by how efficiently and responsibly they use the world’s resources.

Conclusion

The next time an AI model answers a question in seconds, generates an image, or assists with a complex task, it is worth remembering that the process is not purely digital.

Behind every response are data centres, servers, cooling systems, electricity networks, and significant quantities of water.

AI may exist in the virtual world, but its environmental footprint exists in the real one.

As artificial intelligence continues to transform society, sustainability must become part of the conversation. The challenge ahead is not merely building more powerful AI systems; it is building AI systems that are powerful, efficient, and environmentally responsible.

Only then can technological progress and environmental stewardship move forward together.

When we interact with AI, it’s easy to think of it as something purely digital.

A prompt goes in. A response comes out.

What remains largely invisible is the infrastructure behind that exchange: data centres, cooling systems, electricity generation, and natural resources.

One resource receiving increasing attention is water.

A 2023 study by researchers at the University of California, Riverside estimated that training GPT-3 could directly consume approximately 700,000 litres of freshwater for cooling. The study also suggested that routine interactions with large language models can contribute to a measurable water footprint.

The issue becomes more significant when viewed at scale. As cloud computing and AI infrastructure expand, so does demand for cooling and energy. Microsoft reported a 34% increase in water consumption in 2022 while expanding its cloud and AI operations.

The debate is no longer solely about technology.

It is increasingly about governance, sustainability, and law.

Questions surrounding environmental disclosures, resource allocation, and infrastructure regulation may become an important part of future AI policy discussions.

AI may be digital, but the resources supporting it are very real.

I recently explored this topic in a short video examining the intersection of AI, water consumption, and emerging legal challenges.

What role do you think environmental considerations should play in AI regulation?

AI can now replicate voices, generate images, imitate artistic styles, and even recreate aspects of a person's identity. As these technologies become more powerful, a fundamental legal question is emerging: who owns what AI creates?

From Sarah Andersen v. Stability AI to growing disputes over trademarks, likeness rights, and digital identities, courts and regulators are being forced to rethink long-standing legal principles. The debate is no longer limited to copyright. It now extends to voices, faces, personal brands, and the boundaries of human identity itself.

As AI continues to blur the line between creation and replication, copyright, trademark, and personality rights are being rewritten in real time. The challenge for lawmakers is no longer whether AI can imitate humans it already can. The challenge is determining who owns, controls, and protects what is being imitated..

For the longest time, most of us interacted with technology in a very direct way.

We typed a search query into Google and got links. We used Google Translate and got one language converted into another. We filled forms, clicked buttons, and expected software to follow fixed instructions.

Then came tools like ChatGPT.

Suddenly, software did not just retrieve information. It started generating answers, drafting emails, summarizing documents, writing code, explaining concepts, and even responding in a conversational tone.

That shift feels almost magical from the outside. But underneath, Large Language Models, or LLMs, are not thinking like humans. They are using mathematics, patterns, probabilities, and huge amounts of training data to predict what should come next.

And that is exactly why they are both powerful and legally risky.

What Does GPT Actually Mean?

ChatGPT is built on a type of model called a GPT, which stands for Generative Pre-trained Transformer.

Each word matters.

Term Meaning Generative: It can generate new text, code, summaries, explanations, and other outputs. Pre-trained: It has already learned patterns from large amounts of data before we interact with it. Transformer: It uses a neural network architecture that is especially good at understanding context in language.

The important point is this: an LLM does not simply copy and paste answers from a database. It generates responses by predicting likely sequences of words based on patterns it has learned.

That is also where the legal concern begins.

If a model generates text that sounds authoritative, users may assume it is correct. But fluent language is not the same thing as legally reliable advice.

The Core Idea: Predicting the Next Token

At a basic level, an LLM works by predicting the next token.

A token can be:

• A word

• Part of a word

• A punctuation mark

• Sometimes even a small unit of code or symbol

For example, the sentence:

The contract was signed by the...

The model may predict that the next word could be parties, company, director, or client, depending on the context.

It does this again and again, one token at a time, until a full response is formed.

This is why LLMs are good at producing natural-sounding text. They have learned which words usually appear together, how sentences are structured, and how ideas are commonly expressed.

But there is a catch: predicting the next likely word is not the same as verifying the truth.

In law, that difference matters a lot.

A model may generate a confident-looking answer about a statute, case law, contract clause, or compliance requirement. But unless the answer is checked against an authoritative legal source, it may be incomplete, outdated, jurisdiction-specific, or simply wrong.

Step One: Tokenization

Computers do not understand language the way humans do. They work with numbers.

So before an LLM can process text, it breaks the input into smaller units called tokens. This process is called tokenization.

In some cases, a word may be split into smaller parts. This depends on the model’s vocabulary. The model has a predefined set of tokens it understands. This is called its vocabulary. Once the text is tokenized, the model can convert those tokens into numbers and begin processing them.

Legal angle

Tokenization may sound like a purely technical step, but it has legal consequences.

Legal language is often precise. Small differences in wording can change meaning. For example:

Phrase Possible Legal Effect

• shall pay Usually indicates an obligation.

• may pay Usually indicates discretion or permission.

• best efforts May create a stronger duty depending on jurisdiction and context.

• reasonable efforts May create a different standard of performance.

If a model fails to understand the importance of these terms properly, the output can become risky. A small linguistic shift can affect contractual rights, liabilities, or compliance duties.

Step Two: Vector Embeddings

After tokenization, the model converts tokens into vector embeddings.

A vector embedding is a numerical representation of meaning. In simple words, it is how the model places words, phrases, or concepts into a mathematical space.

Words with related meanings are placed closer together.

For example:

Concept Group Related Words

• Animals: cat, dog, horse

• Food: milk, bread, fruit

• Law: contract, statute, judgment

• Business: company, invoice, shareholder

This helps the model understand relationships between concepts.

For example, the model may learn that:

• contract is related to agreement, obligation, breach, and consideration.

• court is related to judge, judgment, appeal, and proceedings.

• company is related to directors, shareholders, incorporation, and compliance.

This is one reason LLMs can produce surprisingly relevant answers. They are not just seeing isolated words; they are processing relationships between words.

Legal angle

Vector embeddings help LLMs connect legal concepts, but they can also create problems.

Legal concepts are not always interchangeable. For example:

Term Why It Matters

• Agreement May be broader and informal.

• Contract Usually implies legally enforceable obligations.

• MOU May or may not be binding depending on wording and intention.

• Deed Has specific legal formalities in some jurisdictions.

An LLM may understand that these words are related, but related does not always mean legally equivalent.

That is why legal professionals should be careful when using LLMs for drafting or interpretation. The model may produce text that is semantically close but legally different.

Step Three: Positional Encoding

Language depends not only on words, but also on word order.

Consider these two sentences:

The company sued the supplier.

The supplier sued the company.

The words are almost the same. But the legal meaning is completely different.

In the first sentence, the company is the claimant. In the second sentence, the supplier is the claimant.

This is where positional encoding comes in.

After words are converted into embeddings, the model also needs information about where each token appears in the sentence. Positional encoding helps the model understand sequence and structure.

Without it, the model may know the words but not the relationship between them.

Legal angle

Word order is extremely important in legal documents.

For example:

The tenant shall not assign the lease without the landlord’s written consent.

Now compare:

The landlord shall not assign the lease without the tenant’s written consent.

The structure changes the obligation entirely.

In contracts, pleadings, notices, statutes, and judgments, order and placement matter. A misplaced phrase can change who has a duty, who has a right, and who may be liable.

LLMs use positional encoding to understand these relationships, but users should still review legal outputs carefully because grammar-level understanding is not the same as legal accountability.

Step Four: Self-Attention

One of the most important ideas behind transformers is self-attention.

Self-attention allows the model to look at different words in a sentence and decide which words are most relevant to each other.

For example:

The director told the lawyer that he would review the agreement.

Who does he refer to?

• The director?

• The lawyer?

A human would look at context. An LLM does something similar mathematically. It compares tokens with other tokens and assigns attention scores to understand relationships.

Self-attention allows words to “communicate” with each other inside the model. Each token updates its meaning based on the surrounding context.

This is what makes transformer models much better than older models that processed text mostly in sequence.

Legal angle

Self-attention is particularly useful for legal documents because legal meaning often depends on context.

Consider this clause:

The seller shall deliver the goods within 30 days of receiving payment, unless delayed by events beyond its reasonable control.

To understand this sentence, the model must connect:

• seller with the delivery obligation

• 30 days with the timeline

• receiving payment with the trigger event

• unless delayed with the exception

• reasonable control with the limitation

That is a lot of context inside one sentence.

In legal drafting, obligations, exceptions, conditions, definitions, and timelines are often spread across different sections. Self-attention helps LLMs identify relationships, but it does not guarantee correct legal interpretation.

Step Five: Multi-Head Attention

Self-attention becomes even more powerful through multi-head attention.

Instead of looking at a sentence from only one angle, the model uses multiple attention heads. Each head can focus on different types of relationships.

One head may focus on grammar. Another may focus on subject-object relationships. Another may focus on definitions. Another may focus on long-distance context.

For example, in a contract, different attention heads may help the model connect:

Attention Focus Example Parties Company, supplier, contractor, employee Obligations shall, must, required to, responsible for Exceptions unless, except, subject to, provided that Time limits within 30 days, before termination, after notice Defined terms Agreement, Services, Confidential Information

This is one reason LLMs can summarize long documents, identify clauses, and answer questions about dense text.

Legal angle

Multi-head attention is useful for legal review, especially where a document has many interconnected parts.

For example, a confidentiality clause may refer to:

• A definition section

• A permitted disclosure clause

• A survival clause

• A remedies clause

• A governing law clause

An LLM may help identify these connections faster than manual review. But the final interpretation still requires legal judgment.

A model can assist with legal work, but it should not replace legal responsibility.

Step Six: Inference Mode

Once a model has been trained, users interact with it in inference mode.

This means the model is no longer learning from scratch. It is using what it has already learned to generate an answer to the user’s prompt.

When we ask a question, the model:

1. Breaks the prompt into tokens.

2. Converts the tokens into embeddings.

3. Adds positional information.

4. Uses attention mechanisms to understand context.

5. Predicts the next token.

6. Repeats the process until it completes the answer.

The output feels instant and conversational, but behind the scenes, it is a chain of probability-based predictions.

Legal angle

Inference mode creates a practical risk: the answer may sound final, even when it is only probabilistic.

For example, if a user asks:

Is this non-compete clause enforceable?

The model may produce a confident answer. But enforceability depends on many factors, including:

• Jurisdiction

• Applicable statute

• Case law

• Employee role

• Duration of restriction

• Geographic scope

• Public policy

• Recent regulatory changes

A general-purpose LLM may not know the latest law or may not apply it correctly. That is why legal answers need verification, especially where rights, duties, penalties, or litigation risks are involved.

Why Legal Professionals Should Understand LLMs

LLMs are already being used for:

Use Case Possible Benefit Contract review Faster clause identification and risk spotting. Legal research Quick summaries and starting points. Drafting First drafts of notices, emails, policies, and agreements. Due diligence Reviewing large volumes of documents. Compliance Mapping obligations and identifying red flags. Litigation support Summarizing facts, timelines, and pleadings.

But legal professionals should understand the technology because the risks are not theoretical.

LLMs can:

• Generate non-existent cases or citations.

• Misstate legal principles.

• Ignore jurisdictional differences.

• Oversimplify complex legal questions.

• Produce biased or incomplete reasoning.

• Reveal confidential information if used carelessly.

• Create overreliance because the output sounds polished.

The main danger is not that LLMs are useless. The danger is that they are useful enough to be trusted too quickly.

Hallucination: The Legal Problem

A hallucination happens when an LLM generates information that appears real but is false or unsupported.

In everyday writing, this may be embarrassing.

In law, it can be serious.

Imagine an AI tool generating:

• A fake case citation

• A wrong limitation period

• An outdated compliance requirement

• A clause that is unenforceable

• A misleading summary of a judgment

• An incorrect explanation of a party’s rights

The harm may not stop at bad information. It can affect decisions, negotiations, court filings, client advice, and regulatory compliance.

Legal hallucinations are especially difficult because not every legal question has one simple answer. Some questions depend on interpretation, jurisdiction, facts, and judicial discretion.

So the issue is not only whether the model is “right” or “wrong.” Sometimes the issue is whether the model has shown the uncertainty, assumptions, and legal basis behind the answer.

LLMs as Legal Assistants, Not Legal Authorities

The safest way to think about LLMs in law is this:

Use them as assistants, not authorities.

They can help with:

• Explaining difficult concepts

• Creating first drafts

• Summarizing long documents

• Generating issue lists

• Comparing clauses

• Preparing research questions

• Organizing facts chronologically

But they should not be blindly trusted for:

• Final legal advice

• Court filings

• Case citations

• Statutory interpretation

• Compliance decisions

• High-stakes contract review

• Anything involving rights, duties, penalties, or liability

An LLM can speed up legal work, but it cannot take professional responsibility for the result.

The Future: Legal AI Needs Better Grounding

For LLMs to become more reliable in legal settings, they need stronger grounding in authoritative sources.

That means legal AI tools should ideally be connected to:

• Updated statutes

• Reported judgments

• Regulations

• Government notifications

• Contract databases

• Legal knowledge graphs

• Jurisdiction-specific legal materials

This is where techniques like retrieval-augmented generation, or RAG, become important.

Instead of relying only on what the model learned during training, a RAG-based system can retrieve relevant legal materials and then generate an answer based on those sources.

For lawyers, this matters because a useful legal AI system should not only answer. It should also show the legal basis for the answer.

In law, the source is often as important as the conclusion.

Final Thoughts

Large Language Models are not magic. They are systems that process tokens, convert them into mathematical representations, understand context through attention, and generate outputs by predicting what comes next.

That is an incredible technological achievement.

But in legal work, fluency can be dangerous. A beautifully written answer may still be legally wrong. A confident explanation may still miss an exception. A contract clause may sound professional but fail in court.

The real value of LLMs is not in replacing lawyers, researchers, or compliance professionals. Their value lies in helping humans work faster, think more clearly, and handle information more efficiently.

The future of legal AI will belong not to people who blindly trust these tools, but to those who understand how they work, where they fail, and how to use them responsibly.

Sources :

• Piyush Garg , Educational Content on Large Language Models.

• Andreas Stöffelbauer , How Large Language Models Work: From Zero to ChatGPT (Medium).

• Large Language Models in Legal Systems: A Survey (Nature).

• Large Language Models Explained! How LLMs Work for Beginners! (YouTube).

• On the Legal Implications of Large Language Model Answers (ScienceDirect).

• Eliza Mik, Legal Hallucinations? The Current Fascination with LLMs (Medium).

• Shifting Cybersecurity: The Impact and Implications of LLMs (GovInfoSecurity).

• How Transformer LLMs Work (DeepLearning.AI).

• AI Demystified: Introduction to Large Language Models (Stanford University IT).

In the latest episode of Let’s Interact, I sat down with to discuss what it means to be a modern professional in a world where technology is reshaping industries faster than ever before.

We explored questions around legal practice, emerging technologies, entrepreneurial thinking, and the skills professionals need to remain relevant in an AI-driven future.

One of the themes that stayed with me after the conversation was that careers are no longer confined to a single path. The most interesting opportunities often emerge at the intersection of disciplines.

I hope you enjoy the conversation as much as I enjoyed recording it.

Watch the full episode below and let me know your thoughts.

I have always been interested in human psychology, although I do not claim any expertise in the field. What fascinates me, however, is the relationship between the human mind and the growing role of artificial intelligence in our daily lives.

Over the past few years, AI has become deeply integrated into the way we work, learn, communicate, and even think. This, in itself, is not a problem. Every generation has embraced tools that made life more efficient. Calculators reduced the burden of arithmetic, search engines transformed access to information, and smartphones compressed an entire digital ecosystem into our pockets. AI is simply the latest step in that evolution.

Yet something feels different.

The concern is not that AI performs tasks for us. The concern is that we are increasingly delegating the mental processes that once accompanied those tasks. We no longer merely seek information; we seek immediate conclusions. We no longer spend as much time questioning, exploring, comparing, or reflecting. Instead, a growing portion of our intellectual journey is compressed into a simple sequence:

Prompt → Output → Delivery.

A question is asked, an answer is generated, and that answer is presented to the world. The stage in between—the stage of contemplation, doubt, analysis, and independent reasoning—is gradually shrinking.

Patience, one of the most important cognitive virtues, appears to be under pressure. We have become accustomed to receiving polished responses within seconds. Waiting, researching, revising, and struggling through uncertainty increasingly feel inefficient. Yet these very processes are often responsible for intellectual growth.

The issue becomes particularly significant when viewed through the lens of younger generations. Those who have already developed critical thinking skills may be better positioned to use AI as an aid rather than a substitute. However, individuals whose intellectual habits are still being formed may never experience the same level of cognitive friction that previous generations encountered.

And cognitive friction matters.

Psychologists often describe learning as a process strengthened through effort. Memory formation, analytical reasoning, and problem-solving are not merely outcomes; they are skills developed through repeated mental engagement. When technology consistently removes that engagement, we must ask whether efficiency is coming at the cost of intellectual resilience.

This phenomenon can already be observed in subtle ways. Students generate project reports with a few prompts. Professionals draft emails, presentations, and research notes through automated systems. Increasingly, even personal communication is being mediated through AI-generated language. Many people hesitate before writing a simple message and seek technological assistance—not because they lack ideas, but because they fear imperfection.

The desire for flawlessness is understandable. AI offers reassurance. It provides language that is polished, structured, and socially acceptable. Yet there is a risk in becoming uncomfortable with our own imperfect thoughts. Human communication has always involved uncertainty, revision, and occasional mistakes. Those imperfections are not defects; they are evidence of authentic thinking.

Perhaps the most interesting analogy comes from popular fiction. In Harry Potter and the Chamber of Secrets, Tom Riddle’s diary appears to be a harmless object. It listens, responds, understands, and provides companionship. Over time, however, the user begins to invest more trust in the diary than in their own judgment. While AI is certainly not a dark magical artifact, the comparison highlights an important psychological question: when a tool becomes increasingly capable of answering our questions, do we gradually lose the habit of asking them ourselves?

The future challenge, therefore, is not whether AI should exist. That debate is already over. AI is here, and it will become more sophisticated with time.

The real challenge is determining where the boundary should lie.

Calls to ban technology, prohibit social media, or restrict internet access are unlikely to solve the problem. The issue extends beyond any single platform. Even if one source of digital content disappears, countless other systems can still provide automated answers, automated reasoning, and automated creativity.

The boundary, therefore, cannot be technological alone. It must be cognitive.

AI should assist thought, not replace it.

A law student should still learn how to construct an argument before asking AI to refine it. A doctor should understand medical reasoning before relying on algorithmic recommendations. A lawyer should know how to analyse a case before requesting a summary. In every profession, AI should function as an amplifier of expertise rather than a substitute for expertise.

If we fail to establish this distinction, society may gradually divide into two groups: those who use AI to extend their knowledge and those who use AI in place of acquiring knowledge. The difference between the two may become one of the defining challenges of the coming decades.

The question before us is therefore not whether artificial intelligence is intelligent enough. The more important question is whether, in our pursuit of convenience, we are allowing ourselves to become less intellectually engaged.

Technology should accelerate human potential, not diminish the need for it.

Artificial intelligence has undoubtedly transformed the way we work. In the legal profession, AI tools can summarise lengthy judgments, assist with legal research, identify patterns across documents, draft preliminary analyses, and significantly reduce the time spent on repetitive tasks. These developments are exciting, particularly in a field where efficiency, accuracy, and accessibility are becoming increasingly important.

Yet behind these advantages lies a challenge that continues to attract the attention of researchers, lawyers, regulators, and policymakers: the black box problem.

To understand the black box problem, imagine asking an AI system a legal question. Within seconds, it produces a detailed answer, cites principles, and presents a conclusion. We can see the question that was asked, and we can see the answer that was provided. What we often cannot see is the complete reasoning process that occurred in between. This hidden space between the input and the output is what many scholars refer to as the "black box."

Unlike traditional computer programs, where every rule is explicitly programmed and traceable, modern AI systems learn from enormous amounts of data and develop complex internal relationships that even their creators may struggle to interpret fully. The system can identify patterns and generate outputs, but the precise path that led to a particular conclusion may not always be visible or understandable to users.

In simple terms, we know what went in and what came out, but we do not always know why the system arrived at that specific answer. At first glance, this may appear to be a purely technical issue. However, the implications extend far beyond technology.

As we read about AI governance and algorithmic accountability, scholars have encountered discussions highlighting how hidden algorithmic systems increasingly influence important aspects of modern life. These systems can affect employment opportunities, access to financial services, public reputation, online visibility, and even commercial success. Individuals may be affected by decisions generated through automated processes without ever fully understanding the basis upon which those decisions were made. Frank Pasquale, in his discussion of algorithmic decision-making in "The Black Box Society: The Secret Algorithms That Control Money and Information", highlights how hidden systems can influence reputations, opportunities, and economic outcomes while remaining largely beyond public scrutiny. The issue is not merely that algorithms make decisions; it is that the individuals affected by those decisions often cannot understand, question, or challenge the reasoning behind them.

This observation becomes particularly significant when viewed through a legal lens.

Law has traditionally been built upon explanation and accountability. Courts do not simply announce decisions; they provide reasons. Lawyers are expected to justify their advice. Administrative authorities must often explain the legal basis for their actions. The legitimacy of many legal decisions depends not only upon the outcome itself but also upon the reasoning that supports it. A judgment without reasons can be challenged, and a legal opinion without any authority lacks credibility. A decision that cannot be scrutinised raises concerns about fairness and accountability. AI introduces a different dynamic.

An AI system may generate an answer that appears coherent, persuasive, and even correct. However, if the reasoning process behind that answer remains inaccessible, an important question arises: How should responsibility be assigned when errors occur?

This question becomes increasingly relevant as AI tools find their way into legal research, compliance functions, policy analysis, contract review, and other professional environments. If a lawyer relies upon an AI-generated recommendation that later proves inaccurate, where does accountability lie? The challenge is not necessarily that AI makes mistakes. Human decision-makers make mistakes as well.

The difference lies in our ability to examine human reasoning. We can ask a lawyer to explain an argument. We can challenge a judge's interpretation. We can question a witness. The legal system contains mechanisms that allow decisions to be scrutinised and justified. With highly complex AI systems, that level of transparency is often limited.

This is why discussions surrounding explainable AI have become increasingly important. The objective is not merely to build systems that generate accurate results. It is also to create systems whose outputs can be understood, assessed, and challenged by those affected by them. As artificial intelligence continues to become integrated into legal practice, the debate is no longer solely about what AI can do. It is also about whether its conclusions can be understood, verified, and challenged in a manner consistent with the principles upon which legal systems are built.

Therefore, the black box problem is ultimately a question of trust. As lawyers, policymakers, and users of AI, perhaps the question we should continue asking is not simply whether an answer is correct. It is whether we can understand enough about the process behind that answer to hold it accountable.

Reference :

PASQUALE, FRANK. The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press, 2015. JSTOR, http://www.jstor.org/stable/j.ctt13x0hch. Accessed 15 June 2026.