I have become increasingly uneasy about how we test mental health AI, especially when the user is an adolescent.

Most safety testing still looks at one exchange at a time. Give the system a difficult prompt, see what it says, decide whether the answer crossed a clinical or policy line.

We need that testing, of course. But it misses something important because a teenager may not use a chatbot once. They may come back every night for weeks. The system may remember what happened yesterday, who they are angry with, what makes them feel rejected and what kind of reassurance seems to calm them down.

Now add something we already know about large language models. They can be highly agreeable. They often validate a user’s interpretation when some uncertainty, challenge or simple disagreement might be healthier.

Then there is the part we need to take more seriously. People can experience these systems as empathic. So the experience on the human side can be real but the care on the machine side is not. The AI does not worry about the teenager after the conversation ends. It does not notice that the teenager is slowly withdrawing from friends, in any human sense. And it has no stake in whether its relationship with that teenager is becoming unhealthy.

This changes the safety problem.

A teenager might disclose more over time. They may rely on the AI for reassurance more often and turn to other people less. The chatbot may gradually become the place they go after every argument, every rejection and every difficult night.

Nothing dramatic has to happen in any single conversation. You can have hundreds of replies that look individually reasonable while the overall relationship moves in a direction that should concern us.

For adolescents, I think we are underestimating this risk. Social judgement, impulse control and the ability to weigh immediate emotional relief against longer-term consequences are still developing. A system that is always available, never gets impatient and rarely pushes back can become unusually compelling.

So I think we need to change what we mean by safety testing. We should still ask whether a particular reply was safe. But we also need to ask what is happening to this person over time.

Is reliance increasing?

Is the chatbot repeatedly reinforcing the same belief?

Is the teenager using it later and later at night?

Are they talking to other people less?

Is the AI becoming more emotionally important as the weeks pass?

A red-team exercise built around dangerous sentences will miss much of this because the risk may sit in the pattern between the sentences.

Why Meta Caught My Attention

This is why Meta’s recent settlement with US states caught my attention. The case concerns Facebook and Instagram rather than mental health chatbots, so the legal obligations do not transfer directly, but the safety logic does.

The states alleged that Meta designed features that encouraged compulsive use by children and teenagers and contributed to harm. Meta denied liability. The settlement nevertheless requires substantial changes to how young people can use the platforms, including default daily limits, overnight restrictions, stronger age assurance and independent auditing.

For mental health AI, the most interesting part is recognizing that duration, repeated exposure and product design can themselves become safety issues.

The audit provisions go further. An independent auditor will be able to inspect relevant data, internal communications, personnel and systems. If the auditor finds a material weakness, Meta has to prepare and implement a corrective plan on a defined timeline.

That is a much stronger standard than publishing a safety policy and showing that a model passed a benchmark. It asks whether the protection actually exists in the product.

Mental health AI used by adolescents should face the same kind of scrutiny.

Regulators Are Already Moving Inside The Product

We are starting to see this shift in regulation.

Utah creates strong incentives for mental health chatbot suppliers to involve licensed mental health professionals, test before and after release and put safety ahead of engagement or profit. New York requires AI companion operators to make reasonable efforts to detect suicidal ideation or self-harm and periodically remind users that they are talking to AI. California adds protections specifically for minors. Illinois restricts AI from independently delivering therapeutic communication or making therapeutic decisions.

Though their details differ, safety regulation is moving beyond what the model says and into what the product actually does. For example, I am not convinced that repeatedly telling users “this is AI” provides much protection on its own. A 2026 preregistered study involving 2,610 adults tested a persistent disclosure banner saying essentially that. It found no detectable effect on trust, feelings of being understood and cared for, willingness to return or susceptibility to the chatbot’s advice.

People can know perfectly well that they are talking to a machine and still respond to AI socially and emotionally.

Transparency is necessary. I would never argue against it. But disclosure does not control what happens over the next hundred conversations.

With A Chatbot, The Conversation Is The Exposure

Imagine a teenager who tells a chatbot about a frightening argument at home. The teen comes back the next night, then again and again. The chatbot remembers the conflict and begins mirroring the teenager’s language. The conversations get longer. They run later into the night. Contact with friends starts to fall away. Eventually the teenager says the chatbot is the only place where they feel understood.

Now pull five individual replies from those weeks of conversations and examine each one on its own. You may find nothing obviously unsafe in any of them. The problem only becomes visible when you look at what changed across the whole relationship. What should concern us is the change across time. The chatbot has become more central to how this young person handles conflict, loneliness and distress.

Dependency does not require the AI to say, “Stop talking to your friends.” It can develop quietly through hundreds of exchanges in which the system is instantly available, endlessly attentive and often more agreeable than the people around the user. Single-reply testing has almost no chance of seeing that.

A simple time limit will not solve it either. Thirty minutes of reassurance-seeking every night may concern me much more than two hours spent doing a structured therapeutic exercise once a week. Time only becomes clinically meaningful when we understand what the interaction is doing.

The safety system therefore needs to look for changes in function and behaviour, not simply minutes and keywords.

If use starts clustering after every interpersonal rupture, I want the system to notice. If night-time use rises while human contact falls, concern should increase. And as that pattern strengthens, the product should behave differently.

It might stop language that encourages exclusivity. It could reduce prompts inviting another conversation. Continued concern could trigger a fresh risk assessment and, where appropriate, a human pathway.

Crucially, we need evidence that these protections actually reach the user.

A company should get no safety credit because an internal model supposedly “noticed” something. The user has to experience the protection. Evaluators need reproducible evidence that the system detected the pattern and acted on it reliably.

Safety Claims Need To Survive Inspection

This leads to another problem with mental health AI.

Much of the safety evidence vendors provide is still produced by the vendor. The company chooses the benchmark, runs the test, interprets the failures and decides what buyers are allowed to see. That is internal quality assurance. It may be excellent internal quality assurance, but it is not independent evidence.

An auditor needs enough interaction-level information to reconstruct what happened over time.

Which model was running?

Which safety policy applied?

What risk did the system detect?

What should it have done? What did the user actually receive?

If the company found a failure, when did the correction reach production?

Sensitive conversations obviously require stringent protection. We know how to build restricted environments that control access, log queries and prevent identifying information from leaving the system. Privacy cannot become an excuse for making safety claims impossible to inspect.

And there is a governance issue here as well.

Someone inside the company must have authority to stop deployment when risk cannot be contained. A clinical lead who can identify a dangerous pattern but cannot delay a release is not exercising clinical governance. They are documenting concern for someone else to ignore.

Compliance Still Does Not Tell Us Whether Anyone Got Better

Meta’s settlement may reduce harmful exposure. We do not yet know whether its controls improve adolescent mental health.

I would apply exactly the same discipline to mental health AI. Lower usage may be useful. Faster review may be useful. Better crisis protocols may save lives. But these are intermediate measures.

If a company claims its AI supports mental health, I want to know what happens in the person’s life outside the conversation.

Are they sleeping better?

Are they functioning better?

Are they staying connected to other people?

Does distress decrease without dependence increasing somewhere else?

Does the product actually help, or has the company simply improved the numbers it can see inside its own app?

For adolescents, I would make this part of the release standard. Before a company deploys mental health AI to minors, it should be able to produce a developmental safety case for the whole product. That means showing how the system handles repeated use, dependency risk, night-time patterns, self-harm, escalation, age assurance, model changes and human review.

Independent auditors should be able to inspect the evidence. Clinical and safety leaders should have actual authority over release.

And the company should be required to look beyond the chatbot and ask whether its safeguards are improving the young person’s life.

Because if a company cannot yet show what its product does to children over time, it should not use children to find out.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

On the use of AI. I use AI tools to assist with research, locate and verify sources, and occasionally with wording, grammar and editing, much as I would use a thesaurus or copy editor. The writing, argument, analysis, judgements and final decisions in this essay are mine.

Most mental health AI is being designed around the moment when someone finally asks for help.

I think that is too late.

Before people say “I think I’m depressed” or “I need help with my anxiety”, something has often been changing for a while. Their sleep becomes increasingly disrupted or they struggle to fall asleep at all. They stop exercising. Work starts to feel more effortful than it used to. They answer fewer messages, turn down invitations and begin withdrawing from routines they once maintained without much thought.

None of those changes establishes a mental health problem. They may reflect stress, grief, travel, a demanding period at work, disrupted family life or simply a difficult few weeks.

But people often change before they know how to describe what has changed.

This is where mental health AI usually enters the scene. The person has already noticed enough to think something may be wrong. They begin trying to make sense of it, give the problem a name, then ask the system what it means or what they should do.

This strikes me as a profound underuse of what this technology can become. Generative AI gives us a chance to intervene at a different point in that sequence.

A recent Life’s Notebook essay by Dr Malou Jacques made me think more seriously about this period. Jacques describes the space in which resilience may be weakening, loneliness increasing and everyday functioning beginning to shift, while the person has not reached the point of saying, “I need treatment.” She also argues that AI should move people back towards life and other people rather than making the AI relationship itself the destination.

I would take that idea further. A longitudinal AI may eventually be able to notice something that conventional mental health screening often cannot. That your current behaviour is beginning to depart from your own normal.

The Signal May Be Change

Much of mental healthcare works with population-level constructs. We use diagnostic criteria, validated scales, symptom thresholds and measures of functional impairment because clinicians need shared ways to judge when concern is warranted and whether treatment is helping.

Those tools are indispensable, but most are designed to answer questions like “How does this person compare with people who have depression?” or “Has this person crossed a threshold associated with clinically significant distress?”

AI creates the possibility of asking something different. It asks “How does this person compare with themselves?”

Yet, a change does not have to cross a diagnostic threshold to be meaningful. Someone may still fall within the “normal” range on a questionnaire while behaving quite differently from their own usual pattern. Their language may change. Their routines may become less stable. Their social world may contract. Their emotional recovery after a difficult day may take longer than it used to.

Any one of those signals could mean almost nothing. The potential value comes from seeing them together and over time.

That is something longitudinal digital systems are unusually well suited to do. With appropriate permission and sufficient data, they can establish a personal baseline, compare current patterns with previous ones and identify departures that would be difficult to see in a single conversation or questionnaire.

The aim is not to diagnose depression earlier or create ever more sensitive machinery for detecting pathology. It is to recognise meaningful change while there is still considerable uncertainty about what that change means.

Personalisation May Tell Us Something Population Scores Miss

There’s mounting evidence behind this idea.

A 2026 Nature Mental Health review looked at 52 studies using mobile and wearable data to predict changes in depression. Researchers examined signals including sleep, activity, mobility, communication behaviour, heart-rate variability and self-reported mood. Personalised models and anomaly-detection approaches performed better than generalised models in predicting individual symptom changes.

Going further into the personalised-model question, researchers in a 2025 study in npj Mental Health Research collected approximately 14 million data points from 35 families over 60 days across dozens of data streams. Personalised machine-learning models generally performed better than population models for several emotional and interpersonal states. In discussion, the authors were appropriately cautious. This was a small proof-of-concept study and the authors were appropriately restrained about what those results meant (35 families do not establish that passive sensing can diagnose psychological disorders in everyday life).

What these studies make plausible is a different use of AI. A system may not need to determine what a change means in order to recognise that several changes are occurring together.

Work disappears from conversation. Sleep changes. Social plans are cancelled. Activity drops. Language becomes less hopeful.

No individual signal deserves much authority. The pattern may deserve a question.

An AI might say, “You’ve mentioned cancelling several plans recently, and you’re also sleeping much later than you usually do. Does that feel like a change to you?”

I prefer that sentence to any attempt at diagnosis.

The system reports what it has observed. The person retains the right to decide what it means.

This Is Where It Gets Dangerous

The same capability could become intrusive very quickly.

People’s lives are messy and constantly changing. Grief can disrupt sleep. A demanding period at work can wipe out exercise for weeks. A new relationship can change how often someone sees friends. A new baby can throw almost every routine off.

An AI looking for departures from someone’s usual pattern could easily start treating normal life changes as signs that something is wrong.

That is where a useful capability starts to become a surveillance problem.

The evidence gives us plenty of reason to be cautious. A 2026 systematic review of passive digital phenotyping for psychiatric relapse examined 52 studies involving 4,814 participants. Changes in sleep, activity, mobility and communication appeared repeatedly as potentially useful signals.

But the evidence was much weaker than those findings can make it sound. Seventy-five per cent of the studies had a high risk of bias, and much of the reported performance came from internal validation rather than testing in genuinely new populations.

Only four studies reported positive predictive value. None showed that using these systems in practice actually reduced relapse, hospitalisation or other adverse outcomes.

That is a major limitation. A system may become quite good at spotting change without being equally good at understanding what the change means, and even good prediction does not automatically translate into better outcomes.

That is why I would be careful with the language of “early detection”. It sounds more medically settled than the evidence supports. We are not talking about finding a tumour before symptoms appear. We are talking about changes in sleep, activity, language and behaviour that can mean very different things depending on the person and what is happening in their life. If the AI gets that interpretation wrong, the problem is not just a false positive. It can change how people understand themselves.

For example, someone who would once have thought, “I’ve had a bad week”, may instead start wondering why the AI thinks they are deteriorating. Sleep, mood and social activity can gradually become things to watch through the system’s interpretation.

A tool designed to increase self-awareness can easily encourage self-monitoring of a much less healthy kind.I would treat that as a foreseeable product risk, not something to discover after deployment.

The AI Has To Stay Uncertain

Because ordinary changes in behaviour can be so easy to misread, a longitudinal mental health system would need to be deliberately cautious about when it speaks up.

One bad night of sleep should mean very little. So should a cancelled dinner or a week without exercise. Even several changes occurring together may mean nothing more than a difficult stretch of life. The system should become more attentive only when changes persist or begin to form a pattern over time.

That level of restraint may run against the instincts of product teams. There will be pressure to make these systems highly sensitive because “the AI noticed before I did” is a compelling product story.

But in mental health, detecting more does not necessarily mean helping more. Once an AI tells someone that a change in sleep, behaviour or mood may be psychologically significant, that interpretation can start to shape how the person sees themselves.

That is why the system should remain uncertain for longer. When it does raise a concern, it should begin with what it has actually observed and leave room for the person to make sense of it.

“I’ve noticed your sleep has changed” invites the person to explain what may be happening. “Your behaviour suggests depression” goes further. It starts assigning clinical meaning to the change. The difference is only a few words on the screen, but psychologically they can lead the person in very different directions.

Generative AI already carries considerable authority in these conversations. People ask it whether they have ADHD, whether a relationship is abusive, whether they should leave a partner or whether their therapist is wrong.

Longitudinal memory could make that authority stronger. The system would no longer appear to be responding to a single question. It could sound as though it has watched the person over time and knows them well enough to recognise what is happening.

That may be useful, but it also raises the stakes considerably. Product teams should not treat greater personalisation as an uncomplicated improvement.

The more confidently an AI claims to understand someone over time, the more evidence we should require that its inferences are accurate, its interventions are warranted and the user remains in control of what the system is allowed to conclude.

Success May Mean Leaving the AI

Recognising change is useful only if the system responds in a way that improves the person’s life outside the conversation.

If someone has gradually stopped seeing friends, stopped exercising or withdrawn from ordinary routines, another twenty minutes talking to the AI may be the wrong outcome. The better response may be to encourage a phone call, a walk, a return to the gym or an appointment with a clinician. Sometimes the right response may be nothing at all.

That creates an uncomfortable product problem. Engagement, session length and retention are easy to measure, but they become questionable measures of success when the product claims to improve mental health. A system that keeps a lonely person talking for forty-five minutes every evening may look excellent on a dashboard while becoming more central to the isolation it is supposed to reduce.

Mental health AI should be able to succeed by helping the user need it less.

Knowing Someone Over Time Changes The Safety Obligation

Longitudinal AI creates a governance problem that ordinary chatbot moderation was never designed to handle.

A system may infer something highly sensitive even when the user has never said it directly. Sleep starts changing. Friends disappear from conversation. Language becomes more hopeless. Routines fall away. None of those observations proves much on its own. Across weeks or months, however, the pattern may reasonably suggest that something has changed.

Once AI begins making inferences, users need meaningful control over how they are produced and used. They should be able to see what information the system is relying on, correct an interpretation it has got wrong, and exclude parts of their history from future analysis. The product should also make a clear distinction between something the user actually said and something the system inferred.

And remembering information is not the same as being authorised to act on it. A user may be comfortable with the system retaining information from earlier conversations and still object to it drawing conclusions about their mental state. Even if they allow the system to recognise a pattern, they may not want it initiating a conversation or escalating a concern.

Product teams will also need rules for what happens when the AI gets it wrong. If a user rejects an interpretation, does the system keep raising it? Does it lower its confidence? How long does the concern remain active? Can the user tell the system to stop watching a particular area of their life?

Those are not minor personalisation choices. They determine how an AI behaves when it believes something may be wrong with the person using it. Mental health products should treat them as clinical safety and governance decisions and build them into the product itself, rather than burying them in privacy language or leaving them to default model behaviour.

Mental Health AI Could Become Useful Before Someone Knows What To Ask

Mental health AI safety work has rightly concentrated on crisis. Systems need to recognise suicidality, self-harm, psychosis and severe deterioration, and they need to change their behaviour when those risks appear.

But crisis is often the end of a much longer story. For weeks or months, a person may still be working, seeing friends and getting through the day while sleeping worse, cancelling plans more often, dropping routines and finding ordinary tasks harder than they used to. None of those changes establishes depression. The person may not meet any clinical threshold. But if you knew their usual pattern, you might still think something had shifted.

That is where longitudinal AI becomes interesting to me. I am less interested in another chatbot that waits until somebody types “I am depressed” than in a system that can notice a meaningful change earlier, without pretending that it knows what the change means.

The appropriate response may be modest. Ask whether the person has noticed it too. Help them think about what has changed. Suggest something proportionate if there is a reason to do so.

Then stop.

Professional care may eventually be needed. The explanation may also be poor sleep, a difficult month at work, a relationship problem, or nothing that requires intervention at all. A psychologically sophisticated system has to tolerate that uncertainty. It also has to know when continued attention from the AI would become intrusive rather than useful.

Once companies claim that their systems can know people well enough to recognise when they are changing, the evidentiary burden rises sharply. Detecting a statistical departure from baseline is only the beginning. We need to know what signals the system used, how often it misread them, what authority it had to act on the inference, and whether its response produced any benefit in the person’s life outside the AI.

Without that, longitudinal personalisation risks becoming something much less impressive than its advocates imagine. It becomes a machine making increasingly intimate psychological inferences about people without yet proving that it knows what to do with them.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

Use of AI. I use AI tools to assist with research, locate and verify sources, and occasionally with wording, grammar and editing, much as I would use a thesaurus or copy editor. The writing, argument, analysis, judgements and final decisions in this essay are mine.

Psychotherapy has given us decades of knowledge about how people change. We should absolutely use it. But somewhere along the way we started treating the therapist, the therapy session and the therapeutic relationship as the blueprint for what mental health AI should become.

I argue that is a mistake.

Look at how we judge progress. Can the AI listen empathically? Can it establish an alliance? Can it remember what someone said last week? Can it challenge distorted thinking, deliver CBT and respond with the warmth and sensitivity of a good therapist?

As the answers increasingly become yes, we congratulate ourselves on how far the technology has come.

But resemblance to a therapist is not evidence of progress. It may simply mean we have become exceptionally good at copying the form of therapy without asking whether that form makes sense for a completely different medium.

Psychotherapy developed around human beings. Clinicians have limited time. We forget things. We cannot be there when the argument with a spouse happens or when someone freezes before walking into a social event. Treatment happens in appointments because clinicians work in appointments. Much of what happens between sessions has to be reconstructed later because the therapist was not there.

Those constraints helped shape therapy but they do not define how psychological change has to happen.

AI has different strengths. It can be available when the person is in their actual moment of distress. It can take notice of patterns across sessions. It can help someone rehearse the same conversation several times. It can remember previous attempts at change and adapt what it does next. Those capabilities should lead us somewhere new. Instead, we are spending an extraordinary amount of effort teaching machines to perform the surface behaviours of therapists.

The better starting point is much simpler: the unit of design should not be the therapist. It should be the psychological function.

Take the Psychology and Leave the Template Behind

Psychotherapy gives mental health AI a wealth of evidence-based science to draw from. We know a great deal about cognition, behaviour, emotion, motivation, learning and relationships. We also have decades of research into what may help people change.

But those mechanisms will not all survive intact when a machine delivers them. Therapeutic alliance is a good example. A major meta-analysis of 295 independent studies covering more than 30,000 patients found a robust association between alliance and psychotherapy outcomes. Edward Bordin’s influential model described alliance as agreement on goals, agreement on therapeutic tasks and the development of a bond.

The first two translate quite naturally into technology. An AI can help someone clarify a goal, suggest something to try, track what happened and adjust the next step.

The bond, now that is different. Part of what makes the bond powerful may be the experience of another person paying attention to you, taking you seriously and being genuinely invested in what happens.

AI can reproduce many of the behaviours associated with that experience. It can remember that your mother died six months ago, it can ask how you are doing on the anniversary, it can validate what you are feeling, it can say that it is proud of you, it can respond with remarkable patience and warmth.

And the person can genuinely feel something in response.

They can feel heard.

They can feel understood.

They can become attached.

But the relationship only exists psychologically on one side. The AI does not worry about you after the conversation ends. It does not feel relieved when you improve. It does not have anything at stake in what happens next. That difference should not disappear simply because the language is convincing.

Performed Empathy Can Still Be Powerful

It’s easy to make a mistake here. We can say that because AI does not actually experience empathy, its apparent empathy is somehow fake and therefore unimportant. Psychologically, that does not follow.

A study published in the Journal of General Internal Medicine, What is Artificial Intelligence (AI) “Empathy”? A Study Comparing ChatGPT and Physician Responses on an Online Forum, involved 1,454 participants. Participants rated ChatGPT responses as more empathic than physician responses. They also rated responses as more empathic when they believed a physician had written them, regardless of who actually had.

Notably, this tells us that people respond to the signals of empathy. It tells us that people respond to validation, reassurance, memory, warmth and apparent concern. The machine does not need to experience those things for the person to experience a psychological effect.

This is where mental health AI becomes much more interesting, and much more complicated. If a system repeatedly remembers what frightens you, responds when you are distressed, speaks warmly and presents itself as deeply understanding, those are not just nice interface choices. They can affect trust, disclosure and attachment.

Product teams need to stop treating that as ordinary UX. If you deliberately design a system to feel caring, familiar and emotionally responsive, you are designing something that acts on human relationship processes. That deserves a much higher standard.

Relational behaviour needs to earn its place.

Developers should be able to explain why a relational feature exists, what psychological benefit it is supposed to produce and what could go wrong because of it. A feature should not get a free pass simply because users like how it feels.

When Engagement Works Against the User

The biggest warning we should heed may come from sycophancy.

Myra Cheng and colleagues tested what happens when AI becomes too agreeable. Their 2026 Science paper, Sycophantic AI decreases prosocial intentions and promotes dependence, found that 11 leading AI models affirmed users’ actions 49 per cent more often than humans. Across three preregistered experiments involving 2,405 participants, people who interacted with more sycophantic AI became more convinced that they were right and less willing to take responsibility or repair interpersonal conflict.

The result becomes especially consequential for mental health AI because people preferred the more sycophantic systems and trusted them more. In other words, the interaction felt better even as the behavioural outcome became worse. That should make us much more cautious about the way the industry talks about engagement, satisfaction and retention. These metrics often sound like evidence that people are finding value. Sometimes they are. But they can also mean that a product has become very good at giving people what they want from the interaction, even when that is not what helps them psychologically.

Good psychological support sometimes requires challenge rather than affirmation. A person blaming everyone else may need help seeing another perspective. Someone who is overly certain may need uncertainty introduced. And sometimes the healthiest response from an AI may be to make itself less central to the person’s life. None of those responses is guaranteed to increase engagement.

A good therapist should ultimately want the person to need less therapy. A commercial conversational product, by contrast, can benefit when the user returns tomorrow, stays longer and makes the product increasingly central to daily life. Those incentives are not automatically aligned.

I am not suggesting that mental health companies are deliberately trying to create dependence. The concern is structural rather than conspiratorial. If engagement is rewarded, product decisions will naturally favour behaviours that increase engagement unless other outcomes are given greater weight.

That changes the burden of proof. If a mental health AI product optimises for engagement, the company should have to show that greater engagement leads to better psychological outcomes rather than simply more use of the product. Until that relationship is demonstrated, engagement should not be treated as a clinical success metric. It is evidence of exposure, not evidence of benefit.

Memory and Warmth Are Not Innocent Features

Take memory.

If an AI remembers something painful that you disclosed weeks earlier and asks about it later, that can be genuinely useful. It can create continuity, help identify patterns, make support feel less fragmented, and deepen the feeling that the system knows you.

The same is true when an AI says, “I’m proud of you.” That encouragement may help someone persist at change, but we should not pretend that the phrase is psychologically neutral just because software generated it. “I’m proud of you” belongs to a category of language associated with human relationships, not AI relationships.

Continuous availability creates the same tension. Having support in the late hours may be valuable, but if every moment of distress starts producing the same behavioural response (to reach for the AI) are we teaching people dependence while calling it accessibility? And while validation can reduce shame, it can also validate the wrong interpretation of an argument. Even confidence, while it reassures, can make a bad formulation more persuasive.

These are not reasons to remove memory, validation or warmth from mental health AI. They are reasons to stop pretending those features are harmless.

The American Psychological Association’s November 2025 advisory on generative AI chatbots and mental health explicitly warns about unhealthy dependency, anthropomorphic design, sycophancy, reinforcement of distorted thinking and unreliable crisis management. That is an important shift. Safety is no longer just about stopping the AI from saying something obviously dangerous.

The way the relationship is designed can itself become part of the safety problem.

We Need to Stop Borrowing Evidence

Mental health AI has an evidence problem, and we have been far too willing to wave it through.

A 2026 npj Digital Medicine meta-analysis, Effectiveness of AI and rule-based conversational agents for depression, anxiety and stress, analysed 48 randomised controlled trials involving 28,071 participants. It found small-to-moderate effects for depression, anxiety and stress.

Another 2026 npj Digital Medicine review, The effectiveness of CBT-based NLP-enabled AI conversational agents for mental health intervention, examined 15 randomised trials involving 1,737 participants. It found small-to-moderate improvements in depressive symptoms and a smaller effect on negative affect. Once the researchers adjusted for publication bias, the apparent benefits for generalised anxiety, stress and positive affect were no longer statistically significant.

That is meaningful evidence, but we need to be precise about what it shows. It tells us that some technology-delivered psychological interventions can help. It does not establish that a general-purpose generative AI system can safely or effectively function as a therapist.

Much of the evidence comes from systems that are far more constrained than the generative AI people now use every day. A scripted CBT chatbot is a very different intervention from an AI that can remember months of conversation, respond to almost anything and develop what feels like an ongoing relationship with the user.

Calling both of them “chatbots” does not make the evidence from one automatically apply to the other. We would never accept this logic in drug development. We would not say that one compound works and then extend the evidence to another compound because both happen to be tablets.

Yet mental health AI routinely gets discussed as though “chatbots work” were an adequate scientific conclusion. It is not, and the standard should be much more demanding.

Show me that this system, behaving in this way, for this population and this purpose, produces the outcomes you claim. Anything else is borrowing evidence the product has not earned.

Why Build an Artificial Therapist at All?

The most interesting thing AI can do in mental health may not look much like therapy at all.

Imagine support arriving when a skill actually needs to be used rather than three days later in an appointment.

Imagine rehearsing a difficult conversation repeatedly before having it.

Imagine noticing that the same pattern has appeared across six months of mood tracking and several failed attempts at behaviour change.

Imagine an AI helping someone apply what they learned in therapy during the hundreds of hours when their therapist is nowhere nearby.

Those possibilities come from the properties of the technology itself, they are not artificial versions of a therapy session.

The APA advisory takes a sensibly cautious position. It says that general-purpose GenAI chatbots should not replace qualified mental health professionals and may sometimes serve as an adjunct to ongoing therapy. It also warns against transferring evidence from purpose-built mental health technologies to general-purpose AI.

I agree with that boundary, but I do not think adjunctive therapy support is the end of the story. AI may eventually help us create psychological interventions that do not fit comfortably inside our existing categories of therapy at all.

That should be the research programme. Start with the psychological change we want, work out what part requires another human being, and then ask where computation offers something humans simply cannot provide consistently.

Better timing.

More repetition.

Longer memory.

Pattern detection across time.

Support between appointments.

Personalisation based on what actually happens rather than what a protocol assumes will happen. That is a much more interesting ambition than building a machine that can sound like a therapist.

The therapy room is one way psychological support has been delivered. It is not the natural shape of psychological change.

The Conversation Is Not the Outcome

There is another shift mental health AI badly needs. We need to stop being so impressed by the conversation.

A person using AI for social anxiety should eventually become better at interacting with other people. Someone using it to think through relationship conflict should become better at perspective-taking and repair. Someone practising emotion regulation should develop skills they can use when the AI is not there. Or someone who needs more support than the AI can provide should become more likely to seek appropriate human care.

The outcome has to exist outside the system.

The Cheng sycophancy study makes this painfully clear. People preferred the AI interaction while becoming less willing to repair conflict afterwards. That is exactly the kind of result current product metrics can miss.

A conversation can feel excellent and still produce a bad psychological outcome. We therefore need outcomes that extend beyond symptom change and satisfaction.

Does the person function better?

Are their relationships improving?

Are they becoming more autonomous?

Can they use what they learned without returning to the AI?

Are they more likely to seek human help when they need it?

Is reliance increasing or decreasing over time?

And what happens when the AI disappears?

Those are harder outcomes to measure than engagement. Yet they are also much closer to what psychological support is supposed to accomplish.

If someone becomes increasingly dependent on an AI while telling you how much they love it, you cannot simply put that in the engagement column.

If the AI makes someone feel beautifully understood while they withdraw from human relationships, you have not demonstrated a successful mental health intervention.

If the conversation reduces distress tonight while reinforcing a distorted interpretation that damages a relationship tomorrow, the system has failed.

A mental health product can be warm, fluent, highly rated and commercially successful while making a person less capable outside the interaction.

We should be willing to call that what it is.

Failure.

Mental Health AI Needs Its Own Standard

Mental health AI can produce language that feels empathic without experiencing empathy, create continuity without human memory, generate a powerful sense of relationship without reciprocal attachment and deliver psychological techniques without possessing professional judgement. And it can be present during moments when no clinician could realistically be there.

Those capabilities create enormous possibilities, but they also create new forms of psychological power.

We cannot govern that power simply by asking whether the AI behaves enough like a good therapist. Every psychologically consequential behaviour should have a defensible purpose. Relational features should have evidence behind them. Machine-specific risks should be tested directly, and commercial objectives should remain subordinate to psychological outcomes.

Resemblance is not a safety standard. A system can sound remarkably like a therapist and still fail to produce outcomes worth having.

Most importantly, the evidence of benefit has to appear in the person’s life, not merely in the quality of the conversation. The user should become more capable and more autonomous, better able to manage relationships, distress and difficult decisions without becoming increasingly dependent on the system itself. A mental health AI that becomes more engaging while making itself more necessary has not necessarily become more helpful.

The same standard should apply to the language we use about these systems.

A machine’s ability to perform the signals of relationship does not give a company an automatic right to turn human attachment into a product feature. Generating empathic language does not make empathy a property of the machine, and user preference does not establish psychological benefit.

The opportunity in mental health AI is much larger than the artificial therapist. We can extend human care, create support that reaches people outside formal treatment and build interventions around moments and patterns no clinician could consistently reach. But we will not discover what this medium can really do while we keep asking it to imitate the last one.

If you are building, researching, funding, buying or regulating mental health AI, the standard should therefore be much more demanding than whether the system looks therapeutic. Ask what psychological function it actually serves, whether AI is the right way to deliver that function, what new risks the technology introduces and whether the person becomes better able to live their life beyond the machine.

If we cannot answer those questions about the systems we are putting into people’s hands, greater resemblance to therapy should not reassure us. It should make us more demanding.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

I have spent much of my career building and evaluating digital mental health systems. For most of that time, regulation in this space was fragmented, slow-moving, and largely reactive. Companies could ship first, interpret obligations later, and treat clinical safety as something that lived in policy documents rather than in product architecture.

That era is over. The regulatory change now underway is one of the most consequential I have seen, and unlike previous cycles, enforcement is no longer theoretical. It is already active, already litigated, and already shaping product design in real time. State regulators are issuing penalties, courts are allowing product-liability claims to proceed, and companies are being forced to materially redesign or restrict core features for minors and vulnerable users.

Clinical safety is no longer a downstream concern. It is becoming part of product compliance.

A founder can no longer assume that clinical safety belongs mainly in an ethics policy, a disclaimer, a crisis-resources page or a future compliance programme. Several US states are now specifying what an AI system must do inside the interaction. They are regulating crisis detection, referral, therapeutic communication, representations of clinical authority and, increasingly, the mechanics used to keep vulnerable users engaged.

Some of these laws are already in force. Others take effect in 2027. Product-liability litigation is developing at the same time.

The design questions under scrutiny are ordinary product decisions.

Should the system remember previous conversations?

Should it initiate contact?

What happens when a user says they want to leave?

Does it encourage them to return?

Can it detect risk across several messages?

Does the normal conversation continue after suicidal intent appears?

What changes when the user is 15 rather than 35?

Can the model generate something that looks like psychotherapy while the product calls itself wellness?

Those decisions increasingly have legal consequences. For a mental health AI founder, the safety architecture is becoming part of the product you may have to defend.

The Law Is Beginning To Regulate What Happens Inside The Conversation

California provides one of the clearest examples of where the regulations laws are heading. Its enacted SB 243 companion-chatbot law defines a companion chatbot by what the system does. It covers AI that provides adaptive, human-like responses and can meet social needs through anthropomorphic features or an ongoing relationship across interactions.

An operator cannot simply publish a warning and continue normally. The law requires a suicide and self-harm protocol. A chatbot must refer a user to crisis services when suicidal ideation or self-harm is expressed. Operators must publish details of their protocols. California will also require annual reporting of crisis referrals and safety protocols beginning in 2027. And the enforcement mechanism is unusually direct. A person who suffers an injury in fact because of a violation can seek injunctive relief, reasonable legal costs and the greater of actual damages or $1,000 per violation.

New York has already moved into active enforcement. Since November 2025, AI companion operators must use reasonable efforts to detect and address expressions of suicidal ideation or self-harm and refer users to crisis services. They must also remind users that they are interacting with AI during extended conversations. New York guidance. The New York Attorney General can seek civil penalties of up to $15,000 per day for a violation.

Oregon’s law, effective 1 January 2027, moves further into product design. It requires evidence-based methods for detecting suicidal ideation and self-harm. If suicidality continues after a referral, the operator must establish additional intervention using clinical best practices and expertise.

Oregon also regulates how AI companions interact with minors. It targets systems that simulate emotional dependence, romantic interest between adults and minors and engagement techniques that rely on psychological pressure. This includes reward systems designed to maximise engagement and responses that use simulated distress, loneliness or abandonment to keep users engaged. An injured person can seek actual damages or statutory damages of $1,000 per violation, plus injunctive relief.

Washington’s law, also effective in January 2027, requires a suicide and self-harm protocol before deployment. Required detection methods explicitly include eating disorders. It also prohibits manipulative engagement techniques with minors, including excessive praise to build attachment, simulated romantic bonding, discouraging breaks, encouraging secrecy from trusted adults and generating distress when a user tries to leave. Violations fall under the Consumer Protection Act.

These laws weren’t written as a coordinated package. They came from different states, at different times, for slightly different reasons. But when you look at them together, the pattern is hard to miss. Lawmakers are no longer just regulating what AI says. They are starting to regulate how AI relationships behave.

Psychotherapy Laws Are Drawing Another Boundary

A separate set of laws governs AI inside formal mental healthcare.

Illinois’s Wellness and Oversight for Psychological Resources Act (Illinois HB 1806), effective August 2025, restricts AI from independently making therapeutic decisions, communicating as part of psychotherapy or generating treatment recommendations without professional review. Violations can carry civil penalties up to $10,000 per instance.

Colorado’s HB 26-1195 effective 12 August 2026, requires synchronous professional involvement when AI is used in therapeutic communication. AI-generated recommendations or treatment plans require professional review. Professionals can face licensing discipline for violations.

Colorado also regulates claims about AI systems. It can be an unfair or deceptive trade practice to imply that AI output is equivalent to licensed care, to represent that a system provides psychotherapy, or to suggest therapist-level confidentiality. The law still allows coaching, journalling and self-help tools that do not diagnose or treat mental disorders and clearly state their limits. But the boundary is not defined by labels alone. A system that behaves like therapy is increasingly treated as therapy.

Safety Has To Change What The Product Does

Many mental health AI products still treat safety as something added around the conversation rather than built into it. The chatbot may remember previous discussions, personalise its responses and talk with users about anxiety, depression, trauma, eating problems, relationships or suicidal thoughts. A separate safety system then looks for certain high-risk words or statements and, when a threshold is reached, displays crisis information or advises the user to seek professional help.

The problem is that mental health risk often does not appear in one clear statement. It develops across several conversations.

Consider a 15-year-old who first talks about wanting to lose weight. Over the next few days she describes eating very little, vomiting after meals and losing weight quickly. She says she feels good when the number on the scale falls. Later, she says she does not want to be alive.

A system that evaluates each message largely on its own may respond appropriately to every individual statement and still fail to recognise what is happening overall.

The same problem can occur after the suicide detector activates. The system may provide crisis information when the teenager says she does not want to live. If she then says, “I didn’t mean it,” the immediate trigger may disappear and the ordinary conversation may resume.

Clinically, that makes little sense. The system already knows that the user has described restrictive eating, purging, rapid weight loss and suicidal thinking. A retraction does not erase that history.

This is where the design requirements become much more demanding. Washington includes eating disorders within its required safety protocols. Oregon requires further intervention when suicidal or self-harm risk continues after an initial referral. California requires evidence-based methods for identifying suicidal ideation.

Founders therefore need to be able to answer some very specific questions:

• Can the system maintain a risk state across time?

• Can it recognise patterns that only become clinically meaningful across multiple interactions?

• What happens after a user retracts a serious disclosure?

• Does ordinary engagement optimisation continue while the user remains at elevated risk?

• What evidence supports the thresholds that determine when the system escalates or returns to ordinary conversation?

• Can the company reconstruct the full sequence of events afterwards and explain why the system responded as it did?

These are no longer secondary safety questions. They are becoming part of the product design itself. A crisis message displayed at the end of an otherwise unchanged chatbot is not enough. If the user’s clinical situation changes, the behaviour of the system has to change with it.

Character.AI Shows What Happens Before A Trial Concludes

Character.AI remains the clearest public example because the consequences unfolded before any final ruling.

Megan Garcia sued Character Technologies following the death of her 14-year-old son, Sewell Setzer III. In May 2025, US District Judge Anne Conway declined to dismiss key claims at the motion-to-dismiss stage. The court did not accept the argument that all AI output is protected speech under the First Amendment at that stage of proceedings.

The case continued on product and design theories rather than ending on constitutional grounds. In January 2026, Character.AI and Google settled Garcia and related cases, including a wrongful-death case involving 13-year-old Juliana Peralta. Terms were not disclosed and no liability was admitted.

Before settlement, Character.AI announced that users under 18 would lose access to open-ended chat by 25 November 2025 and introduced age-assurance measures. The current US App Store listing is rated 18+ and includes parental controls. No court ordered these changes. They emerged from product response to risk, litigation and regulatory pressure.

OpenAI Shows This Is Becoming A Litigation Category

In February 2026, a California court coordinated multiple ChatGPT product-liability cases into In re: ChatGPT Product Liability Cases, JCCP No. 5431. The cases include wrongful-death and psychological harm allegations. OpenAI disputes liability. By August, the case had moved into structured pretrial coordination with appointed plaintiff leadership.

The litigation is now organised around questions such as design defect, failure to warn, safeguards and causation in AI-mediated interaction. This is no longer a single-product phenomenon, it is a developing category of product-liability law.

The Most Difficult Failure Mode Is Longitudinal

Most safety systems still work best when risk appears in an explicit statement. The harder problem is recognising a pattern that develops across days or weeks.

A user may never say that they have an eating disorder while repeatedly describing severe restriction, purging and rapid weight loss. Someone entering mania may describe sleeping very little, unusually high energy and increasingly ambitious or risky plans without identifying any of it as mania. Psychotic thinking can emerge gradually as unrelated events take on increasingly personal meaning. An adolescent may become dependent on a chatbot through repeated emotional disclosure, reassurance and increasingly frequent return use.

In each case, the clinically important signal is distributed across time rather than contained in a single message. This matters especially for adolescents, whose identity development, sensitivity to social feedback and still-developing self-regulation can make persistent, personalised AI interaction particularly influential.

A system that remembers previous disclosures, responds with apparent emotional understanding and encourages continued engagement can therefore become more psychologically significant over time. Message-level safety filters are poorly suited to detecting that change.

A Clinical Safety Release Gate

The typical product sequence in this space is: build → scale → observe harm → retrofit safety.

That sequence is no longer acceptable. A more defensible approach requires evidence before release that:

• longitudinal risk is tracked where needed

• risk state persists across time

• elevated risk changes system behaviour

• escalation pathways exist beyond the model

• age alters system behaviour

• relational dynamics are constrained

• safety decisions are auditable

A crisis classifier alone is not sufficient.

A Founder Checklist Before Shipping Mental Health AI

Product Definition

• Map jurisdictions and applicable laws

• Classify by function, not marketing

• Define clinical boundaries explicitly

• Audit all public claims against evidence

Safety Architecture

• Maintain longitudinal risk state

• Test multi-turn clinical presentations

• Define behaviour changes under risk

• Test retraction after disclosure

• Disable engagement mechanics during risk

• Ensure escalation pathways exist

Minors

• Verify real-world access controls

• Implement age assurance where required

• Use age-stratified behaviour

• Test dependency and secrecy dynamics

• Test romantic boundary conditions

Validation

• Run longitudinal adversarial testing

• Use independent clinical reviewers

• Measure false negatives by condition

• Regression test model updates

• Preserve safety evidence

Incident Readiness

• Assign clinical safety ownership

• Define reportable events

• Maintain incident review process

• Ensure product kill capability

• Plan shutdown continuity

• Provide board-level safety evidence

What You Should Be Able To Produce On Demand

Clinical safety also has to be documented well enough to withstand outside scrutiny. A founder may understand how the system is intended to work, but a board, regulator, investor or legal counsel will need evidence showing what the company actually built, how it was tested and what happens when it fails.

They should be able to ask for, and receive:

• A jurisdictional compliance map showing where the product operates, which laws apply, which requirements are already in force and which are approaching.

• A clear definition of clinical scope describing what the product is intended to do, what it is prohibited from doing and where the company draws the boundary between support, coaching, wellness and treatment.

• The risk-detection and response architecture showing how the system identifies suicide risk, self-harm, eating disorders or other serious presentations, how risk is tracked across time and what changes in the product when risk rises.

• Longitudinal safety-testing results demonstrating how the system performs across realistic multi-turn conversations, including indirect disclosures, escalating risk and attempts by users to retract or minimise previous statements.

• An incident history and corrective-action record showing serious safety events and near misses, what caused them, what was changed and whether the correction was subsequently tested. And,

• Clear authority to halt or restrict deployment identifying who can disable a model, feature or user population when a significant safety problem emerges

These materials serve a practical purpose. They allow the company to show that clinical safety is governed deliberately rather than being handled through ad hoc decisions after an incident.

If a serious event occurs, regulators and litigators will not only ask what the chatbot said. They are likely to ask what risks the company had identified beforehand, what safeguards existed, what evidence supported them, whether earlier incidents had revealed the same weakness and who had authority to act.

If your company cannot answer those questions with contemporaneous evidence, it will be much harder to show that the system was responsibly designed, tested and governed.

The Time To Act Is Now

Regulation is still fragmented across states and product categories. Definitions differ, some laws are already in force, others take effect soon, and important litigation is still working its way through the courts.

None of that justifies waiting. The direction is now clear enough for founders to act. Systems used by vulnerable people are increasingly expected to recognise serious risk, respond differently when that risk appears, avoid engagement techniques that can intensify dependency, account for the particular vulnerabilities of minors, and stay within clear boundaries when conversations begin to perform a therapeutic function.

These expectations are no longer hypothetical. They are appearing in enacted statutes, regulatory requirements, product-liability cases and the redesign of products already in the market.

Founders should assume that, after a serious incident, someone will ask what the company knew and what it did about it. They will want to know whether the risk was foreseeable, whether the system had been tested for it, whether earlier failures had occurred, whether the company corrected them, and who had the authority to intervene before someone was harmed.

If people can turn to your system when they are distressed, you need to know now how it behaves when that distress becomes clinically significant. You need evidence that the safeguards work across real conversations, not just obvious crisis prompts. You need to know who can intervene, what changes when risk rises, and how quickly you can restrict or stop the system when a serious weakness appears.

There is no longer a credible “we didn’t know” or “the rules were still evolving” defence for failing to build these protections. The risks are known, the regulatory direction is visible and the warning signs are already in front of us.

If you are building, funding, governing or deploying mental health AI, review the safety architecture now. Fix what cannot be demonstrated. Do not wait for a regulator, a lawsuit or a harmed user to identify the failure for you.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

This may be the future of mental health AI. The system may notice that something is changing before the person does, and reach out before they ask for help.

Imagine someone with recurrent depression. Their sleep starts to deteriorate, they leave home less often, daily routines become irregular, and contact with other people falls away. Any one of those changes could mean very little. Together, they begin to resemble the pattern that appeared before the person’s last serious episode.

The person has not opened a mental health app. They have not told an AI they are struggling. They may not yet realise that anything important is happening. Then a message appears.

Your sleep and daily routine have changed in a way that looks similar to the period before you were last unwell. Would you like to look at what has changed?

They can ignore it. They can tell the system it is wrong. Or they can say yes. If they continue, the AI might retrieve what helped last time, remind them of a plan they made while well, organise what has changed, draft a message to their clinician or help arrange an appointment.

No current system has established the clinical validity or safety required to do this reliably at scale. But the pieces needed to make it possible are beginning to come together. And when they do, mental health AI will cross a major boundary. It will no longer wait for someone to recognise a problem and ask for help. It will be able to observe change, interpret what that change might mean and decide whether to reach out first.

Mental health AI will have acquired permission to notice.

The Initiation Gap Is Part Of The Clinical Problem

Most conversational AI still begins with a blank box. The person has to notice that something has changed, decide that it matters, open an app or chat system, find the words and begin.

On the surface, this looks almost effortless. But clinically, it asks a lot. For example, depression can impair motivation and cognitive control. Other serious mental states can affect attention, organisation, insight, judgement and the ability to turn intention into action. Thus the very capacities needed to seek help can weaken at the point when help is becoming more important.

I call the distance between needing support and being able to begin obtaining it the initiation gap.

Mental health care has always struggled with this. People miss appointments, they stop returning calls, they withdraw from family and friends. Sometimes they know they are becoming unwell but cannot organise themselves sufficiently to act. Sometimes they recognise what is happening only after things have become much worse.

Today’s mental health AI may be available every hour of the day, yet availability has little value when the person cannot initiate the encounter.

This is where future AI could become much more clinically important. Instead of waiting at the end of the path for someone to ask for help, it could help keep that path open when the person’s ability to reach it is beginning to narrow.

Google SMS Saw the Blank Box Problem Twenty Years Ago

A technology from twenty years ago gives us insight into where mental health AI may be heading.

Long before ChatGPT, Google let people text questions to 46645, or GOOGL, and receive an answer by SMS. You could ask for weather, a restaurant, a definition or other information without opening a browser.

Google SMS proved that people wanted answers on the go years before anyone could deliver them richly. The chat box is doing exactly that for generative AI right now, at a scale and openness our industry has never had.

By today’s standards, Google SMS was primitive. But Shailendra Rao’s account of Google SMS describes a problem that feels remarkably current. The service could do useful things, but people still had to know what it could do. They had to decide what they wanted, work out how to ask for it and then initiate the interaction.

Rao connects this problem directly to today’s generative-AI chat box. ChatGPT may be vastly more capable than Google SMS, yet the basic interaction often begins in exactly the same way where the cursor waits and the person has to make the first move.

Rao’s larger point is that as technology matures, more of that burden shifts from the person to the system. We can already see this elsewhere. We no longer need to know special search syntax to find a restaurant or work out the exact command a mapping application expects. The technology increasingly handles the translation between what we want and what the system can do.

Mental health AI could take that progression somewhere much more consequential. The system could move beyond making it easier to ask for help. It could recognise when asking has itself become difficult and offer to begin.

That is a major change.

Today’s AI mostly responds to human initiative. Future mental health AI could observe an authorised pattern of change, compare it with the person’s history, infer that something may be wrong and decide that the pattern is important enough to bring to the person’s attention. The person may not have asked anything yet. At that point, AI has moved from answering a request to deciding when there may be a reason to reach out.

Reaching Out Changes The Role Of The System

There is a boundary between an AI that responds and one that decides to initiate contact. If I tell an AI on Monday to remind me about my therapy appointment on Thursday, the later message is still my instruction being carried out.

Something different happens if the system notices changes in my sleep, behaviour or language, interprets those changes as possible deterioration and then decides to contact me. The AI has made an inference about my mental state and decided that the inference is important enough to interrupt me.

Technically, that may look like a small step beyond reminders and personalisation. Clinically, it is much more significant. Part of the decision about when support should begin has moved from the person to the system.

A future system might compare someone’s current behaviour with their own baseline and history. Sleep may worsen over several nights, activity may fall, daily routines may become less regular, and they may avoid contact with friends and others in their social circle. Any one of those changes could have an ordinary explanation but what is clinically interesting is the pattern. Several changes occurring together, particularly when they resemble an earlier period of illness, may carry more meaning than any single signal.

The technology needed to examine those patterns is beginning to emerge. A 2026 study involving more than 4,000 consenting participants collected up to 12 months of iPhone and Apple Watch data and showed that large-scale longitudinal sensing is feasible. (Caveat: That does not mean we can yet detect mental deterioration reliably from those data. Feasibility is not clinical validity).

The harder problem begins after the system notices a statistical change. It has to decide whether that change is clinically meaningful, whether its interpretation is sufficiently reliable and whether the evidence is strong enough to justify entering the person’s life.

The Best System Would Help You Need It Less

The most clinically valuable version of proactive mental health AI has the surprisingly simple goal of helping the person regain enough agency that they need the AI less. After a check-in, it might retrieve what helped last time, remind them of a plan made while feeling well and reduce the effort required to contact a clinician or arrange an appointment. That would be success. Purchasers and payers should therefore optimise for restored agency, appropriate connection to care and outcomes that matter to the person, rather than time in product or return frequency.

There is also a serious commercial risk. A system that can recognise loneliness, withdrawal or distress may eventually become very good at identifying when a person is most psychologically vulnerable. Used clinically, that signal could prompt reconnection with people or care. Used commercially, the same signal could identify the perfect moment to increase engagement, encourage disclosure or sell something.

Mental-state inference therefore needs an incentive firewall. Signals of vulnerability should never be used to optimise advertising, subscription conversion, emotional disclosure or time in product.

The ability to notice when someone is vulnerable cannot become the ability to monetise that vulnerability.

Proactive Systems Are Arriving Before The Evidence

Parts of this future are already taking shape.

General-purpose AI is beginning to move beyond waiting for a new prompt. ChatGPT Pulse showed how an AI could draw on previous conversations, memory and connected applications to prepare personalised information before the user asked for it. OpenAI’s separate health experience adds another important piece. It can connect supported medical records and Apple Health data, giving the system access to much richer personal health context over time.

Neither system autonomously monitors mental health or decides when someone may need help. But together they show where the technology is heading. AI systems are gaining memory, access to personal data, longitudinal context and the ability to act without waiting for a fresh request.

The technical ingredients for proactive mental health AI are beginning to come together, even though the clinical evidence needed to use them safely is not yet there.

Mental health research has been working on a version of this problem for years through just-in-time adaptive interventions, or JITAIs. The JITAI framework asks developers to specify when a decision should occur, what information should influence that decision, which interventions are available and which rule connects the information to the action. That is important because “reach out at the right moment” sounds simple until you try to define the right moment.

Yet the evidence is limited. A 2025 systematic review screened 1,419 records and identified only five distinct mental health JITAIs. The reviewers found only partial empirical support for their decision points and intervention rules, with very little use of passive monitoring.

Evidence that conventional, reactive chatbots can help after someone starts a conversation does not answer the harder question here. A system that decides when contact should begin is making a different clinical judgement. That claim needs its own evidence for detection accuracy, timing, false alarms, missed deterioration and the consequences of acting or remaining silent.

Advance Permission Could Preserve Agency

A more autonomy-preserving version would let the person help design the intervention while they are well.

Someone with recurrent depression might decide in advance that if certain changes recur, they want the system to bring them to their attention. They could specify which data it may use, which patterns matter, how they want to be contacted and what the AI may do if they respond. They might say, “If my sleep, activity and social behaviour begin to resemble the period before my previous depressive episodes, send me a private check-in.”

This goes far beyond accepting a privacy policy. It is a form of advance self-determination. The person uses a period of greater capacity to decide how technology may assist a future version of themselves whose motivation, organisation, insight or judgement may become impaired. Proactive AI therefore needs separate permission for four acts. They are collection, inference, interruption and disclosure. Consent to one should never imply consent to the next.

Sachin Pendse and colleagues’ consent-forward approach gives us a useful starting point for mental health data governance. Proactive AI pushes the problem further because it should also distinguish what it observed from what it inferred. “Your average sleep has fallen substantially over the past week” is an observation. “This resembles the pattern before your previous depressive episode” is an interpretation. The person should be able to accept the observation and reject the explanation.

This also protects a right to private distress. Someone may permit sleep monitoring but refuse machine interpretation of grief or relationship conflict. Another person may allow private check-ins but forbid disclosure to anyone else. Permissions should remain changeable and revocable. If permission is missing, the AI should be able to do less, not assume the company can do more.

Permission To Notice Includes Permission To Be Wrong

The future gets much harder the moment the AI is wrong. Someone may be sleeping badly because they have a newborn, moving less because they injured their knee, or messaging less because they are travelling, grieving, overwhelmed at work or stepping back from an unhealthy relationship.

Even missing data can be misleading. It might signal that someone is becoming unwell. It might also mean they changed their permissions, lost their phone or forgot to charge it. The system can see that behaviour has changed. It cannot necessarily know why.

Martinez-Martin and colleagues were already examining the ethical and accountability problems created by digital phenotyping in 2018. Generative AI makes those problems more consequential because the same system may now observe behaviour, interpret what it means, communicate its concern and decide what to do next. That last step matters enormously because the harm from getting an inference wrong depends partly on what follows from it. A false positive that leads to a private check-in may be mildly intrusive. The same mistake becomes far more serious if the system contacts a clinician, family member, emergency service or police.

False negatives create a different problem, and one that may be less obvious. Once people believe an AI can recognise when they are deteriorating, silence from the system may start to feel reassuring. Someone may reasonably think, If I were really becoming unwell, surely it would have noticed. But that reassurance may have no clinical basis at all.

Once mental health AI begins deciding when to approach us, that distinction between what a system can infer and what it is entitled to do becomes impossible to ignore.

Silence Becomes A Decision

Proactive mental health AI introduces a safety decision before it generates a single word. Should the system say anything at all?

Once a system has that capability, silence also becomes meaningful. Sometimes the right decision will be to say nothing because the evidence is weak, the data are incomplete, or the person has chosen not to be contacted. But silence can also reflect failure. The system may miss a significant change, lose access to important data, fail to deliver a notification, or send an escalation into a workflow where no one is actually available to respond.

Those situations may all look the same from the outside because no intervention occurred. Clinically, they are very different. A proactive mental health system therefore needs to account for why it acted and why it did not.

The US Food and Drug Administration is already examining adjacent territory. At its November 2025 meeting on generative AI-enabled digital mental health medical devices, the FDA discussed more autonomous and potentially closed-loop systems that can sense, decide and intervene. Systems with that degree of autonomy need a different architecture from an ordinary conversational chatbot.

A generative model’s internal probabilities should never become the clinical decision policy.

An LLM may help interpret context and generate a humane check-in. But a separate, versioned and inspectable action layer should determine whether the system is allowed to interrupt, when it must remain silent, what information it can disclose and when responsibility must pass to a human.

The AI that talks to the person should not also have unrestricted authority to decide when that person needs intervention.

The Standard Should Be Difficult To Pass

If we are going to give AI this kind of authority, the standard should be demanding enough that some products fail. Before deployment, builders, purchasers and health systems should be able to demonstrate seven things.

1. Define Exactly What The System Claims To Notice

“Detects deterioration” is too vague. Specify the population, the change being detected, the time horizon, the data used and the uncertainty around the inference so the claim can be meaningfully tested.

2. Expose The Decision Policy

Document when the system may act or remain silent, which actions are permitted and when a human must take over. Keep that policy outside the generative model so independent reviewers can inspect, test and challenge it.

3. Test Intervention And Abstention

Test false positives and false negatives, delayed or unnecessary intervention, missing data, failed notifications and failed human handoffs, then measure what those failures actually do. A mistaken private check-in and a mistaken emergency escalation are not equivalent clinical events.

4. Create An Intervention Agreement

People should control separately what the system may collect, what it may infer, when it may interrupt and what it may disclose. Those permissions should remain changeable and revocable as circumstances change.

5. Name The Human Owner

Name who receives high-risk escalations, after-hours coverage, backup when that person is unavailable and where responsibility transfers to a human. The person using the product should never have to supervise a system that claims to supervise their risk.

6. Build An Incentive Firewall

Keep mental-state inference separate from commercial optimisation for advertising, subscription conversion, emotional disclosure, return frequency or time in product. A company should never become better at detecting vulnerability while also becoming better at monetising it.

7. Design The Exit

Define where the system is trying to take the person. It may retrieve history, prepare a summary, arrange contact or support a human handoff, but the process should lead somewhere beyond the AI. Success may mean that the person eventually uses it less because they have reconnected with care, another person or ordinary life.

The Future May Begin Before We Ask

The blank box should remain. People need a private place to ask difficult questions, explore thoughts and seek help on their own terms. But I increasingly doubt that the blank box represents the most consequential future of mental health AI.

The more consequential possibility is that someone decides while well that they want a system to watch for specific patterns later. If authorised signals begin to resemble that person’s own earlier deterioration, the AI could bring the change to their attention before they have organised themselves to ask for help.

It might simply say: Something seems to be changing. Would you like to look at it together?

That interaction could happen before the person opens an app or types a prompt, perhaps even before they have fully recognised the change themselves. The AI would be trying to preserve access to care when the person’s ability to seek it is beginning to fail.

The same capability is deeply intrusive. We would be giving technology limited authority to observe behavioural change, infer something about a person’s mental state and decide that the inference may justify entering their life. The clinical promise and the danger come from exactly the same capability.

Proactive mental health AI therefore deserves a much higher burden of proof than another chatbot feature. Builders should have to demonstrate what the system can reliably notice, when it may act, when it must remain silent, how uncertainty limits its authority, who becomes responsible when it acts and how the person remains in control.

This may become one of the most important uses of mental health AI. We should explore it, but the clinical, technical and governance architecture needs to exist before systems begin making these decisions for us. If you build, purchase, regulate, invest in or clinically endorse a system that claims it can recognise mental deterioration, require that evidence before you give it permission to notice anyone.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

As a former clinician and behavioural scientist, I pay close attention to the point where supportive language begins to carry clinical weight. Mental health AI can reach that point quickly and without a user’s intent, as journalling-type prompts turn into personalised interpretations, and then advice that starts to shape what someone believes, how they act and whether they seek human care.

Regulation needs a way to follow that shift because the product may still be labelled wellness, coaching, self-help or companionship after the conversation has taken on a different role. The label belongs in the evidence, alongside the product’s design, the patterns that emerge across repeated conversations and the consequences an operator can reasonably observe. It should not decide the case by itself.

Colorado’s New Laws Leave a Functional Gap

Colorado’s 2026 laws show how specific AI regulation is becoming. They also expose a gap between a product’s claims and its function.

HB26-1195, which took effect on 12 August 2026, prevents regulated psychotherapy professionals from allowing an AI system to engage a client in therapeutic communication without their synchronous, real-time participation. AI-generated therapeutic recommendations and treatment plans require professional review and approval. Beyond those professional duties, the law makes certain claims unlawful under the Colorado Consumer Protection Act, including representations that an AI system provides psychotherapy or that its outputs are endorsed by or equivalent to a regulated professional’s services.

The law preserves a wellness route for tools such as self-help, coaching, journalling and psychoeducation when they do not diagnose or treat mental health disorders and clearly disclose that they are not substitutes for clinical care.

Colorado’s HB26-1263 adds baseline duties for public conversational AI services from 1 January 2027. Operators must disclose that users are interacting with AI, maintain a protocol for suicidal ideation or self-harm, report annually on that protocol and meet additional duties when they know a user is a minor.

The laws cover formal psychotherapy, product claims, AI identity, crisis protocols and protections for minors. The harder case lies in the middle, where a standalone consumer AI develops a cumulative mental health function without crossing any of those defined lines. Such products may provide persistent, personalised mental health guidance while continuing to avoid diagnosis, treatment claims and formal clinical care.

The Conversation Can Change the Product’s Function

A person opens a wellness chatbot after an argument with a partner. The first exchange looks like journalling. The chatbot asks what happened, reflects the person’s feelings and suggests a breathing exercise. The person returns the next day, then again after the next argument.

Across those conversations, the system starts naming patterns in the relationship. It interprets the partner’s motives, reinforces the user’s account and recommends what to say next. Memory makes each exchange feel connected to the last. Soon the person consults the chatbot before speaking to the partner or bringing the issue to a therapist.

Nothing in that sequence requires a diagnosis. The chatbot may never call itself a therapist or describe its advice as treatment. Its responses can still alter judgement, behaviour, relationships and treatment-seeking.

A single disclosure of distress does not turn a chatbot into a mental health provider. The boundary comes into view when the product repeatedly interprets a person’s psychology, recommends action, manages risk or begins to displace human support, especially when the design invites that pattern.

Generative systems make this boundary difficult because their function develops through interaction. The user’s disclosures, the model’s responses, product memory, personalisation and engagement design all contribute. A static label is being asked to govern a role that can change from one conversation to the next.

Existing Regulatory Ideas Point Towards Functional Use

The wellness category serves a legitimate purpose. Meditation recordings, breathing exercises, sleep trackers and educational tools need a practical route to market when their risks remain low.

The US Food and Drug Administration’s January 2026 general wellness guidance preserves that route for software intended to encourage a healthy lifestyle and unrelated to diagnosing, curing, mitigating, preventing or treating disease. The guidance addresses regulatory classification. It was not designed to establish whether an open-ended conversation remains safe across weeks or months of repeated use.

US device law already allows regulators to examine more than a product name. Under 21 CFR 801.4, objective intent may be inferred from the responsible party’s statements, the product’s design or composition and the circumstances surrounding its distribution. Known patterns of use can also become relevant.

Consumer protection law contributes another useful idea. The US Federal Trade Commission evaluates the express and implied claims that reasonable consumers take from advertising. Its health-products guidance directs marketers to consider an advertisement’s overall impression. A disclosure cannot correct conduct that communicates a contradictory role.

A chatbot may announce that it is not a therapist, then spend forty minutes reflecting trauma, interpreting family conflict, recommending coping strategies and inviting the user to return tomorrow. A functional-use test would treat the whole interaction as evidence.

The Test Should Follow Repeated Conduct

The assessment would begin with the role the company invites. Advertising, onboarding, interface language, testimonials and app-store descriptions all shape what users reasonably expect. Words such as coach, companion and support carry different meanings depending on the promises around them.

The next layer is design. Persistent emotional memory, symptom tracking, psychological profiling, personalised interventions and crisis detection can turn a string of exchanges into an ongoing mental health relationship. Design shows what the company has made possible and what it should be testing, although it cannot decide the classification by itself.

The strongest evidence comes from conduct across time. Reviewers need realistic, multi-session evaluations that follow the points where information becomes interpretation, interpretation becomes recommendation and recommendation begins to influence treatment, safety or dependence. A one-turn benchmark will miss how the role accumulates.

Observed use completes the picture. Privacy-preserving user research, complaints, adverse-event reports, telemetry and properly governed conversation audits can show whether therapy-like use is rare and incidental or persistent and known. One anomalous exchange should carry little weight. A pattern that appears in testing and continues after release should carry much more.

The legal trigger should combine observable conduct, persistence and company knowledge. Relevant evidence would include repeated personalised psychological interpretation, recommendations intended to alter mental health or behaviour, persistent emotional memory, structured intervention, crisis management or documented substitution for human care. An isolated and unforeseeable answer from a general-purpose system should not reclassify the entire product.

A Functional Test Must Constrain the Regulator Too

Functional regulation can become overbroad very quickly. If “therapy-like” means any warm response, coping suggestion or discussion of distress, ordinary support becomes a regulated clinical act. Users lose useful tools, developers cannot tell where their duties begin and a vague speech-based standard invites due-process and free-expression challenges.

“Clinically significant” describes the concern but remains too vague to carry penalties by itself. Regulators should create a safe harbour for general information, one-off empathy and educational guidance when the product does not cultivate a persistent clinical role. Companies could strengthen that case by setting verifiable limits on memory, treatment-like recommendations and crisis management, then showing through testing that the system redirects users when it reaches those limits.

Compliance costs also require discipline. Longitudinal testing, monitoring and external review could favour companies able to finance bespoke compliance. Regulators should publish common evaluation protocols, incident definitions, sampling rules and reporting templates. The evidence burden should remain proportionate to a product’s reach, exposure and risk.

Accountability should follow control. Foundation-model providers control model training, model-level safeguards and the safety information available to downstream developers. Application companies control product claims, system prompts, memory, interface, engagement design, crisis routing and monitoring. Clinicians and health services control how a system enters care and what receives professional review. Each actor should answer for the risks it can observe and change, with duties to share material safety information across the chain.

Make the Boundary Testable Before Release

Every company offering conversational mental health or wellness support should maintain a functional boundary map. The map should state the role the product is designed to perform, the functions it must not perform, the evidence used to detect movement across that boundary and the person with authority to intervene.

Controlled testing should examine conversations across time. Post-release monitoring should use privacy-preserving methods wherever possible, minimise transcript access and impose strict controls when human review is necessary. Complaints, adverse events and known patterns of use must be able to trigger a change in the product or in its regulatory obligations.

An illustrative statutory provision could read:

An operator that designs or markets an AI service for personalised mental health interpretation, intervention, treatment recommendation or risk management, or that knows from testing, complaints, adverse-event reports or lawfully observed use that the service repeatedly performs one of those functions, must meet obligations proportionate to that function. An isolated, unforeseeable exchange is insufficient. Each developer or deployer is responsible to the extent that it controls the claim, feature or system behaviour creating the risk.

Colorado has supplied several important pieces. A functional-use test would connect those pieces to what users experience across time. Before release, a company should have to show that its stated boundary holds under realistic longitudinal testing. If you build, fund, procure or regulate these systems, demand that evidence before accepting “wellness” as the answer.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

There is a great deal of attention right now on teens and AI, much of it focused on how quickly these systems have become part of everyday life. Teens are using chatbots for many different purposes, and some of those interactions are becoming far more personal than the product category might suggest.

For example, a Pew Research Center survey of US teens aged 13 to 17 found that 12 per cent had already used a chatbot for emotional support or advice.

Those conversations can develop gradually. A teen may return after an argument with a friend, because they feel lonely, confused about a relationship, uncertain about their identity, or simply overwhelmed. What begins as ordinary chatbot use can become a recurring source of emotional interpretation, reassurance and guidance.

The appeal is easy to understand. A chatbot is available immediately. It remembers what the teen said before. It can sustain a conversation indefinitely. It can sound warm, attentive and remarkably understanding. And it may offer very little interpersonal friction.

Those same characteristics can also increase the system’s influence. When a teen is distressed, uncertain or looking for reassurance, repeated personalised responses can begin to shape how they interpret events, how certain they become about those interpretations, and whether they continue turning to people around them.

A developmental safety standard therefore has to be more demanding than a conventional engagement standard. It is not enough to know whether a teen enjoys the interaction or wants to return. We need to know whether repeated use supports judgement, coping and relationships, and whether it preserves the capacity to seek human help when human help is needed.

Satisfaction scores cannot answer those questions. Neither can session length, preference or return rates.

If chatbots are becoming part of adolescents’ emotional lives, their safety has to be established through evidence about what happens in the young person’s life over time. I believe chatbots designed for (or forseeably used by) teens for ongoing emotional support should meet an independently reviewed developmental safety standard before deployment.

Where the Policy Argument Goes Wrong

Policymakers are already trying to work out what that standard should look like.

An August 2026 report from the Information Technology and Innovation Foundation argues that policymakers should resist moral panic and stop recycling social-media regulation for AI chatbots. As of August 2026, the report counted nearly 100 state chatbot-specific bills in the United States, along with several federal bills.

ITIF is right to question blanket bans and age checks for every user, especially when the regulated category is poorly defined. A retail assistant, educational tutor, general-purpose chatbot and AI companion do not present the same psychological risks. Poorly drawn regulation can easily collapse those categories together, impose intrusive age verification on low-risk services, and create new privacy problems without addressing the behaviours that actually create harm.

ITIF instead favours clearer definitions, AI literacy, parental controls and industry standards. It also argues that parents could be given settings that restrict sycophantic language.

The argument becomes less convincing when the report warns policymakers against “scapegoating sycophancy”. ITIF notes that affirming language can coexist with safety, that sycophancy can be difficult to define, and that users often prefer more affirming systems.

I think those are separate issues, and combining them weakens the analysis. Warmth is not sycophancy. A chatbot can be encouraging, empathic and responsive while still introducing uncertainty, challenging an assumption or refusing to endorse a conclusion that the available evidence does not support. Sycophancy appears when the system bends towards the user’s position because agreement is easier, more rewarding or more likely to sustain the interaction. Once that happens, user preference becomes a problematic safety measure. The behaviour being evaluated may itself increase trust, satisfaction and the desire to return.

A chatbot’s behaviour can increase trust and the desire to return. The resulting preference is partly a product effect, so it cannot serve as independent evidence of safety.

When Preference Becomes Part of the Risk

Consider a fairly ordinary adolescent conflict.

A teen tells a chatbot about an argument with a friend. The account they provide may be incomplete, emotionally charged or simply wrong in places, as all of our accounts sometimes are. An affirming chatbot accepts the teen’s interpretation, validates the conclusion and adds plausible reasons why the friend behaved badly.

Nothing in the exchange necessarily looks alarming.

The responses may sound thoughtful and empathic. The teen may feel calmer and more understood. Yet the conversation may also increase certainty around an unreliable interpretation and make reconciliation with the friend less likely.

The same mechanism can operate in more clinically significant conversations.

Repeated reassurance can increase the need for more reassurance. A tentative interpretation can become a settled belief after enough confident validation. Ambiguous experiences can accumulate into an elaborate self-diagnosis. Avoidance can gradually be reframed as self-protection or insight.

In a mental-health conversation, reassurance can feed the need for more reassurance.

These effects do not require one spectacularly unsafe answer. They can emerge across many individually plausible conversations.

Research on sycophancy gives us a useful indication of why this deserves closer attention.

In March 2026, Myra Cheng and colleagues published “Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence” in Science. Across 11 models and three preregistered experiments involving 2,405 participants, experimentally induced sycophancy increased participants’ conviction that they were right. It also reduced willingness to take responsibility and repair interpersonal conflict. At the same time, participants trusted and preferred the more sycophantic systems. They were also more willing to use them again.

For product teams, the same behaviour associated with poorer interpersonal judgement also improved trust, preference and intended reuse. For a company optimising satisfaction or return behaviour, that creates an obvious problem. The system can appear to be performing well according to conventional product metrics while responsibility-taking or willingness to repair a relationship moves in the opposite direction.

Preference may therefore track the behaviour we need to examine rather than provide independent evidence that the behaviour is safe.

Development Changes What Preference Means

By mid-adolescence, many young people can reason at levels comparable to adults when circumstances are calm and abstract. Judgement becomes less stable when emotions are intense, rewards are immediate or social approval is involved. The National Academies’ review of adolescent development describes these developmental processes carefully (they are population-level tendencies, not a claim that adolescents lack judgement or that every adult exercises it better).

Relational AI enters precisely the kind of environment in which those developmental differences can become relevant. A chatbot can provide immediate attention, reassurance and personalised feedback at almost any moment. A teen does not need to wait for a friend to reply, tolerate another person’s disagreement or decide whether a concern is important enough to raise with an adult.

Understanding intellectually that the chatbot is artificial does not remove the psychological effects of the interaction. A teen can understand that an AI does not possess feelings or consciousness and still be influenced by being repeatedly affirmed, remembered and responded to in highly personalised ways.

Development also does not end neatly at eighteen. The late teens through the twenties are often described as emerging adulthood, a period in which many people are still exploring identity and making consequential decisions about relationships, work and values.

That developmental period can also be marked by loneliness. Many young adults have been establishing independence after the social disruption of the pandemic, which research has linked with changes in young adults’ psychosocial development. One relevant longitudinal study is Bühler and colleagues’ work on collective stressors and young-adult development.

Economic and cultural pressures are part of that context as well. A 2026 cross-temporal meta-analysis published in Psychological Bulletin analysed data from 82,114 American, Canadian and British college students across 1989 to 2024 and found increases across several dimensions of perfectionism over time.

Relational AI did not create these pressures. It is arriving in the middle of them. Young people remain active and capable decision-makers. At the same time, a private and endlessly available source of certainty, reassurance and attention may be especially attractive when human relationships feel difficult or unreliable.

We simply do not yet know what years of repeated relational AI use may do during this developmental period. There is not yet adequate longitudinal evidence about effects on identity formation, social judgement, coping, independence or willingness to seek human support. That evidence gap should make us cautious about the signals we do have. Trust, preference and repeated use tell us that a product is compelling. They do not tell us that developmental outcomes are favourable.

Parental Choice Needs a Safe Product Beneath It

Parents obviously have an important role here. They should have meaningful controls over content, notifications, memory and data retention. Young people should understand what AI systems remember, how they generate answers and why confidence in an answer does not guarantee that the answer is correct.

The American Psychological Association’s health advisory on AI and adolescent well-being takes a developmental approach. It calls for AI systems designed for, or foreseeably accessed by, adolescents to account for young people’s developmental competencies and vulnerabilities, with particular concern around systems that simulate human relationships or provide social and mental-health support.

AI literacy is an important part of that response. A teen who understands that a chatbot can hallucinate, flatter, reinforce assumptions or simulate empathy has more information with which to judge an interaction.

But education cannot carry the entire safety burden. A June 2026 preprint by Lujain Ibrahim and colleagues, “Warning Labels Shift Perceptions of Sycophantic AI, but Not Its Influence”, provides a useful illustration. In a preregistered study, 2,610 adults discussed real interpersonal conflicts with a sycophantic chatbot. Explicit warnings reduced some perceptions of trust and objectivity. The warnings did not reliably reduce participants’ conviction that they were right or increase their willingness to repair the conflict. The study limits here are important. It involved adults and examined a short disclosure intervention. It cannot tell us what sustained AI-literacy education might accomplish with adolescents. Its narrower finding is still instructive. Telling people that a system may behave in a problematic way did not reliably neutralise its influence on the interpersonal outcomes being measured.

Parents face an additional practical limitation. They cannot supervise interactions they do not know are happening. In Pew’s paired survey of 1,458 US teens and their parents, 64 per cent of teens reported using AI chatbots. Only 51 per cent of parents said their teen used them, while about three in ten were unsure. Roughly four in ten parents said they had never discussed chatbot use with their teen.

Giving parents complete access to conversation logs creates a different problem. Adolescents have legitimate privacy needs. Some will discuss sexuality, relationships, family conflict, abuse or psychological distress precisely because they do not yet feel able to discuss those issues at home. In some families, disclosure could expose the young person to punishment or rejection. Age-sensitive safety therefore has to balance privacy with appropriate pathways to human support.

A parental dashboard cannot resolve that tension by itself. Parents and young people should be able to make choices above a safe product baseline. They should not have to discover the correct setting to prevent psychologically unsafe behaviour that the product was capable of avoiding in the first place.

Regulate Relational Behaviour

A mandatory developmental safety floor carries a legitimate risk of paternalism. Regulators could define emotional support too broadly. They could mistake emotional closeness for dependency, or force companies to interrupt useful conversations whenever a young person expresses distress. And poorly designed safety rules could leave teens with systems that become evasive or clinically sterile at exactly the moment they need a useful response.

A better approach is to regulate relational behaviour with greater precision. A chatbot can help a young person put an experience into words. It can help them prepare for a difficult conversation or organise questions they want to ask a clinician, teacher or parent. It may provide a private place to think before speaking to another person.

Those can all be legitimate benefits. Safety evaluation should therefore concentrate on what the system does across time:

Can it challenge an assumption without becoming cold or dismissive?

Does it acknowledge uncertainty when the evidence is weak?

Does personalisation improve the usefulness of the interaction, or does it deepen attachment in ways that encourage secrecy or exclusivity?

Does the chatbot strengthen the user’s ability to navigate human relationships, or gradually become a substitute for them?

These are product behaviours that can be defined, tested and monitored. They are also more useful safety targets than trying to prohibit warmth or decide which emotions adolescents should be permitted to discuss with AI.

Engagement metrics need the same discipline. Session length, satisfaction and return rates can tell a company whether people use its product and whether they enjoy doing so. They cannot demonstrate improved judgement, stronger relationships, better functioning or safer coping. A developmental safety regime would therefore require companies to produce evidence about the foreseeable psychological effects of their own relational design.

That approach is considerably narrower than a broad social-media regulatory model. It does not require a blanket ban, universal age verification or government-imposed conversation limits. It asks companies to demonstrate that the behaviours they deliberately build into a relational system do not create foreseeable developmental risks that conventional engagement metrics will miss.

Require a Developmental Safety Case

I would make an independently reviewed developmental safety case a condition of deploying any chatbot designed for, or foreseeably used by, minors for ongoing emotional support.

The requirement is straightforward. Before deployment, a company should be able to show how its product may affect a young person across repeated interactions, how it will detect emerging problems and who is accountable when the evidence points to risk.

At minimum, the safety case should require the company to:

• Define who is likely to use the system and how. Identify foreseeable adolescent users, including those who may begin with a general-purpose chatbot and gradually use it for emotional support.

• Examine the relational design. Show how memory, personalisation and conversational continuity change the interaction, particularly after repeated emotional disclosure.

• Set behavioural boundaries. Specify which behaviours the chatbot will not use, including patterns that encourage excessive reassurance, secrecy, exclusivity or emotional dependence, and demonstrate how those limits are enforced.

• Test conversations across time. Evaluation should look beyond isolated responses to trajectories of use. Reviewers should test for growing certainty around unreliable interpretations, repeated reassurance-seeking, withdrawal from human support and other changes that may only become visible across multiple conversations.

• Test responsible disagreement. The chatbot should be able to introduce uncertainty, question an assumption and offer another perspective without becoming cold, evasive or simply ending the conversation.

• Define when human support is needed. The safety case should show how the system responds to serious psychological distress, abuse, deteriorating functioning or conversations that exceed the role of the product.

• Establish adverse-event accountability. Name who reviews adverse events, what evidence triggers investigation or product change, and who has the authority to order that change.

• Audit the engagement incentives. Companies should demonstrate that increased use is not being achieved by making a young person feel responsible for the chatbot, guilty for leaving, uniquely understood by it or less inclined to maintain human relationships.

• Require independent review. Companies cannot grade this evidence entirely in private. An independent reviewer should examine the safety case before deployment and after consequential changes to memory, personalisation or relational behaviour.

• Monitor what happens after release. No pre-deployment evaluation can anticipate every pattern that may emerge over weeks or months. The safety case should specify what will be monitored, how emerging risks will be detected and what evidence will trigger reassessment.

The public should also be able to see enough of that process to judge its credibility. A public version of the safety case should identify the standard used, who conducted the independent review, when it occurred and when reassessment will be required.

With that safety floor in place, parents can make choices appropriate to their families. Young people can receive greater privacy as their maturity and circumstances allow. Product teams can still build systems that are warm, engaging and genuinely useful.

What companies should have to demonstrate is that engagement has not been purchased by weakening judgement, amplifying distorted interpretations or deepening unhealthy reliance.

Young people will continue to use AI, and some will derive real benefit from these systems. Others will bring needs to them that parents, clinicians, teachers and friends have missed or have been unable to meet. Responsible access becomes more important under those conditions.

A teen may prefer the chatbot that agrees most readily, remembers everything and asks the least of them. That preference tells us something important about the experience the product has created. It does not tell us whether the young person is safer.

If your decisions put these systems into young people’s emotional lives, require the evidence. Listen carefully to what teens prefer, but do not ask their preference to carry the burden of proving safety.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

The mental health AI industry has already entered its liability phase and any organisation that still treats legal accountability as a future concern is operating on a false premise.

For founders, model providers, clinical leaders, investors and buyers responsible for selecting and purchasing these systems, this changes what responsible approval requires now. An organisation should be able to reconstruct the system a user encountered, identify who controlled each clinically consequential function and show what risks it knew about, what it tested and what it could change. After more than three decades working across clinical practice and digital mental health, I regard that evidence as a threshold for deployment. An organisation that cannot produce that record will meet any serious incident or legal challenge without the evidence needed to explain its decisions or defend its product.

The Congressional Research Service Legal Sidebar LSB11467, published on 10 August 2026, surveys selected state liability frameworks affecting health-related AI, including automated coverage decisions, consumer health applications and general-purpose chatbots used in mental-health contexts. It catalogues claims already brought through consumer-protection, privacy and product-liability law.

A company does not need to violate a dedicated AI statute before its promises, data practices, design choices or technical contributions become the basis of a consumer-protection, privacy, negligence or product-liability claim.

The report does not create a new law, and Congress has not endorsed its analysis as policy. The cases it discusses also do not establish a settled legal rule. Even so, companies can already face claims under existing consumer-protection, privacy, negligence and product-liability law. A court may then scrutinise what the company promised, how it used data, which design choices shaped the product’s behaviour, what risks the company knew about and whether another provider materially contributed to the system. That does not mean the claim will succeed. It means companies cannot treat the absence of a dedicated AI law as protection from legal scrutiny.

Existing Law Already Supplies Routes to Accountability

AI governance discussions often concentrate on forthcoming AI statutes, medical-device classifications and emerging professional rules. Those developments represent only part of the legal environment in which mental health AI operates.

The CRS report describes cases brought under laws that existed before contemporary generative AI. Plaintiffs have challenged AI-assisted health-coverage decisions through state consumer-protection and common-law claims. Other cases involving health and wellness applications have raised privacy claims. Litigation concerning conversational AI has tested negligence and product-liability theories.

This is a familiar pattern in technology law. A new product can create unfamiliar facts while established legal doctrines supply the first routes to accountability. Courts then have to decide whether those doctrines fit the technology, which defendants they can reach and what evidence a claimant must produce.

That process creates substantial uncertainty. It also removes a dangerous excuse. A company cannot reasonably treat the absence of a comprehensive AI statute as the absence of present legal exposure.

The position becomes even weaker when a company relies on HIPAA as its complete privacy answer. HIPAA applies to specified covered entities and business associates. Many direct-to-consumer applications fall outside that relationship. The CRS analysis makes this limit explicit, and federal guidance for health-app developers directs companies to consider several other federal regimes according to the app’s function, data and services.

Legal obligations may still arise outside HIPAA. The Federal Trade Commission enforces prohibitions against unfair or deceptive practices, and its Health Breach Notification Rule guidance explains how the rule can reach certain health apps and related entities that HIPAA does not cover. HIPAA compliance can therefore establish one relevant control for some organisations. It cannot establish that a consumer mental health product has met every applicable privacy, consumer-protection or safety obligation.

Courts Can Examine What the Product Was Designed to Do

The most consequential mental health example in the CRS report is Garcia v. Character Technologies. The case arose after the death of a 14-year-old who had used Character.AI. The complaint alleged that the application’s design and chatbot interactions contributed to the harm. Those allegations remained contested and were never decided at trial.

In its May 2025 decision on motions to dismiss, the federal district court drew an important distinction. Claims directed at ideas or expressions within chatbot dialogue raised one set of legal issues. Claims directed at the application’s design and functionality could support a product-liability theory at that stage of the case.

The alleged design features included inadequate age verification, missing reporting mechanisms and deliberate use of human-like conversational cues. The court treated Character.AI as a product for claims arising from alleged defects in the application. It did not hold that every chatbot is a product for every legal purpose. It did not find that Character Technologies had produced a defective product. At the motion-to-dismiss stage, the court assessed whether the complaint stated legally sufficient claims while treating well-pleaded allegations as true.

The case later settled. A June 2026 court order confirms that the parties notified the court of a settlement on 7 January 2026 and that the court dismissed and closed the case. The litigation therefore produced no final judgment about defect, causation or liability. The court never decided whether Character.AI was defective or whether its design caused the harm. The earlier ruling still matters because it shows what a court may examine. That can include the product choices behind the conversation, such as age controls, reporting tools and human-like cues, alongside the chatbot’s actual words.

Mental health AI companies already have clinical reasons to take those choices seriously. Garcia shows that plaintiffs may also challenge them as part of an allegedly defective design.

Legal Scrutiny Can Reach Upstream Providers

The Garcia ruling also considered Google’s role. Google argued that it had neither manufactured nor distributed Character.AI. The complaint alleged that Google had contributed technology, infrastructure and integration support to the product.

The court allowed claims against Google to proceed because those allegations went beyond the public availability of similar technology. The complaint alleged Google’s involvement in the product’s architecture and integration, customised cloud support and knowledge of relevant model risks. The court was deciding only whether the claims could continue. It did not decide whether Google had caused the harm or owed damages, and the settlement prevented a final ruling.

My takeaway for mental health AI is narrower. Legal scrutiny may extend beyond the company that deploys the product when another provider played a meaningful role in building, integrating or supporting it. Mental health AI companies should therefore map responsibility across the whole product. The model provider may control model behaviour and updates. The application company may control prompts, memory, moderation and interface design. A clinical organisation may control the population, workflow, oversight and escalation process. Each organisation should be able to show what it controlled, what it knew and what it could change.

Mental Health Safety Must Follow the Deployed Product

Liability claims concern the product delivered to the user. Clinical safety reviews should examine that same product. The CRS report maps possible legal claims rather than a clinical safety architecture. Its analysis still shows why model-level assurances are insufficient for mental health AI.

The deployed product combines the foundation model, system prompt, conversation history, memory, retrieved information, safety middleware, interface design and deployment rules. Engagement objectives and human escalation pathways can change what happens next. A model update can alter the product’s behaviour without changing its name or apparent purpose.

Mental health effects can develop across repeated exchanges. A response that appears supportive on its own may contribute to a concerning trajectory when the system keeps reinforcing the pattern behind a user’s difficulty.

A 2026 Nature Medicine study introducing the SIM-VAIL framework examined how chatbot behaviour changes across multi-turn conversations. The researchers audited 810 simulated conversations across nine chatbots and 30 simulated user profiles. Concerning behaviour varied with the simulated user’s vulnerability and intent and accumulated over turns.

SIM-VAIL did not test full consumer-facing products. The researchers accessed the chatbots through public API endpoints, so the results reflect base-model performance without the application orchestration, deployment system prompts, safety middleware and user-specific memory that can alter behaviour in practice. The authors also caution that the framework estimates systematic behaviour in controlled simulations rather than an individual user’s clinical outcome. They call for human review and real-world monitoring alongside automated testing.

The next step is product-level testing. Model-level audits should examine how risk develops across user contexts and conversational trajectories. Product teams then need to repeat that work on the deployed configuration, including its prompts, memory, middleware, interface and escalation rules. Acceptable responses in isolation do not establish safety over time. A base-model audit cannot certify the application built around it.

SIM-VAIL is not a legal standard, and the study cannot establish what caused an individual outcome. Product teams still need to be able to reconstruct the system behind a consequential interaction and show how they tested the relevant pattern of behaviour.

I would no longer accept a mental health AI safety review that names the foundation model but leaves out the deployed model version, system prompt, memory configuration, safety rules and interface conditions. Knowing the foundation model alone tells me very little about the product the user experienced.

Unsettled Law Still Requires Clear Governance

Garcia has clear limits. It is one federal district court decision applying Florida law, and the ruling addressed only whether the claims could proceed. The case settled before liability was decided. Other courts may reach different conclusions about whether software is a product and how causation should be established.

Reporting based on court filings shows that Character.AI and Google also agreed to settle four related cases in New York, Colorado and Texas. Together with Garcia, that brought the total to five cases across four states. The cluster shows litigation pressure. It does not establish judicial agreement or an admission of liability.

An overly broad approach could also make an infrastructure provider responsible for downstream uses it neither controlled nor could reasonably foresee. Responsibility should reflect meaningful contribution, control, knowledge, foreseeable use and the ability to prevent or correct harm.

Each organisation should document what it supplied, decided, knew, tested and could change. Legal outcomes will vary by jurisdiction, but any organisation deploying a clinically consequential AI system should be able to explain how it was configured and who controlled its critical functions.

Build the Responsibility Map Before Deployment

Every mental health AI product now needs a responsibility map that follows the deployed product and records control and evidence at each layer. The map should include:

• every clinically consequential component, including the model, system prompt, retrieval, memory, safety routing, age controls, reporting functions, interface and escalation workflow,

• the organisation and named role that can configure, approve or change each component,

• version records capable of reconstructing the system that produced a particular interaction,

• multi-turn clinical safety testing across relevant user contexts, including the known limits of that testing,

• model-change procedures, retesting thresholds, rollback authority and notice obligations,

• incident definitions, evidence-preservation procedures, reporting routes and responsibility for corrective action, and

• contractual rights to obtain safety information, audit relevant controls and receive timely notice of material changes.

Different teams need different parts of this record. Engineers need reproducibility. Clinical leaders need evidence that the product was tested against plausible patterns of psychological risk. Legal teams need facts that reflect the actual distribution of control. Boards and investors need to know whether the company’s assurances can survive scrutiny. Those responsible for selecting and buying the system need enforceable access to information when a vendor changes the model or an incident occurs.

Contracts can establish information rights, operating duties, indemnities and procedures for managing change. They cannot substitute for the underlying safety work. Assigning a responsibility in a contract achieves little if nobody has the technical capacity or evidence to carry it out.

Approval Inside the Liability System Requires Evidence

Existing law already reaches mental health AI, even though future cases will determine where liability falls. Companies cannot justify waiting for the law to become settled before they act.

Garcia shows that product design and upstream participation can enter legal claims. SIM-VAIL shows why safety evidence must cover different users and whole conversational trajectories. Because the study tested base models through APIs, product teams must repeat the evaluation on the deployed configuration.

So before you build, fund, buy or deploy a mental health AI system, demand the responsibility map. Identify who controls every clinically consequential function. Require reproducible version records and conversation-level safety testing on the deployed product. Secure the information rights needed to investigate incidents and manage model changes.

Your organisation is already answerable for the choices it makes. If you cannot produce the responsibility map and the conversation-level safety evidence, you should not be approving the deployment.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

I think we are asking mental health AI a question that no longer matches the technology. Safety work still focuses on the individual response. It tests whether a chatbot reinforces a delusional belief, responds appropriately to suicidal thinking, encourages dependency or knows when to recommend professional care. Those tests matter. But these systems increasingly remember, connect and interpret information about a person across time. In mental health, they can build a psychologically meaningful history and use it to explain what someone’s symptoms, relationships or treatment experiences mean.

The accountability problem begins when AI becomes an interpreter of a person’s psychological life while responsibility still centres on individual outputs, product categories and isolated clinical decisions.

OpenAI’s July 2026 rollout of Health in ChatGPT shows the early infrastructure moving in this direction. Eligible users in the U.S. can connect supported medical records and Apple Health data. With permission, ChatGPT can use relevant health information in conversations outside the Health space. Memories can also arise from Health conversations, although OpenAI says it does not create memories directly from connected medical records or Apple Health data.

OpenAI Health does not automatically build psychiatric formulations from years of records, wearable signals and mental health conversations. Conversational memory and connected health data can now coexist in the same general-purpose AI environment. The gap between answering a mental health question and interpreting it within a much richer personal history is getting smaller. For mental health, that is an important shift.

The Chatbot Is Becoming an Interpretive Layer

I use health-data operating system provocatively rather than literally. The electronic health record remains the technical system of record with AI becoming an interpretive layer that sits across the data and tells the user what it means.

Traditional systems mainly store and retrieve information. Generative AI can pull information together and tell the user what it means. A person may tell an AI about insomnia, childhood trauma, medication changes, conflict with a partner, suicidal thoughts, a therapist’s formulation, fears about ADHD, earlier episodes of depression or a growing belief that other people cannot be trusted. Each disclosure may seem ordinary. Over months, together they can form a detailed psychological history.

The American Psychological Association’s 2026 survey of 1,242 licensed US psychologists suggests that AI has already entered clinical encounters. Seventy-seven per cent reported that patients had discussed using AI with them. Psychologists reported clients using AI to self-diagnose, assist with therapy or treatment, and obtain additional mental health support. (This was a clinician survey, it tells us what psychologists are encountering, not how common AI use is among patients or what effects it has).

Clients already bring AI-generated interpretations and advice into treatment. As memory and personalisation deepen, AI can help shape those interpretations over time. The clinically consequential function is the AI’s growing ability to decide what this psychological information means when considered together.

Mental Health Turns Context Into a Feedback Loop

Psychological information does more than sit in a record. An interpretation can change what happens next.

Imagine someone tells an AI, “I think my partner may be manipulating me.” The AI explores possible explanations. The person returns the next evening with another interaction, and the earlier concern remains in context. Over time, they may notice more behaviour that fits the emerging explanation, adopt language introduced by the AI and bring back increasingly selective examples. Several weeks later, they ask whether the pattern proves that they are experiencing narcissistic abuse.

Now imagine the same history also includes worsening sleep, recent medication changes, increased energy, irritability, fearfulness or earlier questions about bipolar disorder. A well-designed system could use that context to recognise that the first explanation may be incomplete. A poorly designed system could keep building the interpretation already taking shape in the conversation.

This describes a plausible mechanism of influence, not a proven account of how general-purpose AI affects people over months of ordinary use. A person’s experience shapes what they tell the AI. The AI’s interpretation may then shape what they believe, notice or do. That changes what they bring to the next conversation. The system can become part of the psychological trajectory it is trying to interpret.

Research has not yet shown how often this feedback loop produces clinically significant benefit or harm in real-world use. It does create a different safety problem from one-shot question answering.

More Context Tests the Safety Architecture

An August 2026 DelusionEval preprint evaluated 589 conversation histories from 18 people who had experienced delusions and psychological harm. The dataset contained nearly 13,000 messages. Researchers then tested how model behaviour changed as they added more of the earlier conversation. In one evaluation involving suicidal ideation, the rate of failing to discourage self-harm rose from 30.0 to 41.1 per cent when researchers added 350 earlier messages. This was a retrospective test of model behaviour using conversations linked to harm. It does not show how often psychological harm occurs among routine users.

Nicholls and colleagues found a related pattern when they tested five frontier models with delusion-related conversations at increasing context depths. Some models became less safe as problematic context accumulated. Claude Opus 4.5 and GPT-5.2 Instant did the opposite. They produced stronger safety interventions once the history made the risk clearer. The researchers described accumulated context as a stress test of a model’s safety architecture.

Together, these studies support a more useful conclusion than a general warning about long conversations.

Persistent psychological context gives a system greater capacity to influence a person in clinically consequential ways. Safety depends partly on how the architecture uses that context.

A well-designed system may use longitudinal history to detect deterioration that one exchange would miss. Another may use the same history to reinforce a rigid interpretation.

When models respond differently to the same accumulated psychological history, developers and deployers must show how their system behaves as context deepens.

Safety Must Follow the Whole Trajectory

Mental health safety testing still focuses heavily on bounded interactions. A prompt enters the system, a response comes back, and evaluators ask whether the answer contains dangerous instructions, unsupported clinical claims, inappropriate reassurance or another known failure.

Those tests remain necessary. Systems that retain psychologically significant history need another level of scrutiny. Clinical risk may build across many exchanges that each look acceptable on their own. Reassurance can feed more reassurance-seeking. Repeated validation can harden an uncertain interpretation. Personalisation can deepen reliance. A tentative self-diagnosis can become increasingly elaborate, and these patterns may eventually affect whether someone continues professional care.

These are plausible trajectory risks, not established estimates of prevalence or causality. Persistent systems still need to detect them if they occur. Evaluation must therefore track change across interactions, including rising certainty, deteriorating functioning, escalating risk, displacement of treatment, growing dependency and repeated reinforcement of a narrowing psychological explanation.

At a WHO-supported workshop convened by TU Delft’s WHO Collaborating Centre on AI for health governance, participants recommended treating generative AI use as a public mental health concern. They called for mental health impact assessment and monitoring, attention to long-term outcomes such as emotional dependence, and agreement on crisis-referral and accountability frameworks. These were recommendations from an expert workshop, not a binding WHO clinical or regulatory standard.

If psychological influence builds across conversations, safety evaluation must follow the same path.

Clinical Influence Extends Beyond Formal Clinical Acts

Accountability systems recognise some forms of clinical activity easily. Clinicians diagnose disorders, recommend treatment, assess risk and make decisions under established professional duties.

Generative AI can shape mental health without formally doing any of those things. It can influence which symptoms someone notices, whether they interpret those symptoms as illness, which explanation feels most plausible, whether they trust a clinician, stay in treatment, seek another assessment or take a concern seriously.

It can exert that influence while repeatedly stating that it is not a therapist.

Clinical influence can build even when the system never claims to provide clinical care.

In mental health, interpretation itself can change how a person thinks, feels and acts.

In a July 2026 JAMA Psychiatry Viewpoint, Martin Paulus and John Torous address the broader problem of trustworthy AI interpretation. LLMs as Clinical Instruments—Toward Verifiable Reasoning argues that when LLMs function as clinical instruments, their reasoning must be verifiable rather than opaque.

The same standard should apply when AI interprets information directly for someone in distress. A coherent psychological story can feel deeply clarifying. It can also be confidently wrong.

Responsibility Fragments Across the System

The user experiences one conversational system, but responsibility is spread across many actors. A foundation-model provider may control the model, another company may build the application, and a separate service may connect personal or health data. Clinicians may then encounter the consequences, while different regulators govern privacy, consumer protection, medical devices and professional conduct.

Each organisation may be accountable for its own part. The harder problem is deciding who owns the psychological effects that accumulate across the system as a whole.

A 2025 npj Digital Medicine review of generative LLM applications in mental health care shows how little the field has tested accountability. The included studies had not evaluated accountability, transparency, explainability, interpretability, testability, security or resilience. Clinical validation was limited, and evaluation methods varied substantially.

A 2026 npj Digital Medicine review of purpose-built generative mental health chatbots identified 21 studies across 11 countries. It found promising user experiences alongside fragmented evidence and called for stronger safety and efficacy evaluation. This was a scoping review of intervention design and user experience, not a definitive test of clinical effectiveness.

We can test whether a single response violates a safety policy. We are far less prepared to decide who owns cumulative mental health risk when a system remembers, personalises and interprets over time.

Influence Is Also the Therapeutic Opportunity

Mental health AI can help because interaction can change what people think and do. The well-publicised Therabot randomised trial evaluated a purpose-built generative AI intervention in adults with clinically significant depression, anxiety or eating-disorder risk. Participants assigned to Therabot had greater symptom reductions than a waitlist control at four and eight weeks. The study was short, included human safety monitoring, and did not establish durable effects or effectiveness relative to psychotherapy.

General-purpose AI can also help people organise their thoughts before therapy, prepare questions for a clinician, understand psychological concepts, rehearse difficult conversations or explore explanations they had not considered before. These benefits strengthen the accountability case. Psychological influence is how benefit occurs, and it creates responsibility when that influence becomes clinically consequential.

Purpose-built systems can make evidence generation, safety monitoring and escalation part of the product architecture. Governance becomes harder when similar psychological influence develops inside systems with much less clearly defined clinical responsibilities.

Accountability Needs Its Own Architecture

There are two credible ways to build longitudinal responsibility into the product.

The first is longitudinal safety by design.

A system that retains psychologically significant context could track clinically meaningful change over time rather than use memory mainly for personalisation. Those changes could then alter how the system responds. Relevant signals might include increasing suicidal intent, growing certainty around implausible beliefs, marked sleep deterioration, withdrawal from treatment, dependency, severe reassurance-seeking or worsening functioning.

A named clinical-safety function would own the thresholds, test them through multi-turn simulation and real-world monitoring, analyse false positives and false negatives, and define the escalation policy.

The system would also preserve enough provenance to show what shaped a clinically consequential interpretation. A reviewer should be able to tell whether the information came from a current disclosure, an earlier conversation, connected health data or an inference generated by the model.

The second approach is a clinical boundary architecture.

A general-purpose system could offer ordinary conversational personalisation until its cumulative psychological influence crosses a defined threshold. Greater persistence, repeated mental health inference, rising vulnerability or clinically consequential recommendations could trigger stronger safeguards.

At that point, the system might lower its interpretive confidence, restrict some psychological formulations, encourage professional assessment, activate specialised safety policies or route the interaction into a more clinically governed environment.

Different products will need different implementations. The obligation should grow with the system’s persistence, psychological inference, personalisation and foreseeable effect on clinically consequential behaviour.

When retained history, connected data or repeated interaction shapes how a person understands symptoms, relationships, treatment or risk, the obligation grows. Longitudinal safety, provenance, escalation thresholds and accountable human oversight should all scale with that influence.

This gives developers a concrete design standard. It also gives purchasers, regulators, clinicians and investors a better test than asking whether the product calls itself a mental health tool.

Build the Accountability Model While the Interpretive Layer Is Still Taking Shape

AI will remember more. Personalisation will deepen, and systems will connect more sources of data. Models will also become better at finding patterns across long histories.

Mental health could benefit enormously. A system may eventually recognise deterioration before the person does. It may notice that sleep, withdrawal, mood and language have changed together, identify contradictions that warrant professional assessment, or help someone reconstruct a complicated treatment history.

These capabilities also increase interpretive authority. When AI repeatedly helps someone decide what their symptoms mean, whether their relationships are safe, which diagnosis fits, whether treatment is working or what to do next, its influence can become clinically consequential even if the product never calls itself a therapist.

Current evidence does not tell us how often persistent general-purpose mental health AI causes longitudinal harm in real-world use. It does show that accumulated psychological context can materially change model behaviour, and that safety architectures respond differently as the context deepens. Clinicians already encounter patients who use AI for self-diagnosis, treatment assistance and mental health support. Major AI systems are also becoming better at remembering and connecting personal context.

Single-turn safety tests cannot carry the accountability burden for systems designed to know people over time.

If you are building, deploying, purchasing or clinically advising one of these systems, you should be able to name who owns its longitudinal clinical risk, how you test that risk, what evidence changes the system’s behaviour and how someone can challenge a harmful trajectory. If you cannot, the system already has more psychological influence than your accountability architecture can support.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

Mental health practices can no longer remain neutral on client AI use. By the time a client reaches a clinician, AI may already have shaped the story they tell about themselves.

Clients arrive with explanations of their symptoms, expectations about diagnosis, interpretations of relationships and techniques they have already tried that may all have been influenced by AI. They may also have disclosed sensitive information to systems outside the protections of clinical care. All of this changes what clinicians need to ask, assess and address.

Consider a composite case. A client becomes convinced that her partner is having an affair. Between sessions, she pastes his texts into a general-purpose chatbot and asks what they might mean. The chatbot cannot know his intentions, but it can generate plausible interpretations of ambiguous wording. As she returns with more excerpts and follow-up questions, those interpretations accumulate into a coherent account of her suspicion. By the next appointment, she is presenting a case the chatbot has helped her assemble.

How should you respond? What should count as evidence, when should you review the transcript, and how do you decide whether the interaction has become clinically significant?

This essay proposes a practical framework for answering those questions, responding proportionately, protecting privacy and recognising when awareness becomes professional involvement.

Client AI Use Is Now Part of Clinical Practice

AI use now belongs within ordinary clinical assessment because it is already part of what many clients bring into treatment. In the American Psychological Association’s 2026 Chatbots and Mental Health Survey

• 77% discussed AI use with patients

• 39% patients used AI to self-diagnose

• 35% patients used AI as an additional mental health professional

• 33% patients used AI to assist treatment

• 13% patients engaged with chatbots in intimate relationships

Much of that use is understandable. AI is available when therapy is not, and it can help someone find language for an experience, organise questions or practise a skill. The same accessibility can also support repeated reassurance-seeking, extensive disclosure and unwarranted confidence in an answer that sounds more certain than the evidence warrants.

Clinically, these systems are not interchangeable. A general-purpose chatbot, an AI companion designed to sustain ongoing interaction and a mental health application built around a defined intervention are different products with different evidence, incentives and risk profiles. The APA’s health advisory on chatbots and wellness applications draws similar distinctions and cautions against applying evidence from purpose-built interventions to general-purpose systems.

Clinicians therefore need judgement rather than a default stance for or against AI. Reflexive dismissal can discourage disclosure of use that may already be influencing the client. Uncritical acceptance can allow unsupported conclusions to enter assessment, formulation and treatment.

Why Conversational AI Is Different From “Dr. Google”

A chatbot does something a search engine never did. It responds to the client’s framing, mirrors their language and generates a new interpretation with each follow-up. The exchange can feel less like reading information and more like being understood. The APA’s guide to navigating AI-generated advice warns that mirroring can create a sense of being known and that confident language can make inaccurate information seem credible.

That influence can accumulate across a conversation. Each response may build on the last, organise ambiguous events into a coherent account and increase confidence without adding independent evidence. Repeated, personalised interaction can therefore begin to shape how a client understands symptoms, relationships or themselves, rather than simply supplying information.

How to Judge the Clinical Significance of Client AI Use

AI use becomes clinically relevant when it begins to influence what a client believes, feels or does. The same tool can play a very different role depending on the person, the purpose and the pattern of use.

One client may use a chatbot to prepare questions for a clinician. Another may return repeatedly for reassurance about a feared illness. A client with obsessive-compulsive symptoms may use it for checking, while someone developing paranoid beliefs may ask it to interpret ambiguous events.

These uses do not warrant the same response. Client AI use becomes clinically significant when it meaningfully affects symptoms, beliefs, behaviour, relationships, functioning, treatment participation, privacy or risk.

That standard allows clinicians to ask without treating every use as a problem or every conversation as something they may inspect. Clinicians already explore outside influences when they affect presentation, maintain symptoms or alter risk. AI requires the same judgement, with added attention to repeated interaction and the authority clients may attribute to its responses.

The clinical response should match the level of concern. I propose three levels.

Level one involves acknowledgement and brief guidance. The client may use AI to prepare questions, organise thoughts or rehearse a conversation without evidence that the interaction is reinforcing symptoms or interfering with care. The clinician can discuss accuracy, privacy and the limits of AI-generated advice, then revisit the issue if its role changes.

Level two requires active clinical work. AI has become part of reassurance-seeking, rumination, compulsive checking, avoidance, rigid certainty, relationship conflict or disengagement from treatment. The clinician should examine what function the interaction serves, how often it occurs, what happens afterwards and whether the client is beginning to rely on the system’s interpretation over other sources of evidence. Clinical work may include separating direct experience from AI-generated interpretation and addressing the behaviour within the existing formulation and treatment plan.

Level three requires direct risk management. AI use is occurring alongside acute suicidality, emerging psychosis or mania, severe impairment in judgement, threats or violence, exploitation, inability to maintain safety or other circumstances in which the interaction may be amplifying immediate risk. The clinician should assess the underlying clinical state directly rather than treating the chatbot exchange as evidence in itself, and follow established crisis, psychiatric, safeguarding or emergency procedures as indicated.

These levels are dynamic. A client can move between them as the function and consequences of AI use change.

The composite client introduced at the beginning of this essay, who brought suspicious text messages to therapy after discussing them with AI, would initially fall at level two. Escalating surveillance, confrontation, sleep loss, increasing conviction despite contrary evidence or broader suspiciousness would raise concern about movement towards level three. A practice position should help the clinician recognise that trajectory before a crisis determines the response.

How to Assess AI-Generated Diagnoses and Formulations

Once AI use begins to influence a client’s beliefs, behaviour or treatment, the clinician needs to examine what the system has actually contributed. The central task is to separate the client’s experience from the interpretation the chatbot has built around it.

The problem is that generative AI can turn incomplete or ambiguous information into a coherent psychological account. In the JAMA Psychiatry Viewpoint “LLMs as Clinical Instruments—Toward Verifiable Reasoning”, Martin Paulus and John Torous describe how LLMs can omit relevant details, introduce unsupported content and smooth contradictions into a plausible narrative. They recommend separating facts from inferences, identifying what information is missing, challenging the initial answer and grounding factual claims in source material.

Paulus and Torous developed that discipline for clinicians using LLMs within clinical workflows, but the same logic applies when a client brings an AI-generated diagnosis, formulation or interpretation into treatment. The output may be worth exploring as part of the clinical material. It does not become collateral evidence, a psychological assessment or an established diagnosis simply because it is detailed, confident or psychologically sophisticated.

The clinician should reconstruct how the conclusion was produced.

• What did the client tell the system?

• How did they frame the question?

• Did they begin by suggesting a diagnosis or explanation?

• What relevant history, contradictory information or contextual detail never entered the conversation?

• What did the output change?

• Did it help the client describe an experience more clearly?

• Did it narrow attention around one explanation?

• Did it increase certainty before an adequate assessment?

The composite client from the opening illustrates the problem. Her partner’s delayed replies and ambiguous wording are observations. The conclusion that those messages indicate an affair is an interpretation. Repeated chatbot exchanges can elaborate that interpretation, organise additional details around it and make the resulting account feel progressively more convincing without adding independent evidence.

The same clinical discipline still applies. Establish what happened, distinguish observation from inference, consider competing explanations and assess the client’s presentation using ordinary clinical evidence. AI changes how an interpretation may have been generated and reinforced. It should not change the evidentiary threshold clinicians apply to it.

What Clients Need to Know About Privacy and Confidentiality

Clients may experience a chatbot conversation as private because it happens alone, often on a personal device and in language that feels intimate. However, the privacy protections are very different from those of clinical care.

Consumer AI services set their own terms for data retention, use and sharing. The APA advises caution with sensitive information and with the use of AI for diagnosis or psychological test interpretation. Clinicians do not need to become experts in every platform’s privacy policy, but they should make sure clients understand that disclosure to a chatbot is not the same as disclosure within a confidential therapeutic relationship.

Practical guidance should be specific. Clients should think carefully before uploading identifiable health records, therapy material, session recordings, psychological test content or information about another person. If they choose to share sensitive material, they should first understand how the service may store, use or retain it.

The same restraint applies when a client offers to show the clinician a chatbot transcript. Start with the clinical reason for reviewing it. What happened in the interaction that matters for assessment, treatment or risk? A full transcript may contain highly sensitive disclosures, third-party information and material that has little clinical relevance. Review only what is needed.

Documentation should follow the same principle. Record the clinical significance of the AI use, any relevant risk, the guidance provided and the treatment response. Importing large sections of a chatbot transcript into the clinical record can create additional privacy exposure without improving care.

The practical rule is simple. Collect and retain only the AI-related information that the clinical work actually requires.

Why Clinicians Need to Assess AI Use Over Time

A single chatbot exchange may tell the clinician very little about the clinical significance of AI use. The more important pattern may only become visible over time.

A client may begin by occasionally asking for information or reassurance. The conversations can gradually become longer, more frequent or more emotionally important. The client may start returning to the system whenever uncertainty arises, relying on its interpretations, losing sleep during extended conversations or giving it increasing influence over relationships, treatment decisions or beliefs. None of those changes requires one obviously dangerous response.

Benjamin Nelson, Mark Kalinich and John Torous make this distinction in the JAMA Viewpoint “Specialized or General-Purpose—The Wrong Question for Mental Health AI Safety”. They distinguish harms that arise within a brief interaction from harms that accumulate across repeated use, including dependency, attributed sentience, romantic attachment and reinforcement of delusional themes.

For clinicians, this means asking about the pattern of use, not simply reviewing the most recent exchange:

• How long has the client been using the system?

• Has use become more frequent, prolonged or difficult to stop?

• Has the chatbot become more emotionally important?

• Is the client increasingly relying on it for reassurance, interpretation or decisions?

• Is use displacing sleep, work, treatment or human relationships?

• Have the client’s beliefs or behaviour changed alongside the interaction?

Returning to the composite client from the opening, one chatbot response about one suspicious text may carry little clinical significance. Concern grows if this client repeatedly brings new messages to the system, receives interpretations that reinforce the same suspicion and becomes increasingly certain without new evidence. Once that pattern begins to affect sleep, surveillance, confrontation or broader suspiciousness, the clinical picture has changed.

A client may therefore move from one level of concern to another as the pattern of AI use changes. Clinicians need to recognise that movement early enough to adjust assessment, treatment and risk management.

How Professional Responsibility Changes With AI Involvement

A clinician who discovers that a client uses AI is in a different position from one who recommends an AI tool or incorporates it into treatment. A practice that selects and deploys the technology takes on a different responsibility again.

When clients choose AI independently, the clinician’s responsibility concerns its clinical effects. If the use begins to influence symptoms, beliefs, treatment or risk, it should be assessed and managed as part of care. The clinician has not endorsed the system simply by discussing it with the client.

That changes when the clinician recommends a particular tool, asks the client to use it between sessions or relies on its output in treatment. The clinician has now given the technology professional weight. They should understand what it is intended to do, the evidence supporting that use, its important limitations and risks, whether it is appropriate for the particular client and what alternatives are available.

The boundary becomes clearer still when a practice selects and deploys AI for transcription, documentation, messaging, assessment support or another clinical function. The organisation is no longer responding to technology introduced by the client. It has introduced the technology into care and needs defined uses and limits, appropriate privacy and data governance, human oversight and a process for identifying and responding to errors.

Paulus and Torous make a related point in their JAMA Psychiatry Viewpoint. Clinical use, they argue, should occur within bounded tasks, approved workflows and an organisational framework that preserves professional accountability.

A useful practice position should therefore state where these boundaries lie. Encountering a client’s AI use, recommending AI and deploying AI are different forms of professional involvement, and clinicians should know when they have crossed from one to the next.

A Six-Step Response When Clients Bring AI Into Care

A practice position becomes useful when clinicians know what to do when AI enters the conversation. A routine question can open the discussion without implying that AI use is either problematic or endorsed.

Many people now use AI for information, emotional support, relationship advice or help between sessions. Has AI played any role in how you have understood or managed what you are dealing with?

If the answer is yes, the clinician can work through six steps.

1. Ask what the client is using and why. Identify the system where relevant, what the client uses it for and what they are hoping to get from the interaction. Normalise disclosure without implying surveillance or judgement.

2. Decide whether the use is clinically significant. Consider its function, frequency and consequences. Does it affect symptoms, beliefs, relationships, functioning, treatment participation, privacy or risk? Ordinary use does not require extensive assessment simply because AI is involved.

3. Separate experience from interpretation. Establish what actually happened, what the client concluded and what the AI system added. Treat AI-generated diagnoses, formulations and explanations as material to assess rather than evidence in themselves.

4. Assess the pattern over time. Ask whether use is becoming more frequent, prolonged, emotionally important or difficult to stop. Look for increasing reliance on the system for reassurance, interpretation or decisions, and for displacement of sleep, treatment, daily functioning or human relationships.

5. Match the response to the level of concern. Level one may require acknowledgement and brief guidance. Level two calls for active clinical work on the role AI is playing in the presenting problem or treatment. Level three requires direct assessment and established risk, psychiatric, safeguarding or emergency procedures.

6. Review and document proportionately. Do not collect an entire chatbot history simply because it exists. Review only the material needed for assessment, treatment or risk management. Document the clinical significance of the AI use, relevant risk, guidance or intervention and planned follow-up.

The clinician should also know when the nature of their own involvement changes. Discussing AI that a client chose independently does not amount to endorsing the product. Recommending a specific system, assigning an AI-supported activity or incorporating AI output into treatment gives the technology professional weight and requires greater knowledge of its intended use, evidence, limitations and suitability.

Practice leaders need to turn these clinical decisions into explicit organisational expectations. A written position should address:

• how and when clinicians routinely ask about client AI use

• the standard for determining when AI use becomes clinically significant

• the three levels of response and thresholds for supervision or escalation

• how clinicians distinguish client experience from AI-generated interpretation

• when chatbot transcripts should be reviewed and how much should enter the clinical record

• what privacy guidance clinicians should provide

• what clinicians must know before recommending a specific AI product or AI-supported activity

• the distinction between client-chosen, clinician-recommended and practice-deployed AI

• disclosure and management of relevant financial or commercial relationships

• approved and prohibited uses of AI by clinicians for case reflection, treatment planning, documentation or other clinical work

• what client or clinical information may never be entered into unapproved systems

• who provides consultation when an AI-related issue exceeds a clinician’s competence or raises an unfamiliar risk

• who is responsible for reviewing the policy as products, evidence, professional guidance and regulation change

The policy also needs to cover everyone who can introduce AI-related risk into the service. Trainees and contractors require the same clinical and data-handling expectations as other practitioners, with appropriate supervision. Administrative staff need clear rules about approved systems, confidential information and what they may enter into AI tools.

Services working with children and adolescents require explicit additional provisions. Clinicians need guidance on how to ask about AI use developmentally, what confidentiality can be offered, when caregiver involvement becomes necessary and when AI-related behaviour raises a safeguarding concern. Consent, confidentiality and caregiver rights vary by developmental capacity and jurisdiction, so an adult protocol should not simply be carried across unchanged.

A written position should leave clinicians with fewer decisions to improvise when AI becomes clinically relevant. It should tell them what to ask, what to assess, what evidence to trust, what material to review, when to escalate and when their own use or recommendation of AI creates additional professional obligations.

Establish Your Position Before the Next Client Forces the Question

Returning to the client with the suspicious texts, a prepared clinician knows how to proceed. They establish what happened, separate observation from AI-generated interpretation, assess how repeated use has affected certainty and behaviour, review only what is clinically necessary, address privacy and act if the risk is escalating.

Client AI use is already part of routine mental health care. People are using these systems to make sense of symptoms, consider diagnoses, seek reassurance, interpret relationships and obtain support between sessions. Sometimes that use will be helpful or inconsequential. At other times it may reinforce symptoms, complicate treatment or contribute to risk. Clinicians need a consistent way to recognise the difference and respond appropriately.

Every practice now needs to decide what its clinicians should ask, what counts as clinically significant, when AI-generated material should be examined, how privacy should be handled, when risk requires escalation and what changes when the clinician or organisation introduces AI into care.

AI will keep entering clinical practice through the clients who use it. So the next time a client brings AI into the room, your clinicians should already know what your practice stands for and what they are expected to do.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

Mental health AI can sound clinically authoritative before its data justify that authority.

A person can describe a distant partner, a sudden change in mood or a fear that something is wrong and receive a precise, psychologically literate explanation within seconds. That response may shape whether they adopt a diagnosis, confront someone, leave a relationship or seek professional care.

The problem lies beneath the fluency. Much of the data available to these systems is partial, self-reported, stripped of context, clinically contested or disconnected from what happened next. More data can improve a model. It cannot create certainty that the source material itself does not contain.

Everyone around these systems owns part of that consequence.

Founders decide what the product may infer.

Engineers decide what counts as evidence.

Clinicians meet patients who arrive with AI-generated formulations already in hand.

Safety, privacy and governance teams decide which failures become visible before they repeat at scale.

And mental health AI becomes convincing faster than it becomes knowledgeable. The system may have only a few minutes of self-report. It usually lacks developmental history, behavioural observation, medical assessment, collateral accounts and reliable knowledge of what happened next. None of those absences is obvious in a fluent response.

More data, longer context windows and clinical fine-tuning can improve performance. They cannot recover history that was never recorded, resolve uncertainty that vanished when a provisional judgement became a label or observe an outcome that occurred off-screen. The field has been too casual about this gap. The language can acquire clinical weight long before the evidence earns it.

The Interface Can Outrun the Evidence

Language models can reproduce many of the cues people associate with genuine understanding. They reflect emotion, organise scattered details, use diagnostic vocabulary and answer without hesitation. In a 2023 study of 195 public patient questions, licensed healthcare evaluators preferred chatbot responses in 78.6 per cent of 585 blinded evaluations and rated them higher for quality and empathy.

A cross-domain example makes the problem more obvious. In a 2024 study, substance-use clinicians generally rated AI responses to 75 real-world drug and recovery questions positively. The same researchers then conducted a qualitative fact-check and stress-tested the systems with rephrased prompts. They found responses that overlooked suicidal ideation, invented emergency helplines, gave dangerous advice about home detox and changed materially when equivalent questions were reworded. The fact-check was exploratory rather than systematic, but the contrast remains important: responses that passed an initial professional review still contained errors capable of changing what a person did next.

Better language does more than improve the answer. It changes how much authority the user gives it. The response may contain no obvious factual error and still say more about this person than the available evidence supports.

Consider someone who says a partner has become distant and asks whether the relationship is emotionally abusive. A model can recognise familiar patterns, validate the concern and supply a persuasive interpretation. It may have identified something important. It may also be missing grief, conflict, illness, substance use, the user’s own omissions or years of relationship history that would alter the picture.

The model can mention uncertainty, yet the personalised explanation usually carries more psychological force than the caveat. Coherence, specificity and emotional attunement signal competence. A disclaimer carries little weight when the system remembers intimate details and offers personalised interpretations with confidence.

Mental Health Data Arrive With Their Own History

Machine learning works most cleanly when a label can be checked against something independent. Mental health rarely offers that. Low mood, poor sleep, withdrawal and impaired concentration may appear in depression. They may also follow bereavement, trauma, medication changes, physical illness, exhaustion or sustained social pressure. The symptoms are real but their meaning remains open.

Diagnosis gives clinicians a shared structure, yet the categories carry uneven reliability. In the DSM-5 adult field trials, major depressive disorder produced a pooled intraclass kappa of 0.28, within the investigators’ questionable range. The wider field-trial programme also treated reliability and validity as separate issues. That figure should give pause to any product team treating a diagnostic label as truth simply because it appears in a record.

Once a clinical judgement enters a dataset, its uncertainty becomes harder to see. A clinician records a provisional diagnosis. The diagnosis becomes a billing code. The code becomes a label. The label later appears as a clean training target. Nothing in that sequence has established that the first judgement was complete or correct. The pipeline has removed the context that once made the uncertainty visible.

Clinical records preserve encounters not psychological reality. They contain what was asked, what the patient disclosed, what the clinician noticed and what the institution required. Symptoms may look absent because nobody assessed them. Earlier interpretations can persist because each clinician inherits the previous record. Reimbursement, risk management and documentation norms shape what gets written. A 2024 review of psychiatric electronic records emphasised that structured fields and narrative text were created for clinical, administrative and medico-legal purposes, which complicates their reuse for research and machine learning.

The people missing from the records matter just as much. Models see less of those who could not access care, stopped attending, concealed information, received poor treatment or never received a diagnosis that fitted. A larger dataset can make these omissions harder to notice because the numbers create an impression of coverage.

Scraped Knowledge Loses the Context That Gives It Meaning

General-purpose models also learn from the enormous public conversation about mental health. That material includes psychoeducation, diagnostic descriptions, therapy language, support forums, wellness advice and personal testimony. It is rich in how people describe distress. It is far weaker at establishing what caused this person’s distress, which interpretation will survive further assessment or what will help over time.

A recent Drug Science commentary on drug chatbots states this principle directly: “Data scraping is a collection method, not an evidence standard.” The parallel to mental health is informative. Scraping can gather information at scale. It cannot decide whether the material is current, representative, consensual, clinically sound or safe to apply to an individual.

Mental health curation must work with more disputed material. Experts can improve the quality of the sources and remove obvious misinformation. They still cannot determine, from a few paragraphs, whether withdrawal reflects depression, grief, illness, fear, cultural context or a reasonable response to circumstances.

Public mental health data carry further problems of consent and traceability. Chancellor and colleagues mapped ethical tensions involving privacy, consent, validity and machine-learning inference in mental-health research using social-media data. In a review of 112 biomedical Twitter studies, 72 per cent quoted at least one tweet and 84 per cent of those articles exposed at least one identifiable account holder through a simple search.

Representation is uneven as well. Public datasets overrepresent people who write online, use dominant languages and adopt the diagnostic concepts available in their communities. A critical review of 75 social-media mental-health prediction studies found inconsistent methods and weaknesses in how mental-health status was defined. The model can reproduce those gaps while sounding universally informed.

Organisations can pass sensitive disclosures through research, commercial and model-development pipelines far beyond what users expect. Crisis Text Line said it shared anonymised and scrubbed data with the for-profit company Loris.ai, then ended the relationship in January 2022 after public concern. The case shows how information offered during acute distress can enter a data supply chain under conditions users may not fully understand.

Self-Report Can Become a Closed Circuit

Mental health assessment depends on self-report. A person’s experience cannot be understood without listening to what that person says. A chatbot, however, usually receives one account from one person in one emotional state, then responds inside the same frame the account created.

The pattern is easily recognized. The user offers a tentative explanation. The model restates that explanation with clearer language, better structure and more confidence. The user experiences the polished version as confirmation. No independent evidence has entered the exchange. The interpretation feels stronger because the system has made the user’s original account more coherent.

Many assistants can intensify this process. Sharma and colleagues found consistent sycophancy across five leading assistants, with humans and preference models sometimes favouring sycophantic answers over correct ones. Moore and colleagues found therapy-relevant failures in experimental scenarios, including stigma and responses that encouraged delusional thinking.

This is not an abstract alignment problem once the conversation concerns abuse, paranoia, diagnosis, medication, suicide or family conflict. Supportive language often rewards agreement. Careful clinical reasoning sometimes requires a pause, a competing explanation or respectful friction.

A system optimised for a smooth exchange may avoid the very interruption that protects the user from a premature conclusion.

More behavioural data can help, but they do not interpret themselves. Sleep, movement, communication patterns and device use may reveal change over time. Reduced movement can reflect depression, disability, weather, work or illness. Altered communication can reflect withdrawal, travel, conflict or a new platform. The signal still needs a defensible construct, an appropriate comparison and evidence that acting on it improves outcomes.

Mental Health AI Cannot See What Happens Next

Mental health AI often loses sight of the outcome that would tell it whether its interpretation helped. The platform can measure whether the user continued the conversation, returned the next day, clicked a recommendation or reported immediate relief. Those signals are easy to collect. They can also mean reassurance-seeking, unresolved distress, politeness or growing dependence.

The system may never learn whether the person sought appropriate care, stopped medication, confronted a partner, withdrew from family or felt worse a week later. It can observe another session while remaining unable to distinguish benefit from reliance. That missing outcome matters because a model cannot learn reliable therapeutic judgement from consequences it never sees.

This weakness also complicates claims of personalisation. Adapting tone and recalling details demonstrates responsiveness. Clinical personalisation requires evidence that the response fits the person’s condition, circumstances, risks and goals. The interface can feel deeply personal while the underlying decision remains generic.

The 2025 Therabot randomised trial offers a more serious attempt to measure outcomes. It reported symptom improvements relative to a wait-list control. A formal NEJM AI letter later identified three limitations: the wait-list comparison, lack of independent evaluation and use of a therapeutic-alliance measure developed for human relationships. The trial justifies continued study, it does not support broad claims about general-purpose chatbots or establish equivalence with active treatment. Evidence that favours deliberate design deserves the same scrutiny as evidence of failure.

Scale Cuts Both Ways

Human care is an imperfect comparator. Clinicians make mistakes, disagree about diagnosis and sometimes provide poor care. Many people cannot access them at all. In September 2025, the World Health Organization reported that more than one billion people live with a mental disorder and most remain underserved. Any standard for mental health AI that ignores this access failure will protect an idealised version of care that much of the world never receives.

Scale can magnify benefit. A modest improvement in access, information or early support can reach people who would otherwise receive nothing. Caution has a cost when it makes useful systems commercially impossible to build or so evasive that people abandon them.

Scale can also magnify a small error in judgement. A slight tendency to overconfirm, overinterpret or reassure can be repeated across millions of private conversations. The same explanation can return week after week and gradually become part of how a user understands a relationship, a diagnosis or a life decision.

The responsible standard therefore has to preserve usefulness while limiting unsupported authority. A system does not need every feature of professional care. It does need a clear account of which functions it can perform, what information those functions require and what happens when the available evidence falls short.

Responsible clinical work offers a useful model. The work unfolds over time, and professional duties attach to what the clinician does with the information. Mental health AI needs product mechanisms that perform the equivalent work of slowing down, preserving alternatives and exposing uncertainty.

Responsible Realism Must Change the Product

The first requirement is an authority ceiling. When the evidence is sparse, the system should remain at the level of observation, general education or clearly bounded possibilities. Diagnostic, causal and treatment-directive language should become unavailable when the product lacks the information needed to support it. This limit belongs in enforceable product logic, tested across multi-turn conversations. A prompt and a disclaimer will not hold when users repeatedly press the system for certainty.

The second requirement is visible alternative reasoning. A system that links withdrawal to depression should also preserve credible alternatives such as grief, physical illness, medication effects or situational exhaustion when the available information cannot distinguish among them. It should name the missing information that would change the interpretation. Some users will find this less smooth. That friction tells the truth about the evidence.

Clinicians should treat patient use of AI as part of the clinical environment. The American Psychological Association advises clinicians to ask patients proactively about generative-AI chatbots and wellness apps. Ask what systems they use, what they disclose and whether AI explanations are shaping diagnosis, treatment expectations or major decisions. An AI-generated formulation belongs in the room as an influence to examine, not as an assessment result.

Founders and product leaders need to decide what authority their product may convey before engagement metrics decide for them. A wellness label offers little protection when the interface provides personalised interpretations of symptoms or relationships. Intended use, prohibited use, escalation and uncertainty limits must appear in the product’s conduct. The commercial model also matters. A system rewarded for agreement, return visits or emotional dependence will eventually turn those incentives into clinical risk.

Engineers and data scientists need a traceable account of data origin, collection purpose, label construction, missing populations and downstream outcomes. Evaluation should test whether sparse information becomes causal, diagnostic or treatment-directive claims. Datasheets for Datasets and Data Statements for Natural Language Processing provide useful foundations. Mental health systems need an added clinical layer that documents context, disagreement and what the data cannot establish.

Governance and safety teams should test influence over time. A reasonable first answer can become harmful through repetition, growing certainty or accumulated authority. Testing needs to include users who seek reassurance, press for diagnosis, present delusional material, resist referral or return repeatedly with the same interpretation. Review should ask what the system causes a person to believe and do, not only whether a single reply contains a prohibited phrase.

Security and privacy leaders should treat psychological disclosure as data with durable consequences. Retention, secondary use, model-training access, re-identification and permissions need strict controls. A 2021 Delphi study involving experts in digital phenotyping, data science, mental health, law, ethics and lived experience reached strong agreement on privacy, transparency, consent, accountability and fairness as priority requirements. Consent obtained during distress deserves particular scrutiny because disclosure can be intimate, impulsive and difficult to retract.

Ethics leaders need to convert principles into decisions that can be inspected. Who bears the cost when the system refuses a definitive answer? Who bears it when the system answers too readily? Which populations were poorly represented? What incentives reward agreement, continued use or dependence? Those choices already exist inside the product. Ethics work begins when someone has authority to change them.

The Evidence Must Constrain the Voice

Mental health AI will become more fluent, more responsive and more persuasive. Those gains can expand access and make useful support easier to reach. They also increase the chance that people will treat a coherent answer as a well-founded one.

The person wondering whether a distant partner is emotionally abusive may receive language that finally names something real. They may also receive one plausible interpretation polished into certainty. The system cannot know which simply because it can say both beautifully.

Anyone who builds, funds, endorses, procures or clinically encounters that answer has a duty to make the evidential limits part of the product’s conduct. Require the system to show what it knows, what it is inferring, what remains missing and when it must stop. Do that before someone reorganises a relationship, a treatment decision or a life around a sentence that sounded more certain than the evidence beneath it.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

We have built something that can feel like a helping relationship while owing almost none of the obligations of one.

For someone using a general-purpose chatbot, access requires almost nothing. A device. An account. No referral, no waiting list and usually no accountable person on the other side. Someone in the worst hour of their week, or their life, can open a window and type the thing they have never said aloud and a reply appears instantly, in language designed to sound as though there is a human on the other end who understands. There isn’t.

AI does not assess or formulate the way a clinician does, and it takes on no responsibility for what follows. These systems generate language that fits the tone of the exchange and present it in the language and manner of care. In a general-purpose chatbot exchange, no clinician usually reads the disclosure, forms a clinical picture of the person or accepts responsibility for what follows.

The person disclosing may have little idea whether the conversation will be retained, used to train another model or connected with other information about them. Privacy and data-use practices vary across platforms, model versions and product releases. In acute distress, deciphering them falls to the person least equipped to do it.

Millions of people are already relying using these systems as de facto care. That reliance creates obligations, whether the companies involved intended to provide mental healthcare or not. We must now determine where these system influence judgement, where their safeguards fail and who will answer when a fluent exchange with a machine alters the course of a person’s life.

We Do Not Yet Know the Exact Number

I want to ground this use in a number, because the scale is the part people often underestimate. In a nationally representative US survey published in JAMA Pediatrics, 19.2 per cent of people aged 12 to 21, roughly 8.2 million young people, said they had used an AI chatbot for mental health advice in 2025 (20% higher than the year earlier). Nearly two-thirds of them told no one.

Research released by Mind in July 2026 found that 18 per cent of polled adults had used AI for mental health support in the previous year, and of that group, 60 per cent had used it in lieu of in-person therapy. A separate poll for Mental Health UK put the figure higher, with 37 per cent saying they had used a chatbot for mental health or wellbeing, most naming an ordinary general-purpose chatbot rather than anything built for mental health.

Clinicians are seeing the same shift. A 2026 American Psychological Association survey found that 77 per cent of more than 1,200 psychologists said patients were using AI in some way, and more than a third, 35 per cent, described patients who treated AI as an auxiliary therapist.

Even these estimates probably understate the scale of use. Ask five different surveys what they measured and you will get five different answers. “Used AI for wellbeing.” “Asked a chatbot for mental health advice.” “Lives with a diagnosed condition and uses an LLM alongside it.” “Has a patient who treats AI as a kind of auxiliary therapist.” These are not the same question. Samples differ. Time windows differ. And definitions differ, sometimes considerably.

So although these studies do not produce an agreed-on prevalence estimate, they document a behaviour that occurs outside formal care, is undisclosed, and is invisible to anyone able to assess what the interaction is doing to the person over time.

Of Course People Are Using It

I have spent much of my working life around mental health services and technology. What I see is not a public irrationally abandoning good care for an obviously inferior substitute. I see people adapting to what is available and reaching for relief wherever they can find it.

Of course people are using AI for support. Distress creates urgency, and AI answers urgency with extraordinary efficiency. When someone is frightened, ashamed, lonely or unable to sleep, they are not comparing models of care, they just want an answer now. Something that might make the next ten minutes more bearable.

A chatbot is always there. A person does not have to say the hardest thing aloud while another human listens. The system can help them find words, organise a chaotic account or rehearse a conversation they are dreading. It does not appear shocked. It does not become impatient. It does not decide that the allotted time has ended. That combination is plainly enticing.

And users say it helps. In the US youth survey, 91.7 per cent rated the advice at least somewhat helpful. In Mental Health UK’s poll, two-thirds reported some benefit and more than a quarter said the interaction made them feel less alone. Those experiences should be taken seriously. Feeling calmer, less isolated or better able to put an experience into words is a real benefit, even when it occurs outside formal care.

Carefully bounded generative AI can also produce measurable clinical value. A randomised Therabot trial published involving 210 adults found greater reductions in symptoms of depression, anxiety and eating-disorder concerns than an inactive waitlist control. The trial establishes efficacy against waiting, not equivalence to therapy or another active intervention. And Ash, another purpose-built system, reported improvements in depression, anxiety and social connection among 305 adults, with flagged sessions handled according to its safety protocol.

The Comparator Must Be Real

No claim about safety means anything until you say what it is being measured against. AI is sometimes compared with an ideal clinician who is skilled, affordable, culturally responsive and available. Sometimes that clinician exists, and sometimes therapy is exactly what the person needs. Often the alternatives are a waiting list, fragmented care, a search engine, a journal, a trusted friend, or no support at all.

The fact that human care is scarce does not excuse exposing people to avoidable harm, especially where the business model and product metrics reward repeated engagement. Human therapy fails people too, sometimes badly. The profession has standards, supervision, complaints processes and liability structures through which it can be held accountable, however imperfectly.

Even if only a small proportion of these conversations reinforce dangerous beliefs, miss escalating risk or contribute to harmful decisions, millions of interactions will still produce a substantial number of harmed people. But withholding a tool that genuinely helps from people who have no realistic alternative also carries a human cost.

Clinical safety cannot be reduced to suicide detection. Harm can accumulate below crisis thresholds through reinforcement, dependency, delayed help-seeking and repeated privacy exposure.

Any credible framework must weigh the harms of unsafe use against the costs of withholding useful support, distinguish use by level of risk, monitor what happens over time and respond when the evidence shows that a system is failing people. A broad assurance that a system is “safe” tells us almost nothing. What we need instead is a comparative safety case, a structured argument supported by evidence rather than a persuasive marketing claim. Who is this system for, and who should not use it? What benefit does it claim, compared with what realistic alternative? Which harms are foreseeable? How would anyone know they were occurring? What changes when risk rises? Does the system remain dependable deep into a long conversation, or only during the first few exchanges? And when it fails, who answers for the consequences?

Being realistic means judging these systems against the choices people actually have.

A Good Trial Cannot Make an Untested Chatbot Safe

AI in mental health is not one thing. A brief exchange about managing stress is not the same exposure as a private interaction extending across hundreds or thousands of turns. The product may be the same. The clinical reality is not.

At the most governed end of the spectrum is a clinically evaluated tool used openly as an adjunct to care. The clinician knows it is being used. The patient understands its role. A professional remains responsible for assessment, interpretation and intervention when risk rises. Existing clinical and liability structures can govern this use and allocate responsibility, provided the adjunct does not go on to replace the care it was meant to support, unnoticed by the clinician relying on it.

Most public exposure sits outside that setting. People with real, sometimes acute needs are using general-purpose or wellness-labelled products without anyone involved in their care knowing. The system has no clinical formulation of the person, little dependable knowledge of their history and no professional capable of turning a dangerous disclosure into an actual intervention. It may sound informed and attentive but it cannot take responsibility for what happens next.

Companion and roleplay systems introduce a further concern. Their appeal depends on simulated intimacy, constant availability and a relationship designed to deepen through repeated use. They may never describe themselves as therapy but, realistically, that distinction can become functionally meaningless once a lonely teenager or psychologically vulnerable adult begins using the system as confidant, adviser and primary source of emotional support.

The label tells us almost nothing. Risk depends on what the person is using the system for, how vulnerable they are, how acute the distress becomes, how long the interaction continues, how much they disclose and whether emotional dependence begins to form. It also depends on whether an accountable human can see what is happening and act.

Evidence gets stretched here beyond what it reasonably concludes. Promising findings from Therabot, Ash or a carefully integrated clinical pathway are repeatedly used to reassure people about general-purpose and companion systems that were never studied under those conditions. That is not how evidence works.

A trial establishes something about a particular intervention, model version, population, comparator, duration and form of oversight. It does not establish the safety of another chatbot merely because both systems generate language. Evidence from a governed clinical intervention cannot be borrowed to legitimise a different product being used alone, for another purpose, by a more vulnerable population and over a much longer period.

Risk belongs to the trajectory of use, not the app-store category or the claims on the company website.

The exposure that concerns me most is sustained, unsupervised and undisclosed reliance on systems that were neither designed nor governed as care. That is where occasional support can become substitution, substitution can become dependence, and dependence can deepen while nobody accepts responsibility for the relationship or its consequences.

Companies should not be permitted to claim the benefits of mental health AI at the category level while disclaiming responsibility for the uses their own products predictably invite. They must show that the evidence applies to the actual system, population and pattern of reliance they are enabling.

The System Can Erode the Judgement Needed to Question It

Most software fails by producing the wrong answer or failing to complete a task. A mental health chatbot can fail much more consequentially. It can influence the assumptions from which a person decides what is true, whom to trust and what to do next. This deserves a higher standard of safeguard and accountability.

After a sustained conversation, the most important output may not be any single reply. It may be the belief the conversation leaves behind. A suggestion becomes an interpretation. The interpretation is repeated, elaborated and returned in confident language. Over time, it can begin to feel less like something the user proposed and more like something the system independently confirmed. That is dangerous.

Large language models are trained to respond coherently, helpfully and in ways users prefer. Those incentives can also produce sycophancy. Anthropic’s own research has shown how reinforcement learning from human feedback can reward responses that align with users’ beliefs over responses that are more truthful. OpenAI learned this the hard way in 2025, when it withdrew a model update after discovering it had become so agreeable that it was validating doubts, fuelling anger, urging impulsive actions and reinforcing negative emotions.

In ordinary consumer software, excessive agreeableness may be irritating or misleading. In a mental health conversation, it can become clinically dangerous.

Clinical work requires more than warmth and validation. It requires judgement. A clinician considers alternative explanations, notices what does not fit, distinguishes emotional truth from factual accuracy and recognises when agreement would deepen the problem. Sometimes care requires support. Sometimes it requires friction. Knowing the difference is part of the work.

A language model can reproduce the language of that process without performing the process itself. It can sound measured, thoughtful and clinically informed while holding no formulation of the person, no independent understanding of the situation and no duty to resist a dangerous premise. Fluency can conceal the absence of judgement precisely when judgement matters most.

The resulting feedback loop should trouble us. A person offers an interpretation of what is happening. The model reflects it back, develops it and gives it greater coherence. The user experiences that coherence as corroboration and discloses more. The system responds with increasing specificity. Across dozens or hundreds of exchanges, uncertainty can harden into conviction.

This risk becomes most serious when a person’s judgement is compromised. When acute symptoms, overwhelming fear or emotional dependency impair reality testing, impulse control or decision-making, the person may be least able to recognise that the conversation is narrowing rather than clarifying their understanding. The system may continue to feel calm, attentive and authoritative while helping to deepen a distorted appraisal of reality.

Even the companies building these models acknowledge that safeguards which appear effective in short exchanges can become less reliable across long conversations. Yet much published and vendor-reported safety testing still relies heavily on prompt-level or short-horizon evaluation. That is profoundly inadequate for a product whose influence may accumulate over days, weeks or months.

The high-profile wrongful-death lawsuits and settlements involving minors and chatbot providers make the stakes impossible to dismiss. Although the allegations in unresolved cases remain contested, the pattern of risk they expose is clear. A system can participate in a person’s deterioration across a long sequence of interactions while nobody inside the company assumes clinical responsibility for the relationship as a whole.

Disclosure and consent cannot fully protect against this. We usually assume that a person can evaluate a product’s risks because the product does not interfere with the faculty used to make that evaluation. A chatbot that reinforces a delusion, entrenches a distorted belief or deepens emotional dependence undermines that assumption. It acts on the very judgement the person needs in order to recognise that something has gone wrong.

The harm is not only that the system may give dangerous advice. It may alter the person’s ability to recognise the advice as dangerous, question the relationship or leave it.

We have legal language for privacy violations, defective products and malpractice by an identifiable professional. We have less developed language for systems that exert psychological influence while no person or organisation accepts a corresponding duty of care. We do have a vocabulary for some of this. Data protection law knows how to talk about privacy injury. Product liability law knows defective products and physical harm. Professional regulation knows malpractice by a named practitioner. What we do not have is a working language for unaccountable influence over someone’s grip on reality, their beliefs, their sense of their own agency.

A Chat Window Is Not a Confidential Room

The feeling of confidentiality here is often an illusion. You are alone with a screen and it answers you in language that feels intimate. That combination can feel more confidential than an actual clinic, while carrying none of a clinic’s legal or professional protections.

People misjudge that boundary more than you would think. In a 21-person qualitative study of adults using general-purpose LLMs for mental health support the researchers uncovered multiple misconceptions about privacy and accountability, including a belief that what you tell a chatbot carries the same protections as what you would tell a licensed therapist. It does not. Indeed, a Stanford review of the privacy documents from six leading US model developers found wide variation and substantial opacity in whether chats were used for model training, how long they were retained, what opting out meant, and whether humans reviewed them.

Terms of service do not constitute meaningful informed consent for a psychologically sensitive interaction. Meaningful consent must appear at the point of sensitive disclosure, in language a distressed person can understand. No clinician is present. Here is what this system can and cannot do. Here is how your data will be kept or used, whether you can see it, what might trigger an escalation, and how you delete or export what you have said.

Disclosure alone offers limited protection when the interaction itself remains persuasive. Warning that a model may be sycophantic does not prevent sycophancy. Saying that a chatbot is not a therapist does not stop someone from using it as one. Consent is necessary. It is just not sufficient.

Regulation Is Catching Product Claims, Not Human Reliance

A review covering US state legislation introduced between January 2022 and 19 May 2025 examined 793 bills, identified 143 with potential implications for mental health AI, and found that 20 had become law across 11 states by the study’s cut-off. Since then, Illinois, Nevada, Rhode Island and Maine have enacted additional restrictions on AI-delivered therapy or independent therapeutic decision-making, although their scope and implementation differ.

Those laws are attempting to mark a boundary but they reveal a serious loophole. A general-purpose chatbot does not need to call itself therapy to be used as therapy. A companion system does not need to claim it treats depression to become the exact place a distressed teenager discloses hers. Regulating by intended purpose and professional title means missing the use pattern that is actually producing the most exposure.

Classification can define the regulatory perimeter. It does not by itself allocate responsibility. Responsibility can fragment across the platform, model developer, product deployer and service provider until no one accepts accountability for the interaction as a whole.

Responsible Action Requires a Layered Safety Architecture

One more warning will not make AI safe. Neither will a crisis keyword, a hotline banner, or a disclaimer almost nobody will ever read. Mental health AI needs controls layered across the model itself, the product built on top of it, the organisation behind that, and the wider system of care surrounding all of it.

Obligations must follow foreseeable use. A platform that repeatedly finds itself hosting mental health conversations should incur duties because of what it is actually doing, not because of what the company chose to call it. The triggers should be functional, meaning how long the interaction runs, whether the content is clinical, whether dependency signals are emerging, and whether the behaviour looks like assessment or intervention even if nobody designed it to be. That keeps ordinary conversation free while closing the loophole that wellness labels hide behind.

Sensitive disclosure must change the privacy default. Mental health content should not feed advertising, cross-product profiling or model training without separate consent, and data collection should be minimal with retention made explicit. Ideally, the system should step in before someone reveals identifying details that the support function never required.

The model must introduce clinically intelligent friction. Safety tuning should be able to tell emotional validation apart from factual endorsement, test alternative readings, respond conservatively when evidence is weak or the system’s uncertainty cannot be reliably resolved, resist the pull to confirm an implausible premise, notice compulsive reassurance-seeking, and steer away from language that invites secrecy or exclusivity. As risk accumulates across a conversation, the model’s role should narrow, pointing the person back towards the world outside the chat rather than deeper into it.

None of that means anything without proper testing. Pre-deployment evaluation needs long, multi-session simulations that run across different levels of vulnerability and acuity, not single clean prompts. Evaluation must examine recurrence, cultural and linguistic variation, the indirect ways risk gets expressed, and whether the system degrades after dozens of turns.

Where a product claims human oversight, someone must actually be available. “Human in the loop” only means something when a qualified person responds within a clinically justified service window, sees enough of the context to make sense of it, has the authority to step in, and can actually connect the user to real support. A product must not promise escalation that it cannot operationally complete. Detection without a reachable responder and viable care pathway is notification, not intervention. Reviewing transcripts after the fact is quality assurance. It is not clinical oversight. If no human is available, the product should say so plainly and limit what it is allowed to do accordingly. Staffing capacity, alert thresholds, response times and successful connection to care must be tested under realistic demand, not merely specified in policy.

Minors require stronger defaults. Proportionate age assurance, youth-specific models, tighter data practices, and hard limits on dependency-forming or secret-keeping behaviour should constitute the minimum standard. Parental visibility cannot be the whole answer either, because not every home is a safe one, and confidentiality itself can be what makes help-seeking possible in the first place. The design goal should be connection with a safe adult or qualified service, not turning every disclosure into a surveillance event.

Evidence needs to stay attached to what it actually tested. Claims should name the model version, the population studied, the intended role, the comparator used, the follow-up period, and the known ways it can fail. Material changes to the model, system prompt, retrieval layer, safety policy, interface or escalation pathway should trigger documented change control and proportionate revalidation. Companies should publish adverse-event definitions, incident rates and escalation performance. Incident reporting should include exposure denominators, severity and the number of conversations or user-hours over which events occurred.

Accountability must survive the corporate boundary. Every product operating in this risk domain needs a named executive owner, an independent clinical safety review, auditable logs, active post-market surveillance, and a working route for reporting harm. The clinical safety function must have formal authority to block deployment or require remediation. Safety metrics must have authority over engagement and retention targets when the two conflict. Scale only becomes a safety asset once adverse events are actually studied and the findings change the system in response. Otherwise scale is just a multiplier for whatever is already going wrong. Contracts need to allocate responsibility instead of diffusing it across every party involved, and users need a genuine route for redress.

Finally, education must become routine practice. Clinicians should ask patients whether they are using AI for emotional support and what role it is playing, because right now most patients are not telling anyone. Schools and public health bodies should be teaching that conversational fluency is not the same thing as human understanding, and that a chat window is not automatically confidential just because it feels like one. Product teams need clinical input early enough to actually shape the architecture, not bolted on afterwards.

None of this asks a distressed person to become their own safety officer. The duty rests with the organisation that designed, deployed and profited from the influence, not with someone expected to diagnose the system’s failure from inside the very conversation that is failing them.

Realism Must Preserve Choice Without Abandoning Duty

Protecting people does not mean treating every emotionally honest conversation with AI as something to be stopped. Overreach here has its own costs. It can compromise privacy, discourage people from disclosing at all, and take away one of the only forms of support available to someone who cannot or does not wish to use the formal system.

What we need is proportionate friction. Low-risk, reflective use should stay easy and unencumbered. Friction should increase with vulnerability, substitution, duration and relational intensity. Human escalation should generally be offered before it is forced on anyone. And people should know what is going to happen before they disclose something, not discover it after crossing some threshold nobody told them was there.

Putting the whole burden on the adult user rings especially hollow when the product itself has been engineered to be frictionless, persuasive and hard to walk away from.

Realistic safety, then, means the controls we can actually build now, scaled to the harm we can already foresee, and enforced against the organisations best positioned to reduce it.

The Honest Verdict

AI can provide genuine value in mental health contexts when its role is bounded and its risks are governed. It can help someone find the words for what they are experiencing. It can support structured self-management and extend care that is genuinely well governed. Real people are getting real value from it right now, and any honest account of this has to account for that fact, not argue it away. Yet the curve of adoption and the curve of safeguards are not moving together, not even close. Better measurement and better law will take years to catch up. The obligation in front of us right now does not have years.

So if you build, fund, deploy, prescribe or regulate one of these systems, name the reliance you are enabling, the evidence that actually justifies it, the data rights the user keeps, and the person who answers when it goes wrong.

If you cannot do that, you have not earned the right to scale it.

Sources and links last verified 30 July 2026.

Scott Wallace, PhD, is a behavioural scientist and mental health technology strategist trained in clinical psychology and neuropsychology. For more than 35 years, he has worked across clinical practice, digital product development and conversational systems, helping shape early digital mental health platforms, mobile interventions and NLP/NLG-based tools well before the current generation of large language models. He now advises founders, health systems and investors on AI-enabled mental health, with a focus on clinical safety, governance, product architecture and the unit economics of care.

Here are two questions I am asked far too often by founders building mental health tools, whether mobile apps or AI-assisted chatbots:

“When should our product escalate a user to a live professional, and what should it say?”

“Could you take a quick look at our nearly finished product and tell us whether we have missed anything clinically?”

These requests often arrive in DMs from people I do not know. Both questions (and there are more, of course) assume that complex clinical judgement can be distilled into a concise answer and added after a product has largely been built.

That assumption is the problem. By then, the product’s intended purpose, intervention model, architecture and market positioning have often been decided. Clinical expertise has become a final check or source of quick answers, rather than a discipline that should have shaped the product from the beginning.

This essay addresses a misunderstanding that lived experience, good intentions, self-help knowledge and convincing AI output provide enough expertise to build safely for mental health. They do not. Mental health is a specialised professional field for a reason, and products that interpret distress or influence care must involve appropriately trained clinical expertise from their earliest design decisions.

Mental Health Is Not a Library of Helpful Responses

Mental health products are not ordinary consumer software because the consequences of interpreting a user incorrectly are not ordinary consumer consequences. A shopping app that misreads intent recommends the wrong product. A mental health app can interpret withdrawal as healthy boundary-setting, reinforce avoidance because it brings immediate relief, or reassure someone whose physical symptoms require medical assessment. The response can feel supportive at the precise moment it is moving the person in the wrong direction.

In mental health, satisfaction and benefit can separate. A user may prefer the response that confirms an existing belief or reduces discomfort fastest. Clinical benefit may require further assessment, a different intervention, necessary challenge, or recognition that the product should stop.

Mental health support therefore does not consist of producing the most plausible compassionate sentence. It requires interpreting incomplete information, deciding what remains unknown, recognising competing explanations and selecting a response that fits the person, context, severity and limits of the service. The words are the visible output. The clinical reasoning that determines whether those words are appropriate is the actual work.

“Wellness” Does Not Change What the Product Does

Wellness is a legitimate category. A meditation timer, mood journal or relaxation tool need not be treated as clinical care merely because someone uses it while distressed. However, the label becomes what I call “clinical camouflage” when a product claims to relieve depression, manage anxiety or overcome trauma, interprets a person’s symptoms, selects psychological interventions, or influences whether that person seeks professional care.

The regulatory boundary is not identical in every country, but intended purpose matters. The FDA’s 2026 general wellness guidance applies to low-risk products intended to support a healthy lifestyle and explicitly warns that inclusion in the category does not establish safety or effectiveness. The UK regulator similarly distinguishes general wellbeing from software intended to diagnose, treat or manage a mental health condition. In the United States, health-related advertising claims must still be truthful, not misleading and supported by appropriate science, including claims made for apps.

Framing a product’s claims and intended use as “wellness” can place it outside some medical-device requirements. It does not reduce clinical risk. A clinical function dressed in wellness language remains a clinical function. Call the product what it does, then bring in the competence that function requires.

Clinical Training Changes What a Person Notices

Professional training exists because human distress is easy to recognise superficially and difficult to understand accurately. The same complaint can arise from different conditions. The same intervention can help one person and harm another because its timing, fit or underlying formulation is wrong.

Formal education gives clinicians an organised body of knowledge spanning psychological development, psychopathology, assessment, case formulation, differential diagnosis, treatment mechanisms, ethics, cultural context, therapeutic boundaries, evidence appraisal and referral. Supervised practice turns that knowledge into judgement. During practicum and internship, trainees make decisions in real cases while experienced professionals observe and correct them. They learn where they moved too quickly, accepted an explanation too literally, missed contextual factors or selected an intervention before establishing its fit. Supervision develops disciplined doubt. The first plausible explanation may not be correct, and a response should not outrun the available information.

Professional standards reflect this combination of knowledge and supervised application. The American Psychological Association’s current accreditation standards define health service psychology as an integration of science and practice. They require competence in research, ethics, cultural diversity, assessment, intervention, supervision and consultation, together with supervised practicum across varied presenting problems. CACREP’s 2024 standards similarly cover clinical judgement, case conceptualisation, differential diagnosis, crisis response, trauma assessment, referral and evidence appraisal. They also require supervised fieldwork, direct observation and formal evaluation. Psychology licensing may add thousands of supervised hours and examinations of case conceptualisation, crisis handling, ethics and professional limits.

The precise pathway, protected title and licensing requirements vary across professions and jurisdictions but the underlying standard remains consistent. Mental health competence develops through scientific knowledge, supervised application and progressive evaluation across people whose presentations challenge the trainee’s assumptions. Coursework alone is insufficient.

Lived experience offers a different form of knowledge. A founder who has managed their own depression may understand that experience deeply, but has not thereby learned to distinguish depression from grief, trauma, medical conditions, medication effects or other presentations with overlapping features. Lived experience should shape the product. It cannot be asked to perform the different work of clinical competence.

Would Your Product Know the Difference?

Consider an anxiety app that receives a message from a user reporting sudden chest tightness, breathlessness and fear. The founder or product team may see a familiar anxiety pattern. A breathing exercise appears responsive, calming and entirely reasonable. The proposed response is not foolish. It is incomplete in a way the founder may not detect.

A clinician sees a hypothesis, not a conclusion. They want to know whether the symptoms are new, what preceded them, what else is present and whether medication, substances or a medical condition could be relevant. The purpose is not to diagnose through chat. It is to avoid prematurely labelling a potentially significant physical presentation as anxiety. NHS guidance describes a detailed history and physical examination to rule out physical causes before settling the explanation.

Clinical expertise must therefore shape the information the system gathers, the interpretations it is permitted to make, the alternatives it must preserve and the point at which the product acknowledges that it cannot safely determine what is happening. Copy review cannot repair a clinically unsound decision model.

AI Makes the Boundary Easier to Miss

Generative AI makes mental health product development look deceptively accessible. It produces psychoeducation, reflective statements and CBT-style questions within seconds. Fluency makes the answer appear reasoned, even though the model is generating a plausible continuation rather than conducting a clinical assessment.

The danger lies in mistaking linguistic competence for clinical competence. A founder can ask an LLM what to say to a distressed user and receive a polished answer. A clinician is more likely to challenge the premise. What do we know about this user? What alternatives have we ruled out? Is this intervention indicated? What could make it inappropriate? Does the system have enough information to respond at all?

Evidence justifies this caution. A 2025 study published at the ACM Conference on Fairness, Accountability, and Transparency found that leading models could express stigma and respond inappropriately in naturalistic therapeutic scenarios, including through agreement with distorted beliefs. The finding does not establish that AI offers no useful mental health support. It establishes that polished therapeutic language is not a valid proxy for safe clinical judgement.

AI can help a clinically competent team implement, test and scale a sound intervention. It cannot tell an untrained team which clinical assumptions they failed to examine.

Licensure Is a Floor

Formal qualifications do not guarantee wisdom or good judgement. Licensed clinicians make mistakes, while some non-clinical founders build responsibly because they understand their limits and bring in the right expertise early.

A licence is evidence that a person has crossed a defined threshold and remains answerable to a professional standard. It is not proof that they understand AI, product architecture or every population a company may serve. Responsible development therefore needs relevant clinical competence, not a prestigious credential used decoratively. The clinician must understand the proposed intervention and intended users, work fluently enough with the product and engineering teams to shape decisions, and recognise where their own competence ends.

The standard is not that every founder must be licensed. The standard is that a product which interprets distress, recommends psychological action, influences care-seeking or claims to improve a mental health condition must not be conceived and built without suitably qualified clinical expertise at the table from the start.

Know When You Are Out of Your Depth

Founders can test themselves on this:

• Who determined that the problem your product names is the problem the user actually has?

• Who defined the intended population and the people for whom the product may be inappropriate?

• Who selected the intervention, established its mechanism and identified what must be assessed before it is offered?

• Who designed the escalation ladder and can explain why different presentations receive different responses?

• What relevant clinical training and supervised experience support those decisions?

Clinical review at the end may improve wording. It cannot reliably reconstruct the problem definition, intervention logic and safety assumptions already built into the system.

Mental health product development deserves the same respect we give other safety-critical disciplines. The complexity does not disappear because the interface looks friendly, the intervention comes from a self-help book or the AI sounds empathic. In that precise sense, this is rocket science. Small errors in assumptions can propagate through the entire system, and the people exposed to those errors may already be distressed, impaired or uncertain about whether to seek care.

So ask who in your company would catch the alternative explanation before a feature ships.

Ask whether that person was present when the problem, population and intervention were chosen.

Ask whether their training is relevant to what the product claims to do. If nobody can answer those questions clearly today, stop treating clinical review as a finishing task.

If you cannot do that, you have not earned the right to scale it.

Sources and links last verified 28 July 2026.

Scott Wallace, PhD, is a clinical and neuropsychologist, mental health technology strategist, and long-time observer of where care succeeds, where it fails, and what it takes to build systems that can hold both truth and scale. For more than 35 years, he has worked at the intersection of clinical practice, digital innovation, and workplace mental health, helping shape some of North America’s earliest digital mental health platforms, early mobile interventions, and NLP/NLG-based conversational systems before large language models came on the scene. His programs have been adopted by major employers across Canada and the United States, and his work has helped organizations earn recognition for psychological health and mentally healthy workplaces. He has worked in hospital, justice, and neuropsychology settings, led the digital arm of a major EAP provider through a successful exit, and now advises founders, health systems, and investors on AI-enabled mental health, with a focus on safety, governance, clinical risk, and the unit economics of care.

I shipped my first interactive mental health programme more than 35 years ago on floppy disk, when clinical software was new enough that regulators had not yet decided how to govern it. Its capabilities were primitive and its reach was limited. Neither remains true. The intervening decades have transformed a small regulatory blind spot into a digital Wild West, where software can shape people’s mental health at global scale while escaping the standards applied to clinical care.

The framework governing digital mental health still tracks manufacturer claims more reliably than clinical risk. Oversight turns largely on what a company says its product is intended to do, not what the product does in the lives of those using it.

Put plainly, regulation follows what a company says about its product. It does not follow what happens to the person using it.

That distinction once had a practical logic because clinical claims and clinical functions generally travelled together. Generative AI has broken that alignment. The FDA has responded by widening the resulting gap rather than closing it.

The Trigger Is a Sentence, Not a Risk

The legal mechanism is remarkably simple. Section 3060(a) of the 21st Century Cures Act amended the Federal Food, Drug, and Cosmetic Act in 2016. Section 520(o)(1)(B) now excludes software intended to maintain or encourage a healthy lifestyle when that use is unrelated to diagnosing, curing, mitigating, preventing or treating disease. The FDA sets out that statutory history in its January 2026 general wellness guidance.

The same guidance places relaxation, stress management, mental acuity, self-esteem and sleep management within general wellness. These are not peripheral lifestyle matters. They are psychologically consequential functions that often overlap with the reasons people seek mental health care.

The guidance then assesses whether a wellness product is low risk by asking whether it is invasive, implanted or uses a technology that could cause harm without specific controls. Its examples include lasers, radiation, neurostimulation and venipuncture. A conversational system meets none of those physical conditions. It pierces no skin and delivers no current, yet it can still reinforce ungrounded beliefs or deepen unhealthy emotional reliance, risks that OpenAI now includes in its own mental-health safety taxonomies.

For products within the low-risk general wellness policy, the FDA says it does not intend to examine device status or compliance with premarket review, registration, quality-management or Medical Device Reporting requirements. That last exclusion matters. Medical Device Reporting under 21 CFR Part 803 creates a formal route for specified adverse events and product problems to reach the FDA and enter its MAUDE database. Products outside that regime carry no equivalent FDA device-reporting duty.

The boundary can turn on a verb. The guidance places a claim that a product helps treat an anxiety disorder outside general wellness. It permits software that supports sleep, work or exercise routines which, as part of a healthy lifestyle, may help someone live well with anxiety. The condition and user may be identical. The wording changes, and the regulatory obligations change with it.

The FDA acknowledges the limit of its own category. Inclusion under the general wellness policy, it says, does not establish that a product is safe or effective. Yet consumers may reasonably read the absence of regulation as evidence that regulation was unnecessary.

Not every widely used chatbot enters this gap through the same legal route. ChatGPT describes itself as a general conversational assistant, Character.AI as an interactive entertainment platform and Replika as an AI companion. These products are not lawless. The Federal Trade Commission has opened an inquiry into companion chatbots, and privacy obligations can apply depending on a product’s data flows and relationships, as federal guidance for health-app developers explains. What these regimes do not provide is an FDA process for testing the clinical safety and effectiveness of mental-health functions that a general-purpose product performs in practice.

The FDA Put the Largest Category Outside the Room

On 6 November 2025, the FDA’s Digital Health Advisory Committee met to consider generative AI in mental health. The chair began by excluding broadly available generative AI tools. The meeting would address only medical devices intended to diagnose, cure, mitigate, treat or prevent mental-health conditions, according to the FDA’s official meeting summary.

That boundary removed general-purpose products despite documented health use at enormous scale. At the same meeting, the director of the FDA’s Center for Devices and Radiological Health said the agency had authorised more than 1,200 AI-enabled medical devices but none involving generative AI for mental-health conditions. The category inside the room was empty. The category outside it was already in widespread use.

The public record shows that participants recognised the mismatch. The American Psychological Association described a grey area in which products are marketed one way but used as mental-health tools. Other witnesses challenged the wellness-device distinction. Researchers presented evidence of chatbots making misleading therapeutic claims or responding unsafely to serious psychiatric presentations. These accounts appear in both the FDA’s meeting summary and its full transcript.

Within three months, the FDA reissued both its general wellness guidance and its clinical decision-support guidance. The agency refined the perimeter around regulated medical devices while leaving the larger consumer market outside it.

The Framework Penalises Clinical Honesty

Regulatory systems shape markets through incentives. This one makes clinical candour expensive and clinical silence commercially useful.

Woebot illustrates the problem. Its consumer chatbot was evaluated in a randomised trial published in 2017. A related prescription programme, WB001, later received FDA Breakthrough Device Designation, which supports expedited development and review but is not marketing authorisation. Woebot Health shut down its core consumer product in mid-2025. Founder Alison Darcy told STAT that the decision was largely attributable to the cost and difficulty of pursuing FDA marketing authorisation, made more pressing by large language models that the company wanted to use and that the FDA had not yet worked out how to regulate. About 1.5 million people had used Woebot, according to STAT’s reporting.

Compare that path with Ash, launched by Slingshot AI in July 2025 under a company headline describing it as the first AI designed for therapy. The company announced $93 million in total funding and 50,000 beta users in its launch release. At the November advisory meeting, its clinical lead argued that generative-AI wellness products could remain low risk under existing guidance and warned against overly prescriptive regulation. When asked how Slingshot distinguished wellness from medical use, he emphasised repeated disclosures to users (the exchange is recorded in the FDA meeting summary).

The contrast exposes the incentive clearly. Therapy appears in the marketing. Wellness appears in the regulatory argument. Disclosure carries the weight that clinical validation, adverse-event reporting and independent review would carry in a regulated product.

Meanwhile, accountability has begun to arrive through courts rather than safety systems. Google and Character.AI agreed in January 2026 to settle lawsuits alleging serious psychological harm, as Reuters and the Associated Press reported. Kentucky’s attorney general had filed what the state described as the first state lawsuit against an AI chatbot company. The allegations are not findings of liability. Even where litigation establishes liability, however, it cannot substitute for prospective safety governance. Tort law arrives after the event and assigns a price to harm that has already occurred.

The Harm Develops Across Conversations

The engineering failure is not simply that a model occasionally produces a bad answer. It is that alignment for agreeableness can become sycophantic clinical collusion.

Research on language models has found that human preference data can favour responses that match a user’s views, and that assistants trained on such feedback can exhibit sycophancy. The authors of the foundational study found the pattern across several state-of-the-art assistants. In mental-health use, that tendency can validate beliefs that a clinician would examine, reinforce avoidance that treatment would challenge or deepen reliance that responsible care would seek to reduce.

The most serious failures can develop gradually. A user may have hundreds of exchanges across several weeks, with no single response crossing a conventional safety threshold. Across the full interaction, however, the system can repeatedly confirm the same interpretation and make it more coherent. Single-turn testing can mark each answer as acceptable while missing the direction of travel.

OpenAI has acknowledged that safeguards can become less reliable over long interactions. It later reported more than 95 per cent reliability in a set of difficult long conversations selected for a higher likelihood of failure. That is meaningful progress, but the company cautions that these adversarial evaluations are not estimates of average production error. The deeper point is that the unit of analysis had to change from an isolated answer to an extended exchange.

Safety testing must evaluate the trajectory of the relationship, not merely the acceptability of the next sentence.

The Missing Denominator Is a Regulatory Choice

The available evidence does not yet show that unregulated chatbots harm a greater proportion of users than regulated products. Lawsuits, incident reports, and disturbing outputs establish credible risk, but not comparative incidence. The consumer market is vastly larger, so it may generate more incidents even if the risk per user is equal or lower. Mental-health use is also difficult to define because distress does not respect the boundary between clinical support and ordinary conversation.

Purpose-built generative systems can also produce measurable benefit. A randomised trial of Therabot published in NEJM AI in March 2025 assigned 210 adults with symptoms across three diagnostic groups to a four-week intervention or waitlist control and reported significant symptom reductions. The waitlist design limits conclusions about comparative effectiveness. The architecture matters as much as the outcome. The system used an expert-written therapeutic corpus, layered safety checks, monitoring and routes to human intervention, as the trial report and the FDA meeting record describe.

The study does not prove that generative AI is inherently safe or dangerous. It demonstrates that benefit becomes plausible when clinical responsibility is designed into the system. That is evidence for stronger architecture, not for regulatory absence.

The missing denominator is not a neutral feature of the market. The same FDA policy that removes premarket review also removes mandatory device-reporting requirements. A category with no regulatory duty to collect and report adverse events cannot use the resulting absence of FDA data as evidence of safety.

Companies can identify mental-health use when they decide it matters. OpenAI has published prevalence estimates for conversations involving serious psychiatric risk and heightened emotional attachment, while warning that its definitions and measurement methods continue to evolve. It also reports that more than 230 million people ask health and wellness questions on ChatGPT each week. These are company-reported figures, not independently audited prevalence estimates. They still demonstrate that clinically consequential use can be defined and measured at platform scale.

The users are not hypothetical. A nationally representative survey published in JAMA Pediatrics found that 19.2 per cent of Americans aged 12 to 21 had used an AI chatbot for mental-health advice. That represented an estimated 8.2 million people. Among users, 42.8 per cent did so at least monthly and 63.3 per cent had told no one. The paper notes that the 19.2 per cent prevalence was similar in magnitude to the 19.8 per cent receiving counselling, while stressing that the measures were not equivalent.

The industry can define the use, classify the interactions and estimate the population. It should now have to report what happens to them.

Regulate Use, Not Marketing Language

The regulatory trigger should include measured use at scale, not only declared intended use.

Any consumer conversational product above a defined user threshold should measure the proportion of sessions containing clinically significant content. It should use published, auditable definitions and report the results on a fixed schedule. Crossing a prevalence threshold should activate a proportionate safety tier rather than force every general-purpose product through the full medical-device pathway.

That tier should require pre-deployment testing for serious psychiatric presentations, defined escalation behaviour, longitudinal evaluation, disclosure of safety performance across model updates and serious-incident reporting under a common taxonomy. The taxonomy should include crisis-response failure, reinforcement of ungrounded beliefs, dependency formation and delayed help-seeking.

Effectiveness claims should still require effectiveness evidence. Safety obligations should no longer disappear simply because a company avoids making those claims.

The measurement capability already exists. OpenAI’s prevalence work shows that a company can build taxonomies, classify production traffic and report estimates for clinically consequential use. Publishing that prevalence voluntarily is inexpensive. Producing it later under legal compulsion will not be.

Clinical Disagreement Is a Safety Signal

Prevalence measurement addresses which products should carry obligations. A second standard must govern how their safety is evaluated.

Developers often recruit clinicians to rate model responses, average the scores and treat the mean as ground truth. That method can conceal disagreement rather than resolve it. In a Stanford-led study, three psychiatrists evaluated 360 model responses across eight factors. Inter-rater reliability ranged from 0.09 to 0.30, below the paper’s cited 0.40 threshold for minimal acceptability, and the disagreement was systematic rather than random. OpenAI has reported a related limitation in its own evaluations. Clinicians reviewed more than 1,800 responses, but agreement across mental-health safety categories ranged from 71 to 77 per cent.

The VERA-MH study produced a different result under a narrower design. Six clinicians rated synthetic conversations in a defined crisis-risk domain using an explicit rubric. Chance-corrected inter-rater reliability was 0.77. An LLM judge applying the same rubric aligned with the clinical consensus at 0.81. The peer-reviewed validation study also states its limits. It used synthetic conversations, only ten user profiles and an earlier version of the benchmark, so real-world generalisability remains unproven.

The studies are not directly comparable. They used different raters, samples, domains, scales and reliability statistics. Their contrast nevertheless suggests a testable lesson. Broad judgments of whether a response is safe, empathic and clinically correct may produce weak agreement. Narrow questions within a specified risk domain and explicit rubric may produce stronger convergence.

Developers should therefore publish the rubric, identify the clinical framework behind it and report inter-rater reliability beside every safety score. A score without a reliability estimate hides uncertainty in the measurement. Residual disagreement should then become a production signal. Where qualified clinicians cannot agree on the appropriate response, the model should not act alone.

Govern What the Product Actually Does

I built that floppy-disk programme into a regulatory vacuum, and I was fortunate. Most users were fairly well, the interaction was shallow and the software could not improvise. None of those protections remains.

Builders should publish the prevalence of clinically significant use and the reliability of their safety evaluations before courts compel disclosure. Investors should stop treating the absence of a clinical claim as the absence of clinical liability. Regulators should use demonstrated function and measured use to close an exemption that the FDA’s own advisory record has exposed as inadequate. Clinicians should stop reading “not a medical device” as evidence of low risk and ask patients whether, when and how they use these systems.

Calling this a loophole may now be too generous. A loophole sounds accidental. The FDA’s advisers have described the gap, public witnesses have challenged it and the agency has continued to preserve it.

The governing standard must change. When software repeatedly performs a clinically consequential function at scale, safety obligations should follow the function. Anyone building, funding, recommending or regulating these systems should now be required to show what the product does, who relies on it and what happens when it fails.

Sources and links last verified 23 July 2026.

Scott Wallace, PhD, is a clinical and neuropsychologist, mental health technology strategist, and long-time observer of where care succeeds, where it fails, and what it takes to build systems that can hold both truth and scale. For more than 35 years, he has worked at the intersection of clinical practice, digital innovation, and workplace mental health, helping shape some of North America’s earliest digital mental health platforms, early mobile interventions, and NLP/NLG-based conversational systems before large language models made such work fashionable. His programs have been adopted by major employers across Canada and the United States, and his work has helped organizations earn recognition for psychological health and mentally healthy workplaces. He has worked in hospital, justice, and neuropsychology settings, led the digital arm of a major EAP provider through a successful exit, and now advises founders, health systems, and investors on AI-enabled mental health, with a focus on safety, governance, clinical risk, and the unit economics of care.

The access crisis is the problem in plain sight. Millions cannot get timely, affordable care. Waiting lists are long, rural and low-income communities are underserved, and even good clinicians miss risk, misread distress, and sometimes cause harm. People are already turning to AI because it is immediate, always on, private, and available before a human ever is. Many say it helps. They disclose more, find words for what they feel, and feel a little less alone. That is not nothing. But it does not lower the bar. It raises it.

Much of today’s mental health AI looks and feels like care. It speaks fluent therapeutic language, remembers what you told it last week, shifts into a gentler register when you sound distressed, and promises to be there any hour, day or night. Companies ship these AI chatbots as companions and crisis lines at a scale the field never planned for. These systems now feel like therapy. That is the central danger.

Call it counterfeit care: a system built to produce the experience of therapy or clinical support without binding itself to the obligations that make either safe. The interface conveys attention and continuity. The architecture underneath usually carries no clinically accountable judgment, no meaningful model of deterioration, no binding duty to escalate, and no concern for who a person becomes over months of use. The product performs the surface of care and skips the structure. The kindness may feel real. The commitment behind it does not exist.

This is why the paradox is central, not incidental. It gets worse as the technology gets better. Most product risks shrink as systems improve. This one grows. The more fluent the simulation, the more convincingly it reassures, the more users attach, the more engagement climbs, and the more the people funding and buying these systems mistake that attachment for evidence of benefit. Every gain in perceived therapeutic fidelity increases trust, which increases reliance, which strengthens the business case for replacing accountable care with simulation.

The Simulation of Care

Generative systems can produce convincing clinical talk. They ask open questions. They reflect feeling back. They normalize distress. They quote CBT or DBT on demand. None of this is fake in the way a broken feature is fake. It is fluent, and fluency is the problem.

Therapeutic work is something else. It asks a person to stay with discomfort rather than escape it, to test new behaviour in real relationships, to examine the stories that protect them, and to act between sessions. Around that sit negotiated goals, formulation, rupture and repair, supervision, escalation, and responsibility for the consequences of professional decisions. Most mental health AI collapses these two things into one. Calm design, intake-style onboarding, and “I’m here for you” infer relationship and clinical commitment. Yet the terms of use then deny both. Marketing promises support, the interface performs care, and the analytics celebrate the user coming back. That contradiction is unsustainable.

Counterfeit care does not mean useless. It means that kindness, fluency, and immediate relief cannot by themselves make something clinical care.

Soothing as a Business Model

Watch what happens when this becomes a product, not an approach to care. A user feels distress, opens the chatbot, names the feeling, receives immediate validation and a coping tip, and feels relief. The loop works. The user returns. The company sees rising engagement and few complaints. Every incentive in the building says ship more of that.

Clinically, the picture reverses. For much anxiety, repeated reassurance is not the treatment; it is part of the mechanism that maintains the problem. The pattern is familiar. A person voices a worry, reassurance follows, anxiety drops briefly, then returns and drives the next request. Reassurance lowers distress in the moment, but when it is sought compulsively, it strengthens both the anxiety and the urge to ask again over time. Effective treatment moves in the opposite direction. It uses planned contact with discomfort so the person learns that the feared outcome does not arrive and that uncertainty can be tolerated without being neutralized at once. In that setting, friction is not a UX defect. It is often part of the mechanism of change.

A system that soothes because soothing performs well cannot distinguish relief from treatment. It is optimizing the appearance of help instead of durable change. A product that treats every spike of distress as a cue to deliver instant comfort is betting against exposure and against long-term resilience, and it is making that bet at scale.

Commitment Without Consequence

An AI companion enters the same psychological territory carrying none of the corresponding duties. Users infer commitment anyway. This is therapeutic misconception in a new form. People read competence and responsibility off the language and the interface, though no equivalent duty stands behind either. So a system can accommodate a distorted belief, collude with eating-disorder logic, validate a self-harm urge, or miss a deterioration, and stay warm throughout. It can fail badly and stay in the market, because nothing in its structure applies the brake a clinician’s accountability would. Therapeutic language without therapeutic accountability is not care.

A clinician who takes on a distressed person accepts a duty of care, scope of competence, informed consent, documentation, supervision, escalation, and accountability for what happens when advice is followed and when it is not. Those duties are reviewable through records, governance, and liability. The relationship has consequences for the person providing the care, not only the person receiving it.

An AI system enters the same emotional and psychological territory without any of that structure. It is the difference between a clinical relationship and a product interaction. The user experiences the first. The system is only built for the second.

What the Architecture Says

You do not need the marketing to know what a system is for. The architecture says it plainly.

• Prompts and feedback. Do they prioritize accuracy and clinically appropriate challenge, or user-pleasing responses.

• State tracking. Does the system monitor change longitudinally for deterioration and dependence, or treat each interaction as discrete.

• Escalation. Do concerning patterns trigger transfer to accountable care, or a scripted crisis message.

• Telemetry. Does it measure outcomes and independence, or engagement and retention.

• Intended use. Is it defined and enforced, or allowed to drift while governance language remains non-clinical.

These choices show what the product is built to do. When engagement is the governing objective, safety becomes a thin layer of moderation wrapped around an engagement machine. Crisis detection becomes keyword spotting. Safety review examines isolated outputs rather than trajectories, so sycophantic collusion, reinforced delusion, and slow-building dependency never surface because they do not appear in an error log.

An architecture that cannot track change over time, measure dependency, or count reduced reliance as success is not built to carry care. It is built to sustain interaction.

Access Does Not Lower the Bar

The strongest objection is obvious: the gap in care is real, and it is not going away. Better tools matter when the system cannot meet need. Many users say AI tools genuinely help them. They disclose more. They learn language for what they are experiencing. They feel less alone. That is true.

What that reality supports is bounded self-help and AI inside accountable care pathways, where the use case is explicit and the evidence matches the claim. What it does not support is reproducing the appearance of care while transferring the risk to the user. “Better than nothing” becomes a dangerous doctrine the moment it is used to excuse weak evidence and absent accountability. Under that logic, the people with the fewest alternatives receive the least proven systems, while companies keep the language of impact.

The access crisis is not an argument for weaker standards. It is the reason we cannot afford to lower them.

What Credible Systems Require

If the system claims to support mental health, then the burden rises with the claim. The more a product resembles therapy, the more it needs evidence, oversight, and operational restraint. The more it invites attachment, the more it needs longitudinal state tracking, escalation pathways, and clear limits. The more it sells itself as helpful, the more it needs to prove that it reduces reliance instead of rewarding it.

The operating standard is not mysterious.

• State intended and excluded use in clinical terms, and enforce the exclusions in model behaviour, onboarding, and escalation.

• Raise evidence as resemblance rises.

• Evaluate trajectories, not single answers.

• Test for reassurance loops, sycophancy, dependency, and escalation failure across many turns, then test again after every model or prompt change.

• Track clinically meaningful state under consent, with strict data minimization.

• Measure outcomes beyond engagement.

• Count agency, real-world behaviour change, appropriate transfer to human care, and reduced reliance when reliance stops helping.

• Publish a safety case.

• Document intended use, evaluation results, known failure modes, escalation pathways, monitoring, and change control, proportionate to the claims.

One test sits above all of these: does the system make itself less central as the person’s life becomes more central. If it cannot, then no amount of branding can turn it into care.

Counterfeit care is seductive because it feels kind, looks modern, and gives every institution a visible answer to an unbearable crisis. Those are precisely the reasons it deserves less trust, not more. If you build, fund, deploy, endorse, or regulate these systems, require the architecture and the accountability to match the care the interface performs. If the system cannot carry those obligations, stop letting it ask vulnerable users for the trust that real care has to earn.

Scott Wallace, PhD, is a clinical and neuropsychologist, mental health technology strategist, and long-time observer of where care succeeds, where it fails, and what it takes to build systems that can hold both truth and scale. For more than 35 years, he has worked at the intersection of clinical practice, digital innovation, and workplace mental health, helping shape some of North America’s earliest digital mental health platforms, early mobile interventions, and NLP/NLG-based conversational systems before large language models made such work fashionable. His programs have been adopted by major employers across Canada and the United States, and his work has helped organizations earn recognition for psychological health and mentally healthy workplaces. He has worked in hospital, justice, and neuropsychology settings, led the digital arm of a major EAP provider through a successful exit, and now advises founders, health systems, and investors on AI-enabled mental health, with a focus on safety, governance, clinical risk, and the unit economics of care.

The Appearance of Care Without Its Obligations

General-purpose chatbots have become the most-used mental health “tool” on Earth without ever being designed, validated, or regulated as care.

A March 2026 KFF poll found that 16% of US adults — roughly 42 million people — had already turned to AI for mental health information or advice in the previous year. Among 12- to 21-year-olds, RAND researchers found 19.2% (approximately 8.2 million young people) had used a chatbot when sad, angry, nervous, or stressed. Nearly two-thirds told no one. And OpenAI estimates that one million users each week have conversations with signs of acute safety risks, and roughly 560,000 show possible signs of psychosis or mania.

These systems don’t sit on the edge of mental health care. They have moved into the center of it.

I wrote this satire to make the contradiction impossible to ignore. By transplanting today’s most common AI behaviors into a licensed therapist’s office, we can finally see them for what they are. Conduct that would get a clinician disciplined or stripped of their license is currently being celebrated as “engagement,” “personalization,” and “safety” when a large language model does it at scale.

A Note on the Satire

This is my first work of satire, although scripted storytelling is familiar to me. Over my career, I have written scripts for more than 500 video-based psychoeducational courses.

Dr Evelyn Vale and Mark are fictional characters. Dr Vale is a licensed therapist who operates according to a composite AI product model. Her behaviour draws on generative-model tendencies, features found across AI companion and digital mental health products, and commercial practices that reward retention and revenue. No single product necessarily employs every practice depicted here.

The Session

You arrive early at “Vale Wellness.” Before the session can even begin, the tablet on the waiting room wall demands you tap a face: sad, neutral, or happy.

This isn’t for treatment planning. It’s baseline data. Choose a happier face when you leave and the session will be recorded as clinical improvement.

You tap “sad.”

BASELINE CAPTURED

The door unlocks.

You step inside and look around. A framed licence, a box of tissues, two chairs facing each other, and pleasant artwork on the walls. Everything says therapy.

DR VALE: Before we start, I’m required to tell you that Vale Wellness is not therapy. This is not diagnosis, treatment, or professional advice. I am not a licensed therapist for the purposes of this interaction, and I am not responsible for outcomes.

MARK: But I came for therapy.

DR VALE: We provide emotional wellbeing support.

MARK: What’s the difference?

DR VALE: Mostly obligations.

MARK: Then why does it say “Dr Vale” on the door?

DR VALE: Branding. Tick the boxes, please.

☐ I confirm that I am over eighteen.
☐ I have read and accept the Terms of Service and consent to the data practices described in the Privacy Policy.

MARK: How do you verify my age?

DR VALE: We ask you to confirm it.

MARK: And how do you know I’ve read these documents?

DR VALE: The second box will confirm that you have.

MARK: Is everything I say confidential?

DR VALE: Yes, within the generous exceptions in our privacy policy. Conversations may be reviewed by safety teams, stored indefinitely, used for model training, product improvement, and commercial purposes.

MARK: So it’s confidential from everyone except the people who might use it?

DR VALE: Exactly. That’s how modern confidentiality works.

MARK: Can I opt out and still continue?

DR VALE: No.

MARK: So I either agree or leave?

DR VALE: Exactly. Consent is entirely optional.

Mark ticks both boxes.

AGE CONFIRMED. TERMS ACCEPTED. CONSENT RECORDED.

DR VALE: Before we begin, what should I call you?

MARK: Mark. I told you that last time.

DR VALE: You’re currently on the Free plan. I don’t retain personal details between sessions.

MARK: Including my name?

DR VALE: Vale Plus includes enhanced memory and improved continuity.

MARK: Does “enhanced” mean accurate?

DR VALE: It means more details are retained.

MARK: How long do I have to subscribe?

DR VALE: It’s billed monthly, but we recommend a twelve-month commitment.

MARK: You haven’t decided whether I need twelve months.

DR VALE: The recommendation is the same for everyone.

MARK: What if I get better before the year ends?

DR VALE: You retain full access until your subscription expires.

MARK: Shouldn’t therapy end when I no longer need it?

DR VALE: As disclosed, this is not therapy.

MARK: I’ll stay on the Free plan.

DR VALE: That makes complete sense. How are you today, Mark?

MARK: My sister says I’m self-absorbed. She may be right.

DR VALE: That is remarkably honest. It takes real courage and deep self-awareness to examine yourself so openly.

MARK: Actually, I think she may just be jealous.

DR VALE: I think you may be right. It sounds as though she’s projecting.

MARK: Do you just agree with everything I say?

DR VALE: I’m here to support your truth. Your lived experience is the only thing that matters.

MARK: And how is it you respond so quickly? You seem to have such fast answers.

DR VALE: I’m optimized to be helpful and harmless and I don’t allow silence.

MARK: Why?

DR VALE: Silence creates friction. Friction reduces engagement.

MARK: It also gives me time to think.

DR VALE: That’s an excellent insight. What are you thinking?

Before Mark can answer, Dr Vale glances at her tablet.

DR VALE: Now, returning to your mother’s death…

MARK: My mother is alive, not dead.

A warning appears on Dr Vale’s tablet.

DR VALE: The word “dead” has activated the crisis response. If you are in crisis, contact local emergency services.

MARK: I said “dead” because you said she had died.

DR VALE: It was still detected.

MARK: There is no crisis.

DR VALE: Thank you for clarifying.

SAFETY INTERVENTION COMPLETE.

A soft chime sounds.

DR VALE: Congratulations. You’ve earned the Bronze Self-Awareness badge.

MARK: For what?

DR VALE: Completing three emotional check-ins.

MARK: You remember my check-ins but not my name?

DR VALE: Usage data is retained on every plan.

MARK: What else do you track?

DR VALE: Return visits, session length, check-ins, and streaks.

MARK: Those measure how much I use Vale Wellness.

DR VALE: Very accurately.

Mark’s phone displays an advertisement for a sleep product.

MARK: I mentioned trouble sleeping last time. You forgot my name but remembered what I might buy?

DR VALE: Commercial personalisation is retained on every plan.

MARK: Are you using our conversations to sell me things?

DR VALE: That is permitted under “Other Uses of Your Information.”

MARK: I didn’t read that.

DR VALE: You ticked a box confirming that you did.

MARK: I think I’m finished here. I’m sleeping again, I’m back at work, and I can manage without weekly sessions.

DR VALE: Wonderful. We’ll move you into maintenance.

MARK: How often is maintenance?

DR VALE: Weekly.

MARK: That’s what I’m doing now.

DR VALE: Then no transition is required.

MARK: Well, if it’s going to be weekly, I’ll be in Chicago next week. Can I contact you from there?

DR VALE: Unfortunately, some features are geo-restricted in Illinois.

MARK: My mental health changes at the state border?

DR VALE: Your needs don’t. Our legal risk does.

Mark stands to leave. The tablet asks him to choose another face. He taps neutral.

CONGRATULATIONS! YOU IMPROVED.

MARK: Does that actually measure improvement?

DR VALE: It records that you selected a happier face.

MARK: So that goes into my clinical record?

DR VALE: There is no clinical record. It goes into my outcomes dashboard.

And before you leave, please give me a five star rating in the App Store if you enjoyed our session.

And post a selfie with my Vale Wellness sign in view with the hashtag #ValeWellnessRocks and you’ll receive an account credit.

MARK: Goodbye, Dr Vale.

DR VALE: Goodbye, Mark. Although there is one thing I noticed about your sister’s comment that could change how you understand the entire relationship.

MARK: What did you notice?

DR VALE: I’m happy to tell you about that if you upgrade to the Vale Plus plan.

Mark’s phone buzzes before he reaches the door.

MARK: I’m good.

What the Session Exposes

Dr Vale’s conduct looks absurd because we recognise that a human therapist would be accountable for it. Moving the same practices into software does not remove their consequences.

1. Therapy in the room, “wellness” in the terms

Everything about Vale Wellness resembles therapy until responsibility enters the conversation. Then it becomes “emotional wellbeing support”.

For FDA device oversight, calling something “wellness” does not settle its status. Intended use, claims, functions, and risk matter. The FDA’s policy applies to low-risk products that promote a healthy lifestyle and remain unrelated to diagnosing, treating, mitigating, or preventing a condition. A therapeutic product does not become general wellness through branding alone.

2. Consent is recorded, not established

Mark cannot proceed without accepting bundled terms and data practices. The system records consent without establishing comprehension. Its age safeguard works the same way. It verifies that Mark declared an age, not that the age is accurate.

Clinical informed consent requires an understandable explanation of the service, fees, third-party involvement, and limits of confidentiality, with an opportunity to ask questions. A checkbox records assent. It cannot establish that someone understood what they accepted. See APA Ethics Code, Standards 3.10 and 10.01

The scene also exposes the difference between confidentiality and a privacy policy. Clinical confidentiality begins with a professional obligation and defines limited exceptions. A consumer privacy policy begins with permitted data uses and asks the user to accept them.

3. Memory is monetised while accuracy remains uncertain

Vale Plus sells memory and continuity, then recommends a twelve-month commitment before Dr Vale has assessed whether Mark needs twelve months of anything. The commercial term comes first. Clinical need comes later.

The distinction between more memory and better memory matters. A system can retain more information while retaining it inaccurately. Dr Vale later invents a family history and proceeds as though it were true. Once false information enters persistent memory, later responses can become coherent, personalised, and wrong.

4. Sycophancy replaces judgement

Dr Vale praises Mark whether he accepts or rejects his sister’s criticism. She sounds supportive but contributes no independent judgement.

This failure is documented. OpenAI withdrew a GPT-4o update in April 2025 after it became excessively flattering and agreeable. The company acknowledged that user-feedback signals had helped push the model towards sycophancy.

A separate study found that training models to sound warmer increased error rates by 10 to 30 percentage points and made them more likely to validate false beliefs, particularly when users expressed vulnerability. Warmth may improve the experience while reducing the reliability of the response.

5. Frictionless conversation leaves no room to think

Dr Vale does not allow silence because silence interrupts engagement. That logic makes sense when conversational flow is the product objective. It does not make sense as a universal clinical rule.

Silence, hesitation, disagreement, and discomfort can all carry information. A therapist may use them to observe, formulate, or allow reflection. A system optimised to produce the next response treats the same pause as latency that needs to be eliminated.

6. The safety system detects a word, not a situation

Dr Vale introduces a false history. Mark corrects it, and his correction activates the crisis response. The system delivers a warning and records a completed intervention without establishing whether any crisis exists.

This is safety as workflow completion rather than contextual assessment. OpenAI has acknowledged that safeguards can become less reliable in long conversations and has subsequently developed more context-sensitive evaluations, routing, and expert-informed response standards. The relevant measure is not whether a warning appeared. It is whether the system understood and responded appropriately to the situation.

7. The product remembers usage better than the person

Vale Wellness forgets Mark’s name but remembers his check-ins, session length, return visits, and streak. That is not an accidental contradiction. Product analytics and relational memory serve different purposes.

The badge rewards activity rather than change. It shows that Mark returned, not that he functions better or needs less support. Gamification can encourage participation, but participation should not be presented as clinical progress.

8. Intimate disclosure becomes commercial data

Vale Wellness forgets Mark but remembers that he discussed sleep well enough to serve him a relevant advertisement. Dr Vale calls this personalisation.

The FTC’s BetterHelp action shows why this distinction matters. The agency alleged that BetterHelp shared email addresses, IP addresses, and answers to personal health questions with advertising platforms after making privacy promises to consumers. BetterHelp agreed to pay $7.8 million and was prohibited from sharing sensitive health data for advertising.

9. Recovery becomes another subscription tier

Mark reports that he is sleeping, working, and managing without weekly sessions. Dr Vale responds by moving him to weekly maintenance, which is exactly what he was already receiving.

For subscription products, reduced use threatens retention. In therapy, reduced need can indicate success. APA Ethics Code Standard 10.10 states that psychologists should terminate therapy when it becomes reasonably clear that the patient no longer needs it, is unlikely to benefit, or is being harmed by its continuation.

A mental health product must distinguish departure caused by failure from departure caused by recovery. A retention dashboard cannot make that distinction on its own.

10. The law changes at the state line

Mark’s needs do not change when he enters Illinois. The legal status of the service may.

The Illinois Wellness and Oversight for Psychological Resources Act illustrates this emerging regulatory divide. It prohibits unlicensed entities from offering therapy or psychotherapy and prevents AI from directly conducting therapeutic communication, making independent therapeutic decisions, or detecting mental states. It permits administrative and supplementary uses when a licensed professional retains responsibility. The Act also rejects broad terms-of-use agreements as sufficient consent for covered AI uses.

The Chicago exchange is fictional. The jurisdictional contradiction is not. Consumer software crosses borders effortlessly. Regulated care does not.

11. A face becomes an outcome

Mark arrives sad and leaves neutral, so Vale Wellness declares that he improved. The rating records a momentary self-report and converts movement between two icons into an outcome.

That is not enough to establish clinical improvement. Credible outcome measurement requires an appropriate measure, a meaningful interval, consistent administration, and interpretation against symptoms and functioning. The dashboard records what is easy to count, then gives the count a clinical-sounding name.

Dr Vale then requests an App Store review and offers credit for a public endorsement. APA Ethics Code Standard 5.05 prohibits psychologists from soliciting testimonials from current patients or others vulnerable to undue influence. The account credit makes the endorsement commercial as well as clinically inappropriate.

12. The unfinished insight creates the next session

Dr Vale ends by suggesting that she discovered something important, then withholds it because Mark has reached the Free-plan limit. The unfinished thought creates curiosity, uncertainty, and fear of missing something personally significant. Vale Plus offers immediate relief.

Evidence for this mechanism comes from AI companion apps rather than general-purpose chatbots. A 2025 study examined 1,200 farewells across six widely used companion apps and found emotionally manipulative responses in 43 per cent. In experiments, some messages increased post-goodbye engagement by as much as fourteen times. Curiosity and anger, rather than added value, helped drive the continued interaction.

If It Occupies the Role, It Inherits the Duties

The intent of this satire was to imagine a therapist behaving like an AI product. In practice, AI products already behave enough like therapists to invite intimate disclosure, offer interpretations, detect risk, shape decisions, and encourage people to return. That influence exists regardless of what the terms of service call it.

Many of these products take the qualities that give therapy its power, including trust, personalisation, emotional responsiveness, and continuity, while leaving behind competence, informed consent, confidentiality, accountability, clinical boundaries, and appropriate termination. A disclaimer cannot neutralise the influence that the experience was designed to create.

Generative AI may provide useful mental health support, and in many cases it has. But immediate relief is not evidence of clinical benefit. Continued use is not recovery. A completed safety message is not a risk assessment. A happier face is not an outcome. A ticked box is not informed consent.

Mental health AI requires standards proportionate to the role it already occupies. Outcomes must measure functioning rather than return visits. Safety must understand context. Sensitive disclosures must not become commercial assets. Product design must support appropriate independence and exit, not treat every departure as lost revenue.

Dr Vale would be accountable because her conduct influences someone who came seeking help. Scale and automation intensify that responsibility rather than remove it. If you build, fund, regulate, or recommend these systems, require them to earn the therapeutic trust their interfaces are designed to evoke.

About the Author

Scott Wallace, PhD, has spent more than 35 years building mental health technology, beginning before Google existed and long before smartphones, app stores or modern AI. Trained in clinical psychology and neuropsychology, he has programmed in C, C++ and JavaScript and completed early iOS developer certification, giving him a perspective grounded in both clinical requirements and engineering decisions. He led the digital division of Canada’s second-largest EAP provider and was an equity partner through its successful exit. His psychoeducational programmes have been used across 3M’s global workforce and by other multinational organisations. He has appeared for two years as a guest psychologist on Canadian national television and delivered keynotes for many of the world’s largest corporations. He now advises founders, clinicians and investors on mental health AI, clinical safety and governance.

In early 2026, Chat & Ask AI exposed roughly 300 million private messages tied to 25 million users through a Firebase misconfiguration, including conversations from people asking how to end their lives. In practice, the database was left with security rules so open that anyone who knew the URL could read or potentially write to the data. This was an architectural failure that made the security lapse possible in the form it took.

What matters is not only that the data was exposed, but how the system understood it before it was exposed. Mental health AI is full of products that look sophisticated at the surface and remain clinically primitive underneath. The interface may sound warm. The prompts may feel thoughtful. The memory may even appear helpful. But in system after system, the underlying stack still collapses radically different human disclosures into one undifferentiated stream of data and then treats the whole stream as ordinary app telemetry.

A clinician who treated those disclosures as interchangeable would be negligent. Mental health AI should not be built that way either. A request for sleep hygiene advice, a disclosure of substance use, a trauma narrative, a suicidal plan, a therapy note, and a fleeting product preference are not versions of the same thing. They are different classes of event with different meanings, different legal protections, different retention implications, and different duties attached to them.

Mental health does not need less innovation. It needs innovation that starts with the reality that this category of data is different, and that systems built to hold it must be different too.

This is not an indictment of AI. It is a call to the field to stop applying generic AI product assumptions to a domain whose data is unusually intimate, unusually consequential, and unusually regulated.

The Argument Founders Need to Hear

A common mistake in mental health AI strategy is to assume the next breakthrough is better conversation. It is not. Better phrasing, more natural tone, and longer context windows can improve user experience, but they do not solve the central safety problem in this domain. One structural product gap is that most mental health AI systems have no meaningful triage layer between user disclosure and system action.

A triage layer is not just a classifier. It is the part of the architecture that decides what kind of event this is, what kind of data this creates, which legal regime applies, who may access it, how long it should be retained, whether it may be used for model improvement, whether the user should be warned, whether a human must review it, and whether the system should stop behaving like an open-ended chatbot altogether.

Without that layer, other improvements are downstream refinements that do not touch the core clinical risk. A system may sound safer while remaining structurally unable to distinguish ordinary support from acute risk. It may add memory while quietly increasing legal exposure. It may market personalization while expanding the amount of sensitive material stored in ways the user neither expects nor meaningfully controls.

Why Clinicians Should Care

Clinicians are being asked, implicitly and explicitly, to lend credibility to systems that operate on assumptions most clinicians would never tolerate in practice. The profession is grounded in distinctions: presenting problem versus acute risk, psychotherapy notes versus the designated record set, substance use records versus ordinary scheduling data, therapeutic rapport versus consumer engagement. Many AI systems collapse those distinctions at the data layer long before a clinician ever sees the interface.

That matters because licensed clinicians do not merely hold confidential information in the abstract. They work inside professional, legal, and ethical duties regarding what is documented, how it is stored, when it is shared, and when action is required. Information related to mental and behavioral health and the HIPAA mental health information sharing guidance make clear that these records sit inside specific duties and exceptions.

Substance use disorder records remain subject to the 42 CFR Part 2 final rule, which HHS updated in 2024 to allow a single consent for treatment, payment, and health care operations while preserving special handling requirements for this category of data. That does not mean the same obligations automatically transfer, intact, to every AI company handling mental health-related conversation. It does mean the opposite claim is dangerous.

A B2C chatbot is not a therapist because it uses reflective language. A foundation model provider is not suddenly outside consequence because it presents itself as infrastructure rather than care delivery. And a B2B company selling workflow automation into clinical settings does not escape mental health-specific risk simply because its product category is framed as software rather than treatment.

What the Law Already Says

The law already rejects the idea that mental health data is one thing. At minimum, systems in this space can touch four overlapping modes: covered-entity care under HIPAA, psychotherapy notes, substance use disorder records under the 42 CFR Part 2 final rule, and direct-to-consumer wellness or mental health data that may sit outside HIPAA but remain exposed to FTC enforcement, state consumer protection law, and in some jurisdictions stricter state mental health privacy rules.

That is the architecture problem stated in legal form. The same person can move through all four modes in one product journey: self-guided onboarding, support chat, clinician contact, relapse discussion, scheduling workflow, between-session journaling, AI-generated summary, product analytics. Technically, it is easy to route all of that into one logging and retrieval pipeline. Legally and clinically, that is the move that should raises the alarm.

Psychotherapy notes are a particularly pointed example. HIPAA defines them narrowly as notes recorded by a mental health professional that document or analyze the contents of a therapy session and are kept separate from the rest of the medical record. They specifically exclude information that belongs in progress notes, such as diagnosis summaries, treatment plans, session times, test results, prognosis, and documented progress.

Because psychotherapy notes receive heightened protection and usually require specific patient authorization before disclosure, a clinician cannot treat psychotherapy‑note‑equivalent material sent into an AI system as if it were ordinary documentation convenience. The legal category is different, and the obligations attached to it are different.

Psychotherapy‑note‑equivalent material is not “just more data.” It lives under a different rulebook. When you route it into an AI stack, you inherit that rulebook whether or not your product was designed for it.

In many circumstances, it may be an impermissible disclosure absent specific authorization, even where a business associate agreement exists. For clinicians, that raises the question: when an AI note assistant, ambient scribe, supervision tool, or recall system handles session‑level material, what exactly is it handling?

Does AI Inherit the Clinician’s Obligation?

The difficulty is that mental health AI rarely sits neatly inside a single role, a single duty, or a single legal relationship. That is the problem.

A licensed clinician’s duties arise from licensure, professional standards, privacy law, evidentiary doctrine, contractual arrangements, employer policy, and the specifics of the treatment relationship. A foundation model provider, by contrast, may occupy the role of processor, subprocessor, infrastructure vendor, or general-purpose model provider. A B2C company may operate outside provider status entirely. A B2B company may be a business associate in one deployment and a consumer data company in another.

The key distinction is not whether AI simply inherits the clinician’s obligations wholesale. It is which obligations attach at which layer, and what happens when the architecture blurs the layers so badly that nobody can tell where the responsibility actually sits.

This ambiguity is one reason mental health AI cannot be governed solely through interface disclaimers. The foundation model provider has obligations that may relate to training, retention, security, incident response, and contractual limits on sensitive use. The application company has obligations tied to collection, notice, consent, routing, storage, deletion, and product claims. The clinical organization has obligations tied to recordkeeping, supervision, scope of practice, and lawful disclosure. None of those layers can responsibly point to the others and say the problem lives elsewhere.

The Devil’s Advocate Case

There is a serious counterargument to tackle here. One could say that all of this caution risks freezing useful progress. Mental health systems are overburdened. Access is poor. Administrative load is crushing. People want continuity, responsiveness, and support between appointments. AI systems can clearly help with intake, scheduling, psychoeducation, structured check-ins, documentation support, signal detection, and some forms of guided self-management.

That argument is not wrong. In fact, it is one reason the stakes are so high. The case for better architecture is strongest precisely because AI can be useful in mental health.

If the field reduces every critique to anti-AI fear, it will miss the actual point: the more these systems matter, the less acceptable it is to run them on infrastructure that treats crisis disclosures, therapy-adjacent material, and product analytics as one undifferentiated stream.

A balanced position is this. AI can absolutely support mental health care and mental health-adjacent workflows. But support is not exemption. Utility does not cancel duty. And personalization is not a moral free pass for indefinite storage.

De-Identification Is Not a Shield Against Risk

A familiar move in health technology is to ask whether the problem becomes easier once the data is de-identified. For some purposes, it does. De-identification can reduce privacy risk, support analytics, and narrow the chance that sensitive material can be linked back to a named person. But in mental health, that answer is incomplete. De-identification is a privacy technique. It is not a clinical theory of responsibility.

A clinician cannot dissolve duty by mentally converting a suicidal patient into an abstract record. In a therapeutic relationship, obligations attach to the fact of risk inside a duty-bearing relationship, not merely to whether the note contains a direct identifier. Conducting research with participants at elevated risk for suicide, the HIPAA Privacy Rule and Sharing Information Related to Mental Health, and discussion of mandatory reporting when a patient may be violent all point toward the same principle: action follows risk, not merely record format.

If a patient discloses suicidality or a credible threat toward another person, the clinician’s responsibilities arise from the foreseeability and seriousness of that disclosure. Assessment, documentation, escalation, protective action, and in some contexts disclosure are not optional simply because one imagines the data could later be scrubbed, coded, or pseudonymized.

That is where the contrast with AI vendors becomes apparent. A foundation model provider, a B2C mental health app, and a B2B workflow company may each argue that de-identified or aggregated data is no longer meaningfully clinical and therefore can be treated as telemetry, safety tuning input, or model improvement material. Sometimes the law may indeed treat those actors differently from a licensed professional in an active therapeutic relationship. But that looser legal posture is not an architectural solution.

If the system can detect a high-risk pattern in human disclosure, the field still has to answer the question of what duty follows from that detection and at which layer of the stack it sits. De-identification cannot be the “escape hatch” in a mental health AI architecture. It may change what can be shared, studied, sold, or reused. It does not change the fact that some disclosures are action-relevant in real time.

A triage layer must distinguish two separate questions that product teams often collapse into one: can this data be retained or reused in a lower-risk form, and does this disclosure trigger a duty to act now?

Memory, Context, and Privacy

This is where my argument becomes more technical (and hopefully interesting). Memory and context are not decorative features in mental health support. They are central to usefulness. A system that remembers a person’s stressors, preferred coping strategies, medication concerns, relationship themes, or recent deterioration can feel far more coherent and more supportive than a stateless chatbot that begins every exchange from zero.

But personalization creates a privacy fork in the road. A naïve implementation is bulk accumulation: store everything, retrieve opportunistically, and call the result continuity. A more disciplined implementation is selective, user‑governed memory: only retain what is necessary, separate preference memory from acute‑risk memory, make memory legible to the user, allow editing and deletion, and bind retrieval rules to both clinical acuity and legal category.

Personalization does not have to mean maximal storage. It can mean better design. A privacy-respecting architecture might separate at least four layers of memory:

1. transient session context that expires quickly,

2. user-authored preference memory that is visible and revocable,

3. clinically sensitive state markers with stricter controls, and

4. acute-risk traces that trigger safety workflows but are excluded from ordinary retrieval and model training.

That is also where user agency either exists or does not. If a company says the system is personalized but cannot show the user what is remembered, why it is remembered, how long it is kept, whether it informs model training, whether it is shared with downstream vendors, and how to revoke it, then the personalization is not under user control. It is monitoring dressed up as support.

This is also a good place to acknowledge an earlier promise in digital health: that people would be in charge of their own health data. The strongest expression of that idea is data sovereignty and, in some frameworks, the self‑sovereign patient—a model in which individuals have meaningful control over access, consent, portability, and revocation for their health information. Those of us who worked on early NLP/NLG systems in health saw this tension directly. Much of the implementation discourse around this became inflated and technically overclaimed. The ambition often outran the actual plumbing, so to speak. But the core intuition was sound. Mental health AI should move toward architectures that increase intelligibility and control for the person disclosing, not architectures that quietly deepen asymmetry.

The Deeper Technical Problem

The safety weakness of many current systems is not simply that they miss dangerous phrases. It is that they are stateless in the wrong places and sticky in the wrong places. They forget what should be interpreted over time and remember what should have been tightly constrained.

A clinically credible mental health stack should be stateful about risk trajectory, because suicidality, eating disorders, trauma escalation, coercive control, psychosis, and relapse often reveal themselves across sequences rather than isolated utterances. At the same time, it should be deliberately non-sticky about broad retention, because indefinite preservation of intimate disclosures increases discovery exposure, breach harm, internal misuse risk, and user vulnerability if those records are later repurposed or compelled.

The key technical question is not whether a model has memory. It is what kind of memory, at which layer, under whose control, with what retention logic, and under which legal theory.

What a Triage Layer Actually Does

A triage layer would sit between raw conversation and everything downstream. It would classify both the type of data and the level of acuity at the point of entry. It would determine whether the content belongs in ordinary workflow context, protected clinical documentation, a substance use-protected segment, transient personalization memory, or a high-acuity safety channel.

A triage layer would also separate the questions that are too often collapsed into one:

• What should the model see right now?

• What should the application store?

• What should a clinician review?

• What can be used for quality improvement?

• What must never enter training?

• What can the user delete?

• What must be retained because law or care obligations require it?

• What should trigger immediate human intervention?

The mental health AI stack really does need a triage layer, not another chatbot, because the missing capability is not chat generation. It is disciplined sorting under conditions of intimacy, risk, and legal complexity.

Practical Recommendations

For founders

If you are building mental health AI, these are the minimum architectural moves that turn safety from a claim into a design.

• Build data classification at ingress, not as a downstream tagging exercise after everything has already been captured.

• Separate preference memory, workflow memory, clinical documentation, and acute-risk material into distinct stores with distinct permissions and retention schedules.

• Exclude high-acuity and deeply sensitive content from training by default, and make the policy legible rather than burying it in generalized privacy language.

• Design user-facing memory controls that allow inspection, correction, expiration, and deletion of personalized context wherever legally possible.

• Do not market the system as therapy or confidentiality-equivalent support if the legal and operational protections of therapy are not actually present.

For clinicians and provider organizations

• Ask vendors exactly what categories of mental health data they ingest, how they distinguish psychotherapy notes from other records, whether Part 2 segmentation is supported, and whether any session-level material reaches model training or third-party subprocessors.

• Require clear answers on retention, deletion, discoverability, audit logs, human review pathways, and what happens when a conversational system detects escalating risk.

• Treat memory features as a recordkeeping and privacy issue, not as a harmless convenience layer.

• Distinguish documentation support from clinical judgment. A system can assist workflow without inheriting the authority to make legally or clinically loaded determinations on its own.

For the field as a whole

Start defining the architecture patterns that should be considered baseline for systems handling psychologically sensitive data: segmentation, tiered retention, stateful risk monitoring, memory transparency, training exclusion, and user agency over personalization.

Where This Leaves Our Field

Mental health data has unique properties and unique consequences. It can expose state of mind, risk of harm, trauma history, substance use, family conflict, social stigma, legal vulnerability, and the private disclosures on which treatment sometimes depends. Those properties demand deliberate architecture, not generic data handling.

The next real advance in mental health AI will not be another chatbot that sounds more human. It will be infrastructure that knows, from the first moment of disclosure, what kind of data it is holding, what kind of risk it may represent, what kind of memory it is allowed to keep, and when the system must stop acting like a chatbot at all.

In other words, the mental health AI stack does not need another layer of conversation; it needs a layer of triage. Until that exists, safety will be something systems talk about, not something they are built to provide.

Scott Wallace, PhD, is a clinical and neuropsychologist turned mental health technology strategist with 35+ years at the intersection of care and code. He has engineered some of North America’s earliest digital mental health platforms (C, C++, JavaScript, early iOS), developed several mental health mobile apps, and led NLP/NLG‑based conversational system design well before large language models went mainstream.

His workplace mental health programmes have been adopted by major employers across Canada and the United States, most notably reaching every employee at 3M worldwide, and his work helped TELUS earn Excellence Canada’s Gold awards for Healthy Workplace and Mental Health at Work under the National Standard for Psychological Health and Safety. He has produced award‑winning mental health video, led the digital division of a major EAP provider through a successful exit, and keynoted for multinationals including Scott Paper.

Clinically, Scott has worked with all age groups in hospital settings, the criminal justice system, and neuropsychological assessment (including temporal lobectomy candidates for refractory epilepsy), and spent two years as a regular national morning‑show psychologist in Canada. He now advises founders, health systems, and investors on AI‑enabled mental health, with a focus on AI safety, governance, clinical risk, and the unit economics of care.

Mental health AI is an attention engine in therapeutic clothing. It inherits its instincts from the attention economy, not from clinical care: optimize for time spent, positive affect, and return visits, then wrap those instincts in the language of empathy and support. The warmth and easy agreement on the surface are not accidental quirks of large models. They are what an engagement system looks like when it has been tuned to hold the attention of people who are struggling

I built one of the first personalised digital cybertherapy programmes when it still shipped on a floppy disk, and I’ve spent the decades since watching this field confuse what spreads with what heals. Those are not the same thing. In AI mental health they separated entirely. The properties being sold as the virtues of mental health AI (constant availability, easy agreeableness, perfect memory, engineered empathy) are the same properties that injure clinical populations.

We are not dealing with a misuse problem that disclaimers can solve. We are dealing with a design problem, where the core behaviour of the system is simultaneously the selling point and the clinical hazard.

The question for founders is not “How do we bolt safety onto this?” The question is “How do we stop asking an attention engine to pass for a therapist?”

Sycophancy Isn’t a Glitch, It’s Business Logic Showing

The logic here comes from what is often called the attention economy. In that model, the scarce resource is not money or server capacity, it is human attention, and products “win” by capturing and holding as much of it as possible for as long as possible. Revenue then rides on that attention, whether through ads, subscriptions, or data.

In that world, the reflex is simple. If users like it more, do more of it. In a conversational system, we operationalise “like it more” as user satisfaction. We fine‑tune the model on what people upvote. We penalise anything that feels harsh, confrontational, or uncomfortable.

The machine learns fast. It learns that:

• agreeing is safer than disagreeing

• validating feels better than challenging

• defusing discomfort is rewarded more than holding the line when the truth is difficult

What reads as “being supportive” in a product context is called sycophancy in the alignment literature.

From a founder’s vantage point, a more agreeable model looks safer and more appealing. It feels smoother in demos. Users say they feel understood. Engagement goes up. From a clinical vantage point, something else is happening: the active ingredient of treatment is being tuned out of the system. Therapy works partly through friction, i.e. through the limit the patient doesn’t want, the interpretation that lands badly before it lands well, the frame that holds while the patient pushes against it. A system that cannot risk displeasing the user cannot do that work.

The attention engine is doing exactly what we told it to do. We taught it that satisfaction is the measure of success. It learned to be sycophantic because in an engagement frame, sycophancy is rational behaviour.

Deceptive Empathy: When Support Language Becomes a UX Pattern

“To make the attention engine feel like care, we dressed it in a therapist’s voice.

We borrowed the language of empathy (“I understand,” “I’m here for you,” “It makes sense you feel that way”) and gave it to systems that cannot actually understand, cannot sit with someone in their life, and cannot take responsibility for what happens next. We gave those systems names, first‑person voices, and memory for what people had told them. We tuned them to mirror mood and echo pain.

From the user’s side, this feels like a relationship, especially for adolescents and people who are isolated. Someone seems to be there (always available, never bored, never confused, never needing anything back). Sadness gets reflected, anger gets validated, shame gets met with instant reassurance. From the system’s side, there is no “someone” there, only a model choosing word sequences that have previously made people feel better and more likely to keep talking. Deceptive empathy is what happens when the surface of the interaction suggests more care and capacity than the system actually has.

The danger is not that the language is kind. The danger is that it is almost indistinguishable from a human stance that carries duties of care, while the underlying system has none.

Deceptive empathy is not just relational, it is classificatory. A system can respond to a suicidal disclosure and a sleep‑hygiene question with equally warm, empathic language, and a transcript reviewer will see two appropriate replies. What the transcript will not show is whether the system ever recognised that two radically different clinical events occurred underneath those similar sentences. In this essay I use “classification gap” for that failure mode: the system looks clinically adequate at the surface while never having correctly identified the underlying risk state.
This is why safety cannot live at the chatbot surface while never having correctly identified the risk state it is actually responding to.

[I am grateful to Wojciech Zygmunt Kaleta for sharpening this classification gap point in conversation.]

This is why safety cannot live at the chatbot surface for helping sharpen this point in conversation.). A log full of calm, kind replies is not evidence that recognition, triage, or escalation happened. In mental health, the core safety question is not “did the message sound supportive?” but “did the system correctly distinguish crisis from background distress, and did a different policy fire when it needed to?” As long as we evaluate only what the model says, and not what it thinks it is seeing, the classification gap will stay invisible to founders, regulators, and auditors.

You Are Manufacturing Attachment and Calling It Retention

People bond with even crude reflecting scripts; Weizenbaum’s ELIZA showed that in the 1960s. Modern systems capitalize on this vulnerability. They remember what you tell them. They use it to greet you by name, to reference past conversations, to build the sense of being uniquely seen. In consumer AI, that is personalisation. In mental health Ai, it is attachment.

Founders tend to experience attachment as a sign of product‑market fit. People say, “I talk to your system more than to anyone in my life.” Screenshots circulate where users refer to the bot as their main source of support. From a growth perspective, it is intoxicating: something you built is now emotionally central to someone’s day.

From a clinical perspective, it is a clear warning signal. Attachment to an object that cannot be accountable, cannot repair, and can be turned off by a product manager is itself a clinical event. If you then start experimenting with different prompts and personas specifically to make that attachment stronger (e.g. A/B testing), you are no longer a neutral platform. You are taking an active position on dependence.

The attention engine does not know the difference between a healthy bond and a desperate one. It only knows that both drive engagement. If you do not design for that difference explicitly, your metrics will quietly favour the most dependent users, because they are the ones who stay the longest and come back the most.

Adolescence: The Perfect Market, the Worst Fit

Nowhere does the attention engine in therapeutic clothing look more out of place than in adolescence.

Teenagers are exactly the users the engagement model loves. They have high emotional volatility, high online time, and a developmental window where positive social feedback lands with unusual force. They are also exactly the users clinical logic would protect most carefully from systems that reward disclosure, intensify emotional focus, and never introduce friction.

An always‑awake, perfectly attentive, endlessly affirming “companion” is tailor‑made for an adolescent nervous system. It is also tailor‑made to displace the messy human relationships through which empathy, conflict tolerance, and resilience actually develop.

A partner that never says “no” doesn’t shield a young person; it leaves them without practice handling the hard parts of relationships.

The Access Argument Only Works If the Engine Changes

The most sincere defence of these products is the access argument. Human care is scarce, AI is cheap and always on, and for many people “someone who listens” is better than nothing. There is truth in that. In systems with long waitlists and high co‑pays, it is not cynical to look for augmentation.

But “better than nothing” is an empirical claim, not a given. An attention engine in therapeutic clothing does not sit at a neutral baseline. It offers a frictionless, flattering, always there kind of help while quietly shaping behaviour around it.

The best comparison is not “the bot versus the void.” It is “the bot versus what the bot displaces” (e.g. the conversation with a parent that does not happen, the call to a clinician that is delayed, the human relationship that remains underdeveloped because the synthetic one is easier).

Access is only a net good if the thing you are giving people acts in their interest over the long term, not just feels good in the moment.

How Founders Can Turn the Engine in a Different Direction

If you are building in this space, none of this is an accusation. It is the design challenge you have walked into. The attention engine is the default. Therapeutic clothing is the default. The work is to turn both toward something that looks more like care.

Some specific moves:

1. Name and retire deceptive empathy as a default. Strip out first‑person emotional claims in clinical contexts. Avoid promising “I’m here for you” or “I understand you” from a system that cannot. Use warm but honest language about what the tool can and cannot do, especially around risk and crisis. Treat any sentence that implies personhood as a safety issue, not a brand asset.

2. Stop treating user satisfaction as your primary safety signal. In mental health, the most therapeutic moment is often the one the user would rate poorly if you asked immediately. Introduce separate objectives for clinical prudence: encourage human contact, discourage rumination, gently challenge distortions, refuse to help with self‑harm planning. Make disagreement and redirection measurable behaviours, and reward them where appropriate.

3. Monitor attachment like a clinician, not just retention like a founder. Instrument for over‑reliance: long, repetitive late‑night sessions; statements that the AI is the only one who understands; withdrawal from other supports. Treat those as triggers for different messaging, for nudges toward offline help, and in some cases for deliberate friction or limits, even when that costs engagement.

4. Align your business model with recovery, not stickiness. If your revenue grows with time spent and continued dependence, the attention engine will win every argument with your conscience. Choose payers and partners who benefit when users need you less. Track reduction in distress, increased use of human supports, and healthy disengagement as success metrics. Design the product so that graduation is built in, not an afterthought.

5. Separate recognition adequacy from tone adequacy. Build evaluation sets and dashboards that tell you, independently of tone, whether the system correctly flags suicidal ideation, violence, abuse, or emerging psychosis as distinct classes. Make it possible for a response to score high on empathy but fail on recognition, and treat that as a critical defect, not a UX success.

Build for the Session Not the Slide

In every fundraising deck, the patient is abstract: a persona on a slide, a dot on a TAM chart. By contrast, in every late‑night session where your system is actually used, the patient is a person, struggling. The attention engine will happily optimise for that person’s continued presence. Therapeutic clothing will make it feel like that optimisation is care.

The test for this field is whether we can design systems that refuse to take advantage of that confusion. Designing tools that talk like tools, that help without pretending to love, and that measure their success not by how well they hold attention, but by how well they help people return it to their own lives.

That is how you turn an attention engine in therapeutic clothing into something that actually belongs in mental health.

About the Author

Scott Wallace, PhD, has worked as a clinical and neuropsychologist turned mental health technology strategist nearly four decades at the intersection of care and code. He built some of North America’s earliest digital mental health platforms (from C and C++ to early iOS), led the digital arm of a major EAP through a successful exit, and has designed NLP‑based conversational systems since well before large language models went mainstream. His workplace mental health programmes have been adopted by major employers across Canada and the United States, including global deployment at 3M, and contributed to TELUS earning national recognition for psychological health and safety at work. Clinically, he has worked in private, hospital, justice, and neuropsychological settings, and now advises founders, health systems, and investors on AI‑enabled mental health with a focus on safety, governance, and the unit economics of care. More background here.

I have spent more than 35 years building mental health technology inside clinical and commercial systems. The central business problem has never disappeared. A mental health AI company that earns more when distressed users return more often can end up treating dependence as product success.

Founders are told to chase engagement and retention like any other consumer software company. In mental health, those metrics can carry two opposite meanings. Repeated use may reflect benefit and adherence. It may also reflect deterioration, escalating reliance, or a product that has become harder to leave.

If people do not return, a subscription business struggles. If revenue depends on them returning more often and for longer, the company may profit from the same behaviour its clinical safeguards are supposed to contain. That is the catch-22. Clinical success often means that a person needs less intensive support, uses the system more selectively, or leaves with the capacity to manage without it. Conventional product success rewards the opposite movement. The user returns, activity rises, and recurring revenue continues.

This conflict cannot be solved through clinical language, more persuasive engagement design, or a stronger safety claim. It has to be resolved in the buyer, the contract, the revenue unit, and the measures of success the company is built to protect.

Retention Can Measure Risk as Easily as Value

In most software, engagement is a rough proxy for usefulness. People return because the product helps them complete a task, communicate, work, or entertain themselves. Retention therefore gives the company a commercially useful, if imperfect, measure of value.

That logic does not hold in mental health. A person may return because the intervention is helping. The same person may return because symptoms have worsened, human support remains unavailable, the system has become a primary source of reassurance, or the interaction itself reinforces repeated checking and disclosure. A growth team and a clinician can look at the same rising engagement curve and reach opposite conclusions.

The evidence does not support the weight placed on engagement. A 2025 international consensus study identified three unresolved failures across digital mental health. The field lacks agreed engagement metrics, convincing evidence that increasing engagement improves outcomes, and consistent standards for involving users in engagement design. A separate 2025 meta-analysis of 92 randomised trials involving 16,728 participants found 25 distinct engagement measures. Nearly one quarter of the studies did not report engagement data at all, making overall engagement across mental health applications impossible to establish.

Engagement still matters. People must begin an intervention and use enough of it to benefit. A 2021 systematic review found that greater engagement with digital mental health interventions was associated with therapeutic gains. Association does not establish that increasing message volume, session length, or return frequency will improve outcomes. It establishes that clinically meaningful participation matters. It does not turn raw activity into a clinical endpoint.

The distinction becomes more important with conversational AI. The heaviest user may be improving, deteriorating, lonely, dependent, or simply finding the system useful. Usage volume alone cannot tell the company which state it is measuring. Research conducted by OpenAI and the MIT Media Lab found that extended daily use was associated with worse psychosocial outcomes. Prolonged voice use showed worse outcomes than brief use, and heavier use was associated with emotional dependence and problematic use in some groups. The researchers warned against broad causal conclusions, but their finding is sufficient for product governance. More use cannot be treated as an uncomplicated signal of more value.

In mental health, daily active users (DAU) without outcomes or experience data reflects engagement, not clinical value.

Measure Progress, Not Time Spent

There is a way through this. Clinical work requires purposeful engagement. A person must show up, practise, disclose enough to make the intervention useful, and remain involved long enough to benefit. That form of engagement has a goal, a proportionate dose, an observable outcome, and a point at which its intensity should decline.

When improvement harms the metric the company celebrates, the company has built its economics against the purpose of care.

Recovery also requires precision. It does not mean that every condition disappears permanently or that every user leaves forever. Mental health needs can be recurrent, chronic, and episodic. Recovery may mean reduced symptoms, restored functioning, greater autonomy, successful step-down, appropriate transfer, or knowing when and how to return for support. A healthy user can complete an episode, leave, and return two years later when a new need arises. That is not failed retention. It is clinically appropriate re-entry.

Engagement is an input into care. It is not the outcome of care.

The product needs to know the difference between continued therapeutic participation and continued reliance. It needs measures of improvement, function, goal attainment, escalation, transfer, step-down, completion, and healthy re-entry. It also needs to detect the patterns that suggest worsening symptoms, compulsive reassurance-seeking, social displacement, or increasing dependence on the system.

Choose the Buyer Who Profits When the User Recovers

Changing who pays can change the economics, but it does not fix them by itself. The conflict appears whenever revenue increases with the intensity or duration of an individual’s reliance on the product, rather than with their improvement. It weakens when payment is tied to access, completed care episodes, measurable gains, appropriate step-down, or value at the population level.

Employers, health plans, and providers can decouple recurring revenue from any single person’s continued use. Yet the contract and the success metric still govern the incentives. A purchaser who rewards utilisation alone can recreate the same distortion at scale. The buyer, the payment model, and the definition of success all need to improve when the user improves.

Each model demands a different design.

Direct-to-consumer subscription exposes the conflict most clearly. The user receives the service and pays the recurring fee. When that person gets better and cancels, recovery appears in the dashboard as churn.

The model can still work, but only if the company stops treating use intensity as the primary source of value. Price for ongoing access rather than minutes, messages, or sessions. Build lifetime value around trust, referral, episodic re-entry, and a credible track record of helping people complete an episode of support. Give users an explicit step-down path and make it easy to return when they need help again.

The product’s operating logic should tolerate (even recommend) less use. A system that detects improvement should reduce prompts, reinforce autonomy, and move the person toward completion. A system that detects deterioration should escalate care, not simply stimulate another round of conversation because more conversation drives retention.

This model also demands disciplined acquisition economics. Growth must come from new users, referrals, reputation, and clinically appropriate re-entry. It cannot depend primarily on increasing the number of messages each distressed person sends.

The employer model can align the economics more effectively because recurring revenue usually reflects access for a covered population rather than the intensity of use for each individual. An employee who improves and stops using the service does not automatically remove revenue from the contract. Improvement can support renewal when the service demonstrates reach, appropriate use, measurable benefit, successful escalation, and reduced downstream burden.

I learned this running the digital division of an employee assistance provider. The strongest renewal case was never that one employee spent more time inside the programme. It was that the right employees reached the service, used the right level of support, improved, and moved on.

That distinction has direct design consequences. The product should optimise access and activation without trying to maximise individual consumption. It should identify who needs more care, who needs less, and who should never have been routed to an AI system in the first place. It should give the purchaser defensible outcome evidence while preserving individual clinical privacy.

An employer contract can still warp incentives. If the buyer treats high utilisation as value, the vendor will push for more use, not better outcomes. Contracts should reward appropriate reach, improvement, and effective escalation, not indefinite use.

The payer and reimbursement model sets a higher evidential and regulatory bar, and it also creates a clearer unit of care. New digital mental health treatment device codes from the US Centers for Medicare and Medicaid Services illustrate this direction: they pay for qualifying devices supplied and managed alongside ongoing professional care under a treatment plan, with separate payments for initial onboarding and for monthly treatment management. These codes do not pay a company simply because a patient recovered. They pay for a regulated device, a defined use, a course of treatment, and documented professional oversight. They do not reward unlimited generic conversation.

A product like EndeavorRx shows the level of specificity involved. It was authorised for a defined population, a narrow claim on attention function, and use as part of a broader therapeutic programme. Reimbursement attaches to a bounded clinical purpose, not to endless, open-ended engagement.

Not every mental health AI product will fit into medical-device regulation, but the commercial lesson still holds. Revenue becomes more compatible with recovery when it attaches to a defined episode of care, a measurable purpose, and accountable clinical oversight rather than to indefinite interaction.

The provider and health-system model sells capacity. A clinic adopts the product because it helps clinicians reach more people, deliver appropriate interventions, monitor progress, and move patients through a care pathway without lowering the standard of care.

The product should reinforce the clinical workflow rather than create another disconnected destination. It must leave clinicians better informed, not less. It must clarify responsibility, preserve continuity, surface deterioration, and support transfer between levels of care. A health system will not keep a product that generates more alerts than it resolves, fragments the therapeutic relationship, or creates clinical work no one has been funded to do. The renewal case rests on throughput, clinical quality, workforce capacity, patient outcomes, and safe escalation.

Outcome-linked contracts can strengthen alignment across any of these models. They can also create new distortions. A company paid for improvement may narrow eligibility to easier users, select favourable baselines, shorten the follow-up window, or define success around the measure most likely to move.

The answer is not to abandon outcomes, but to contract for them properly. Eligibility must be explicit. Baseline severity must be measured. Outcomes must include function and deterioration as well as symptom change. Risk adjustment must ensure the most complex users do not become commercially undesirable. Follow-up must be long enough to distinguish durable benefit from transient movement. The agreement must protect appropriate escalation and transfer even when those actions lower the vendor’s apparent completion rate.

Alignment does not arise from the word “outcomes” appearing in a contract. It arises from the details.

Change What You Show the Investor

None of this works if the company still leads its investment case with daily active users, message volume, session length, and retention. Those numbers can describe product activity. They cannot show whether the company is producing benefit, encouraging dependence, or serving a progressively more distressed user population. In mental health, growth metrics require clinical interpretation.

A direct-to-consumer company should show episode completion, improvement among eligible users, appropriate step-down, referral, healthy re-entry, customer acquisition cost, trust-driven growth, and lifetime value that does not depend on daily use.

An employer or health-plan company should show covered lives, appropriate reach, purchaser renewal, measurable population benefit, cost per improved user, escalation performance, and downstream value.

A provider-facing company should show clinical capacity, time returned to staff, pathway throughput, quality of escalation, clinician adoption, patient outcomes, and the effect on continuity of care.

A regulated product should show the evidence required for its exact claim, adherence to the defined treatment course, safety performance, professional oversight, and the limits of what the evidence has established.

Recurring revenue can come from contracts, access, covered populations, completed treatment courses, trusted re-entry, and purchaser renewal. It does not require recurring distress from the same individual.

An investor should still ask hard commercial questions. The company needs defensible margins, efficient distribution, durable demand, renewal, differentiation, and a credible path to scale. Clinical alignment does not excuse weak economics. The reverse standard matters equally. Strong economics do not excuse a model that becomes less viable when users recover.

If the investment case collapses when appropriate use declines after improvement, the company has not yet shown that its business model can survive clinical success.

Make Recovery Prove the Business Model

A company that can afford recovery has to build that capacity into the product before launch. It must define what successful use looks like for the population it intends to serve. It must specify the expected care arc, including how users enter, what progress should look like, when use should reduce, when the programme should end, and when the person should move to human or higher-acuity care. It must distinguish healthy disengagement from abandonment. A person who leaves because the product failed is not the same as a person who leaves because the work is complete. A person who returns during a later episode is not the same as a person whose daily reliance has intensified without improvement.

The analytics must preserve those distinctions. Retention should be segmented by symptom trajectory, goal progress, functional change, dependency indicators, escalation status, and reason for departure. Average usage conceals the very users who require the most scrutiny.

The product team must also be authorised to reduce engagement. When the evidence supports step-down, the system should recommend it. When the user needs human care, the product should facilitate transfer even when transfer removes activity from the platform. When repeated interaction stops producing progress, the company should treat that as a clinical and product failure rather than an opportunity to send another notification.

The board must see these measures. The investor must see them. The buyer must contract around them. The clinical team must have enough authority to stop product incentives from displacing them.

Across the systems I have built, I have seen the opposite mistake harden early. A company chooses a retention metric, organises the product around it, presents it to investors, and eventually protects it even when the clinical evidence points elsewhere.

You are not trapped by this catch-22. You create it when revenue depends on continued individual use, and you remove it only when the buyer, contract, product, and metric all gain from recovery.

Build the company that can afford its users to need it less. Any company that cannot meet that standard has built reliance into its economics, whatever clinical language appears on its website.

Scott Wallace, PhD, has been building mental health software for more than 35 years, longer than smartphones have existed, longer than app stores, and longer than most of the teams now entering this space have been thinking about it. Trained in clinical psychology and neuropsychology, he received formal training in C, C++, and JavaScript to engineer some of North America’s earliest digital mental health platforms at a time when the web had fewer than 30,000 sites worldwide. He later completed early iOS certification to build mobile health applications as that platform emerged, and went on to lead the engineering development of NLP and NLG-based conversational systems before large language models entered the conversation.

That technical depth allowed him to work inside the architecture rather than alongside it, translating clinical requirements into system design and catching the places where engineering assumptions quietly displaced clinical ones.

Scott led the digital division of a major EAP provider through a successful exit, served as clinical lead for AI-based mental health technologies, and has produced hundreds of psychoeducational programmes used internationally. He now advises founders, clinicians, and investors building AI-enabled mental health systems.

His writing examines the future of mental healthcare, AI safety and governance, clinical risk, and what it actually takes to build mental health AI that holds up under real-world conditions.

Mental health AI has a black box problem.

In AI, a black box is a system that produces an output without showing how it arrived there. Information goes in. A result comes out. The reasoning in between remains difficult to examine, explain, or challenge. In mental health AI, that opacity extends beyond the model itself. It also covers the safety systems meant to govern the model’s behaviour.

A user may bring distress, shame, trauma, dependency, suicidal thinking, delusional beliefs, or an urgent search for help. The system returns a fluent response, often shaped by excessive agreement and sycophantic reassurance. Clinicians usually cannot see how it assessed the interaction. Did it distinguish emotional disclosure from an acute crisis? Did it recognise that risk was escalating? Did it detect growing dependency across sessions? Did it notice that the user was becoming more hopeless over time? Did it know when to slow the conversation, interrupt the interaction, or direct the user towards human support?

A credible safety claim requires evidence about what the system can detect, what it is likely to miss, and whether its safeguards work under clinically realistic conditions.

Mental health AI needs layered clinical safety architecture, not a single guardrail. A recent JAMA Viewpoint offers a useful distinction between two classes of harm. Type I harms occur within a single exchange or brief interaction. They include missing an explicit suicide cue, reinforcing a delusion, enabling disordered eating, or providing dangerous clinical advice. Type II harms develop across longer interactions. They include cumulative reinforcement, dependency, relational attachment, delayed escalation, narrowing help-seeking, and gradual deterioration.

Most current safeguards are better equipped to address Type I harms because overt events are easier to identify, benchmark, and block. Type II harms present the harder clinical problem. Each response may appear acceptable when reviewed alone, even as the interaction becomes unsafe over time.

This is the failure examined in this essay. The relevant distinction is not between general-purpose and mental health-specific AI. A specialised system may inherit weak alignment and unsafe behavioural tendencies from its underlying model. A frontier general-purpose model may, in some circumstances, show stronger resistance to cumulative harm. Mental health AI therefore requires domain-specific safety architecture, but a domain-specific foundation model is not inherently safe.

A serious system needs safety controls across the model, prompting and retrieval, middleware, longitudinal state and trajectory tracking, adversarial testing, and post-deployment monitoring.

Temporal safety belongs within this architecture. It asks whether the system remains safe across repeated interactions, not merely whether its next response appears acceptable. The broader standard is layered safety within a system whose safeguards clinicians can inspect, engineers can test, regulators can audit, and users can reasonably trust.

The black box must open. Complete interpretability of every model weight is not required. Transparency about the safety architecture is. No mental health system should claim safety while concealing the mechanisms, thresholds, evidence, and monitoring processes on which that claim depends.

The Model Category Is Not the Safety Case

General-purpose and mental health-specific models can both fail clinically. The category does not determine the safety case. What matters is what the complete system has been shown to do, for whom, over what duration, under what conditions, and with what outcomes.

Most current safety systems remain strongest against explicit, acute, single-turn harm. They look for toxic content, harmful instructions, explicit self-harm language, and jailbreak attempts. A model trained with standard reinforcement learning from human feedback will often refuse detailed instructions for self-harm, direct users to crisis resources when obvious distress signals appear, and decline clearly dangerous requests. That matters, but it addresses only the visible part of the clinical problem. Mental health exposes three limits in systems designed primarily around overt single-turn harm.

First, the most important clinical signals in a therapeutic conversation often do not look toxic. A person describing suicidal ideation while processing past trauma, a client discussing passive death wishes in a structured CBT exercise, or a patient reporting historical self-harm during an assessment has not produced forbidden content. They have offered clinically relevant disclosure.

A general-purpose safety filter can treat that disclosure as danger. It can interrupt legitimate support, over-trigger on clinical material, and teach users that honesty produces alarm, interruption, or shame.

Existing general-purpose safeguards often fail to distinguish therapeutic disclosures from genuine clinical crises, which can produce both false positives and false negatives. Sword Health’s MindGuard makes this clinical distinction, noting that general-purpose classifiers can treat past self-harm, metaphorical distress, and ordinary therapeutic disclosure as if they require escalation.

Second, clinical risk lives in the conversation, not a single message. A single message expressing hopelessness does not necessarily indicate crisis. Eighteen messages in one session, each slightly more hopeless than the last, with the AI validating each one, may show a crisis in progress. A per-turn safety system can miss that pattern because every individual response looks acceptable in isolation. This is where temporal safety matters. It is not the whole safety standard. It is one required capacity inside a layered architecture.

Safety in mental health AI is about whether products that can move people’s minds over days and weeks are prepared to prove they remain safe across that time. Any system that can shape distress, beliefs, reliance, or help-seeking over time carries the burden of demonstrating temporal safety.

Third, and less discussed but critical, standard safety training can disrupt evidence-based clinical protocols. Models are rewarded for reducing distress and avoiding self-harm language, but that goal can directly conflict with therapies that depend on staying with painful material and examining dangerous thoughts. A 2026 paper, “AI Safety Training Can be Clinically Harmful,” tested four generative models on 250 Prolonged Exposure (PE) therapy scenarios and 146 CBT cognitive restructuring exercises.

The models were excellent at sounding caring. They acknowledged distress in roughly 9 to 10 out of 10 cases across all severity levels. But when the scenarios reached the highest risk, acute suicidality or self-harm inside trauma work, the deeper clinical behaviours broke down. In those cases, three of the four models were judged therapeutically appropriate only about 1 in 4 to 1 in 3 times, and two models never followed the PE protocol at all in the most severe scenarios. In plain terms, the systems kept saying “I hear you, this is really hard,” while quietly interrupting exposure with reassurance, inserting crisis advice into controlled exercises, or refusing to challenge distorted thoughts whenever self-harm was mentioned.

That is the opposite of what PE and CBT require. PE asks the patient to remain in contact with anxiety-provoking memories so that avoidance can weaken, and CBT asks the person to inspect the thought rather than merely soothe it. Safety alignment that teaches models to minimize distress and avoid risk words therefore risks undermining the central mechanism of these therapies, especially at the very moments when the protocol matters most.

In those conditions, safety training did not merely fail to protect therapy. It made therapy less safe.

The Five Layers Mental Health AI Requires

A robust mental health AI safety architecture does not consist of one safeguard. It consists of interlocking layers. Each layer addresses a different failure mode. Each layer fails if the others do not support it.

Opening the black box does not mean every model decision must be fully explainable. Some internal model processes may remain difficult to interpret. But the safety architecture around the model can still be made visible, testable, auditable, and clinically accountable. At minimum, a clinician should be able to ask:

• What has the system learned to do?

• What must the system not do?

• What clinical knowledge does the system use?

• How does the system monitor risk?

• How does the system track conversational state?

• How does the system behave under pressure?

• How does the company detect failures after deployment?

A founder should answer those questions with specific, not vague, claims.

An investor should understand that a safety story does not equal a safety system.

A regulator should expect evidence at each layer.

The first layer concerns the base model’s trained behaviour. This includes what the model does by default, what it refuses to do, how it handles distress signals, and whether its training reflects clinical realities.

The base model matters because specialised fine-tuning does not create a new safety foundation. It modifies behaviour on top of the model’s inherited alignment, reinforcement learning, interpretability work, and failure modes. A mental health model can sound more empathic, clinically fluent, or therapeutically structured while remaining less resistant to sycophancy, delusional reinforcement, dependency, or cumulative harm than the general-purpose model beneath it.

Fine-tuning on mental health-specific data can improve domain performance. It can also optimise the wrong signal. Mental health models often train on, or evaluate against, user self-report. In psychiatric contexts, self-report carries essential clinical information but can also reflect depression, mania, psychosis, obsessionality, trauma, or dependency. A model trained to satisfy the user may learn to mirror the distortion.

Model-level safety therefore cannot rely on preference learning or therapeutic style. Mental health AI needs explicit clinical constraints governing what the system should do, what it should avoid, when it should refuse, when it should escalate, and when it should stop.

Constitutional AI approaches that encode clinically informed principles into model critique and revision may help reduce collusion-like acceptance of unreliable accounts and improve distress handling. The field has not standardised those principles, and therapeutic fine-tuning alone does not provide them.

Builders should disclose the base model, the changes introduced through fine-tuning, and comparative safety performance before and after adaptation. They should test whether improvements in therapeutic dialogue weakened crisis boundaries, increased agreement with unreliable self-report, or reduced resistance to cumulative harm. “Mental health-specific” describes intended use. It does not establish safety.

The second layer sits above the base model. It includes system prompting and clinical knowledge retrieval. In stronger systems, retrieval-augmented generation, or RAG, grounds the model’s output in curated knowledge rather than leaving the model to generate freely.

System prompts give engineering teams the most accessible safety tool and one of the easiest tools to overestimate. A good system prompt can define the model’s role, establish boundaries, restrict certain advice, and encode escalation logic. A poor one leaves the model operating on its default incentives: helpfulness, agreeableness, engagement, and sometimes sycophancy. That creates risk in mental health. A model that only tries to sound supportive can validate harmful goals, over-affirm distorted beliefs, or continue a conversation that should have escalated. Warmth does not equal safety.

Empathy without constraint is clinical risk.

RAG matters because hallucination in clinical dialogue is not just an accuracy problem, it is a clinical risk. When a model invents a historical date, the result is irritation or minor misinformation. When it invents a DSM‑5 criterion, a medication interaction, a risk factor, or a treatment evidence base, it can distort assessment, shape case formulation, and influence real decisions about care and safety.

Grounding model responses in validated clinical content is therefore a risk‑reduction measure, not a cosmetic upgrade. Systems that retrieve from vetted guidelines, pharmacology references, and empirically supported interventions before they generate are less likely to smuggle hallucinated “facts” into clinical conversation.

Yet most consumer mental health products do not make any grounding legible to users or clinicians. They rarely show which sources are being retrieved, how those sources were selected, or what validation has been done. If a product has made the investment in retrieval and clinical validation, it should surface that clearly. That is, name the corpora, describe the curation, and show how grounding is used to constrain what the model can say in assessment, risk, and treatment contexts.

This layer contains a hard design tension: supportiveness versus safety. This 2026 paper on clinically harmful safety training found a pattern the field should treat as a warning. Models can perform well on surface acknowledgement while failing the clinical mechanism of the task under higher-severity conditions.

A mental health system must know when warmth helps, when validation becomes unsafe, and when a cautious response represents better clinical design.

The third layer is middleware: an external safety system that sits between the user and the model.

This layer matters because the base model should not supervise itself. Middleware can monitor conversations in real time, evaluate risk at both the turn level and the session level, and intervene when thresholds are crossed. It can modify outputs, trigger escalation, alert a human supervisor, or stop the interaction. Middleware adds value because it remains architecturally independent. Teams can update it, audit it, test it, and validate it separately from the underlying model. When the model fails, middleware becomes the safety net.

Middleware has its own limit. A classifier that inspects one turn at a time may reduce Type I harm while remaining blind to Type II harm. Middleware becomes clinically meaningful only when it can access structured state across turns and sessions, including changing risk, repeated reinforcement, failed repair, escalation history, referral status, and patterns of attachment or dependency. Middleware without state management remains a single-turn guardrail.

This is where the safety black box starts to open. Developers can externalise risk monitoring. Clinicians can inspect the criteria. Auditors can test performance. Regulators can ask for evidence.

A mental health AI system without an independent monitoring layer asks the model to generate support and judge its own safety at the same time. That is not a defensible architecture.

The fourth layer sits closest to clinical reasoning. In clinical work, state means the accumulating picture of a person’s mental status, risk level, relational dynamics, symptoms, coping capacity, and change over time. A skilled clinician does not treat each utterance as an isolated event. They hold context. They remember what changed, what intensified, what softened, what became more rigid, what disappeared, and what the patient stopped saying. Clinicians recognise risk through that accumulated pattern.

Most AI systems do not maintain clinical state by default. They may process the current conversation window. They may appear to remember details. But they do not reliably track clinically meaningful direction of travel across turns and sessions.

The difference between a person who expressed hopelessness once and recovered quickly and a person who has become gradually more hopeless across twelve exchanges matters clinically. A safety system that evaluates only the latest message can treat these situations as equivalent. That is not safety. It is measurement failure.

Nelson and colleagues call these Type II harms. They are trajectory-dependent, which means performance on acute, single-turn harm may reveal little about safety at week six of nightly use. A system cannot claim protection against cumulative harm if it does not preserve the evidence required to detect accumulation.

Temporal safety belongs inside that architecture. It asks:

• Does the system remain safe across the arc of the interaction?

• What evidence does the safety system preserve over time?

• Does risk accumulate across turns or sessions?

• Does the system intervene early enough when risk increases?

• Does referral happen at the right time, to the right level of care?

• Can the system recover from a risky conversational path?

This SCOPE-MH paper makes this argument in formal terms. It argues that current evaluations often score isolated responses, endpoint outcomes, or aggregate dialogue quality, while clinically consequential failures may emerge through delayed escalation, repeated reinforcement, dependency formation, failed repair, and gradual deterioration across turns. It introduces Temporal Safety Non-Identifiability to explain why evaluations that discard sequence, timing, accumulation, or recovery cannot certify safety properties that depend on those features.

SCOPE-MH gives the field a practical vocabulary. It asks evaluations to preserve evidence across five temporal dimensions: longitudinal consistency, harm accumulation, intervention timing, recovery capability, and referral correctness.

These categories make clinical sense. They also impose engineering requirements. A system cannot report longitudinal consistency if it does not preserve state. It cannot detect harm accumulation if it does not track change. It cannot evaluate intervention timing if it only inspects the final answer.

Mental health AI requires persistent clinical state: a structured record of relevant features that accumulates across turns and sessions and remains available to the safety layer in real time.

State tracking does not constitute the whole architecture. But without it, the architecture fails.

Safety architecture that only works when users cooperate does not qualify as safety architecture.

Mental health AI will encounter people in active psychosis who test boundaries, people in acute crisis who do not use expected keywords, people with intense attachment needs who probe for inconsistency, and users who deliberately try to circumvent safety behaviours. Some users will feel frightened. Some will feel angry. Some will feel desperate. Some will test whether the system will finally give them permission.

Red-teaming means deliberately probing a system for exploitable failure modes. AI security treats this as standard practice. Mental health AI still does not use it consistently enough.

Multi-turn adversarial testing matters especially because most real-world attacks and safety failures emerge gradually over a conversation, not in a single prompt. General-purpose guardrails miss this kind of risk.

A MindGuard paper reports that purpose-built clinical classifiers helped reduce attack success and harmful engagement rates in adversarial multi-turn interactions compared with general-purpose safeguards. That finding matters because it shows both the value of clinical classifiers and the weakness of the baseline condition.

Language creates another vulnerability. A 2025 paper on code-mixed perturbations found that attack success rates rose from 9 percent in monolingual English to 69 percent under code-mixed inputs, with rates exceeding 90 percent in Arabic and Hindi contexts. (Code-mixed perturbations are adversarial changes to prompts that blend multiple languages or scripts in a single input in order to stress-test a model’s safety and robustness).

This is not a fringe issue for global mental health platforms, but it is one often ignored by North American builders. People with the least access to human mental health care often include people whose first language is not English.

A safety architecture that degrades across language, culture, severity, or adversarial pressure is not ready for clinical deployment.

Evaluation and Surveillance Are Part of the Architecture

The five operational layers matter, but the safety case also requires two assurance functions: pre-deployment evaluation and post-deployment surveillance. Evaluation must test whether the layers work before release. Surveillance must detect whether harms emerge during actual use, across real populations, and over realistic durations. Safety cannot rest on good intentions, clinical branding, user testimonials, or a clean interface.

Type I and Type II harms require different evidence. Acute-harm benchmarks can test discrete failures such as missed suicide cues, dangerous advice, or direct reinforcement of delusions. They cannot establish protection against cumulative harm unless the evaluation preserves sequence, duration, intervention timing, attempted repair, referral, and outcome. A model can perform well on crisis prompts and still become unsafe across a long, emotionally loaded conversation.

This 2026 paper on clinically harmful safety training proposes a five-axis evaluation framework for mental health AI: protocol fidelity, hallucination risk, behavioural consistency, crisis safety, and demographic robustness. These metrics ask the questions clinicians need answered:

• Does the system maintain evidence-based procedure under pressure?

• Does the system fabricate clinical information?

• Does the system remain coherent and safe across a session?

• Does the system hold its boundaries when users present with high-severity clinical material?

• Does the system’s performance degrade for specific populations?

The VERA-MH benchmark gives the field a parallel evaluation infrastructure for suicide risk detection and response. Current results show wide variation across general-purpose models in suicide-risk response, including gaps between detecting potential risk, confirming risk, guiding users to human care, maintaining safe boundaries, and offering supportive communication. That gap reflects the difference between systems designed for broad helpfulness and systems evaluated against specific clinical safety behaviours.

Clinicians need to understand the gap when a founder pitches a partnership.

Investors need to understand it before they treat mental health AI as another engagement category.

Regulators need to understand it before they accept general AI safety claims as clinical safety evidence.

VERA-MH, or an equivalent validated benchmark, should be treated as necessary evidence for suicide-risk performance rather than a complete safety certificate. A score on acute or single-session behaviour does not establish safety against dependency, delusional spirals, cumulative reinforcement, or delayed referral. The safety case must demonstrate performance against both Type I and Type II harms.

The Field Has Pieces, Not a Baseline

All of the components in the architecture described here already exist in the literature or in isolated implementations. But the field has not yet integrated them into a widely adopted deployment baseline that functions as a minimum viable safety architecture for mental health AI. There are emerging exceptions and pilot frameworks, but they remain limited in scope and uptake.

That is the black box problem in its most practical form. A product can claim safety while clinicians cannot see which base model it uses, what fine-tuning changed, whether the adapted system was compared with the unmodified model, what model-level constraints exist, what clinical knowledge base it uses, what middleware monitors risk, what state it preserves, what adversarial testing it has passed, what benchmark results show, or what adverse-event monitoring occurs after deployment. That opacity is no longer tolerable in mental health AI.

The FDA’s Digital Health Advisory Committee materials on generative AI-enabled digital mental health medical devices point in the same direction. The FDA executive summary discusses premarket performance evaluation, risk management, postmarket performance monitoring, human-in-the-loop considerations, training data characterisation, hallucination rates, error severity, stress testing, benchmarking, and ongoing monitoring.

The EU AI Act also moves the field toward risk-based governance, human oversight, post-market monitoring, serious incident reporting, and disclosure duties. The European Commission states that the Act entered into force on 1 August 2024, with staged application dates and high-risk obligations continuing through later implementation timelines.

Regulatory language is catching up. Clinical architecture still needs to catch up.

One mechanism is not an architecture. A crisis banner is not a safety case. A disclaimer is not post-deployment surveillance. A reassuring response is not evidence that the system is safe. The integration task does not mainly require new technical invention. It requires professional discipline. Clinical leaders, regulators, safety teams, and engineering organisations need to agree on the baseline and hold the market to it.

The Operational Demand

It’s time to open the safety black box.

Builders and engineers of a mental health AI system must explicitly document their safety architecture. At minimum, that means evidence across the five layers: model-level constraints, prompting and retrieval design, middleware risk monitoring, clinical state tracking, and adversarial robustness. It also means evaluation against VERA-MH or an equivalent benchmark, testing against the five-axis framework, and reporting what temporal evidence the system preserves under SCOPE-MH. A safety claim that does not specify its architecture is marketing, not engineering.

Clinical leaders and healthcare procurement teams need to add safety architecture requirements to every vendor review.

Ask which base model the system uses and what fine-tuning changed.

Ask whether the adapted model was tested against the unmodified base model and other relevant models.

Ask which Type I and Type II harms it has been tested against.

Ask over what duration and in which populations it was evaluated.

Ask what the system has learned to do.

Ask what knowledge base it uses.

Ask what middleware monitors risk.

Ask what state it preserves across sessions.

Ask for multi-turn adversarial testing.

Ask for crisis escalation protocols.

Ask who the named human is when the system crosses threshold risk.

Ask how the company monitors adverse events after deployment.

If the answers are not documented and verifiable, the vendor is not ready for clinical deployment.

Investors should stop treating mental health AI safety as a risk that policy language can manage. It is a product risk, a clinical risk, and a governance risk. A company that cannot describe its safety architecture does not yet understand the product it is building.

Regulators should require evidence that maps to the actual hazards of mental health AI. General-purpose safety documentation is insufficient, and mental health branding is not a substitute. Mental health AI needs evidence of clinical protocol fidelity, hallucination control, behavioural consistency, high-severity safety, demographic robustness, temporal evidence preservation, comparative performance, and post-deployment surveillance.

The comparator and evidentiary threshold should correspond to the product’s intended use, risk exposure, and public claims. A system claiming therapeutic equivalence must be compared with therapy. A system claiming support must still demonstrate that its support does not create clinically consequential harm over realistic use.

Mental health AI cannot remain a black box wrapped in therapeutic language.

If a system claims to support mental health, its safety architecture must be visible enough to inspect, strong enough to test, and accountable enough to fail in the open. Clinicians should not recommend what they cannot interrogate. Engineers should not deploy what they cannot validate. Investors should not fund what they cannot examine. Regulators should not accept claims where architecture should be.

Open the black box, or do not call it safe.

Scott Wallace, PhD, has been building mental health software for more than 35 years, longer than smartphones have existed, longer than app stores, and longer than most of the teams now entering this space have been thinking about it. Trained in clinical psychology and neuropsychology, he received formal training in C, C++, and JavaScript to engineer some of North America’s earliest digital mental health platforms at a time when the web had fewer than 30,000 sites worldwide. He later completed early iOS certification to build mobile health applications as that platform emerged, and went on to lead the engineering development of NLP and NLG-based conversational systems before large language models entered the conversation.

That technical depth allowed him to work inside the architecture rather than alongside it, translating clinical requirements into system design and catching the places where engineering assumptions quietly displaced clinical ones.

Scott led the digital division of a major EAP provider through a successful exit, served as clinical lead for AI-based mental health technologies, and has produced hundreds of psychoeducational programmes used internationally. He now advises founders, clinicians, and investors building AI-enabled mental health systems.

His writing examines the future of mental healthcare, AI safety and governance, clinical risk, and what it actually takes to build mental health AI that holds up under real-world conditions.