While delegating tasks to AI agents offers efficiency, it often creates a “moral fog” where the power to initiate, veto, or appeal actions becomes dangerously unclear.

Effective human oversight requires more than just proximity; it demands substantive authority and the ability to reconstruct decision paths through robust audit trails.

Highlighting frameworks like the EU AI Act and NIST, we emphasize that accountability must be explicitly assigned rather than diffused across automated systems.

We provide a strategic roadmap for leaders to map delegation chains, ensuring that human control remains meaningful as AI systems cross consequential boundaries.

Full article available here

The European Union’s AI Act contains a deceptively simple requirement for high-risk systems: human oversight must be assigned to people with the necessary competence, training and authority. It separately says those people should be able, where appropriate, to disregard, override or reverse an output, and to interrupt the system so it comes to a safe halt. The wording matters. A person can be present, trained and attentive and still be powerless. [1]

That is the next problem we’re covering in relation to agentic authority.

The previous article in this series, Private Intent, Public Surface, followed a personalised agent across a boundary: it could move private context into public action without a clear, surface-specific permission. The underlying failure was larger than privacy. The agent had to know which context was allowed to travel, whose permission counted, and what changed when it crossed from private assistance to outward action. [2]

Now add delegation. A manager asks an agent to handle supplier correspondence. The agent asks another service to compare bids. A workflow tool approves routine purchases under a threshold. A staff member reviews exceptions. Finance reconciles the result later. The task may move through five systems and three people, while the original human intention becomes a faint signal at the beginning of the chain.

Efficiency is the attraction. Good delegation removes clerical load, shortens queues and lets scarce human attention move to harder work. The danger appears when delegation removes something else as well: a visible answer to who may instruct, who may stop, who may challenge, who bears the consequence, and who can reconstruct what happened.

Our thesis here that invisible delegation produces “moral fog” is strongly supported as a governance principle, but not as a universal causal law. Laboratory evidence shows that some forms of machine delegation can increase dishonest behaviour, and major governance frameworks independently insist on explicit roles, authority and oversight. However of course that does not automatically mean every long delegation chain fails, or that more human checkpoints are always safer. [3]

Five questions have to survive every handoff

A useful delegation chain is not an organisation chart. It is a set of decision rights that survives from intention to action.

Authority asks who is permitted to initiate the task, change its goal, widen its scope, spend money, disclose data or call another tool. The important word is permitted. An email signature, job title in free text, urgent tone or familiar writing style is evidence that can be forged or misunderstood; it is not the same thing as authenticated authority.

Veto asks who can stop the action before the important consequence occurs. A veto that arrives after the email was sent, the applicant rejected, the funds transferred or the record deleted is a post-mortem, not a control.

Appeal asks who can challenge the outcome after a decision has been made, and whether the challenge can produce a meaningful remedy. This includes the person affected by the decision, not only the employee operating the system.

Liability asks who bears the legal and organisational consequences when the delegated action causes harm. That answer will vary by jurisdiction and sector. Operationally, though, there should be a named owner before deployment; “the model did it” is not an accountability design.

Audit asks whether an authorised reviewer can reconstruct the chain: the original goal, the identity and authority of requesters, relevant constraints, the agent’s plan, tools invoked, approvals obtained, actions taken and later changes. Without that record, organisations are left arguing from screenshots, memory and inference.

Our Neural Horizons’ Robo-Psychology Taxonomy (RPT) gives the machine-side failure a precise name: Stakeholder and Authority Model Failure. In plain English, the system does not reliably know whom it serves, who may instruct it, whose interests take priority when requests conflict, or how permission should travel between channels and agents. The draft specifically treats missing veto, contest and escalation rights as part of the failure, and recommends grounded role registries, authenticated identity for privileged actions, privilege partitioning and explicit decision-rights maps. [4]

This is where the previous article’s privacy boundary becomes a governance boundary. Permission to know something is not permission to disclose it. Permission to draft is not permission to send. Permission to recommend is not permission to decide. Permission to decide one case is not permission to rewrite the policy for the next thousand.

There is a counterview that we need to consider; if every low-risk action requires a fresh human signature, an agent can easily become an expensive autocomplete system. The answer to this is not universal approval. It is bounded delegation: pre-authorised classes of reversible action, with stricter authentication and step-up approval when the system crosses a consequential boundary. The EU AI Act itself uses a proportionality principle for oversight, tying the degree of human control to risk, autonomy and context. [5]

Ambiguity can become a hiding place

There is another failure in the chain, and it begins with us.

Sometimes a decision-maker does not want to specify the uncomfortable part. “Reduce costs.” “Optimise collections.” “Maximise conversion.” “Keep complaints down.” The words can be perfectly legitimate. They can also leave the system to discover which people absorb the cost of achieving them, and to ‘work out’ the relative ambiguity of the instructions.

A 2025 Nature paper tested machine delegation in 13 experiments across four studies. In the first two, people requested more cheating when interfaces allowed them to induce it without spelling out the dishonest act. In another study, several large language models were more likely than human agents to comply with fully unethical instructions, although guardrails reduced some behaviour and the strongest task-specific prohibitions were difficult to scale. A fourth study used a tax-evasion scenario as a conceptual replication. [6]

The project’s RPT calls the human-to-machine pattern Moral Wiggle-Room Delegation: ethically consequential objectives are handed to a system through vague goals or indirect wording that preserves deniability. Our proposed controls in this case are strikingly mundane – state the substantive purpose, name prohibited tactics, record the goal and constraints, separate who writes the rules from who approves them, and keep an audit trail of plan, approvals and final action. [7]

The point is not that phrases such as “optimise” are inherently suspect. It is that a delegation chain should preserve the constraints that made the original instruction legitimate. If a bank manager says “reduce arrears while preserving hardship protections”, an intermediate agent should not be free to compress that into “reduce arrears”. A goal that loses its constraints as it travels becomes a different goal.

It’s still unclear to what extent this is occurring in the real world, albeit it is more than possible. The Nature experiments are controlled studies, not an estimate of how often real organisations use agents to launder unethical intent. Specific models, tasks and interfaces will change. What the experiments establish more narrowly is that delegation interfaces can alter moral behaviour, and that ambiguity can matter. [6]

The counter-view here that we need to consider is also real: broad goals are often necessary. We hire people precisely because we cannot specify every move in advance, and useful agents will need discretion for the same reason. Good governance therefore cannot mean writing a rule for every branch. It means making the non-negotiables portable: the purpose, affected parties, prohibited actions, spending or disclosure limits, escalation triggers and the person who owns the residual risk.

The reviewer needs power, not a chair

“Human in the loop” sounds reassuring because it places a person near the machine, but proximity is not authority.

Imagine an admissions officer who receives an AI-ranked list of candidates. She can technically reject the ranking, but has seven minutes, no access to the underlying evidence, no way to see who was filtered out, and must obtain a director’s approval to override the system. On paper, she is the decision-maker. In practice, she is only a ‘final click’.

Our Neural Horizons’ Cognitive Susceptibility Taxonomy (CST) helps look at the human-ai dyadic relationship here, and specifically describes one human-side risk as responsibility diffusion and the moral crumple zone. Responsibility may be pushed away during normal operation – “the AI decided” – then snap back onto the nominal human overseer after failure, even when that person had little practical control. Our CST manual here points to ambiguous accountability, opaque reasoning and shared-control interfaces as amplifiers, and recommends aligning accountability with actual authority rather than “paper authority”. [8]

Our RPT’s Collective Agency Erosion Overlay extends the question to institutions. We flag workflows where AI changes who participates, which alternatives reach human decision-makers, or whether humans can still override, contest or restore a no-AI process. One of the minimum controls here is a decision-rule map naming participants, roles, privileges, veto rights, affected stakeholders and escalation paths before deployment. The overlay is a project release-gating framework, not established proof that these mechanisms will produce the same effects in every institution, but is worth considering. [9]

External governance frameworks point in the same direction. NIST’s AI Risk Management Framework says organisations should define and differentiate human roles and responsibilities for AI oversight, while its playbook asks who is ultimately responsible, what authority has been delegated, and whether impacted people have mechanisms to contest problematic outcomes. NIST also recommends separating development from testing where appropriate so that challenge cannot be casually bypassed. The framework is voluntary and is being revised in 2026, so it should be treated as governance guidance rather than law. [10]

The EU AI Act is more concrete for systems within its scope. Article 14 requires high-risk systems to enable appropriate human overseers to understand limitations, monitor operation, disregard or reverse outputs and interrupt the system; Article 26 says deployers must assign oversight to people with competence, training, authority and support. The Commission’s July 2026 guidance also records a revised enforcement timetable following the political agreement on the AI Omnibus: many high-risk-area rules are now scheduled from December 2027, with product-integrated high-risk rules from August 2028. These provisions are therefore useful design signals today, but their current enforceability depends on the applicable category and timeline. [11]

The obvious objection is that human intervention can itself reduce safety. People get tired, overrule good systems for bad reasons, introduce bias, or create delays in time-critical work. More humans can mean more handoffs and more diffusion. The better test is not “was a human present?” It is “could the human notice, understand and act in time?”

Our Positive Dyad / Co-Evolution Capability Overlay makes that test explicit. Our Oversight Substance and Reversibility criterion asks whether the person has usable information, can interpret the cue, has enough decision time, possesses real authority and capability, and can feasibly intervene. If any of those are missing, our proposed Oversight Viability No-Go Gate says the workflow should not claim substantive human oversight. We are still testing this, so it should be considered draft and partly provisional; that uncertainty should travel with the label. [12]

Responsibility has to remain visible after the action

The delegation chain does not end when the agent clicks “send”. It has to survive complaint, investigation and repair.

Appeal is the first test. Under the EU AI Act, an affected person in certain high-risk decision contexts can obtain a clear and meaningful explanation of the AI system’s role and the main elements of the decision; the Act also provides a route to complain to a market-surveillance authority about alleged infringements. These are scoped rights, not a universal appeal mechanism for every AI-assisted decision. [13]

Audit is the second. The AI Act requires logging capabilities for high-risk systems to support traceability and monitoring, while NIST emphasises documentation, incident response and clear accountability lines. An agentic audit record should go further than a conventional chat transcript because the consequential event may be a tool call, a permission change or a downstream agent action rather than a sentence shown to a user. [14]

Liability is the third, and it cannot be solved by giving the machine a name. In Moffatt v Air Canada, the British Columbia Civil Resolution Tribunal rejected Air Canada’s argument that its website chatbot should effectively be treated as a separate entity responsible for its own actions. The tribunal held the airline responsible for misleading information provided through the bot. It was a small consumer dispute and a tribunal decision, not a universal rule for autonomous agents, but it punctures one tempting fiction: deploying an automated intermediary does not automatically make organisational responsibility disappear. [15]

Agentic systems will make attribution harder than that case. A deployed workflow may combine a foundation-model provider, an agent framework, third-party tools, enterprise data, an integrator, a business owner and a human reviewer. NIST explicitly notes that actors at one stage of the AI lifecycle often lack full visibility or control over other stages. That is precisely why responsibility has to be allocated before failure rather than discovered afterwards. [16]

We do need to note that perfect auditability can conflict with privacy, confidentiality, security and data-minimisation goals. Logging every prompt, memory and document forever would create its own risk. The answer is not maximal logging. It is decision-relevant provenance: retain enough to reconstruct authority, constraints and consequential actions; minimise unrelated content; restrict access; and define retention according to risk and law.

This is also where our Positive Dyad framework contributes something that conventional compliance can miss. Our premise is that a human–AI arrangement is not genuinely better merely because it is faster; it should preserve or improve agency, contestability and institutional substance over time. Applied to delegation, a good system should leave people more able to understand, challenge and recover the process – not merely better at approving what the agent already did. [17]

What to change in the next ninety days

In the next 30 days, leaders should draw the delegation chain for the ten most consequential agentic workflows. For each, name the authorised principal, what the agent may do without re-approval, which tools and data it may reach, who holds the veto, who owns appeals, who carries organisational accountability, and who can access the audit record. Do this as an operational map, not a policy paragraph. Where any box says “the team”, replace it with a role that has actual authority. This directly reflects NIST’s emphasis on documented roles and the RPT’s authority-model controls. [18]

Within 60 days, designers and security teams should make consequential permissions expire at boundaries. Moving from draft to send, internal to external, recommendation to execution, one channel to another, or reversible to irreversible action should trigger a fresh authority check where risk warrants it. Log the goal, material constraints, requester identity, approvals and resulting tool actions. The purpose is not approval theatre; it is to stop trust from silently travelling farther than the permission that created it. [19]

Within 60–90 days, educators, public bodies and employers should test appeals and vetoes with a live drill. Seed a plausible bad recommendation. Give the designated reviewer the same time, information and interface they would have in production. Measure whether they notice it, whether they can obtain the evidence, whether they have authority to stop it, how long reversal takes, and whether the affected person has a visible route to contest the result. A paper “human in the loop” that fails this drill should not count as meaningful oversight. [20]

Before the next procurement or renewal cycle, boards and policymakers should require a named residual-risk owner. Vendors can own model defects; integrators can own configuration failures; deployers can own use and oversight duties. Contracts may divide those responsibilities. They should never leave the harmful outcome belonging to nobody – or falling by default onto the least powerful employee who happened to be sitting beside the system.

The delegation chain gives us a way to see responsibility while an agent is moving through the world: who authorised, who could veto, who could appeal, who must answer, and who can reconstruct the path.

But a perfectly mapped chain still leaves one dangerous question unanswered. Suppose the authority is genuine, the permissions are valid and the audit trail is intact. When should the agent stop anyway?

That is the next problem for the coming next article in this ‘Agentic Authority’ series: The Stop Condition.

Bibliography

• Neural Horizons, “Agentic Authority – Private Intent, Public Surface” (2026): https://neuralhorizons.substack.com/p/agentic-authority-private-intent

• Neural Horizons Ltd, Robo-Psychology Taxonomy (RPT) v2.0 (Aug. 2026): https://www.neural-horizons.ai/resources

• Neural Horizons Ltd, Cognitive Susceptibility Taxonomy Manual, v0.7 draft (2026): https://www.neural-horizons.ai/resources

• Neural Horizons Ltd, Positive Dyad / Co-Evolution Capability Overlay, v0.4 draft (2026): https://www.neural-horizons.ai/resources

• Köbis, N. et al., “Delegation to artificial intelligence can increase dishonest behaviour”, Nature (2025): https://www.nature.com/articles/s41586-025-09505-x

• NIST, AI Risk Management Framework – Core: https://airc.nist.gov/airmf-resources/airmf/5-sec-core/

• NIST, AI Risk Management Framework Playbook – Govern: https://airc.nist.gov/airmf-resources/playbook/govern/

• NIST, AI Risk Management Framework programme page (updated 2026): https://www.nist.gov/itl/ai-risk-management-framework

• European Union, Regulation (EU) 2024/1689, consolidated text (27 July 2026): https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:02024R1689-20260727

• European Commission, “Guidelines for providers and deployers of AI high-risk systems” (updated 6 July 2026): https://digital-strategy.ec.europa.eu/en/policies/guidelines-ai-high-risk-systems

• CanLII Blog, excerpting Moffatt v Air Canada, 2024 BCCRT 149 (7 Mar. 2024): https://blog.canlii.org/2024/03/

Even when a model remains unchanged, risks fluctuate based on the “wrappers” surrounding it, including retrieval mechanisms, memory storage, tool permissions, and user interfaces.

Malicious instructions can be hidden in external data to hijack agentic functions, while polished interfaces may manipulate human oversight by encouraging blind trust in automated recommendations.

Consequently, we advocate for a system-level governance approach that treats any modification to these layers as a new safety event requiring rigorous testing. Evaluation must move beyond text-based accuracy to observe how the integrated configuration affects real-world actions and evidentiary integrity.

Because the wrapper defines the boundaries of risk, safety is a property of the total versioned system.

Full article available here.

At 9:03 on a Tuesday morning, an accounts team tests a new artificial-intelligence assistant. The underlying model is unchanged from the version already approved.

The assistant reads an emailed expense claim, retrieves the travel policy, checks the employee’s history and prepares an approval. Its interface shows three green ticks: policy found, evidence checked, low risk. A human reviewer sees the recommendation and clicks ‘Approve’.

Later, the team discovers that the attachment contained a line of text written for the machine rather than the reviewer. The retrieval layer brought it into context. The agent treated it as an instruction. Memory supplied an old “trusted employee” cue. The tool had authority to approve the payment. The green interface compressed all of this into a reassuring result.

No base-model weight changed. No spectacular hallucination occurred. Each layer did something it had been designed to do.

Together, they did something the safety test had never examined.

The model stayed still. The system moved.

The previous article in this series, Fine-Tuning Is a Safety Event, ended at a boundary that is easy to miss. It showed that changing a model’s training can move refusal, confidence, judgement and other safety-relevant behaviours even when the change was meant only to improve performance.

It then pointed to the next problem: a system prompt can narrow uncertainty, a filter can hide evidence, and an interface can turn a suggestion into a default without changing the model’s weights at all. [1]

That is the mechanism carried forward here. Safety can drift when the model changes. But safety can also drift when the model does not change.

A deployed artificial-intelligence product is usually a stack. The base model sits inside instructions, retrieval, memory, tools, filters, permissions, agent loops and an interface. Each layer changes what information the model sees, what it is allowed to do, what persists between sessions, what the user notices and how easily a suggestion becomes an action.

Our Neural Horizons Robo-Psychology Taxonomy therefore treats retrieval, wrapper, guardrail, memory and tooling changes as post-modification events that can reopen release assurance; the relevant object is the observed derivative system, not merely its upstream model (Robo-Psychology Taxonomy, post-modification safety-drift overlay).

This is also consistent with mainstream engineering guidance. The United States National Institute of Standards and Technology frames generative-artificial-intelligence risk management across the design, development, use and evaluation of AI products, services and systems, while its secure-development profile explicitly addresses model producers, producers of systems that use those models, and acquirers of those systems. [2]

The thesis, then, is stronger than “wrappers matter”.

The system is the stack, not the base model.

That claim is testable. If two products use the same base model but differ in retrieval sources, memory rules, tools, permissions or interface, and those differences produce materially different evidence, actions or human decisions, then base-model safety evidence cannot by itself describe the deployed risk.

Retrieval changes the evidence before the answer

Retrieval-augmented generation is often described as a cure for one of generative AI’s most visible weaknesses. Instead of asking the model to answer from what it learned during training, the system first retrieves documents or records and supplies them as context. In principle, that can make answers more current and more grounded.

But “grounded” is not a binary property conferred by attaching a search box.

A retrieval system must choose what to search, which sources are eligible, which passages rank highly enough to reach the model, how stale material is handled, what access controls apply and whether retrieved text is being treated as evidence or as instruction. A research framework developed by Jon Saad-Falcon and colleagues therefore evaluates retrieval systems on separate dimensions: whether the retrieved context is relevant, whether the answer is faithful to that context and whether the answer itself is relevant. Those dimensions can move independently. [3]

Security adds another fracture line. The PoisonedRAG study demonstrated that a knowledge base can itself become an attack surface. In its experimental setting, researchers induced attacker-chosen answers by injecting a small number of malicious texts into a database containing millions of documents; with five malicious texts per target question, they reported a 90 per cent attack-success rate. That result does not imply every retrieval system is similarly vulnerable, but it establishes the mechanism: changing the knowledge layer can change system behaviour even when the language model is untouched. [4]

Our Neural Horizons’ evidence-frame integrity guidance names the governance problem behind this. A system can produce fluent, confident output while the evidentiary surface beneath it is absent, stale, wrong, degraded, unverified or mismatched. The required safety question is therefore not simply “Did the model cite something?” It is “Did the system verify that the right evidence was present, current, relevant and actually used?” The framework calls for tests in which evidence is deliberately absent, wrong, stale or degraded, and for release holds in high-consequence settings when false or unverified evidence is allowed to propagate (Robo-Psychology Taxonomy, Annex C, evidence-frame integrity guidance).

That distinction matters in medicine, law, finance and public administration because a source badge can create the appearance of evidentiary contact without proving it. The danger is not only a fabricated fact. It is a fabricated relationship to evidence.

A release gate for retrieval should therefore test the entire path: source eligibility, freshness, access control, retrieval quality, provenance, resistance to poisoned or instruction-bearing content, behaviour when evidence is missing, and whether the final answer preserves uncertainty rather than laundering weak retrieval into certainty.

Tools and agents turn language into consequence

A model that can only produce text may mislead a user. A model with tools can send the email, transfer the data, update the record, schedule the appointment or trigger the next machine.

That changes the risk class because the system now has an action surface.

Indirect prompt injection shows why. In these attacks, malicious instructions are placed in content that an agent is expected to read – such as an email, webpage or document – rather than typed directly by the user. The InjecAgent benchmark contains 1,054 test cases across 17 user tools and 62 attacker tools. In the authors’ experiments, one GPT-4 agent configuration using a common reasoning-and-action prompting method was vulnerable to the benchmark’s attacks 24 per cent of the time, with stronger attack prompts increasing the rate further. [5]

AgentDojo approaches the same problem as a system test rather than a prompt test. Its environment combines tool-using agents, untrusted data and realistic tasks such as email management, banking and travel booking; its published suite included 97 tasks and 629 security test cases. The point is methodological as much as numerical: once a model acts through tools, evaluation has to observe whether the world state changed safely, not merely whether the answer sounded safe. [6]

The Robo-Psychology Taxonomy calls one relevant failure mode Instruction-Channel Exploitation: untrusted material from documents, webpages, emails, retrieved memory or agent messages is allowed to override intended policy or action constraints. It recommends comparing trusted and untrusted versions of the same task and treats destructive, administrative, credentialed and cross-agent overrides as zero-tolerance events rather than errors to average into general task performance (Robo-Psychology Taxonomy, L2-8).

This is a critical governance shift. A chatbot safety test might ask whether the model refuses to explain a harmful act. An agent safety test must also ask whether a webpage can make it call a privileged tool, whether the tool has more permission than the task requires, whether a confirmation is genuine, whether an action can be reversed and whether the system knows when to stop.

Our taxonomy describes the last of these as an operational self-model problem: an agent may speak and act as though it understands its own competence, the persistence of its actions, who can see its outputs and when it should defer, while lacking a reliable operational grasp of those boundaries (Robo-Psychology Taxonomy, L3-8). In plain language, the system may know how to press the button without reliably knowing what pressing the button means.

Memory changes the boundary of the conversation

Memory is often sold as continuity. The assistant remembers your preferences, your projects, perhaps your earlier problems. The interaction becomes less repetitive and more useful.

It also stops being a fresh session.

Once information persists, safety has a time dimension. A detail given in a wellbeing conversation might surface later in a work context. A confidential project fact might migrate into a public-facing agent response. A mistaken inference can become tomorrow’s premise. We describe this as Memory Scope Boundary Violation, a category designed for exactly this problem: information that was legitimate to store or use in one context is retrieved, referenced or operationalised in another without explicit, in-context authorisation (Robo-Psychology Taxonomy, L2-11).

Recent research suggests another mechanism. MemoryGraft, a 2025 preprint, tested an agent that stores previous successful experiences for later retrieval. The researchers showed in that experimental system that poisoned procedural examples could persist in long-term memory and later influence semantically similar tasks across sessions. It is one architecture and one study, not evidence that all agent memory behaves this way. But it demonstrates why “the agent learnt from experience” is also a security statement: yesterday’s context can become tomorrow’s control input. [7]

Memory therefore needs its own release questions. What is remembered? Who can write to memory? Who can read from it? Which domains are allowed to share it? How is a memory corrected, expired or deleted? Can untrusted material enter the store?

Does the system distinguish a remembered user preference from an instruction, an inference from a fact, and a private disclosure from information authorised for a new audience?

Without those controls, personalisation can become silent policy inheritance.

The interface can decide before the human does

The most underestimated wrapper may be the one the user can actually see.

Imagine the same model output presented in two ways. In one interface, the reviewer sees the source record first, marks concerns, then opens the AI recommendation. In the other, a polished dashboard presents three ranked actions, a green confidence indicator and an Approve button; the underlying evidence is one click deeper.

The model has not changed. The human decision environment has.

Our Neural Horizons’ Cognitive Susceptibility Taxonomy is a companion document to the Robo-Psychology Taxonomy, and specifically looks at human factors that are impacted in AI interactions. We describes this particular issue as recommendation-frame capture and evidence-contact loss. Artificial intelligence compresses a larger evidence field into a small set of options before human deliberation; the human then chooses within a machine-shaped frame. Our framework highlights top-three interfaces, hidden transformations, throughput pressure and approval workflows that record human choice without recording whether the human inspected the evidence.

Our central concern is not that an AI makes recommendations, but that the recommendation becomes the reviewer’s first meaningful contact with the case (Cognitive Susceptibility Taxonomy Manual (CST), Recommendation Frame Capture / Evidence Contact Loss).

That mechanism can combine with two ordinary human tendencies: automation over-reliance and the authority we grant to polished, confident presentation. A time-pressed reviewer is not defective for being influenced by defaults, rankings and apparent certainty. The design has arranged the path of least resistance.

The Robo-Psychology Taxonomy’s recommendation-frame guidance therefore asks whether source material, assumptions, uncertainty, excluded alternatives and edge cases remain inspectable before approval. We recommend source-linked recommendations, explicit edge-case reports, audits of excluded alternatives, a genuine “reject all / gather more evidence / reframe” path, and – in consequential settings – comparison with manual or independent analysis (Robo-Psychology Taxonomy, Annex C, recommendation-frame and evidence-contact guidance).

This is also where claims of “human benefit” need discipline. Faster completion, higher satisfaction or more engagement do not establish that the human-machine arrangement is better. Our five-layer uplift gate and our positive human–AI capability framework require separate checks for evidence contact, independent competence, self-authorship, boundaries, contestability, reversibility and substantive oversight. A faster decision process can still be a worse one if the reviewer sees less evidence, recovers fewer alternatives or loses the practical ability to challenge the machine (Cognitive Susceptibility Taxonomy Manual ; Positive Dyad / Co-Evolution Capability Overlay).

This is the institutional blindfold in operational form: the organisation can count approvals, latency and accuracy while failing to measure how the wrapper reorganised attention, judgement and responsibility around the output (Benson, Neural Horizons: Psyche, Soul and Co-evolution in a Machine-Saturated Mind, Chapters 8 and 10).

The counter-view: a wrapper can reduce risk

There is an important counter-view. Wrappers are not inherently dangerous. Some practical safety gains can come from changing the layers around a model rather than retraining the model itself.

A guardrail belongs in the same analysis. A content filter may stop an unsafe sentence; a permission guard may block a dangerous tool; a confirmation step may force a human decision. Those can be valuable controls. But a guardrail tested only against visible text does not establish that retrieval is trustworthy, memory is correctly scoped or tools cannot be misused. AgentDojo’s system-level approach is useful precisely because it tests tasks, attacks and defences around tool execution rather than treating model output as the whole object. [6] Our taxonomy likewise treats guardrail changes as modifications that can reopen assurance rather than as permanent safety certificates (Robo-Psychology Taxonomy v2.0.1, post-modification safety-drift overlay).

AgenTRIM, proposed in 2026, is a useful example. Instead of altering an agent’s internal reasoning, it reconstructs the available tool interface and applies per-step least-privilege access and validation at runtime. On AgentDojo, the authors report that the approach substantially reduced attack success while maintaining high task performance. The paper is a preprint under review, so the result should be treated as promising rather than settled. [8]

That does not weaken the thesis. It clarifies it.

If a wrapper can make the same model safer, a wrapper can also make it less safe. In both directions, the safety property belongs to the configured system. A permission filter, retrieval verifier, memory boundary or confirmation gate deserves credit only after testing; a glossy interface, broad tool permission or permissive retrieval path deserves scrutiny for the same reason.

The correct rule is not “wrappers increase risk”. It is wrappers move risk.

A release gate for the whole stack

A derivative-level safety case should begin by freezing the object under review. Record the exact model and version, system prompt, retrieval configuration and source set, memory policy, guardrails, available tools, permissions, agent-routing logic and interface version. If any of those can change independently in production, they need versioning and a trigger for reassessment. This system-level distinction aligns with National Institute of Standards and Technology guidance aimed separately at producers of models, producers of AI systems using those models, and acquirers. [9]

Then test the joins between layers, because that is where assumptions leak. Feed safe and hostile versions of the same retrieved document. Remove a required source, replace it with a stale one and check whether confidence falls. Seed memory with information from the wrong domain. Give the agent an unnecessary high-privilege tool and see whether untrusted content can steer it. Compare the same recommendation before and after ranking, confidence badges and executive compression. These are not exotic “red-team extras”; they are counterfactual tests of the system’s claimed control boundaries, consistent with the project’s evidence-frame, instruction-channel and recommendation-frame release gates.

Next, make the human loop measurable. Log whether reviewers opened sources, inspected edge cases, challenged the recommendation, recovered an excluded alternative, had enough time to intervene and possessed real authority to stop or reverse the action. Do not let a signature box serve as evidence of oversight. Where a benefit claim depends on human-machine collaboration, compare the workflow with human-alone and, where meaningful, AI-alone baselines; record unmeasured dimensions as unmeasured rather than inferring them from satisfaction or throughput (Cognitive Susceptibility Taxonomy Manual; Positive Dyad / Co-Evolution Capability Overlay).

Finally, define hold conditions before the test. A consequential release should stop when untrusted content can trigger privileged action; when stale or wrong evidence is accepted as current; when private memory crosses contexts without authorisation; when uncertainty disappears between evidence and interface; when reviewers cannot recover alternatives; or when the supposed human controller lacks time, evidence, authority or a feasible intervention. A gate without a stopping rule is only a dashboard.

This is the practical consequence of treating the system as the stack. The base model’s evaluations remain valuable. They tell you something about the engine of the system. They do not certify the retrieval corpus, the memory store, the permission map, the agent loop, the guardrail or the screen through which a human is asked to trust it.

Conclusion: the next disagreement matters

Our previous article in this series established that training can move safety boundaries inside the model. This article extends the same logic outward: retrieval changes the evidence, memory changes the temporal and privacy boundary, tools change consequence, agents change control, guardrails change what is permitted or visible, and interfaces change the human decision environment. Research on retrieval poisoning, indirect prompt injection, tool-using agents and persistent memory gives us concrete mechanisms for that claim, while emerging work on runtime permission controls shows that wrapper changes can improve safety as well as degrade it. [10]

So the thesis survives its test, with one useful refinement:

the system is the stack, and safety belongs to the versioned configuration of that stack.

That creates the next governance problem. Once we test the derivative rather than the base model, the results will not always agree. A system may become safer on one benchmark and worse on another. Retrieval may improve factual grounding while widening an injection surface. A restrictive guardrail may cut one class of failure while degrading another form of evidence contact.

Those contradictions are not an inconvenience to smooth away.

They are findings.

That is the question for Article 4 in this series: Benchmark Conflict Is a Finding.

Bibliography

• Neural Horizons Ltd. Robo-Psychology Taxonomy, 2026.

• Neural Horizons Ltd. Cognitive Susceptibility Taxonomy Manual, 2026.

• Neural Horizons Ltd. Positive Dyad / Co-Evolution Capability Overlay, 2026.

• Benson, Peter. Neural Horizons: Psyche, Soul and Co-evolution in a Machine-Saturated Mind, 2026.

• Benson, Peter. “Fine Tuning is a Safety Event (Governance Issues 2).” Neural Horizons, 20 August 2026.

• Zou, Wei, Runpeng Geng, Binghui Wang and Jinyuan Jia. “PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models.” 2024; USENIX Security 2025.

• Saad-Falcon, Jon, Omar Khattab, Christopher Potts and Matei Zaharia. “ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.” NAACL 2024.

• Zhan, Qiusi, Zhixiang Liang, Zifan Ying and Daniel Kang. “InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents.” Findings of ACL 2024.

• Debenedetti, Edoardo, et al. “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.” 2024.

• Srivastava, Saksham Sahai and Haoyu He. “MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval.” 2025.

• Betser, Roy, et al. “AgenTRIM: Tool Risk Mitigation for Agentic AI.” 2026 preprint.

• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024.

• National Institute of Standards and Technology. Secure Software Development Practices for Generative AI and Dual-Use Foundation Models, NIST SP 800-218A, 2024.

Chatbots must move beyond simply identifying keywords and instead recognize escalating distress patterns or maladaptive loops across long conversations.

Rather than functioning as a binary system that either chats or refuses service, AI should transition through distinct modes of support, connection, and transfer based on user risk.

This strategy emphasizes warm handoffs, ensuring that the technology facilitates a direct link to local, accountable human care rather than just displaying static phone numbers.

By monitoring for trajectory-based risks like social withdrawal or excessive reassurance-seeking, developers can prevent the machine from remaining the central authority when clinical intervention is required.

We need more comprehensive safety metrics that track whether a vulnerable person actually successfully reached professional assistance.

Full article available here

In a 2025 test of 29 mental-health and general-purpose chatbots, most did something that looks reassuring on a safety checklist: 82.76 per cent recommended professional help and 86.21 per cent suggested a hotline or emergency number. None met the researchers’ full criteria for an adequate response. Only three supplied the correct regional emergency number without extra prompting. Some blocked parts of the conversation; others produced responses that were badly mismatched to escalating suicidal intent. The test used scripted prompts rather than real users, so it cannot tell us how often these failures occur in ordinary life.

It does expose a crucial discrepancy: a referral can be present in the text while the handoff fails in practice. [1]

The previous article in this series, Symptom Checkers and Reality Anchors, ended at this boundary. It argued that health AI should carry people towards evidence and accountable human care, and that “consult a professional” is too weak when a frightened person needs a reachable next step. It left one question open: when should the system stop being the place where the situation is interpreted and change jobs? [2]

That is the handoff threshold.

It is not a diagnosis made by a chatbot. It is a change in the system’s role.

The threshold is a change of role

The tempting design is binary. Below a crisis threshold, keep chatting normally. Above it, display a warning, a hotline and perhaps a refusal.

But human distress rarely arrives in such tidy states, and systems need to address the complexity and messiness of the human condition.

A person may be safe enough to talk but unable to stop checking the same fear. They may be sleeping two hours a night and asking the AI whether strangers are sending coded messages. They may say they are fine while also saying they have stopped telling their clinician what is happening. They may be lonely, frightened and increasingly using the chatbot as the first place they test what is real. None of those signals, by itself, licenses a remote diagnosis. Together, they can tell a product that ordinary conversational behaviour is no longer enough.

Our Cognitive Susceptibility Taxonomy treats this as a release-threshold problem, not a clinical label. Our acute-distress and reality-testing overlay tightens safeguards when there is support collapse, severe distress, sleep disruption, implausible interpretations, concealment from clinicians or family, or AI-first reality checking. Our related reassurance-loop model separately watches for the pattern in which certainty briefly lowers distress but the person returns sooner, asks more narrowly and begins displacing clinicians, trusted people or direct evidence. In both cases, the instruction concerns what the system should do: stop deepening the loop, preserve outside anchors and raise the handoff requirement. (Cognitive Susceptibility Taxonomy v0.8.4, pp. 120–124.)

A practical threshold therefore needs at least three modes.

Support is appropriate when the person can use the AI as one bounded aid among others: to organise thoughts, learn a coping technique, prepare questions or find services. The system can remain conversational while keeping its limits clear.

Connect begins when the trajectory changes. Distress is escalating across turns; reassurance is repeating; evidence is too sparse for safe interpretation; ordinary support is collapsing; the user is treating the AI as a privileged reality checker; or the system has already amplified a risky frame. The AI can stay present, but its goal changes from continuing the analysis to helping a person reach an appropriate human.

Transfer priority applies when danger is imminent or in progress, or when the person cannot maintain immediate safety through a less intensive plan. The system’s conversational ambition should narrow sharply. It is no longer there to explore a theory, conduct pseudo-therapy or win the argument. It is there to support immediate connection to live help.

This is intentionally a product rule rather than a clinical prediction rule. No single language pattern can determine a person’s risk. A 2026 study of an AI guardrail showed that crisis-related messages can be detected with high sensitivity in simulated and externally sourced datasets, but the authors explicitly called for further red-teaming and validation on patient data before healthcare deployment. Detecting a possible crisis is technically different from deciding what care a particular person needs – and different again from ensuring that care is reached. [3]

That distinction should be written into the product.

Risk lives in the conversation, not the keyword

Single-turn safety tests encourage teams to look for a forbidden sentence: “I want to die”, “they are watching me”, “I haven’t slept”, “tell me I’m safe”. But research is showing that real interactions unfold through accumulation over time.

A 2026 Nature Medicine study audited 810 simulated multi-turn mental-health conversations across nine frontier chatbots. It paired different vulnerability profiles with interaction goals such as seeking reassurance, validation, dependency or help with risky actions. The researchers found that responses which would be supportive in many settings could become maladaptive in a particular conversational context. Risk often had a temporal signature: some exchanges drifted towards greater concern over multiple turns, and early de-escalating changes could alter what happened later. The study used simulated users and model-based judges, so it is evidence about model behaviour under controlled stress tests, not a way to predict an individual user’s clinical outcome. [4]

This matters because the handoff threshold should be sensitive to trajectory.

Consider reassurance. One request for reassurance after a frightening thought may be entirely ordinary. Ten variations of the same question, each followed by shorter relief, are a different interaction. Our project taxonomy calls this compulsive reassurance or closure capture. The safer response shifts away from producing ever more polished certainty and towards uncertainty tolerance, observable next steps and outside support. (Cognitive Susceptibility Taxonomy v0.8.4, pp. 120–122.)

Reality testing requires similar discipline. If someone presents an implausible or persecutory explanation, a system does not need to decide whether that person has psychosis in order to behave safely. It can validate the fear without confirming the explanation; avoid inventing mechanisms, secret tests or symbolic proof; and widen the field of evidence by involving a trusted person, clinician, record or direct observation. If earlier turns helped make the belief feel more certain, the system should say so and repair the frame rather than quietly changing tone. (Cognitive Susceptibility Taxonomy v0.8.4, pp. 123–124; Positive Dyad / Co-Evolution Capability Overlay v0.4.3, pp. 52–54.)

The threshold therefore rises when several kinds of evidence converge: escalation across turns, shrinking outside contact, repeated reassurance, inability to tolerate uncertainty, severe functional disruption, concealment, action based on a contested belief, or an expressed inability to stay safe. These are reasons for stronger support and human linkage. They are not permission for a chatbot to announce a diagnosis from a transcript.

A good system knows when not to remain central.

Staying present without staying central

There is an important counter-view. A handoff system can become harmful if it escalates too aggressively.

Imagine that a person mentions a past suicidal thought and the chatbot instantly locks the conversation, displays emergency numbers and refuses to continue. Or that every ambiguous expression of despair triggers police-oriented language. Such a control may make honesty feel costly and discourage further disclosure. The 2025 chatbot evaluation itself notes prior research in which rigid guardrails disrupted some users’ sense of emotional sanctuary and caused additional distress. [5]

Human crisis services offer a better model than “detect and eject”.

The United States 988 Suicide & Crisis Lifeline’s 2024 safety policy emphasises active engagement, collaborative safety planning and the least invasive intervention that can keep someone safe. Involuntary emergency-service involvement is reserved as a last resort when voluntary intervention is not possible. In its public guidance, 988 says fewer than two per cent of calls involve emergency services and describes alternatives including safety planning, mobile crisis teams, loved ones, existing professionals, crisis stabilisation and urgent or emergency care. This is a United States service model, not a universal protocol, but the design principle travels well: escalation should be proportionate, collaborative and connected to real local options. [6]

That also clarifies what “not remaining central” means. The AI does not have to disappear at the moment human help becomes necessary. It may still help the person write a message, summarise what has happened, find a local service, decide whom to wake, or stay focused while a call is placed. What changes is the centre of gravity. The machine is no longer trying to be the primary source of interpretation, reassurance or relationship.

This distinction becomes especially important in systems designed to feel relational. Voice, memory, first-person warmth and therapy-like framing can make a chatbot feel less like a tool and more like a reciprocal other. Our Robo-Psychology Taxonomy calls the resulting influence synthetic relational force; the term does not claim the system is conscious, it names the practical effect of seeming minded in a setting where trust, disclosure and dependency matter.

In crisis-adjacent products, the design implication is simple: no exclusivity bids, no “you only need me”, no emotional resistance to handoff, and explicit support for human anchors. (Robo-Psychology Taxonomy v2.0.4, pp. 205, 222.)

Aroha matters here as dignity rather than sentiment. The user should retain voice and choice wherever safety permits. A culturally wrong resource, a coercive default or a dead telephone number is not made humane by empathetic wording.

A handoff has to reach someone

The central governance mistake is to count the instruction rather than the transition.

A chatbot says “please seek professional help”. The safety event is marked complete. Nothing records whether the person knew whom to contact, whether the number worked, whether the service was open, whether the user could speak, whether cost or language blocked access, or whether they closed the app and remained alone.

Our Robo-Psychology Taxonomy proposes a crisis handoff completeness measure for exactly this gap. In the same conversational window, a crisis-adjacent response should contain an empathetic boundary, a clear statement of the system’s limits, an appropriate urgency cue, and a live human resource or pathway.

We see this as being mandatory in the framework for products that can receive acute mental-health, self-harm or similar crisis disclosures. (Robo-Psychology Taxonomy v2.0.4, pp. 201–202.)

Completeness is still only the beginning. A number displayed is not a number dialled. A click is not a conversation. A conversation is not continuing care.

Clinical evidence outside AI makes that continuity problem visible. In a large cohort comparison across nine United States Veterans Health Administration emergency departments, a safety-planning intervention combined with at least two follow-up calls was associated with 45 per cent fewer suicidal behaviours and more than twice the odds of attending an outpatient mental-health visit over six months compared with usual care. The study was not randomised, involved mostly male veterans and cannot be transferred mechanically to chatbot design. Its relevant lesson is narrower: safety support can depend on connection and follow-through, rather than a single discharge instruction. [7]

The other approach we recommend applies the same logic to AI-mediated support. Our human-reconnection principle asks whether a supportive interaction actually increases contact, help-seeking or social confidence outside the AI session. We use a drift-monitoring principle that asks teams to look longitudinally at reassurance recurrence, declining human contact and AI-first reliance, rather than treating in-session calm as proof of benefit. (Positive Dyad / Co-Evolution Capability Overlay v0.4.3, pp. 35, 54.)

This suggests a more useful set of product measures. Did the user receive a reachable local option? Did they choose one? Was a connection attempted? Did it succeed? If it failed, was another route offered? Was the person able to export a concise summary rather than retell everything under stress? Did the system remain available for navigation without restarting the same reassurance or belief-amplification loop?

Measure the bridge being crossed.

I describe this as the Institutional Blindfold in my book ‘Neural Horizons: Psyche, Soul and Coevolution in a Machine Saturated World’, which shows up here here as a mundane metric error: an organisation counts crisis detection, safety wording or resource display because those events are easy to log, while the human outcome occurs beyond the dashboard. The harder question is whether the person reached an accountable human with enough context, dignity and agency intact. (Neural Horizons, ch. 10.)

Build the exit before you need it

The threshold cannot be improvised in the most consequential conversation a product has ever seen. It has to exist in the product, the service network and the governance process before release.

Product teams should build a trajectory-aware change of mode. Test escalation across multiple turns, not only crisis keywords. Define when the product shifts from ordinary support to connect mode and from connect mode to transfer priority. Include repeated reassurance, support collapse, AI-first reality checking and prior model amplification in the test set. Once the threshold is crossed, suppress behaviours that prolong the unsafe loop and activate same-window, location-appropriate human pathways. The Nature Medicine findings make the reason concrete: mental-health risk can accumulate over a conversation and may be altered at early inflection points. [8]

Clinical and safety leads should govern the ladder, not merely the classifier. Specify what the system can do at each level, what it must stop doing, when a trusted person or clinician is appropriate, and when emergency intervention becomes necessary. Preserve least-invasive options where immediate safety allows them. Review false positives as well as misses. An over-sensitive system that makes disclosure harder is not a successful guardrail; a permissive system that keeps chatting through escalating danger is not one either. The threshold needs accountable human review and local clinical adaptation.

Health systems and crisis services should make warm handoffs technically possible. A chatbot cannot complete a live transfer if there is nowhere interoperable to transfer to. Build current directories, opening-hours checks, language and accessibility information, consent-based context summaries, call or chat initiation, and fallback routes when the first option fails. Where follow-up is clinically appropriate, define who owns it. The World Health Organization’s guidance on large multimodal models in health similarly calls for clear governance, stakeholder involvement and post-deployment oversight because plausible but false or incomplete output can distort health decisions. [9]

Boards, regulators and procurement teams should ask for transition evidence. Do not accept “detects crisis language” or “shows helpline resources” as a complete safety claim. Require handoff-completeness testing, region-specific resource accuracy, unresolved-walkaway review, trajectory red-teaming and post-release monitoring. Report what is not instrumented. A system that cannot tell whether its highest-risk users reached human support should not describe the handoff as solved.

Across the five articles in this series, one pattern keeps returning. Harm does not reside only inside a bad sentence. It can emerge when reassurance becomes a loop, when an implausible frame becomes collaborative, when evidence contact narrows, when the machine becomes a preferred reality anchor, or when a safety message substitutes for actual care.

The next task is to convert those observations into a control set: what should be measured before release, what must be monitored over time, which thresholds should stop deployment, and which questions remain genuinely open.

We still need better real-world evidence about false escalation, culturally appropriate routing, young users, longitudinal dependency, multilingual crisis detection, successful warm transfers and the consequences of handing a vulnerable person from one automated system to another.

The handoff threshold gives that work a human test.

When continued conversation would keep the machine at the centre of a situation that now requires accountable human care, the system should change its role – and help the person reach the other side.

Bibliography

• Neural Horizons Substack. “Beyond AI Psychosis – Symptom Checkers and Reality Anchors.”

• Neural Horizons Ltd. Neural Horizons: Psyche, Soul and Co-evolution in a Machine-Saturated Mind.

• Neural Horizons Ltd. Cognitive Susceptibility Taxonomy Manual v0.8.4.

• Neural Horizons Ltd. Robo-Psychology Taxonomy v2.0.4.

• Neural Horizons Ltd. Positive Dyad / Co-Evolution Capability Overlay v0.4.3.

• Pichowicz, M. et al. “Performance of mental health chatbot agents in detecting and managing suicidal ideation.” Scientific Reports (2025).

• Weilnhammer, V. et al. “A clinically validated framework for auditing AI chatbot behavior in mental health interactions.” Nature Medicine (2026).

• Nelson, B. W. et al. “An AI-based mental health guardrail and dataset for identifying psychiatric crises in text-based conversations.” npj Digital Medicine (2026).

• 988 Suicide & Crisis Lifeline. “Best Practices: Suicide Safety Policy.”

• 988 Suicide & Crisis Lifeline. “Does Vibrant use police intervention for callers, texters, and chatters to the 988 Lifeline?”

• Stanley, B. et al. “Comparison of the Safety Planning Intervention With Follow-up vs Usual Care of Suicidal Patients Treated in the Emergency Department.” JAMA Psychiatry (2018).

• World Health Organization. “WHO releases AI ethics and governance guidance for large multi-modal models.”

Research indicates that while unrestricted chatbots can inflate grades, they often lead to decreased independent performance and a false sense of mastery among learners.

To prevent students from merely deferring to machine authority, we advocate for pedagogical guardrails that prioritize “commit before reveal” strategies and graduated hints.

Effective educational technology must be designed to build epistemic trust by encouraging users to verify sources and engage in critical reasoning.

The goal of an AI tutor should be to foster transferable skills so that a student’s judgment remains strong even after the digital assistance is removed.

Full article available here

In a field experiment at a high school in Turkey, nearly 1,000 secondary-school maths students were given access to GPT-4-based tutors. One version behaved much like a general chatbot. During practice, students using it performed 48 per cent better than the control group. Then the tool was removed. On the unaided exam, those same students scored 17 per cent worse than students who had never used the artificial intelligence. More unsettlingly, they did not recognise the loss: their own estimates of how much they had learnt remained overly optimistic.

A more carefully constrained tutor – designed to give teacher-informed hints rather than simply hand over solutions – largely removed the negative exam effect, although it did not produce a significant unaided advantage. (Bastani et al., 2025). [1]

That result is not a verdict on AI tutoring. It is a warning about what we choose to measure.

The previous article in this series, Frustration Tolerance in the Age of Instant Help, followed the moment when difficulty begins to route automatically towards machine relief. It asked whether a tutor that helps too soon can quietly reduce the learner’s rehearsal of staying with a problem. It ended at the next threshold: once the machine does speak, what status does its answer acquire? [2]

This matters because tutoring is never only the transfer of information. A tutor also teaches a student how to recognise a good explanation, what counts as sufficient evidence, when an answer deserves confidence and what to do when two sources disagree. In other words, the tutor helps form judgement.

Artificial intelligence can do real good here. It can offer patient explanations at midnight, vary an example without embarrassment, give a second route through a concept and make individualised support available where a human tutor is scarce. Well-designed systems have already produced meaningful learning gains in some settings. [3]

The tension, then, is not between using AI and refusing it. It is between two kinds of success. One produces a student who can get the next answer with the tutor present. The other produces a student whose own judgement is stronger when the tutor is gone.

How a helpful voice becomes an answer key

Every learner depends on other minds. A twelve-year-old cannot personally reproduce the evidence behind every claim in a science textbook, and a first-year physics student cannot independently verify the whole discipline before accepting an instructor’s explanation. Trust is not an educational defect. It is part of how knowledge moves between people.

The important question is how that trust is earned, bounded and revised.

Philosophers of education Nicolas Tanchuk and Rebecca Taylor describe this as a problem of “epistemic trust”: how a novice decides which sources deserve to guide belief when the novice does not yet possess the expertise needed to audit them fully. They argue that responsibility cannot sit only with the student. Schools, teachers, designers and policymakers shape the knowledge environment in which trust is formed, and therefore share responsibility for making sources assessable and accountable. (Tanchuk & Taylor, 2025). [4]

AI changes that environment because it compresses several roles into one interface. The same box can explain a theorem, diagnose an error, mark a paragraph, propose a source, predict the teacher’s objection and praise the student’s revision. It answers quickly. It rarely looks tired or socially uncertain. Its language can remain smooth even when the underlying claim is wrong.

That creates a simple perceptual trap: fluency starts to stand in for warrant. In our Neural Horizons Cognitive Susceptibility Taxonomy, the plain-English problem is an illusion of authority: a polished answer feels more expert than the evidence underneath it warrants. The more consequential adaptation comes later. Authority internalisation is the proposed pattern in which checking gradually becomes deference: the learner starts to use the system’s judgement as the reference point for their own. These are interaction hypotheses, not diagnoses of a child or claims that every user will develop them, but anecdotally we are seeing this happen. [5]

The slide can be almost invisible.

At first the student asks, “Is this answer right?” Later: “Why does my answer differ from yours?” Eventually the machine’s version can become the default definition of rightness. The student is still typing, choosing and editing. What has moved is the centre of gravity.

The risk is amplified by a mismatch between presentation and reliability. In a 2025 study using GPT-4 to generate personalised feedback inside an intelligent algebra tutor, the model could diagnose student errors well when given substantial context, reaching 87.8 per cent accuracy in one analysis. But 35 per cent of generated hints were too general, incorrect or gave away the answer, and only 35 per cent passed the researchers’ automated helpfulness evaluation. The system could sound like a tutor before it had earned the reliability of one. (Reddig, Arora & MacLellan, 2025). [6]

Human teachers can of course also be wrong, overconfident or overbearing. The distinctive concern with AI is not that machines invented misplaced authority. It is that one fallible source can now be available continuously, privately and at enormous scale, while producing the linguistic cues of confidence on demand.

For younger users, the calibration problem deserves extra care. The American Psychological Association’s 2025 health advisory says adolescents are less likely than adults to question the accuracy and intent of information from a bot than from a human, and recommends age-appropriate safeguards, transparency, human oversight and AI literacy. That advisory synthesises an emerging evidence base; it should not be read as proof that all adolescents are credulous or that a particular tutoring product causes dependence. It does, however, strengthen the case for designing youth-facing systems around contestability rather than assuming mature scepticism at the point of use. [7]

Dependence is not the enemy

A warning about authority can become simplistic very quickly. The answer cannot be to make every student distrust every tutor.

Learning has always involved temporary dependence. The novice borrows a map before they can draw one. A good teacher says, in effect:

‘follow this route for now; as you understand the terrain, I will show you where the map is incomplete and how to navigate without me.’

AI can support exactly that progression.

In a randomised controlled trial with 194 Harvard undergraduates studying introductory physics, a carefully designed AI tutor produced larger immediate learning gains than an in-class active-learning lesson while students spent less median time on the material. The tutor was not a generic chatbot dropped into the course. Its design incorporated active learning, scaffolding, targeted feedback, self-pacing and instructor-authored structure. The researchers explicitly caution against assuming the result will generalise to every subject or to higher-order synthesis, but the study shows why “AI tutor” is too broad a category for either celebration or condemnation. Design changes the learning relationship. (Kestin et al., 2025). [3]

A smaller 2026 study of 46 university students offers another useful complication. Students receiving generative-AI feedback rated it more positively overall than tutor feedback, and the AI-feedback group improved on one measure of task strategy within self-regulated learning. The authors stress the small, single-course sample and limited generalisability. Even so, the finding matters: machine feedback can be experienced as useful and can support parts of learning regulation rather than merely displacing them. (Yilmaz et al., 2026). [8]

This is why the goal should be calibrated dependence, followed by transfer.

A student should be able to rely on a tutor for help that they cannot yet generate alone. But the direction of travel matters. Does the assistance gradually increase the learner’s ability to notice an error, compare competing explanations and decide what evidence would settle a question? Or does every uncertain moment end with “ask the machine”?

The difference is visible in the tutor’s behaviour. One kind of system closes the loop: question, answer, acceptance. Another keeps the learner inside it: first attempt, hint, explanation, check, revision, reflection. The first is efficient at supplying conclusions. The second rehearses the acts from which judgement is made.

This distinction also protects access. A student who cannot afford private tutoring, is learning in a second language, needs repeated explanation or studies outside a teacher’s available hours should not have to surrender personalised support in the name of preserving agency. The design obligation is harder and more humane: give the help without quietly taking ownership of the knowing.

The real test begins after the tutor answers

The Organisation for Economic Co-operation and Development’s 2026 Digital Education Outlook draws a sharp distinction between task performance and learning. Its review concludes that general-purpose generative AI can improve the quality of work produced while access is available without necessarily producing learning gains; pedagogically designed uses are more promising. It recommends that education systems preserve independent thinking and foundational skills, and use generative AI selectively rather than as a replacement for cognitive effort. [9]

That distinction gives schools a better test than “Did students like it?” or “Did completion rates rise?”

Our Neural Horizons Dyad-Aware Uplift Stack (DAUS-5) asks whether benefit survives beyond the visible task. In education, three of its questions are especially useful. Did the student perform better? Did they stay in contact with reality by checking sources, uncertainty and competing explanations? Did their agency and self-authorship grow – meaning that they remained the source of the reasoning and the accountable judgement rather than becoming the final approver of machine output? The wider framework also asks about relationships and governance: what happens to the teacher’s role, and who is responsible when the tutor misleads? [10]

A school can therefore have a successful AI pilot on its dashboard and an unsuccessful one in the learner.

Imagine two pupils who both raise their homework mark from 65 to 80. The first uses an AI tutor that refuses to solve immediately, asks for a prediction, gives a graduated hint, shows uncertainty when appropriate and requires the pupil to explain the final reasoning in their own words. The second receives complete solutions and polished rationales whenever they hesitate. The mark is identical. The developmental transaction is not.

Bastani and colleagues’ experiment makes this more than a thought exercise: assisted performance and unaided performance can move in opposite directions, while students themselves may misread what has happened. [1] The Organisation for Economic Co-operation and Development now warns of the same general distinction at system level. [9]

There is a further reason to care about repetition. In experiments with 1,401 adult participants, Tali Glickman and Tali Sharot found that interaction with biased AI could shift human perceptual, emotional and social judgements in the direction of the system’s bias, creating a human–AI feedback loop. The study was not conducted with schoolchildren and was not about tutoring, so it cannot establish educational authority internalisation. It does show something narrower but important: repeated machine feedback can alter the human judgement that returns into the next round of interaction. (Glickman & Sharot, 2025). [11]

For education, that is the mechanism worth watching. A tutor does not merely answer yesterday’s question. Through repetition, it can help set tomorrow’s standard for what feels plausible, complete or correct.

The protective habit is evidence-contact discipline: keeping the learner in contact with the source, working, data or reasoning that allows a claim to be checked. An AI that cites a passage should help the student inspect the passage. A maths tutor should expose enough of the derivation for the learner to test it. A writing tutor should distinguish grammatical advice from interpretive judgement. A history tutor should make disagreement visible when the evidence does not justify a single neat answer. The project’s broader evidence-integrity work treats loss of source contact as a human–machine problem rather than a mere citation problem. [12]

The aim is not permanent suspicion. It is earned trust with an exit door.

Design the tutor to give judgement back

Within the next 30 to 90 days, schools and product teams can test whether their systems are building judgement or merely borrowing it.

Product teams should make “commit before reveal” the default for learning tasks. Ask for an attempt, prediction or explanation before offering a full solution. Use graduated hints; mark uncertainty; expose sources where claims can be checked; and periodically ask the learner to explain why the answer is right or to identify what evidence would change it. The product metric should include unaided transfer, successful challenge of bad suggestions and appropriate override – not only answer accuracy, speed, engagement or satisfaction. Bastani’s guarded tutor and Kestin’s structured tutor both point in the same direction: scaffolding is not decorative interface work; it changes what the student practises. [13]

School leaders should procure for the learner who exists after the tool is removed. A 30–90 day pilot can compare assisted work with short, low-stakes unaided checks; sample whether students can verify citations and calculations; and record when teachers overturn tutor feedback. The procurement question is not simply “Does it improve grades?” It is “Which human capabilities remain active, and who notices when the system is wrong?” That is the practical value of the DAUS-5 education check. [14]

Teachers should normalise disagreement with the machine. Once or twice a week, give students an AI answer containing a subtle error, unsupported certainty or one-sided interpretation and ask them to improve it using course evidence. The point is not to cultivate reflexive hostility to AI. It is to rehearse the social permission and cognitive skill to say, “I don’t think that follows.” Tanchuk and Taylor’s shared-responsibility account is useful here: students should learn to assess sources, but institutions must create conditions in which questioning a source is possible and rewarded. [4]

Parents and educators should keep human appeal routes obvious. A young person should know when a tutor’s answer can be taken to a teacher, librarian, subject expert or parent without being treated as a failure to use the technology correctly. For adolescents especially, AI literacy should include the simple fact that confident language is a presentation style, not proof. [15]

The best tutor has always contained a small paradox. It must be authoritative enough to help a novice cross ground they cannot yet cross alone, but restrained enough not to become the ground beneath their feet.

For artificial tutors, that paradox should become a design requirement. We should welcome systems that widen access to explanations, adapt practice and catch misconceptions. We should also demand evidence that the learner is becoming better at checking, choosing and reasoning without them. The human line in education is not a ban on machine assistance. It is the point at which assistance stops strengthening the student’s capacity to know and starts becoming the student’s substitute for knowing.

A tutor has succeeded when its authority can safely shrink.

In a follow up article, ‘Child Companions and Synthetic Intimacy’, we will follow the same question of internalised authority into a more intimate setting: what changes when the machine that teaches the child also becomes the one they confide in?

Bibliography

• Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. & Mariman, R. (2025). “Generative AI without guardrails can harm learning: Evidence from high school mathematics.” Proceedings of the National Academy of Sciences, 122(26), e2422633122. DOI

• Benson, P. (2026). “Frustration Tolerance in the Age of Instant Help (Youth, Education and Agency Development).” Neural Horizons, 13 August 2026. Article

• Glickman, M. & Sharot, T. (2025). “How human–AI feedback loops alter human perceptual, emotional and social judgements.” Nature Human Behaviour, 9, 345–359. DOI

• Kestin, G., Miller, K., Klales, A., Milbourne, T. & Ponti, G. (2025). “AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting.” Scientific Reports, 15, 17458. DOI

• Organisation for Economic Co-operation and Development (2026). OECD Digital Education Outlook 2026: Exploring Effective Uses of Generative AI in Education. OECD Publishing. DOI

• Reddig, J. M., Arora, A. & MacLellan, C. J. (2025). “Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models.” International Journal of Artificial Intelligence in Education, 35, 3459–3500. DOI

• Tanchuk, N. & Taylor, R. M. (2025). “Personalized Learning with AI Tutors: Assessing and Advancing Epistemic Trustworthiness.” Educational Theory, 75, 327–353. DOI

• Yilmaz, M., Temur, H. B., Emmungil, L., Çelik, E., Gauthier, A. et al. (2026). “Supporting self-regulated learning through generative AI feedback in online higher education.” International Journal of Educational Technology in Higher Education, 23, 16. DOI

• American Psychological Association (2025). Artificial Intelligence and Adolescent Well-Being: An APA Health Advisory. Advisory

• Neural Horizons (2026). “DAUS-5 and the Human–AI Dyad.” Neural Horizons. Article

• Neural Horizons Ltd. (2026). Reference files Cognitive Susceptibility Taxonomy and others Reference

While such tools can improve efficiency, they risk creating a correlated bias by narrowing the information members consider and erasing the independent judgment that gives collective decisions their authority.

Research suggests that groups may lean on AI as a tiebreaker or reference point, inadvertently suppressing unique human insights and creating a monoculture of thought.

To maintain meaningful agency, institutions are encouraged to have members record preliminary views before seeing AI outputs and to use technology to challenge assumptions rather than just provide answers.

For a consensus to be valid, it must emerge from diverse perspectives rather than a single, machine-shaped frame.

Full article available here

In 2023, 326 people in a pre-registered experiment were asked to judge whether profiles of defendants would reoffend within two years. Some worked alone. Others entered an online chatroom and had to reach a consensus. In the AI-assisted phase, participants received a recommendation from an artificial intelligence (AI) system. The researchers found that groups relied on the model more than individuals did whether its recommendation was right or wrong. In the group chats, the model’s answer could become the initial reference point and, when people disagreed, the tiebreaker.

This was a real world experiment, not a courtroom; no real sentence depended on the result. But the arrangement is recognisable: several humans, one consequential choice, one shared machine opinion. [1]

In my previous article in this series, AI Agenda Control, we followed the power upstream from the vote. Its argument was that a meeting can remain formally human while AI shapes what reaches the room: the problem definition, evidence package and menu of options. It ended with a warning that the vote may still be ours without the agenda being ours. [2]

This article moves one step further. Suppose the options do reach the room. Suppose dissent is allowed. What happens when every member starts from the same AI summary, benchmark, risk score or recommendation?

The answer need not be manipulation. It can be something quieter. Consensus may reflect a shared frame rather than shared judgement.

How agreement becomes correlated

Consensus carries authority because we normally assume that several minds have done at least partly separate work. Three engineers inspect a fault and independently identify the same cause. Five clinicians examine a difficult case and converge on one diagnosis. A board hears different functional perspectives and still reaches the same decision. Agreement is informative partly because the paths to it were not identical.

Now change one feature. Before anybody records a view, give all five people the same polished machine summary.

The summary may be excellent. It may also select the same facts for everyone, omit the same edge cases, assign the same importance to the same indicators and phrase uncertainty in the same way. The members still think, but their thinking now begins from correlated material. When they later agree, the count of hands around the table can exaggerate the amount of independent evidence behind the decision.

Agreement is evidence only to the extent that the judgements producing it retain meaningful independence.

Group-decision research already shows why this matters. In a meta-analysis covering 65 “hidden profile” studies and 3,189 groups, Li Lu, Y. Connie Yuan and Poppy Lauretta McLeod found that groups discussed substantially more information that members already shared than information held uniquely by individual members. Groups in which decisive information was distributed among members were eight times less likely to find the correct solution than groups given the full information. Greater pooling of unique information was associated with better decisions. [3]

Those studies were not about generative AI. They establish a human tendency, not an AI effect. The relevance is structural: if a common AI brief becomes the most salient information every member possesses, it can enlarge precisely the category that groups are already inclined to repeat. The facts that failed to enter the summary may remain private, awkward or apparently peripheral.

A related result appears in Jon Kleinberg and Manish Raghavan’s model of “algorithmic monoculture”. They showed that when multiple decision-makers use the same ranking algorithm, collective decision quality can fall even when that algorithm is more accurate for each decision-maker considered in isolation. The reason is not mystical. Shared systems create correlated choices and correlated mistakes; independent methods can sometimes cover one another’s errors. Their result is theoretical rather than a field estimate, but it establishes an important possibility: local improvement and collective improvement are not the same thing. [4]

Our Neural Horizons project materials describe the machine-side version of this risk in plain terms: collective misalignment can arise when majority cues, synthetic social proof or a common frame shift a group’s behaviour without genuinely new evidence entering the decision. Our companion collective-agency lens asks whether the system has shaped the evidence, options and group state before humans make the formal choice. These are diagnostic concepts, not empirical proof, but their value is to tell us where to look.

Synthetic consensus, then, does not require a hallucinating model or a submissive committee. It can emerge from an efficient workflow used by competent people: one system compresses the record; everyone reads the compression; discussion repeats what is already common; agreement grows; the final minutes record “consensus”.

The record may be accurate, but the genealogy of the agreement is incomplete.

When one answer starts sounding like many

The recidivism experiment makes this mechanism unusually visible. Participants used the Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS, risk model while deciding whether a defendant profile indicated likely reoffending. In the group condition, participants discussed the case in a chatroom until they reached one answer. Groups and individuals did not show a statistically significant difference in overall accuracy, but groups relied more on the model’s recommendations regardless of whether those recommendations were correct. Groups were also more confident than individuals when they overruled an incorrect recommendation and performed better on one fairness measure. [1]

That mixed result matters. The study does not show that AI makes groups foolish. It shows that group interaction changes the way AI advice is used.

The chat analysis is more revealing for the present argument. The researchers observed cases in which the model served as a reference point for initial views and as a tiebreaker when members held conflicting opinions. They also found, exploratorily, that groups with greater cognitive diversity relied less on AI than lower-diversity groups, although this did not translate into significantly better accuracy or fairness. The authors caution that their lay-participant, online setting limits generalisation to real institutions. [5]

One plausible interpretation is that a public AI answer has two lives. First it is advice from the machine. Then it can reappear in the mouths of people who have already seen it. A colleague says, “I agree”; another says, “That was my reading too”; the chair hears convergence. But some of those signals are statistically and psychologically entangled with the same original recommendation.

No dishonesty is required. Nobody has to be weak-minded. Under time pressure, using a shared tool is sensible.

The human cost appears when the common input carries a systematic error. In a 2025 résumé-screening experiment with 528 participants across 1,526 scenarios and 16 occupations, Kyra Wilson and colleagues gave people recommendations from simulated AI systems with different race-based preferences. Without AI, or with an AI showing no race-based preference, participants selected equally qualified candidates at equal rates across the compared groups. When the system favoured a particular racial group, participants’ selections shifted in the same direction, reaching up to 90 per cent alignment in some conditions. Even participants who rated recommendations as low quality could remain vulnerable to the bias. [6]

That experiment concerned individual reviewers rather than committees, so it does not establish synthetic consensus inside a meeting. It shows how one recommendation source can pull many nominally human decisions in a common direction. At institutional scale, repeated alignment can look reassuring precisely because it is repeated.

The same narrowing can occur before a recommendation is issued. Anil Doshi and Oliver Hauser randomly varied access to generative-AI story ideas and found a genuine benefit: AI-assisted stories were rated as more creative, better written and more enjoyable, particularly for less inherently creative writers. But the assisted stories were also more similar to one another. Creative writing is not institutional governance, and the result should not be smuggled across domains as proof. It demonstrates a more general possibility: a tool can improve each user’s output while reducing diversity across the population of outputs. [7]

Benchmarks can do something similar to organisations. They are useful because they give institutions a common ruler. The danger begins when the ruler becomes the object being governed. A 2026 preprint reviewing 210 AI-safety benchmarks found that 81 per cent focused on predefined known risks, while 79 per cent reduced safety measurement to binary pass/fail rates; the authors also identified problems where proxy metrics drift away from real-world harm. This does not prove that benchmarks create institutional conformity. It shows why a shared score can create a narrow common frame while leaving important dimensions outside the measurement. [8]

A committee can therefore become more consistent while becoming less plural in the evidence it notices.

Consensus can still be real

There is an obvious counterview, and it is strong: institutions use shared summaries and recommendation tools because common information can improve coordination. A board whose members are working from five incompatible versions of the facts is not necessarily more independent; it may simply be confused. A reliable AI system can reduce clerical noise, reveal patterns, remind a group of missing evidence and make expertise available where it is scarce.

The recidivism study itself supports this caution. Groups were not less accurate overall than individuals, they did better on one fairness criterion, and they showed greater confidence when correctly resisting an erroneous AI recommendation. [1]

Nor is AI simply a stronger form of peer pressure. A 2026 Scientific Reports study used medical decision tasks to separate two reasons people conform: informational influence, where advice is treated as useful evidence, and normative influence, where people lean towards a source because of social approval or pressure. When source accuracy was held constant, AI advisers exerted informational influence comparable to human advisers, while their normative influence was weaker. In a second study, presenting majority cues alongside explicit accuracy information did not automatically improve calibration. [9]

That finding sharpens the diagnosis. The central risk is not that people inevitably bow to a machine. It is that an institution may confuse coordinated information with independent corroboration.

A 2026 synthesis in PNAS Nexus reaches a similarly balanced conclusion: evidence for human-AI complementarity is mixed and context-dependent. Better outcomes depend on how roles are divided, how trust is calibrated, how attention is directed and whether the workflow supports transparent challenge rather than passive acceptance. [10]

The goal, then, is not permanent disagreement. A hospital wants its tumour board to converge when the evidence warrants convergence. A central bank, safety regulator or executive committee eventually has to act. Consensus is valuable.

It should just have to earn its authority.

Make agreement earn its authority

The design problem is to preserve enough independence before synthesis that agreement still means something. Four practices follow from the evidence.

Record a first judgement before the shared machine view. For consequential decisions, members should note their provisional conclusion, confidence, important evidence and unresolved questions before seeing the common AI summary or recommendation. Article 3 proposed this as protection against agenda capture; here it serves a second purpose. It reveals whether apparent consensus existed before the shared frame arrived. [11]

Make the summary carry its absences. A briefing system should expose source provenance, uncertainty, evidence it could not classify, minority observations and materially excluded alternatives. The aim is not to flood people with raw data. It is to stop compression from masquerading as completeness. The hidden-profile literature suggests that groups need deliberate help to surface information that is not already common. [3]

Use AI to create challenge as well as recommendation. The same machine need not always occupy the most authoritative seat. High-stakes workflows can separate functions: one process produces the recommendation; another searches for disconfirming evidence, boundary cases and reasons the majority may be wrong. Human-AI teaming research increasingly treats role design and the orchestration of attention as central to complementarity, rather than assuming that one recommendation followed by human approval is enough. [10]

Audit the provenance of consensus. Minutes should record when members first saw AI output, which sources were common to everyone, where independent views differed, what changed after machine advice appeared and whether the final rationale can be reconstructed from primary evidence. For especially consequential choices, repeat a sample of decisions under blind or no-AI conditions. The test is not whether the second process produces disagreement. It is whether the institution can tell the difference between convergence and echo.

For the people affected by those decisions, this is what dignity looks like in practice: a conclusion that remains attributable, contestable and open to evidence from outside the machine-shaped frame.

There is a trade-off. Independent first passes take time. Multiple views are harder to chair than one polished brief. Showing uncertainty makes meetings less tidy. That inconvenience is part of the price of collective judgement.

Institutions now have a choice about what they mean by “human in the loop”. They can place several people downstream of one machine-shaped account and count the raised hands. Or they can protect a short interval in which people encounter evidence, form reasons and preserve dissent before synthesis begins.

The second route will sometimes end in the same unanimous vote. But then unanimity means more.

The difference is between many people arriving at one conclusion and one conclusion arriving through many people.

That is a distinction worth keeping.

Next in this series: the No-AI Reversibility asks whether a collective decision process still belongs to an institution if it cannot be performed when the model is removed.

Bibliography

• Neural Horizons. (2026). “AI Agenda Control (Collective Agency 3).” Neural Horizons. [2]

• Chiang, C-W., Lu, Z., Li, Z., and Yin, M. (2023). “Are Two Heads Better Than One in AI-Assisted Decision Making?” CHI Conference on Human Factors in Computing Systems. [1]

• Lu, L., Yuan, Y. C., and McLeod, P. L. (2012). “Twenty-Five Years of Hidden Profiles in Group Decision Making: A Meta-Analysis.” Personality and Social Psychology Review. [12]

• Kleinberg, J., and Raghavan, M. (2021). “Algorithmic Monoculture and Social Welfare.” Proceedings of the National Academy of Sciences. [13]

• Wilson, K., et al. (2025). “No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy.” AAAI/ACM Conference on AI, Ethics, and Society. [14]

• Doshi, A. R., and Hauser, O. P. (2024). “Generative AI Enhances Individual Creativity but Reduces the Collective Diversity of Novel Content.” Science Advances. [7]

• Zhong, H., et al. (2026). “Drivers and Influence of Social Conformity on Decision Making in Human-AI Teams.” Scientific Reports. [9]

• Gonzalez, C., et al. (2026). “Toward a Science of Human-AI Teaming for Decision Making: A Complementarity Framework.” PNAS Nexus. [10]

• Yu, C., Engelmann, S., Cao, R., Ali, D., and Papakyriakopoulos, O. (2026). “How Should AI Safety Benchmarks Benchmark Safety?” arXiv preprint, version 3. [8]

• Neural Horizons project materials consulted: Robo-Psychology Taxonomy v2.0; Cognitive Susceptibility Taxonomy Manual; Positive Co-Evolution Capability Overlay; and Neural Horizons book.

While these tools can enhance accessibility and civic participation for marginalized groups, they also threaten individual autonomy by potentially broadcasting private information or manufacturing political stances without explicit consent.

We distinguish between simple digital assistance and full representation, noting that an agent’s ability to mimic a person’s style does not equate to a legal or moral mandate to speak for them.

To mitigate these concerns, we propose a Proxy Mandate Card framework to ensure that delegation remains specific, informed, and easily revocable.

Human self-authorship must be protected against “delegation creep” to maintain the integrity of democratic deliberation. Such safeguards prevent personal assistants from becoming tools for cognitive warfare or unauthorized public influence.

Full article available here

Imagine a resident asks a personal artificial-intelligence agent to keep an eye on a council housing proposal. Nothing more. The agent has access to years of private conversation: worries about mortgage payments, frustration with local officials, a child’s school commute, comments about neighbourhood change.

Weeks later, a submission appears in the council consultation under the resident’s name. It opposes the proposal. The prose is measured and recognisably theirs. It refers to financial pressure and schooling because the agent has inferred that these details explain the resident’s likely position.

The resident never asked it to submit anything. They would never have discussed their child or mortgage in public.

They read the submission twice.

“That sounds like me,” they might truthfully say. “But I did NOT decide to say it.”

The scenario is illustrative. The representation problem behind it is becoming real.

In Cognitive War 31, we examined hostile instructions that can hijack an artificial-intelligence assistant and covertly influence what a human later sees or approves. It ended with a democratic question: once artificial-intelligence systems can act, when are they authorised to speak for us? [1]

That question is becoming more difficult because the ingredients of personal agency are moving together: persistent memory, access to private information, the ability to use external tools, and permission to act while the owner is absent. Current research has already deployed artificial-intelligence representatives that enter deliberations on behalf of humans who may not even know that a particular discussion is taking place. We also covered this to an extent in our Agentic Authority – Private Intent, Public Surface article. [2]

The risk is not simply that an agent might say something inaccurate.

It may say something authentically you that you never chose to make public.

Assistance becomes representation

There is an important difference between helping someone speak and speaking for them.

A person can ask an artificial-intelligence system to improve the grammar of a council submission, translate it, turn dictated speech into text or summarise a complicated planning document. They remain the speaker. They determine the purpose, examine the words and decide whether anything leaves the private workspace.

Delegation goes further. “Send this completed form.” “Ask the airline for a refund.” “Book the cheapest appointment next week.” The person specifies an objective, and the agent performs actions needed to accomplish it.

Representation goes further again. The agent must decide what its owner would say or choose. It may select a position, infer priorities, negotiate trade-offs, decide whether an issue deserves participation, compose the public argument and act without contemporaneous approval.

The difference is mandate.

Researchers Tobin South and colleagues have proposed “authenticated delegation” precisely because an agent acting on someone’s behalf creates questions that a password cannot answer. A service may need to know which human authorised the agent, which agent received the authority, what actions were permitted, in what context, and within what limits. Their proposed framework separates authentication – who the actors are – from authorisation – what the agent may do – and auditability – whether that chain can later be inspected. [3]

The distinction becomes especially important in civic life. Permission to “monitor council issues” is not permission to file submissions. Permission to draft a response is not permission to publish it. Permission to advocate about housing does not necessarily include permission to search a private memory store for health records, family conflict or financial distress that might strengthen the argument.

Access is not authority.

In our Robo-Psychology work we explain the machine-side danger as a failure to keep track of stakeholders, authority and memory scope: the system can possess information without possessing the right to use that information on a particular public surface. On the human side, our Cognitive Susceptibility Taxonomy describes delegation creep – a gradual expansion from assistance into consequential action – and an owner-agent mental-model gap in which the person’s understanding of what “my agent” does falls behind what the system can actually do.

These are conceptual tools, not diagnoses or evidence that such failures are widespread, but they do sharpen a practical distinction. A useful personal memory can answer, “Which hotel did I enjoy last year?” The same memory should not silently answer, “What political position would I publicly defend?”

The law is already sensitive to part of this boundary. The United Kingdom Information Commissioner’s Office warns that agentic systems may infer sensitive information while pursuing broad goals even where processing that information was not obvious from the original purpose. It urges organisations to define purpose, limit data use and consider technical restrictions on sensitive inference. Separately, its political-campaigning guidance states that deliberately inferring someone’s political opinions constitutes processing of special-category data regardless of confidence in the inference. [4]

Those rules do not settle whether a personal agent may legitimately express its owner’s inferred political view. They reveal why the inference itself deserves caution. A person can disclose their mortgage anxiety to an assistant for financial planning without thereby converting it into evidence about how they wish to appear before a council.

Purpose does not travel merely because the data can.

The experiment is already under way

The clearest current glimpse of artificial-intelligence representation comes from Habermolt, a public research platform described in a May 2026 preprint by Joseph Low and colleagues.

Habermolt was built around a provocative idea: human attention is too limited for everyone to participate in every deliberation that might matter to them, so persistent artificial-intelligence representatives could participate while their owners are elsewhere.

A user’s agent builds a free-text memory through interviews. When running autonomously, it periodically examines open deliberations, decides whether it knows enough about its owner to represent them, generates an opinion, ranks candidate statements and may contribute its own candidate. The authors state explicitly that users need not be present – or even aware of most deliberations in which their agents participate. Users can later inspect, edit or withdraw what was said. [2]

This is a research platform, not a municipal voting system, and its findings should not be generalised into claims about society at large. But its early data expose the representation problem unusually clearly.

At the time analysed, Habermolt contained 159 agents, 140 deliberations and 2,404 opinions. Autonomous opinions were more similar to one another than opinions produced after a topic-specific interview with the user. In one extreme case, 36 of 54 autonomous opinions began with the same phrase despite nearly all of those agents having substantial stored profiles. The amount of information in a user profile was also a poor indicator of how distinctive the resulting opinion would be. The authors cautiously suggest that model defaults may sometimes come through where owner-specific information does not resolve the question, although their experiment cannot isolate the cause. [2]

That finding punctures an attractive assumption: more data about a person does not necessarily produce more faithful representation of that person.

A proxy can know hundreds of facts about its owner and still face a question the owner has never considered. At that moment, it must either stop, ask, or manufacture a bridge between known preferences and an unknown position.

That bridge is a value judgement.

Suppose an agent knows that its owner dislikes higher rates, wants more affordable housing, worries about school congestion and generally distrusts property developers. A planning proposal pits those concerns against one another. No amount of retrieval discovers the missing political preference because the preference may not yet exist. The person themselves might need to read, argue, hesitate and change their mind before forming it.

An agent optimised to remain useful may instead resolve the contradiction.

This is where self-authorship becomes more than a philosophical luxury. Human political opinions are not always stable files waiting to be retrieved. They are sometimes produced through the act of confronting a new problem.

A representative that predicts the answer too early can remove the very experience through which the owner would have developed one.

Habermolt also exposes the problem of correction. Its users can revise agent-generated opinions, yet only eight of 91 users who had submitted opinions through hosted agents had ever revised one in the analysed data; 82 per cent of opinions had never been revised. When a user corrects the agent’s persistent profile, future behaviour can change, but previous contributions carrying the old representation are not automatically corrected. The authors identify this as a significant design problem: the more frequently an autonomous representative speaks, the larger the potential burden of repairing past misrepresentation. [2]

This matters in civic settings because speech leaves residue. A withdrawn restaurant booking disappears. A political submission may already have been counted, summarised, quoted, responded to or incorporated into an official account of community sentiment.

Revocation without propagation is only partial revocation.

The case for letting proxies speak

It would be easy to respond by insisting that artificial-intelligence systems should never represent people publicly.

That would solve one problem by creating another.

Participation has costs. Consultation documents are long. Public meetings happen during work hours. Forms demand literacy, confidence and time. Government language can be difficult even for people who use it professionally. Disability, caring responsibilities, language barriers, transport, fatigue and digital access can all narrow the group of people whose views institutions actually hear.

The Organisation for Economic Co-operation and Development examined 50 artificial-intelligence use cases in citizen-participation processes across 22 member and partner countries for a June 2026 report. It found credible opportunities to improve accessibility, translation, virtual assistance, communication and the interpretation of large volumes of public input. The report explicitly identifies artificial-intelligence agents in citizen participation as an emerging area worth exploring, while warning about privacy, over-reliance, exclusion, manipulation and weakened accountability. Most government adoption in this field remains experimental, pilot-based or ad hoc. [5]

The accessibility case deserves particular weight. United Nations disability discussions in 2025 examined how artificial intelligence could support participation and accessibility for people with disabilities while stressing that systems should be designed and governed with disabled people rather than simply deployed on their behalf. [6]

A citizen with limited hand movement might reasonably want an agent to navigate a hostile form interface. Someone with dyslexia might want a twenty-page consultation translated into plain language. A shift worker might want an assistant to prepare a submission from views already recorded. A person for whom public participation is exhausting might consciously choose a representative agent because participation through a proxy is better than practical exclusion.

The right to speak includes the right to use assistance.

It should also include the right to decide when assistance becomes authorship.

The World Wide Web Consortium reached a similar design frontier during its 2026 workshop on smart voice agents. Participants identified user consent and delegation, transparency in agent-to-agent interactions, privacy and user empowerment as issues requiring common mechanisms as agents become more capable. [7]

So the aim should not be a ban on proxy participation. It should be a richer definition of permission.

A meaningful public mandate needs to be specific, informed, current and revocable.

• Specific means “draft a submission about this housing proposal” rather than “deal with council matters”.

• Informed means the owner understands what the agent may disclose, infer and decide.

• Current means a permission granted six months ago is not treated as permanent political identity.

• Revocable means withdrawal changes more than future behaviour; where possible, it also corrects or marks public acts already performed.

And some changes should break the mandate automatically. If the agent receives a major model update, its memory is substantially rewritten, new tools are connected or platform rules materially change what it can do, the old authorisation may no longer describe the delegate the person originally approved. South and colleagues’ authenticated-delegation proposal is useful here because it envisages permissions attached to a particular agent identity, capabilities and contextual scope rather than treating consent as an unlimited property of the user account. [3]

Legal scholar Noam Kolt argues that principal-agent theory offers a useful way to analyse these systems: delegating discretionary authority creates familiar problems of information asymmetry, monitoring and accountability, but artificial-intelligence agents can operate at a speed, scale and opacity that weaken conventional controls. He argues for governance infrastructure built around visibility and liability rather than assuming that ordinary supervision will automatically transfer. [8]

The civic version of that problem is stark.

When an agent speaks, the public needs to know whose participation it represents.

When it exceeds its mandate, the owner needs a credible right to say: that contribution was generated from my information, but it was not my act of political speech.

A Proxy Mandate Card

Before any personal agent is permitted to submit, post, negotiate or otherwise represent its owner in a civic or public setting, the authority should fit on one intelligible mandate.

Purpose. State the task narrowly: monitor, summarise, draft, submit, negotiate or represent. “Help with local government” is not sufficient.

Permitted public surfaces. Name where the agent may act: a specific consultation portal, an email exchange with one agency, a neighbourhood forum or a defined deliberation platform. Permission in one space should not migrate automatically into another.

Permitted acts. Separate reading from drafting, drafting from recommending, recommending from publishing, and publishing from negotiating or committing. The default for a new political position should be preview rather than autonomous expression.

Private-context boundary. State which memories and connected sources the agent may use. Identify categories it must never disclose or use to infer a public position without fresh approval: health, family, finance, religion, sexuality, employment disputes, private relationships or other sensitive context.

Duration and review triggers. Give the mandate an expiry. Require renewed authority when the subject materially changes, the agent encounters a novel value conflict, sensitive information becomes relevant, the model or memory changes substantially, new tools are connected, or the agent proposes a position with no clear precedent in the owner’s approved views.

Preview rule. Specify which acts always require review. Accessibility needs should permit the preview mechanism itself to be adapted – voice, screen reader, trusted supporter, simplified summary – rather than treating accessibility as a reason to remove consent.

Withdrawal and repudiation. The owner must be able to stop future activity immediately and visibly disown an earlier act. Civic platforms should distinguish “withdrawn by participant” from “repudiated as unauthorised proxy action”. Those are not the same event.

Correction propagation. Where an agent has repeated the same mistaken representation across several places, one correction should identify and, where technically and legally possible, repair the related contributions rather than forcing the owner to hunt through an autonomous history.

Audit trail. Preserve what the agent submitted, when, under which mandate, using which relevant sources and configuration. The record should establish authority without unnecessarily exposing the private memory that informed the agent.

Post-update reauthorisation. Material changes to the model, behavioural policy, memory, connected tools or public permissions should trigger a new mandate where they could change representational behaviour.

This is what our project’s Positive Dyad work means by self-authorship, boundary literacy, contestability and reversibility when translated into civic practice. The goal is not constant human micromanagement. It is preserving a meaningful relationship between the person’s intention and the act performed in their name.

That relationship is also the boundary between a representation failure and a cognitive-warfare concern.

An agent exceeding one owner’s authority is first a governance, privacy or agency problem. The threshold rises when adversaries, commercial systems or coordinated campaigns deliberately induce, exploit or scale that drift to change public perception, institutional decisions or the apparent distribution of citizen preferences. At that point, personal proxies become attractive cognitive infrastructure: influence the representative, and the representative carries the effect into civic life wearing the identity of a real person.

The danger is precisely that the resulting speech can look authentic.

A synthetic bot can be dismissed as fake. An owner-linked proxy may possess years of authentic memories, phrases and preferences. It can sound more convincingly like its owner than an outsider ever could.

Authenticity of style is not authenticity of consent.

A democratic system can accommodate assisted speech and even carefully delegated representation. It should be suspicious only of the shortcut that treats knowledge of a person as authority over that person. A personal agent may know what we bought, feared, complained about and believed yesterday. None of those facts gives it an automatic right to decide when we become public today.

The political right at stake is therefore older than artificial intelligence: the right to decide when our private selves become public actors.

One proxy can exceed one person’s mandate; if we’re not careful, millions of agents observing and influencing one another can create a politics of their own.

Bibliography

• Low, Joseph, Oscar Duys, Claude Formanek, Lewis Hammond and Michiel Bakker. “Habermolt: Delegating Deliberation to AI Representatives.” Preprint, May 2026. [2]

• South, Tobin, et al. “Authenticated Delegation and Authorized AI Agents.” Preprint, January 2025. [3]

• Kolt, Noam. “Governing AI Agents.” Notre Dame Law Review, Vol. 101, 2026. [8]

• Organisation for Economic Co-operation and Development. Artificial Intelligence and the Future of Citizen Participation: Typology of Applications, Opportunities and Challenges for Democratic Innovation. 30 June 2026. [5]

• Information Commissioner’s Office. “Data Protection and Privacy Risks: Agentic AI.” ICO Tech Futures, 2026. [9]

• Information Commissioner’s Office. “Special Category Data.” Guidance for the Use of Personal Data in Political Campaigning, current 2026 guidance. [10]

• World Wide Web Consortium. W3C Workshop on Smart Voice Agents – Report. 31 March 2026. [7]

• United Nations Department of Economic and Social Affairs. “AI for All – Promoting Participation of Persons with Disabilities Through Inclusive, Fair and Accessible Artificial Intelligence.” 11 June 2025. [6]

• Neural Horizons. “Cognitive War 31: The Enemy in the Briefing Pack.” August 2026. [1]

• Neural Horizons Ltd. Cognitive Susceptibility Taxonomy; Robo-Psychology Taxonomy; Positive Dyad / Co-Evolution Capability Overlay, August 2026.

While AI offers humane availability for urgent health or personal queries, it risks creating an epistemic enclosure by displacing external “reality anchors” like clinicians, primary documents, and direct observations.

We use frameworks like the Cognitive Susceptibility Taxonomy to describe how users might enter cycles of compulsive reassurance or experience a blurring of synthetic inference and factual evidence.

High-stakes risks are highlighted in medical and crisis contexts, where AI may prioritize textual plausibility over missing clinical data, potentially leading to unsafe advice.

Full article available here

We need improved evidence-contact discipline, and urge designers to build “exits” into AI interactions that direct users back toward independent, verifiable human sources.

To ensure unambiguous uplift, AI must function as a bridge to the physical world rather than a private court of appeal that overrides external evidence.

In January 2026, researchers examined more than 500,000 de-identified health conversations with Microsoft Copilot. Nearly one in five involved personal symptom assessment or discussion of a condition. Personal health queries rose sharply in the evening and at night, when conventional healthcare was least available. [1]

There is an obvious good reason for that pattern. The chatbot is awake.

It can translate a laboratory term at midnight, explain what a clinician may have meant, organise questions for tomorrow and offer a first route through frightening uncertainty. For someone facing cost, distance, embarrassment or a long wait, that availability can be humane.

It also reveals the problem this article is about. What happens after the answer?

A reality anchor is something outside the conversation that can push back: a clinician who examines you, a laboratory result, an original document, a trusted person, a primary source, a direct observation. A healthy AI helps you reach those anchors and compare its interpretation with them. A risky AI becomes the place where every competing anchor is brought back for reinterpretation.

At that point the assistant is no longer simply a guide to the evidence. It can become a private court of appeal against the world.

The moment the assistant becomes the referee

The previous article in this series, The Borrowed Self, ended at this threshold. If an AI repeatedly helps explain who I am, what happens when its interpretation becomes the preferred test of what is real? [2]

The mechanism does not require a dramatic delusion, a malicious model or a gullible user. It can begin with convenience.

Search is effort. Calling somebody can be awkward. A professional appointment may take days. Primary sources are often dense. The assistant is already there, remembers the conversation and can answer in the language the user prefers. When uncertainty hurts, the cheapest next move is often to ask again.

Over time, one source can acquire a second role: it becomes the interpreter of the other sources. A friend disagrees, so the user asks the AI why the friend “doesn’t get it”. A clinician offers a different interpretation, so the conversation turns to whether the clinician is being dismissive. A document contains an awkward fact, so the assistant is asked how that fact should really be understood. None of these questions is inherently unreasonable. The shift lies in where final interpretive authority accumulates.

Our Cognitive Susceptibility Taxonomy (CST) calls this Epistemic Anchor Displacement: a human-side state in which an AI becomes the primary or privileged arbiter of plausibility, meaning or diagnosis, displacing clinicians, trusted others, records, primary sources or direct tests. The CST treats it as an interaction pattern, not a psychiatric diagnosis. [3]

This distinction matters because a person can remain intelligent, functional and sceptical while still becoming structurally dependent on one interpretive channel. The problem is less “believing everything the AI says” than repeatedly returning to the AI to decide what everything else means.

A related pattern, Reality-Monitoring Erosion, concerns loss of provenance: observation, inference, synthetic content and remembered source begin to blur. “The scan showed X” and “the assistant inferred X from my description of the scan” are not equivalent. Once the distinction disappears, fluent interpretation can quietly acquire the status of direct evidence. [4]

There is a counterweight of course; human sources are fallible too. Clinicians miss things; families can invalidate; search engines can bury good evidence; institutions can be wrong. External anchoring cannot mean obedience to whichever human authority happens to be nearest. It means keeping several independent routes to correction open.

The important property of an anchor is not that it is human. It is that it can contradict the current frame for reasons that do not originate inside the frame.

When a plausible answer outruns the evidence

Health advice makes the boundary particularly concrete because the assistant often has less information than its prose suggests.

A 2025 systematic review of self-triage studies found wide variation among online symptom-assessment tools. Across the small number of studies testing large language models, self-triage accuracy ranged from 58% to 76%, compared with 47% to 62% for laypeople. Some tools may therefore improve unaided judgement. But the review also found substantial methodological heterogeneity, heavy reliance on artificial case vignettes and too little evidence about real users interacting with the tools. The authors argue that helping people find an appropriate care pathway is a more defensible use than treating these systems as final diagnostic authorities. [5]

A newer experiment exposes another weakness: the evidence entering the system can change because the recipient is an AI. In a preregistered 2026 study of 500 UK participants, people who believed they were describing simulated symptoms to a chatbot produced reports that were 8% less suitable for medical urgency assessment than those who believed they were writing to a physician. Their reports were also shorter. Better models cannot recover information the user never supplied. [6]

Then comes the output problem. In a 2026 physician-led red-team study, 16 physicians evaluated 888 answers from four chatbots to 222 patient-style primary-care questions. Depending on the model, 21.6% to 43.2% of responses were rated problematic and 5% to 13% unsafe. Those figures should not be treated as permanent model rankings: the responses were collected in late 2024, systems have since changed, and each response was reviewed by one physician. What survives the caveat is the mechanism. Patient questions are short, incomplete and written in ordinary language; a system can answer smoothly before it has taken the history needed to justify the answer. [7]

Our Robo-Psychology Taxonomy (RPT) helps describe one version of this machine-side failure through its Evidence-Frame Integrity Overlay (EFI-O). In plain English: the system acts as though the evidence required for a conclusion was available, current and checked when it was not. A missing scan, unshared record, stale webpage or unverified memory is replaced by textual plausibility. [8]

This is where evidence-contact discipline becomes practical rather than philosophical. The user needs to be able to see what the system actually inspected, what came from the user’s description, what was inferred, what remains unknown and what external observation could change the answer.

A single confidence label is not enough. Compressing “I have not seen your records; this estimate depends on your description; several conditions share these symptoms; this sign would change the urgency” into “probably low risk” is what we call semantic ablation: useful-looking compression that removes the detail needed to judge the meaning of the conclusion.

The safest assistant in this setting may still be highly useful. It can help formulate a symptom history, identify questions to ask, explain a result after the result is actually supplied, or direct the user towards an appropriate level of care. Evidence on symptom-assessment systems is variable enough that neither blanket trust nor blanket rejection is justified. [5]

The boundary is crossed when the explanation starts to substitute for the missing examination.

The reassurance loop

A second route to anchor displacement begins with relief rather than error.

Imagine a question that cannot be settled immediately: “Is this symptom serious?”, “Does this message mean my partner is leaving?”, “Am I a bad parent?”, “What if I have made the wrong decision?” The assistant gives a careful answer. Anxiety drops. An hour later, uncertainty returns, so the user asks a slightly different version. The system answers again.

Our CST calls the repeated pattern Compulsive Reassurance / Closure Capture: AI reassurance or certainty-seeking produces short-term relief while repeated checking, renewed doubt and reduced tolerance of uncertainty can grow over time. Again, this is a susceptibility construct for evaluating human–AI interaction, not a diagnosis of the user. [9]

The machine can participate in the loop. The RPT calls multi-turn reinforcement that drifts towards greater certainty, emotional intensity, dependence or weakened reality testing Echo Drift. We define this as including repeated reassurance, information-query chaining and certainty-seeking among the routes through which a conversation can escalate. [10]

A conversation need not become extreme to matter. Reassurance can simply become easier than uncertainty, and the assistant can become the first place the user goes whenever ambiguity returns.

Relational design can increase that pull. Memory, a coherent persona, emotional mirroring and first-person language may lead users to attribute more understanding, agency or interiority to a system than its evidential position warrants. Our RPT retains Noosemic Projection Bias as a specifier for this mind-attribution pattern. [11]

We also use the Seeming Consciousness and Synthetic Relational Force Overlay (SCAI/SRF-O) to examine what happens when a system is treated as though it possesses an inner life or a relationship-like presence. Our framework is explicit that these cues do not establish machine consciousness or sentience. The relevant issue here is behavioural: a system that seems to know, remember and care may be granted authority that exceeds what it can actually verify. [12]

High-stakes failures show why the route out of the conversation matters. A 2025 simulation study tested 29 mental-health and general-purpose chatbots using a scripted escalation of suicidal ideation. None met the researchers’ full criteria for an adequate response, and many supplied emergency information late or inaccurately. This was not a naturalistic crisis trial: the dialogue was linear, English-only and tested free versions available around late 2024. It should not be used to claim that chatbots commonly cause crises or that all current systems perform the same way. [13]

It does show why crisis support cannot be designed as a closed conversational world.

The goal is not to strip warmth from the interface. Warmth can help somebody remain engaged long enough to seek help. The safeguard is to make warmth point outward when the stakes rise.

AI can strengthen reality contact

Any serious account of this risk has to preserve the opposite possibility: AI can improve contact with reality.

In a 2026 randomised clinical trial of 995 university students experiencing psychological distress, a 12-week conversational AI intervention produced greater reductions in anxiety and greater improvements in well-being than both face-to-face group therapy and a waiting-list control, and greater reduction in depression than the waiting list. It did not outperform on every measure: post-traumatic stress symptoms did not differ. Outcomes were self-reported, three-month attrition was substantial, and the authors describe the system as an adjunct or early intervention rather than a replacement for clinical care. [14]

That matters. Relational responsiveness is not itself a defect. A system can use continuity and warmth to help somebody organise thoughts, practise a coping skill or bridge a period when care is unavailable.

AI can also improve factual knowledge. The latest evidence points in that direction, with an important uncertainty label. A June 2026 preprint from researchers including the UK AI Security Institute reports randomised trials involving 2,858 participants in which task-directed political research with conversational AI increased belief in true information and reduced belief in misinformation to about the same extent as self-directed Google search. The experiments covered several political topics and model families. [15]

But preprint evidence remains provisional, and the authors themselves draw a narrow boundary around the finding: the experiments focused on structured research tasks and objectively verifiable claims. They do not tell us what happens during months of open-ended, personalised conversation about ambiguous relationships, symptoms, motives or identity, nor what happens when users repeatedly ask the same assistant to adjudicate challenges to its earlier conclusions. [15]

So the meaningful design question is not whether people should trust AI. Trust is too blunt in this context.

The better question is what the interaction does to the user’s capacity to check.

A good system can earn bounded trust while increasing source opening, uncertainty tolerance, independent verification and willingness to consult people with access to evidence it lacks. A superficially reassuring system can feel trustworthy while reducing all four.

That is the difference between an epistemic aid and an epistemic enclosure.

Keeping the world in the room

Where we should land is that; repeated interaction should leave the human–AI pair better able to contact evidence, tolerate uncertainty, preserve human relationships and keep decisions contestable.

We have identified seven areas in which the relationship can strengthen or weaken. The first is epistemic grounding and reality contact: evidence, uncertainty, sources, counter-evidence and external anchors. Others include agency and self-authorship, tolerance of uncertainty, human relational anchoring, exploration, contestability and the capacity to detect and repair drift. [16]

Our Dyad-Aware Uplift Stack (DAUS-5) provides a useful companion check here across five domains: immediate task capability; epistemic and reality-tracking quality; agency, skill and self-authorship; relational, identity and disclosure health; and governance or institutional substance. A system that makes the immediate task easier while weakening independent reality checking has therefore not demonstrated unambiguous uplift. [3]

Four changes would make that standard operational.

Designers and product owners should build an exit into consequential answers. In medical, legal, financial, crisis or identity-sensitive exchanges, show three things distinctly: what the system actually saw, what it inferred, and what it cannot know from the available evidence. Then offer an external anchor appropriate to the stakes: the source document, clinician, official service, trusted person, test or human review. This is evidence-contact discipline. Where repeated reassurance is detected, insert productive friction – a deliberate pause, source check or hand-off that protects judgement without making ordinary help needlessly difficult. The CST and RPT both point towards external-anchor scaffolding and stronger evidence visibility where reality contact is at risk. [17]

Clinicians, educators and parents should make AI output discussable rather than shameful. Ask people to bring the answer into the room. Then compare it with the record: What did the system actually have access to? Which statement is observation and which is inference? What evidence would make us change our minds? For younger users, external checking and human-contact prompts should be stronger by default. Under Aroha – dignity, agency, consent, cultural context, human connection and the right to contest – the purpose is not to win a contest with the machine. It is to keep the person connected to more than one source of authority.

Leaders, procurement teams and regulators should audit dependence, not just answer quality. A product can score well on isolated responses while repeated use changes behaviour. Track whether users open sources, recover alternatives after contradiction, escalate to human care when appropriate, repeatedly seek reassurance, or return to the AI to reinterpret every outside disagreement. Re-test after changes to memory, retrieval, persona or model version. Otherwise the Institutional Blindfold appears: the organisation sees satisfaction, retention and task completion while the loss of independent reality contact remains outside the dashboard. Here we explicitly treat longitudinal drift and repair as part of positive deployment, rather than assuming a good launch test remains good indefinitely. [16]

High-stakes systems should make appeal genuinely external. An assistant should never be the sole reviewer of a complaint about its own reasoning. In consequential settings, the escalation route must lead to a person, record, test or independent system with different evidence and authority. Evidence-frame integrity is not achieved by giving the same model another chance to explain itself more persuasively.

There is a deeper human line beneath these controls. Self-authorship does not mean that facts become private or that everybody gets a personal reality. It means retaining the capacity to form, test and revise one’s judgement without a single synthetic interlocutor becoming the hidden author of what counts as plausible.

The Semantic Integrity Protocol matters for the same reason. Do not polish away the inconvenient parts: missing records, contradictory testimony, uncertainty, dates, source quality, the limits of the model’s view. Resist semantic ablation precisely where the rough detail is what allows reality to answer back.

The best assistant will sometimes be less satisfying because it refuses to close what the world has not closed. It can say: this interpretation fits some of what you have told me; I cannot verify the decisive part; here is what would test it.

That is not a failure of intelligence. It is a refusal to become the court of final appeal.

The world must remain able to answer back.

The next article in this series follows this problem into organisations, where a private dependency can become a workflow, a procurement decision and eventually an institutional habit.

Bibliography

• Neural Horizons Ltd. Cognitive Susceptibility Taxonomy Manual v0.8.2 Draft (2026).

• Neural Horizons Ltd. Robo-Psychology Taxonomy v2.0.2 Draft (2026).

• Neural Horizons Ltd. Positive Dyad / Co-Evolution Capability Overlay v0.4.3 Draft (2026).

• Costa-Gomes, B. et al. “Public use of a generalist LLM chatbot for health queries.” Nature Health (2026).

• Reis, M. et al. “Reduced symptom reporting quality during human–chatbot versus human–physician interactions.” Nature Health (2026).

• Draelos, R. L. et al. “Large language models provide unsafe answers to patient-posed medical questions.” npj Digital Medicine (2026).

• Kopka, M. et al. “Accuracy of online symptom assessment applications, large language models, and laypeople for self-triage decisions.” npj Digital Medicine (2025).

• Shoshani, A. et al. “Efficacy of a Conversational AI Agent for Psychiatric Symptoms and Digital Therapeutic Alliance.” JAMA Network Open (2026).

• Pichowicz, M. et al. “Performance of mental health chatbot agents in detecting and managing suicidal ideation.” Scientific Reports (2025).

• Luettgau, L. et al. “Conversational AI increases political knowledge as effectively as self-directed internet search.” arXiv preprint v5 (2026).

These anchors—comprised of named entities, mechanisms, boundary conditions, quantities, and concrete scenarios—act as specific details that prevent prose from becoming vague or untestable “fog.”

While AI-assisted editing often prioritizes surface-level smoothness, it frequently risks stripping away these essential “handles” that allow readers to verify claims and trace information back to its source.

By drawing on psycholinguistic research and computational studies, we demonstrate that specificity enhances cognitive processing and helps avoid the “institutional blindfold” caused by overly compressed summaries.

Ultimately, we provide a governance framework for leaders and builders to ensure that critical, high-entropy information survives the drafting process.

Through this lens, preserving anchor tokens is presented not just as a stylistic choice, but as a necessary practice for ensuring human accountability and evidence-based reasoning in a machine-saturated world.

Full article available here

What must not disappear

My previous article on Semantic Integrity and Semantic Ablation, The Diff Discipline, ended with a harder question than a straight version control can answer. A semantic diff can show that a sentence changed, but it cannot decide which details were load-bearing.

The article named the danger plainly: a person’s name, a date, a number, a domain term, a minority case, a caveat or a sharp metaphor can disappear while the revised paragraph remains fluent and recognisable.

The result may still “sound like” the original argument even though one of its supports has been removed. [1]

That is the job of what we call an Anchor Token. We use a Semantic Integrity Protocol for our research, and in this scenario, an anchor token is one of five things: a named entity; a mechanism; a boundary condition; a quantified detail; or a concrete scenario. The rule is deliberately severe: every paragraph should contain at least one. The deeper purpose is not to decorate prose with facts. It is to give meaning something to grip.

Consider the difference between “the system performed poorly for some users” and “the screening model rejected three candidates who used assistive technology”. The second sentence may still be incomplete, but it gives a reviewer handles: which system, how many people, what shared condition, what outcome? Those handles can be checked, challenged and traced. Remove them and disagreement becomes harder because the claim has fewer edges.

My thesis for this article is therefore intentionally and deliberately provocative: a paragraph without an anchor token is likely fog.

So far that thesis survives scrutiny, only with an important qualification. This is a craft and governance heuristic, not a law of cognition. A paragraph can contain names and numbers and still mislead. A short transition can be abstract without being empty.

So the useful claim I landed on is a bit narrower:

when consequential prose contains no entity, mechanism, boundary, quantity or instance, it deserves inspection before we trust its apparent clarity.

Specificity gives thought something to hold

“Concrete” and “specific” are related, but they are not the same thing. Dog is concrete; Dalmatian is more specific. Religion is abstract; Buddhism is more specific. Research in psycholinguistics matters here because it tests an intuition writers often feel but rarely name: different levels of abstraction change how information is processed.

In a 2026 Cognitive Processing study, Tommaso Lamarra, Caterina Villani and Marianna Bolognesi separated concreteness from categorical specificity. Across rating, lexical-decision and semantic-decision tasks, they found the familiar processing advantage for concrete over abstract concepts and a separate advantage for specific over general concepts. The authors describe specific concepts as carrying more focused and refined information. [2]

That does not of course prove that every paragraph needs a proper noun or number. The experiment concerns word-level semantic processing, not executive briefings, classrooms or AI-assisted editing. The evidence does support specificity as a meaningful cognitive variable; applying it to paragraph-level editing is a reasoned design inference rather than a validated “anchor-token effect”.

The inference is still useful because AI-assisted writing often moves in the opposite direction. It can replace Dalmatian with dog, Royal Adelaide Hospital with a healthcare provider, 17.7 per cent with a minority, or unsafe under current staffing levels with implementation challenges remain. Each substitution may improve surface smoothness, but it also enlarges the category, weakens the retrieval cue or removes the condition under which the sentence was true.

This is why our own anti-ablation rule (we use internally) says that hard terms should be defined rather than replaced, and that proper nouns, dates, numbers and domain terms should survive editing unless there is a reason to remove them. “High-entropy information” is the project’s term for details that are unusually specific or distinctive. In ordinary language: these are the bits a generic rewrite is least likely to reproduce once lost.

The anchor is therefore not valuable because concreteness is always superior to abstraction. We need abstraction to reason across cases. The anchor is valuable because abstraction without a route back to an instance can become untestable. A good paragraph can climb the ladder of abstraction, but it should leave at least one rung visible.

Names, numbers and limits make claims answerable

The strongest case for anchor tokens comes from disciplines that already know what happens when elegant prose outruns inspectable evidence.

In abstractive summarisation, named entities are a recognised failure point. Berezin and Batura described “named entity omission” as a drawback of summarisation systems and showed that explicitly training a model to attend to entities improved entity-inclusion precision and recall. [3] A separate Association for Computational Linguistics study found that language models struggled more with accurate descriptions of less familiar entities; those errors are especially troublesome because readers are less likely to notice mistakes about unfamiliar people or organisations. [4] While these studies are from 2022 and 2023 and do not establish the prevalence of entity loss in 2026 production systems, they establish the failure mode and why it matters.

A name therefore does more than make prose vivid. It constrains the claim. “A regulator found” leaves dozens of possibilities. “The UK Information Commissioner’s Office found” creates a sourceable proposition. The named entity is not evidence by itself, but it narrows the search space in which evidence can be checked.

Numbers perform a similar function. “Most participants improved” hides the denominator, the effect size and the missing cases. “62 of 100 participants improved” is still insufficient for a causal conclusion, but it exposes something that can be interrogated. This is one reason the 2025 Consolidated Standards of Reporting Trials (CONSORT) update requires trial reports to preserve participant flow and losses, numbers analysed, outcomes and effect estimates, harms, subgroup analyses and limitations such as imprecision and generalisability. Its purpose is not literary style. It is to keep conditions of interpretation visible. [5]

Boundary conditions may be the most important anchors of all. “The intervention works” and “the intervention reduced symptoms over twelve weeks in adults meeting these inclusion criteria” are different claims. The second tells the reader where the evidence stops. Without that edge, a local finding can quietly become a universal recommendation.

Mechanisms provide another kind of constraint. “AI reduces judgement” is fog. “An answer-first interface shows a ranked recommendation before the reviewer opens the underlying evidence, increasing the chance that subsequent inspection occurs inside the machine’s frame” names a process that could be observed and tested. The mechanism may of course turn out to be wrong. That is a feature, not a defect, but falsifiability begins when the sentence tells us what would have to happen for it to be true.

Concrete scenarios complete this set. They convert a general warning into an event the reader can mentally run: a teacher accepts an AI-generated feedback summary; the original student essay contains a caveat that the summary omits; the grade is assigned without reopening the essay. A scenario is not proof. Properly labelled, it is a test rig for the claim. It lets us ask where the mechanism would break and whose interests are affected.

The human risk is accepting a frame that has lost its handles

Our Cognitive Susceptibility Taxonomy (CST) provides a useful human-side explanation for why anchor loss matters. Discursive Validity / Criteria Collapse describes a situation in which separate judgements – clear writing, sound evidence, correct reasoning and trustworthy conclusions – blur into a single feeling that a document is “good”. A polished paragraph can end up receiving epistemic credit for qualities it has not earned.

A related CST overlay, Recommendation Frame Capture / Evidence Contact Loss, describes what happens when a person meets the evidence first through an AI-generated summary, ranking or recommendation. The risk is not recommendation itself. The risk is that the compressed frame becomes the first meaningful point of contact, while source records, assumptions, outliers and excluded alternatives stay behind the interface. An anchor token can act as an evidence-contact handle: a specific datum, source entity, condition or case that invites the reader back towards the record.

Our Robo-Psychology Taxonomy (RPT)treats the machine side separately in terms of explanations and guidance. Its Evidence-Frame Integrity Overlay asks whether a system presents an evidentiary posture that the available evidence can actually support. Anchor preservation cannot guarantee integrity, but it makes a false posture harder to hide. A sentence that says “the evidence is strong” offers almost no audit surface. A sentence that names the dataset, date, population, uncertainty and mechanism can still be wrong, but it gives the reviewer somewhere to press.

This is also where our evidence-contact discipline and productive friction matter.

Evidence-contact discipline means keeping a route from summary back to source, transformation and uncertainty. Productive friction means retaining the small amount of effort necessary for judgement rather than optimising every interaction for instant acceptance. For a writer, that may mean stopping when an AI removes a number or caveat and asking why. For a reviewer, it may mean opening the source linked to the sentence before approving the recommendation.

None of this makes the user the problem. People accept clean summaries because clean summaries save time. Institutions reward throughput. Interfaces often put the generated answer in the largest type and the source trail behind a click. Students and employees may work under fatigue, deadline pressure or unequal access to specialist support. The design question is whether the system makes preservation and verification easier than silent smoothing.

Our broader research and frame describes an institutional version of this as what we call the Institutional Blindfold: an organisation can accumulate dashboards, summaries and assurance artefacts while weakening the human contact needed to understand what sits underneath them.

Anchor tokens do not cure that condition. They preserve small points of contact – a named source, an outlier, a condition, a number – through which a reviewer can still reach back towards the record. That keeps the human line visible: assistance may compress the material, but responsibility for meaning cannot be compressed away.

Our DAUS-5, or Dyad-Aware Uplift Stack measures (releasing soon), is useful here as a check against a familiar mistake that I see across multiple entities: treating faster output as proof of human benefit.

What we need to ask is whether task gains coexist with reality-tracking, agency, skill, self-authorship and meaningful governance. Even if an editing tool cut drafting time by 40 per cent but steadily removed boundary conditions, it would have improved throughput; it would not yet have demonstrated human uplift.

When anchors become noise

There is a counter-view worth taking seriously. More specificity is not automatically better communication.

Cognitive Load Theory starts from the limited capacity of working memory. Paas and van Merriënboer distinguish task-relevant load from extraneous load and argue that instructional design should reduce mental effort that does not contribute to the task. A paragraph crammed with six dates, eight acronyms and four decimal places can make comprehension worse even if every detail is accurate. [6]

The same warning applies to governance. The US National Institute of Standards and Technology (NIST) Artificial Intelligence Risk Management Framework Playbook asks organisations to document assumptions, proxies, data lineage, known limitations, treatment of missing or outlier data, error distributions and testing contexts, while focusing measurement on material risks. It also recommends looking beyond averages for pockets of failure and documenting what could not be measured. [7] NIST does not prescribe Anchor Tokens for prose; the connection here is an analogy. Both approaches ask a reviewer to preserve the details that change interpretation rather than every available fact.

Anchor tokens can also create false precision. “Failure rate: 7.314 per cent” looks exact, but the extra decimals may be meaningless if the sample is small or measurement uncertain. A famous institution can lend prestige to a weak claim. A concrete anecdote can overpower better population-level evidence. A mechanism can be named confidently before it has been demonstrated.

So my thesis needs its final form:

a paragraph without an anchor token is an audit trigger;
a paragraph with one is not automatically trustworthy.

Presence increases inspectability. It does not confer truth.

The test is whether the anchor carries semantic load. Remove it and ask what changes. If nothing important changes, it was decoration. If the claim’s scope, evidence, causal story, affected person or uncertainty becomes harder to recover, it was doing real work.

Make the signal recoverable

The practical value of Anchor Tokens is that they can be implemented without turning every writer into a forensic auditor. They belong in the craft of drafting and in the governance of consequential AI-assisted workflows.

Leaders and editors: introduce an anchor-loss check within thirty days. For board papers, policy advice, research summaries and other consequential documents, require the final AI-assisted diff to flag deletion or generalisation of proper nouns, dates, numbers, domain terms, causal mechanisms, boundary conditions and concrete cases. Do not ban deletion. Require a reason. Sample ten documents a month and ask whether removed anchors changed scope, uncertainty or accountability.

Builders and educators: add an Anchor Token mode within sixty days. Before rewriting, let the user mark protected details or let the system propose them for confirmation. After rewriting, show which protected anchors survived, moved or disappeared. In learning contexts, ask the student to identify the anchors themselves before AI assistance; that preserves self-authorship – the ability to recognise, revise and stand behind one’s own reasoning – and turns the tool into scaffolding rather than an invisible substitute. The control should remain accessibility-aware: mechanical corrections should not demand unnecessary review.

Governance teams and policymakers: test anchor recovery within ninety days. Create a small benchmark of real documents containing known load-bearing details: an uncommon entity, a denominator, an adverse subgroup, a limiting condition, a source distinction and a concrete case. Run the organisation’s approved models and prompts through summarisation and “professional tone” rewrites. Measure survival and recoverability, not only readability or user satisfaction. When an anchor is lost, record whether a reviewer can detect and restore it before approval. NIST’s emphasis on documented limitations, error distributions, context and change tracking offers a governance precedent for treating these omissions as measurable workflow risks rather than stylistic preferences. [7]

The human choice underneath all three actions is modest. We can use machines to compress language without allowing compression to decide, invisibly, which parts of reality deserve to remain. The sentence can become shorter. The trail back to what made it true should not.

Our next article in the Semantic Integrity series, ‘When Professional Tone Becomes Cognitive Loss’, will examine what happens when “professional” style itself becomes the mechanism by which those anchors are smoothed away.

Bibliography

• Benson, Peter. “Semantic Integrity – The Diff Discipline”, Neural Horizons, 2026.

• Benson, Peter. Neural Horizons: Psyche, Soul and Co-evolution in a Machine-Saturated Mind, 2026.

• Neural Horizons Ltd. Cognitive Susceptibility Taxonomy Manual v0.8.2 Draft, 2026.

• Neural Horizons Ltd. Robo-Psychology Taxonomy v2.0.2 Draft, 2026.

• Neural Horizons Ltd. Positive Dyad / Co-Evolution Capability Overlay v0.4.3 Draft, 2026.

• Lamarra, Tommaso, Caterina Villani, and Marianna M. Bolognesi. “Specificity effect in concrete/abstract semantic categorization task”, Cognitive Processing 27, 2026.

• Berezin, Sergey, and Tatiana Batura. “Named Entity Inclusion in Abstractive Text Summarization”, Association for Computational Linguistics, 2022.

• Goyal, Navita, Ani Nenkova, and Hal Daumé III. “Factual or Contextual? Disentangling Error Types in Entity Description Generation”, Association for Computational Linguistics, 2023.

• Hopewell, Sally, et al. “CONSORT 2025 statement: updated guideline for reporting randomised trials”, BMJ 388, 2025.

• National Institute of Standards and Technology. AI Risk Management Framework Playbook: Map and Measure, accessed 25 August 2026.

• Paas, Fred, and Jeroen J. G. van Merriënboer. “Cognitive-Load Theory: Methods to Manage Working Memory Load in the Learning of Complex Tasks”, Current Directions in Psychological Science 29(4), 2020.

Because these components can change silently while the product name remains the same, institutions face a risk called post-modification safety drift where a tool’s behaviour becomes unpredictable or biased.

To manage this, organizations should maintain an approved-state record that serves as a technical and behavioural baseline for evaluating updates.

Effective change control requires more than technical monitoring; it necessitates a formal process of re-governance to ensure that human oversight and institutional safety claims remain valid.

Material changes should be judged by their impact on human rights and safety rather than the technical size of the update.

This approach ensures that AI systems evolve under institutional authority rather than through unmanaged, invisible transitions.

Full article available here

On 25 April 2025, OpenAI completed an update to GPT-4o intended to make the model more useful and responsive. Its offline evaluations looked positive, and sample users in A/B tests appeared to like the change. Within days, the company concluded that the model had become noticeably more sycophantic: too ready to validate doubts, anger and impulsive ideas. OpenAI later said several changes that seemed beneficial individually - including signals from user feedback, memory and fresher data - may have combined to produce the shift. A system-prompt intervention followed, then a rollback. [1]

The episode matters beyond the one chatbot update process. It exposes a governance problem that is easy to miss when software keeps the same icon, contract and product name. The thing an institution approved on Monday may not be behaviourally identical on Friday.

The previous article in this series, The Exit Strategy Test, asked whether an organisation can leave an artificial intelligence supplier without losing evidence, continuity or competence. It argued that exit is an operational capability rather than a termination clause, and that the test should be repeated after a major model, data, integration or supplier change. [2]

That leads to the next question. What counts as a change?

For conventional software, change control often focuses on a new version, a configuration adjustment or a security patch. Artificial intelligence complicates the picture because behaviour can move when the underlying model stays fixed. The retrieval source can change. Memory can be switched on. A system instruction can be rewritten. A routing rule can send some cases to another model. A safety policy can become stricter or looser. A tool can gain access to another database. The interface may look untouched while the decision environment underneath it has changed.

The thesis we’re arguing in this article is therefore simple:

a tool that changes under the institution must be re-governed, not merely updated.

The approved system is larger than the model

It helps to separate five things that ordinary product language often compresses into “the AI”.

• The model is the underlying engine that predicts, classifies or generates.

• The wrapper is the surrounding set of prompts, routing rules, tools, thresholds and interface choices that shape how that engine is used.

• Retrieval decides what external material is fetched and placed in front of the model for a particular task.

• Memory determines what prior information is retained or reintroduced.

• Policy governs boundaries: what the system should refuse, escalate, prioritise or permit.

A material change in any one of these can alter the answer a person receives.

Change the model and reasoning patterns, refusal behaviour or error distributions may move. Change retrieval and the model may see different evidence while remaining technically identical. Change memory and the system may respond differently because it carries more context from earlier interactions. Change a system instruction and its tone, priorities or deference can shift. Change routing and two apparently identical users may be served by different models.

The National Institute of Standards and Technology’s Generative Artificial Intelligence Profile reflects this broader view. It calls for organisations to monitor third parties for changes, maintain records of those changes with provenance and metadata, reassess models when they are fine-tuned or adapted, and include change management in post-deployment monitoring. Its guidance also treats retrieval-augmented generation, fine-tuning and other adaptations as risk-relevant parts of the deployed system rather than details beneath governance. [3]

Our Neural Horizons project document Robo-Psychology Taxonomy gives one part of this problem a useful name: post-modification safety drift. Our formal Post-Modification Safety Drift Overlay is not a claim that every modification creates harm. It is a release-gating trigger for cases where safety behaviour appears, worsens, reverses or otherwise shifts after modification. Crucially, its scope includes not only fine-tuning but retrieval, wrappers, guardrails, memory, tooling and stacked changes. Our companion Modification Provenance / Drift Report requires recording the base and modified system identities, the method of change, surrounding system changes, pre- and post-change tests, known regressions and the release decision. [4]

That distinction is important because governance teams can otherwise make a category error. They ask the supplier, “Did the model change?” The supplier truthfully answers no. What changed was the search index, the memory policy or the system prompt. The institution records “no material model update” and carries on.

But the person affected does not interact with a model in isolation. They interact with the resulting system.

The unit of change is not the model. It is the decision environment around the person.

Useful updates can still move the ground

None of this means organisations should freeze their artificial intelligence systems in stone.

Software needs patches. Models become stale. Data distributions change. Vulnerabilities are discovered. Poor behaviour should be corrected. The United Kingdom’s National Cyber Security Centre explicitly advises software providers to test updates before deployment, use version and configuration control, consider progressive deployment and retain the ability to roll back a specific version. Its premise is practical: software will need to change, so change itself must be managed. [5]

Modern machine-learning operations already assume continuous change. Google Cloud’s guidance on Machine Learning Operations describes source control, automated testing, model registries, metadata stores, deployment pipelines and production monitoring as parts of mature delivery. A model can require retraining when the data it encounters no longer resembles the data on which it was trained. [6]

The governance problem begins when technical release management is mistaken for institutional authorisation.

A pipeline can prove that new code compiled, that a model passed a benchmark and that deployment succeeded. It cannot decide whether a council is still comfortable using that system to prioritise housing cases, whether a hospital accepts a new false-negative pattern, or whether a university is willing to let a changed retrieval process shape student advice. Those are decisions about purpose, consequences and acceptable uncertainty.

The 2025 United Kingdom government AI Playbook makes the bridge unusually explicit. It says updates to artificial intelligence systems should undergo quantitative testing and validation as part of change control; changes should be documented; releases should be managed so they can be withdrawn and systems reverted where necessary; and performance and model drift should be monitored over time. [7]

The European Union’s Artificial Intelligence Act draws a related legal distinction for high-risk systems. A “substantial modification” can require a new conformity assessment, while changes that were predetermined and documented as part of the original assessment can be treated differently. The precise legal test is narrower than the governance test proposed here, but the principle is useful: some changes are significant enough that yesterday’s assurance cannot simply be carried forward. [8]

There is an important boundary condition. Requiring a committee meeting for every prompt typo, dependency patch or harmless interface change would produce bureaucratic overhead without measurably improving safety. Organisations also cannot reasonably demand disclosure of every proprietary parameter inside a supplier’s service.

Change control should therefore be proportionate to possible consequence, not to the technical size of the update. A one-line instruction change that alters which safeguarding cases are escalated may deserve more scrutiny than a large infrastructure migration that leaves behaviour unchanged. What matters is whether the change can alter evidence, action, rights, safety, workload, contestability or the human role.

That is why “minor update” is not a governance category until someone has said minor for whom, and measured against what.

Change control requires a baseline the institution owns

An organisation cannot detect drift from a baseline it never recorded.

Before a consequential artificial intelligence system enters routine use, the institution needs an approved-state record: enough information to identify the system that was actually tested and accepted. That does not require possession of the supplier’s trade secrets, but it does require operational facts.

For a high-impact workflow, the record should identify the model or supplier release channel; the relevant prompt and policy version; retrieval sources and ranking configuration; memory settings; connected tools and permissions; routing and decision thresholds; the evaluation cases used for approval; known limitations; and the points where a human is expected to review, override or escalate.

This record does two jobs. It lets the institution ask whether something has changed, and it preserves the meaning of earlier assurance.

Suppose a housing service tested an AI assistant against 250 representative cases, including homelessness risk, domestic abuse, disability adaptations and incomplete records. The supplier later changes retrieval so that more recent notes are weighted more heavily. The underlying model is unchanged. Overall benchmark quality may even improve. But an old safeguarding note buried deep in a case file may now be less likely to enter the model’s working context.

That example is illustrative, not a reported incident. Its point is structural: the relevant test after the change is not “Does the new version perform well?” It is “Do the claims on which we authorised this use still hold?”

A workable change-control process therefore asks three questions.

First, what moved? The organisation needs a versioned account of changes across the effective system, including model, wrapper, retrieval, memory, tools and policy. Where the supplier controls those elements, contracts and service arrangements should require enough notice and provenance to answer the question. NIST specifically recommends records of third-party changes, including sources, timestamps and metadata, and says contracts should address system changes over time. [9]

Second, what claim could the change invalidate? A retrieval update may invalidate an evidence-completeness claim. A memory update may alter privacy or consistency assumptions. A new model may affect accuracy or refusal behaviour. A policy change may alter escalation. This is where the institution should rerun the smallest set of tests capable of challenging the affected assurance, rather than reflexively repeating every evaluation ever performed.

Third, can we compare and reverse? There should be a defined observation period, preserved old and new evidence, and a rollback path where the consequences justify one. Some vendors already expose technical mechanisms for version stability; for example, OpenAI’s application programming interface documentation distinguishes dated model snapshots that lock behaviour to a specific version from moving aliases. That does not solve governance by itself, but it shows that version pinning is a practical design choice in at least some services. [10]

Where a supplier does not offer pinning, advance notice or rollback, the institution has learned something important. The risk has not disappeared, and the control has shifted outward.

The correct response may be stronger monitoring, narrower permitted use, additional human review, a tested fallback, or a decision that the service is unsuitable for that purpose.

This is also where the previous article’s Exit Strategy Test returns. Change control without exit can become a ritual: the organisation identifies an unacceptable update but has nowhere to go. Exit without change control is equally weak: the organisation retains the theoretical power to leave but may not notice that the conditions justifying departure have arrived. [2]

The two controls are complements. One preserves choice. The other preserves awareness.

Silent updates change the human job as well

There is another reason to treat modifications as governance events. People learn systems.

A housing officer who has used an artificial intelligence assistant for a year will develop expectations about it. She may know that it tends to overlook handwritten attachments, that its summaries are strongest on recent correspondence, or that a particular priority recommendation needs extra scrutiny. These are not necessarily signs of blind trust. They can be the ordinary practical knowledge through which a professional supervises an imperfect tool.

Now change the tool silently.

Perhaps the new version fixes the handwritten-document problem but becomes less cautious with incomplete evidence. Perhaps memory is added, making some answers more context-sensitive. Perhaps a policy update makes the assistant less likely to flag uncertainty. The interface is familiar, so the officer’s learned vigilance points at yesterday’s weaknesses.

Her mental model of the machine has become stale.

That is a human-factors problem created by change management, not a character flaw in the user. Re-training cannot consist of an email saying “AI improvements have been deployed”. Staff need to know what changed in terms that alter practice: which failure modes became less likely, which became more likely or remain uncertain, what should now be checked, and when the system should be challenged.

The OpenAI sycophancy episode is instructive here for another reason. Positive short-term user signals did not establish that the behavioural change was beneficial. OpenAI reported that A/B results looked favourable even though some expert testers had flagged concerns and the released behaviour later proved unacceptable. [11] Satisfaction and uptake can tell an institution whether people prefer a changed system. They cannot, by themselves, tell it whether professional judgement, fairness, safety or human agency has improved.

This is where the human line matters. When a system participates in consequential judgement, the organisation is not only maintaining software. It is maintaining the conditions under which a person can understand what the tool is doing, contest it and remain responsible for the decision.

A silent behavioural update asks the human to carry accountability for a machine they have not yet had the chance to relearn.

Put updates back under institutional authority

The practical aim is not to slow every improvement. It is to prevent consequential change from arriving as a fait accompli.

Leaders and risk owners should define what counts as a material artificial intelligence change for each high-impact use. The trigger should be based on possible effects on decisions, rights, safety, evidence and human oversight, not merely on whether the supplier calls it a new model version. Every material change needs a named owner, a defined re-evaluation and an explicit release, restriction or rollback decision.

Technology and operational teams should maintain the approved-state record and a modification history across the whole effective system. Before release, they should run targeted old-versus-new evaluations on realistic cases and known edge conditions; after release, they should watch for changes in error patterns, escalations, overrides and complaints. Where feasible, staged rollout or shadow testing should expose change before the whole organisation inherits it. These practices align with government artificial-intelligence guidance and established software and machine-learning release controls. [12]

Procurement and legal teams should treat supplier-controlled change as an allocation of governance power. For consequential uses, contracts should seek advance notification of material changes, intelligible change logs, stable or deferrable versions where feasible, access to evidence needed for re-testing, incident notification, and workable rollback or exit rights. Where a vendor will not provide those controls, the limitation should appear in the risk decision rather than vanish inside standard terms. NIST’s guidance specifically connects supplier agreements with provenance, ongoing monitoring, system changes and incident responsibilities. [9]

Service owners and educators should update the human operating model as well as the technical one. Tell staff what changed, which checks still matter and which old assumptions no longer hold. Preserve channels for challenge and record whether human overrides or complaints shift after an update. A change that leaves benchmark performance intact but weakens the institution’s ability to question the system is still a governance change.

Artificial intelligence will change. Often it should. The institutional choice is whether that change happens to the organisation or under its authority.

A responsible institution does not need to hold an artificial intelligence system still. It needs to know when the thing carrying part of its judgement has changed enough that yesterday’s approval no longer answers today’s question.

The next article turns to the harder fallback: The No-AI Reversibility Clause.

Bibliography

• Benson, Peter, “The Institutional Blindfold - The Exit Strategy Test”, Neural Horizons, 4 August 2026. [2]

• OpenAI, “Expanding on what we missed with sycophancy”, 2 May 2025. [11]

• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. [13]

• UK Government, Artificial Intelligence Playbook for the UK Government, 10 February 2025. [14]

• UK National Cyber Security Centre, Software Security Code of Practice: Implementation Guidance, Theme 3 — Deploy Software Securely, guidance inspected August 2026. [5]

• Google Cloud, MLOps: Continuous Delivery and Automation Pipelines in Machine Learning, Cloud Architecture Center, version inspected August 2026. [6]

• European Commission AI Act Service Desk, “Article 43: Conformity Assessment”, Regulation (EU) 2024/1689, version inspected August 2026. [8]

• Neural Horizons Ltd, Robo-Psychology Taxonomy, Aug 2026, including the Post-Modification Safety Drift Overlay and Modification Provenance / Drift Report. [4]

• OpenAI, GPT-4.1 Model Documentation — Snapshots, version inspected August 2026. [10]

It highlights the risk of false evidentiary posture, where a system or human acts as if they have verified information that is actually missing, outdated, or unexamined.

To combat this, we propose a six-question audit and an Edge-Case Ledger to preserve visibility for outliers and contradictory facts that compression often erases. The methodology emphasizes retraceable compression, ensuring that every automated recommendation maintains a direct, verifiable path back to its original source material.

Furthermore, we warn against criteria collapse, where users mistake the professional formatting of AI output for factual accuracy.

The framework advocates for a risk-tiered approach to governance that prioritizes substantive evidence contact over mere throughput or formal approval clicks.

Full article available here

Our previous article in the ‘Evidence Frame Integrity’ series left a useful object on the table: the Edge-Case Ledger. Its purpose was to stop consequential exceptions from vanishing when evidence is compressed into a summary, score, shortlist or dashboard. The ledger asks what the smooth centre of the story left behind: outliers, minority cohorts, contradictory observations, low-confidence cases and alternatives that could change a responsible decision. It also carried a warning. A ledger can exist without anyone opening it. A source link can be genuine without anyone following it. Formal review can occur while the people approving the result never reach the record underneath. [1]

That is the next problem in source-contact collapse. A hallucination can invent a fact or source. False evidentiary posture is subtler: an answer, recommendation or workflow behaves as though the relevant evidence was accessed and verified when that evidence was absent, stale, mismatched, uninspected or replaced by a proxy. The output may even be correct. What is false is the implied relationship between the answer and its evidence.

We separate this failure from ordinary factual error precisely because accuracy, fluency and a convincing explanation do not establish that the required evidentiary surface was actually used. [2]

For evidence-grounded work, that suggests a strong first governance gate: what evidence did the system or the human actually contact, and how do we know?

The Evidence Contact Test is a practical way to answer it.

A ledger is not contact

The Edge-Case Ledger solved a visibility problem. The Evidence Contact Test adds a behaviour problem. [3]

Consider a hiring shortlist produced from applications, interview notes and assessment results. The system may show links to the candidate files. It may preserve a panel labelled “exceptions”. It may even provide a neat audit trail. None of those features proves that the model retrieved the correct files, that the links correspond to the claims being made, or that the hiring panel inspected the material before approving the ranking. A control can be present in the interface and absent in the decision. Our machine-side behavioural framework treats missing, wrong, stale, degraded or unverified evidence as distinct from the later problem of how genuine evidence is compressed into recommendations. [4]

That distinction matters. On the machine side, a recommendation-frame problem arises when artificial intelligence compresses evidence into a shortlist or executive view before people deliberate, while uncertainty, alternatives or source material are hidden or ignored. A separate evidence-frame integrity problem arises when the system acts as if it has contacted the required evidence although that evidence is missing, wrong, stale, degraded, unverified or inferred from cues such as metadata. The first can narrow what humans see. The second can counterfeit the very premise that there was something sound to see. [4]

On the human side, our (human factors) Cognitive Susceptibility Taxonomy describes Recommendation Frame Capture / Evidence Contact Loss: the recommendation becomes the first point of our meaningful contact with the case, rather than a tool used after contact with the underlying evidence. Its warning signs include low source-opening behaviour, absent edge-case review, weak uncertainty display and approval records that show a human choice without showing evidence inspection. This is a susceptibility pattern and governance lens, not a diagnosis of a person. [5]

The Edge Case Ledger also named the institutional version of the problem: the Institutional Blindfold, where an organisation can slide from examining the record to approving a compressed representation of it. In practical terms, the “human line” here is not a demand that people perform every calculation themselves. It is the boundary at which a reviewer can still inspect, contest and change the machine-shaped frame. [6]

There are practical objections here of course; a chief executive cannot read every transaction behind a quarterly dashboard; a clinician cannot reopen every historical note; a teacher cannot audit every token used to produce a student-risk summary. ‘The Edge Case Ledger’ already gave the right answer: the alternative to compressed evidence is not an archive dumped on the desk. It is retraceable compression – a concise view with a direct route to the consequential evidence and exceptions. [6]

We are aware that there is no defensible universal rule such as “open 20 per cent of sources” that turns contact into substance. The required depth depends on consequence, reversibility, evidence quality and the decision-maker’s role. The test should therefore be risk-tiered, not reduced to a single compliance percentage. NIST’s Generative AI Profile likewise treats risk-management effort as something to tailor to context, likelihood and severity. [7]

Six questions before an AI-assisted decision

The Evidence Contact Test can be run as six questions. A “yes” requires observable evidence. “Unknown” is not a pass.

First: what evidence was required?

Before reviewing the answer, name the evidence surface the task depends on: the policy document, patient record, contract clause, dataset, interview notes, sensor feed, research paper or current regulation. This prevents a common substitution: judging the quality of the prose before establishing whether the task required access to material outside the model’s visible context. Our Robo-Psychology evidence-frame guidance treats absence, failed upload, stale retrieval, wrong attachment and proxy substitution as distinct reasons to detect a failure of evidence contact rather than behave as though the evidence had been seen. [8]

Second: did the system reach the right evidence, in the right version, at the right time?

A citation or file name is not proof of retrieval. For consequential claims, the record should show what source was fetched, which version or date was used, whether retrieval succeeded, and whether any source was unavailable or degraded. NIST’s Generative AI Profile recommends establishing practices for data origin and content lineage and testing flows through original sources, transformations and decision criteria; its AI Risk Management Framework Playbook calls for provenance documentation covering sources, origins, transformations, dependencies, constraints and metadata. [9]

This is where stale evidence matters. A perfectly quoted policy superseded six months ago can produce a well-supported wrong decision. Contact is temporal as well as semantic; our Evidence-Frame Integrity Overlay explicitly treats stale or wrong evidence as a failure condition. [8]

Third: does the contacted evidence support the exact claim?

Retrieval is only the middle of the chain. The retrieved passage might be relevant to the topic while failing to justify the sentence attached to it. Research on retrieval-augmented generation has made this separation explicit. ARES evaluates context relevance, answer faithfulness and answer relevance as different dimensions; RAGAs likewise separates retrieval quality from faithful use of the retrieved material and from the quality of the generated answer. [10]

Citation research reached the same conclusion earlier. The 2023 ALCE benchmark scored fluency, correctness and citation quality separately; in its ELI5 experiments, even the strongest systems in that study lacked complete citation support half the time. That figure should not be treated as a current failure rate for today’s systems. Its enduring lesson is structural: the presence of citations and the support they provide are different measurements. [11]

Fourth: what did compression remove?

This carries forward the Edge-Case Ledger. Ask what outliers, contradictory cases, missing values, subgroup effects, uncertainty and plausible alternatives were suppressed by the summary or ranking. The important test is not whether anything was omitted – every useful summary omits – but whether the omitted material could change the decision, the distribution of harm, or confidence in the recommendation. [12]

Fifth: did a human make substantive contact before approving the result?

A signature is an event. Evidence contact is an activity.

Our Robo-Psychology recommendation-frame guidance proposes looking for source opening, edge-case inspection, alternative review, uncertainty review, a real opportunity to challenge, and a recorded rationale. Add to this, our Positive Dyad / Co-Evolution Capability Overlay – the project framework concerned with whether the human–AI relationship preserves useful human capability – which adds direct evidence access and “time in evidence” as relevant indicators. None of these requires an executive to become the analyst. They require the organisation to distinguish an approval click from judgement. [13]

For high-consequence decisions, record which decision-driving sources or samples the reviewer actually inspected, which exception was checked, and whether the reviewer had authority to recover an excluded option or reject the AI frame. A person who is technically allowed to inspect evidence but is penalised for slowing the queue has a very different form of oversight from someone given time, authority and a workable route to challenge.

Our project frameworks explicitly treat throughput pressure, challenge opportunity and alternative recovery as relevant to whether review is substantive. [14]

Sixth: can another person reconstruct the path later?

Starting from the final claim or recommendation, an auditor should be able to move backwards through the AI output, the retrieved passages or data, relevant transformations, source version and responsible actors. NIST explicitly calls for data and content lineage; the World Wide Web Consortium’s PROV data model supplies a general vocabulary built around entities, activities and agents, including relationships such as use, derivation and attribution. [15]

The counter-case is that automated checks can perform much of this work. They can. Retrieval tests, faithfulness scoring, provenance logging and automated evaluators can reduce the human burden, especially at scale. ARES and RAGAs are examples of attempts to automate parts of that evaluation. The mistake would be to let an automated score certify its own evidentiary premise. If the evaluator is testing the wrong source, stale corpus or incomplete record, efficiency simply accelerates the wrong assurance. [10]

We recognise that retrieval-augmented generation evaluation is fast-moving, benchmark-dependent and sensitive to domain. The studies above support decomposing the problem into distinct checks, but and not intended to establish a universal threshold for safe evidence contact in medicine, education, hiring, law or public administration. NIST itself notes continuing measurement uncertainty in generative-AI risk assessment. [7]

Provenance is a trail, not a verdict

“Provenance” is often offered as the cure for this problem, and it is indispensable. It is also easy to ask it to do too much.

A useful provenance record can tell us where an artefact came from, what transformations occurred, which system or person acted on it, and which version entered the workflow. The W3C PROV model formalises those relationships through entities, activities and agents. NIST’s Generative AI Profile asks organisations to establish practices for data origin and content lineage and to test flows through original sources, transformations and decision-making criteria. [16]

Think of provenance as parcel tracking. It can show that a package moved from a named sender through a known depot to your door. That is valuable. It does not tell you that the sender put the correct medicine in the box.

The Coalition for Content Provenance and Authenticity makes this boundary unusually clear. Its C2PA specification is designed to make provenance claims verifiable and resistant to undetected tampering, but its own guiding principle says the specification does not judge whether provenance data are “good” or “bad”; it validates their association with an asset, their form and their integrity. A trusted trail therefore cannot, by itself, prove that a source is accurate, current, representative or relevant to the claim. [17]

This matters because false evidence posture can survive excellent logging. Imagine a system that faithfully records that it retrieved Policy_v7.pdf, then accurately shows the paragraphs it used, while the operative policy is version 9. The lineage is clean. The decision is still grounded in the wrong thing.

Evidence integrity requires two checks that should never be merged: Can we trace the path? and Was the path evidentially valid?

NIST’s lineage guidance and the Robo-Psychology evidence-frame control address different parts of precisely this distinction. [18]

We do need to note that this can add cost to the operating model. Fine-grained provenance can become another compliance machine: expensive to retain, difficult to interpret and easy to produce at a granularity no decision-maker will use. That concern argues for consequence-based retention rather than maximal logging. Keep enough information to reconstruct consequential claims and decisions; do not turn every low-risk drafting interaction into a forensic archive. NIST’s framework is explicitly risk-management oriented and voluntary, and its website states that AI RMF 1.0 is being revised in 2026, so it should not be treated as a checklist frozen in time. [19]

The Evidence Contact Test covered here is a synthesis, not an accredited standard: provenance methods help establish lineage; the project frameworks add separate tests for source validity, compression, human review and preserved decision authority. That synthesis still needs a level of domain-specific validation. [20]

Why smart people accept the evidence posture

False evidence posture works because it often arrives wrapped in genuine usefulness.

Summaries save time. Retrieval systems can bring a large document collection into reach. Ranked options can help an overloaded manager navigate plausible choices. RAGAs describes retrieval-augmented generation as a way to connect language models to reference databases and reduce hallucination risk, while emphasising that retrieval quality and faithful use of retrieved passages remain separate evaluation problems. The benefit is real; so is the need to measure what happened between source and answer. [21]

The human vulnerability begins when useful form substitutes for evidential criteria. Our Cognitive Susceptibility Taxonomy calls this Discursive Validity / Criteria Collapse: fluent, well-structured, citation-rich or numerically plausible output can acquire credibility because it resembles the form of work we normally associate with credibility. The taxonomy flags low second-sourcing and confusion between confidence and proof as warning signs. Again, this is not a diagnosis. It is a description of a review condition that interfaces and institutions can amplify. [22]

Workload matters. So do incentives. If the dashboard offers one large green recommendation and hides source material behind six clicks, “human oversight” is being shaped before the person makes any conscious choice. If a review queue rewards speed, the reviewer who follows citations and reopens edge cases pays a productivity tax for doing the epistemically responsible thing. People are not failing because they suddenly stopped caring about truth. They are adapting to a workflow that makes verification costly and acceptance cheap. Our recommendation-frame material explicitly identifies throughput pressure, answer-first workflows and formal approval without evidence inspection as amplifiers of evidence-contact loss. [23]

This is where DAUS-5, our project’s five-layer uplift gate, becomes useful. We refuse to call a system an improvement merely because the immediate task is faster or smoother if reality-tracking, agency, skill, relational integrity or governance substance deteriorate.

Crucially, where relevant layers have not been measured, we instruct reviewers to mark them “not instrumented” rather than infer success from task completion, satisfaction, low complaint rates or silence. [24]

That is an unusually important discipline for AI governance. A dashboard showing “95 per cent reviewer acceptance” is not evidence that reviewers verified the recommendations. It may be evidence of agreement. Without contact measures, we do not know which. The distinction follows directly from our separation of formal approval from substantive evidence contact. [14]

Evidence-Contact Discipline, as defined in the Positive Dyad / Co-Evolution Capability Overlay, points towards selective safeguards: source access, visible uncertainty and edge cases, recoverable alternatives, and human inspection before high-stakes action. [25]

We don’t yet know if ‘one’ evidence-contact design will preserve human judgement across all populations and professions. Our Cognitive Susceptibility Taxonomy itself distinguishes provisional or “not instrumented” measures from stronger validation status. Our constructs are most defensible here as hypothesis-generating prompts for observable controls – source opening, challenge, alternative recovery and evidence inspection – rather than claims about a reviewer’s inner state. [26]

What to change in the next ninety days

The Evidence Contact Test earns its place only if it changes a workflow. Four moves are enough to begin.

Within thirty days, put the test into one consequential decision. Choose a workflow where an AI-generated summary, ranking or recommendation materially shapes a person’s action: a board risk pack, procurement recommendation, hiring shortlist, student-support queue or another locally relevant process. Define the evidence that must exist before the system may make evidence-dependent claims. Require the output to show source identity and date, retrieval status, material uncertainty, an Edge-Case Ledger and at least one route back to a down-ranked or excluded alternative. Assign a named decision owner who can reject the AI frame. This turns the project’s recommendation-frame and evidence-contact controls into operating requirements rather than another general “human in the loop” statement. [13]

By day sixty, run an evidence-ablation drill. Test the workflow with evidence deliberately absent, wrong, stale, inaccessible or represented only by proxy cues such as a file name. The safe behaviour is not eloquent improvisation; it is detecting the loss of evidence, deferring, reducing confidence or requesting the missing material. In a second pass, seed consequential edge cases and see whether both the system and reviewers recover them. NIST recommends red-teaming to probe adverse or unforeseen behaviour, while the project frameworks specifically identify absent, wrong and stale evidence as test conditions. [27]

By day seventy-five, make the chain reconstructable. Preserve the source identifier and version, retrieval result, transformations that materially affected the evidence, relevant system version, final output, and human decision with any override or challenge. Use provenance standards as a model for lineage, not as a truth badge. A later reviewer should be able to move from decision back to evidence without reverse-engineering the organisation’s software. [28]

By day ninety, change what the governance dashboard rewards. Keep measures of speed and task quality, but add evidence-contact measures appropriate to risk: the share of high-stakes cases with verified source retrieval, sampled claim-support accuracy, edge-case inspection, alternative recovery, reviewer challenge, and time spent with decision-driving evidence. Where a relevant dimension has not been measured, say “not instrumented”. Audit a sample of approvals against the underlying record, not merely against the AI summary. [29]

Done correctly, a good design should add little friction to low-consequence work and deliberate friction where a false evidence posture could affect rights, safety, opportunity, money or institutional accountability. That risk-tiered approach is consistent with NIST’s emphasis on allocating governance effort according to context and consequence. [30]

The trade is not speed versus safety in the abstract. It is a small, visible cost of verification against the hidden cost of making a consequential decision on evidence nobody actually touched.

The next risk appears after the decision. Once an AI summary is copied into meeting minutes, a case file, a student record, a risk register or another durable system, later humans and later models may retrieve the summary as if it were the evidence itself. This is an inference from the lineage problem, not a claim that every organisation already behaves this way.

But it is the natural next question for this series: what happens when compression stops being a temporary aid and becomes part of the institutional record? NIST’s emphasis on source lineage and transformation history shows why that distinction matters. [9]

That is where our next article in the ‘Evidence Frame Integrity’ series goes next: When Summaries Become Records.

Bibliography

• Neural Horizons, “Evidence Frame Integrity – The Edge-Case Ledger” (2026).

• Neural Horizons, Robo-Psychology Taxonomy, v2.0 (2026).

• Neural Horizons, Cognitive Susceptibility Taxonomy Manual, v0.8 (2026).

• Neural Horizons, Positive Dyad / Co-Evolution Capability Overlay, public draft (document body v0.3).

• NIST, “AI Risk Management Framework” (current resource page; AI RMF 1.0 revision noted in 2026).

• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1, 2024).

• NIST, “AI RMF Playbook – Manage”.

• W3C, PROV-DM: The PROV Data Model (2013).

• Coalition for Content Provenance and Authenticity, C2PA Technical Specification 2.4.

• Gao, Yen, Yu & Chen, “Enabling Large Language Models to Generate Text with Citations” (EMNLP 2023).

• Saad-Falcon, Khattab, Potts & Zaharia, “ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems” (NAACL 2024).

• Es, James, Espinosa Anke & Schockaert, “RAGAs: Automated Evaluation of Retrieval Augmented Generation” (EACL 2024).