When AI makes the claim, who checks it before you act?

The incident does not prove that your AI system is unsafe. It does not establish that every AI-generated error will survive human review. It does not show that the use of AI itself caused every reported failure in the wider decision process.

It does illustrate a failure pattern worth testing in any organisation:

1. a system produces or strengthens a claim
2. the claim acquires authority as it moves through a formal workflow
3. the people downstream see a finished answer rather than the weaknesses underneath it
4. action begins before the evidence is reconstructed

The danger is not simply a model producing a wrong sentence. Organisations have always handled bad information. The change is speed, scale and apparent fluency. A plausible output can pass through familiar documents, dashboards and approval routes before anyone notices that the supporting evidence is weaker than the presentation.

Experts disagree sharply about the probability and timing of long-term loss of control. The International AI Safety Report 2026 says that expert opinion varies greatly and that the likelihood, nature and timing remain unusually ambiguous. It also says current systems show early signs of some relevant capabilities, but not at levels that would enable the loss-of-control scenarios it examines.

That disagreement matters for policy and long-range investment. It is a poor excuse for leaving today's operating boundary vague.

You do not need to settle the existential-risk debate before deciding:

• which information an AI system may read
• which sources it may retrieve from outside your organisation
• which outputs require verification
• what it may create, change, publish or send
• which decisions require named human authority
• how an action can be stopped and reversed

These are ordinary leadership decisions made more urgent by a technology that can compress research, interpretation and execution into one apparently smooth sequence.

Do not begin with the product name or the label attached to it. Assistant, copilot and agent are used too inconsistently to tell you what the system actually does.

Start with one real workflow and trace the first point at which AI touches it.

Ask what triggers the system. Is it responding to a person, monitoring an inbox, reading a document, querying a database or acting when another system changes state?

Then separate four levels of authority:

1. Answer: it produces text or analysis for a person to consider.
2. Access: it can retrieve files, records, messages, websites or connected-system data.
3. Recommend: it interprets evidence and proposes a decision or next action.
4. Act: it can create, change, publish, send, buy, move or delete something.

The commercial and operational exposure changes at each level. Read-only access can still disclose sensitive information or shape a poor decision. A recommendation can carry more authority than its evidence deserves. An action can reach a customer, supplier, employee or financial system before a person understands what happened.

For a consequential decision, you should be able to reconstruct the claim from the output back to its inputs.

That means seeing:

• which internal records were used
• which external sources were retrieved
• when those sources were published or updated
• what the model inferred rather than found
• where conflicting evidence was excluded or compressed
• what confidence or uncertainty was communicated
• which person reviewed the evidence rather than only the summary

This is also where prompt injection becomes an operating issue rather than a technical curiosity. OWASP explains that indirect prompt injection can occur when a model accepts content from external sources such as websites or files and treats material within them as instructions. Retrieval can therefore import both information and an attempt to influence what the system does with it.

A source link alone is not enough. The reviewer needs to know whether the source supported the claim, whether the system transformed it accurately and whether another instruction altered the result.

“A human was in the loop” tells you almost nothing about the quality of the control.

The useful questions are:

• Which human?
• What were they authorised to decide?
• What evidence could they see?
• Did they have time and expertise to challenge it?
• Could they stop the workflow without being penalised for delay?
• Was their approval recorded against the exact version they reviewed?

A person asked to approve a polished recommendation without its evidence is not exercising meaningful oversight. They are absorbing accountability after the system has already framed the decision.

The NCSC's May 2026 guidance recommends starting with tightly bounded, low-risk tasks, applying least privilege and making clear who owns access, monitoring, incidents and the ability to stop a system. Its August 2026 guidance adds practical emphasis on defined red lines, sandboxing, guaranteed approval gates and not relying on prompting alone.

These controls do not require every organisation to build a new governance department. They require the existing owners of technology, data, operations and commercial outcomes to agree where authority begins and ends.

Good control does not mean preventing every mistake. That promise would be dishonest. It means reducing the chance that one weak output becomes an irreversible consequence.

For each material workflow, establish:

1. Reconstruction: can you reproduce which inputs, instructions, tools and model version contributed to the result?
2. Interruption: can an authorised person pause the workflow before the consequence reaches the outside world?
3. Reversal: if an action is taken, can it be undone safely and within a useful time?
4. Escalation: does the system recognise when evidence is missing, contradictory or outside its permitted scope?
5. Ownership: is one named person accountable for the workflow rather than only for the software contract?

The NIST AI Risk Management Framework provides a useful operating sequence: govern, map, measure and manage. Its value is not the vocabulary. It is the insistence that roles, context, evidence, risk response and ongoing review remain connected.

The answer is not to wrap every harmless use of AI in a committee.

A tool drafting an internal meeting summary does not need the same controls as a system that changes a price, sends a customer message, approves a supplier, ranks a candidate or moves money. Treating them as equivalent wastes attention and turns governance into theatre.

Match the control to three things:

• reach: who or what can be affected
• reversibility: how easily the outcome can be undone
• evidence burden: how much confidence the decision requires

Low-consequence uses may need simple disclosure and spot checks. Higher-consequence uses may need constrained access, independent source verification, versioned decision records, explicit approval and a tested rollback route.

The point is not to slow every workflow. It is to stop speed from hiding the moment when a suggestion becomes authority.

Choose one AI-enabled workflow that already affects a real decision. Ask:

1. Where exactly does AI enter, and what outcome is it meant to improve?
2. What internal data, external material and connected systems can it use?
3. What can it create, change, recommend or send without another check?
4. Who sees the underlying evidence and holds authority to stop the action?
5. Can the decision be reconstructed and the consequence reversed?

If the answers are vague, do not begin with a policy document covering every possible use. Tighten the boundary around that workflow first.

You may find that the system is working well and the right decision is to leave it alone. You may find that it needs better evidence, narrower access or one additional approval. You may find that the problem sits outside AI altogether.

The test is whether the evidence helps you make a better decision, including a decision to take no further action.

If you need a practical way to examine one consequential workflow, use our five-question AI authority check. It explains the boundary between answering, accessing, recommending and acting, and what a bounded review can establish without asking you to expose confidential data.

• CNN, “US military had close call after using AI for false intelligence report, sources say”, 18 September 2026. The account relies on unnamed sources and is treated here as reported evidence, not an official finding.
• International AI Safety Report 2026. Used for the state of evidence and expert disagreement about long-term loss-of-control risk.
• UK National Cyber Security Centre, “Thinking carefully before adopting agentic AI”, 15 May 2026.
• UK National Cyber Security Centre, “Managing the cyber risk of agentic AI”, 20 August 2026.
• NIST AI Risk Management Framework.
• OWASP, LLM01:2025 Prompt Injection.