There is a particular kind of silence that precedes a paradigm shift in risk management. It is the silence of industries not yet ready to name what they are seeing. Actuaries and underwriters are watching. Legal departments are running quiet scenario analyses. A handful of specialist insurers are drafting policy language for coverage categories that have no established loss history, no actuarial tables, and no settled case law to draw on. And at the centre of all of it sits a problem that even the most advanced AI labs have not been able to solve: the hallucination.

by Kasun Illankoon, Editor in Chief at Tech Revolt

[For more news, click here]

When a large language model confidently cites a law that does not exist, recommends a drug dosage that would kill a patient, or generates a financial projection built on invented data, the output looks indistinguishable from a correct one. That is the specific danger that enterprises are beginning to reckon with. Not that AI will obviously fail, but that it will fail convincingly, at scale, inside systems that real people are depending on to make real decisions.

The Architecture of the Problem

To understand why this is so difficult to govern, it helps to understand how the failure happens in the first place. Generative AI models are not databases. They do not retrieve information so much as they predict it, constructing responses from statistical patterns learned across enormous volumes of text. The result is a system that is extraordinarily fluent and sometimes breathtakingly wrong, in ways that even its creators cannot reliably predict or prevent.

Toju Duke, Founder and CEO of Diverse AI, is direct about the limits of current governance in addressing this. "The solution to AI hallucinations cannot be addressed by any existing AI governance framework or responsible AI process today," she says. "If there was a solution to it, it would have been addressed by now. The main reason for this is the way generative AI is built."

Duke points to the fundamental architecture of large language models as the source of the problem. By training on vast datasets using neural networks, these systems do not only predict the next word or match pattern sequences. They also fabricate and make up results to provide a convincing answer. And because the neural networks that power these models remain what the AI community calls a black box, opaque and largely resistant to interpretability, even the most rigorous governance framework cannot pinpoint the source or cause of any specific hallucination after the fact.

"This problem will exist for a very long time," Duke adds, "until the AI scientific community finds a solution, which is probably impossible due to the infrastructure that large language models are built from."

"Because neural networks remain a black box, even the most advanced governance framework cannot pinpoint the source and cause of hallucinations."

From Technical Glitch to Legal Exposure

The insurance industry has spent decades pricing risk for systems it can model. Fire spreads in calculable ways. Vehicles fail according to known distributions. Even cyber risk, which seemed intractable a decade ago, has developed enough loss history to support a functioning market. AI hallucination risk is different, and the difference is not simply one of novelty. It is structural.

When an AI system operating inside a legal research platform invents a precedent that a practitioner then cites in court, the chain of responsibility runs from the model developer to the platform operator to the law firm to the partner who filed the brief.

When a medical AI flags an incorrect contraindication and a physician acts on it, liability could attach at multiple points simultaneously. When a financial AI generates a fabricated risk assessment that informs an investment decision, the losses might not surface for quarters. The latency of harm, the distributed nature of the causal chain, and the absence of any clear moment of human negligence make these cases extraordinarily complex to litigate or price.

Enterprises that were early movers on generative AI deployment are already encountering this problem in practice. Several high-profile cases in recent years, including US lawyers who submitted AI-generated court filings containing entirely fictitious case citations, have demonstrated that reputational and legal damage from AI hallucinations is not a theoretical future risk. It is a present one.

The Transparency Deficit at the Heart of AI Decision-Making

Running alongside the hallucination problem is a related and equally urgent issue: the absence of any meaningful requirement for organisations to disclose when consequential decisions are being made by AI systems. Duke argues that this must change, and urgently.

"While several laws require businesses to disclose if customers are interacting with an AI system, no laws currently require businesses to disclose if an AI system made a decision, and this must change," she says.

Her argument is not simply procedural. It is grounded in a long and troubling history of algorithmic decision-making that has caused documented harm across employment, housing, healthcare, and the justice system.

The examples are numerous and, taken together, constitute a serious indictment of ungoverned AI deployment. Amazon's internal recruitment tool, which was trained on a decade of male CVs and systematically discriminated against women. Apple Card's credit limit algorithm, which offered lower limits to women even when their financial profiles were identical to male applicants.

The COMPAS recidivism algorithm, investigated by ProPublica in 2016, which was found to falsely predict that Black defendants would reoffend at twice the rate of white defendants, and which produced accurate violent crime predictions in only 20 per cent of cases.

Duke is unambiguous about the principle at stake: "It is a fundamental human right for people to receive transparency on how a decision was made on their behalf and the factors that influenced that decision, especially if made by an AI algorithm."

"No laws currently require businesses to disclose if an AI system made a decision, and this must change."

What AI Governance Actually Needs to Do

If transparency is the baseline, governance is the structure that makes it enforceable and meaningful. Duke outlines a role for AI governance that goes well beyond policy documents and ethics statements. Effective governance, in her framing, gives organisations the tools to understand, assess, and evaluate their AI systems before, during, and after deployment. It equips businesses with appropriate AI policies and governance structures covering how AI is used both by employees and by customers. And critically, it provides the compliance infrastructure needed to stay ahead of an accelerating wave of global AI regulation.

The regulatory landscape is shifting rapidly. The European Union's AI Act has established the first comprehensive legal framework for AI risk classification in the world's largest trading bloc. Jurisdictions from Singapore to Brazil to the Gulf states are developing their own frameworks. The patchwork nature of this regulatory environment creates both a compliance challenge and, arguably, an opportunity for companies that move first on robust internal governance. Those that can demonstrate documented, auditable AI risk management practices will be better positioned when regulators come knocking, and better positioned still when insurers begin requiring it as a condition of coverage.

Can You Actually Insure Against a Hallucination?

This is the question the insurance industry is now wrestling with in real time. Traditional professional indemnity and errors and omissions policies were never designed to cover AI-generated misinformation. The standard frameworks assume human professional judgment, human error, and human accountability. Generative AI distributes all three in ways that existing policy language simply does not address.

Some specialist insurers and managing general agents have begun developing what might broadly be described as AI liability coverage, typically embedded within technology errors and omissions frameworks or cyber policies. But these products remain nascent and, in most cases, inadequately specific. They tend to cover the consequence, a data breach, a compliance failure, a financial loss, without addressing the causal mechanism that is increasingly responsible for producing those consequences.

Duke sees the evolution of AI insurance as inseparable from the evolution of AI law.

"For this to happen, there will need to be standardised AI governance frameworks across various jurisdictions in the world," she says. "Rather than AI governance standards alone, I see an increased introduction of global AI laws and regulations which will require governance standards and guidelines in the immediate future."

The implication is clear: AI governance standards are not just a responsible business practice. They are likely to become a prerequisite for obtaining coverage at all. Insurers cannot price a risk they cannot model, and they cannot model a risk for which there are no standards, no disclosures, and no auditable compliance trails.

The Enterprise Reckoning That Is Coming

For enterprises currently running generative AI at scale, the question of liability is no longer purely hypothetical. The combination of rapidly expanding deployment, accelerating regulation, and an insurance market that is only beginning to understand the exposure means that the window for proactive action is narrowing.

Companies that have deployed AI in healthcare diagnostics, legal document generation, financial advisory, or any other domain where the consequences of incorrect outputs carry real-world stakes need to be running serious risk assessments now, not waiting for regulatory compulsion or the first class-action lawsuit to force the issue. Duke's prescription is practical: create risk management programmes designed to identify and mitigate discrimination risks, conduct ongoing impact assessments, and maintain continuous documentation to demonstrate compliance and responsibility.

The uncomfortable truth at the centre of all of this is that hallucinations are not bugs that will be patched away in the next model release. They are properties of the architecture, features of a technology that generates language rather than retrieving facts. Enterprises that treat them as temporary inconveniences are taking on risks they have not priced. Regulators that wait for the science to solve it before writing the rules are ceding ground that is already being contested in real courtrooms.

The era of AI hallucination liability has not fully arrived yet. But the infrastructure for it, the legal concepts, the governance frameworks, the insurance products, the regulatory requirements, is being assembled, piece by piece, right now. The organisations that understand this and prepare accordingly will not just avoid the worst outcomes. They will shape what the accountability structures of AI ultimately look like. That may be the most consequential enterprise technology decision of the next decade.

Related Articles:

NHS Medical Records Are Now an Open Book for a US Spy-Tech Firm, Here's How It Happened

Microsoft's Vulnerability Numbers Went Down, That's Actually the Scary Part

GameStop's $56 Billion Bid for eBay, Explained