Brandon Nuttall, chief digital & AI officer at Xceedance explains insurers need to recognise the risks that comes with AI implementation.

Artificial intelligence deployments promise uplifts in speed and consistency of decision-making. However, insurers who transform their operating model with AI are facing heightened scrutiny, particularly where AI makes or supports decisions that affect coverage, pricing, or claims outcomes. However, they also introduce a new category of risk: algorithmic clarity.

When technology makes or supports decisions that affect coverage, pricing, or claims outcomes, insurers face heightened scrutiny from regulators, investors, and customers. The question is no longer whether AI can enhance operations; it is how to ensure it can do so responsibly, transparently, and ethically.

The transparency imperative

As AI becomes more widely adopted across the value chain, transparency is shifting from an ethical aspiration to a regulatory expectation. For example, the EU AI Act can place heavy penalties on organizations who engage in “high-risk” activities with AI. Other regulators will likely follow suit, increasing the importance of explainable, auditable AI.

There are some known challenges. For example, AI systems can:

• Replicate historical biases embedded in training data
• Overfit to patterns that are statistically valid but ethically inappropriate
• Produce nondeterministic outcomes
• Operate as black boxes where logic cannot be easily traced

As an industry, it is imperative that we ensure our models and platforms are developed and rigorously tested to mitigate these risks.

The Emerging Horizon of AI Governance

The risk profile of every automated or semi-automated decision is dependent on data quality, model design, and embedded assumptions. Without proper rigour and oversight, these foundations create blind spots that are hard to detect and even harder to correct. When an algorithm denies a claim, adjusts a premium, or flags a customer as high risk, insurers must be able to answer a single question: Why?

What regulators are signalling

Global momentum is converging around responsible, traceable AI. Common themes across emerging frameworks include:

1. Auditability and Decision Traceability Clear documentation of datasets, model architectures, version histories, and decision logic.
2. Bias and Fairness Testing Periodic assessments to identify disparate impacts on protected or vulnerable groups.
3. Data Accountability Organisations must demonstrate that model inputs are accurate, representative, and ethically collected.
4. Human in the Loop Requirements Fully automated decisions, especially in underwriting or claims denial, are discouraged without human oversight or appeal pathways.
5. Model Risk Management (MRM) Robust validation, independent reviews, and clear governance lines, similar to the banking industry.

Building governance into the algorithm

To access the full benefits of AI and mitigate unintended consequences, governance must be embedded throughout the lifecycle, from data ingestion to decision execution. Effective structures could include:

1. Clear Audit Trails Every output should be traceable to its inputs, model version, and decision context, supporting regulatory defence and internal accountability.
2. Bias Mitigation Protocols Regular testing across demographic segments and the use of fairness constraints to reduce unintended disparities.
3. Data Lineage Tracking Demonstrate where data originated, how it was processed, and how it influenced any given outcome.
4. Cross-Functional Oversight Committees Combine underwriting, data science, compliance, and risk management to avoid blind spots.

Long story short: explainability is a must

As the use of AI accelerates across the board, explainability becomes one of the critical factors for its success. Explainable AI is emerging as a compliance necessity. Regulators want defensible models, customers want fairness, and investors want assurance that automation won’t create hidden liabilities.

Explainability bridges these demands by allowing insurers to articulate:

• How decisions were made
• What influenced them
• Whether they were fair
• Whether they can be trusted.

The path forward: from efficiency to accountability

This issue is not unique to insurance, but tackling it will become vital for industry growth. It will also enable insurers to build products that support the ethical use of AI across industries.

AI is poised to generate substantial and enduring value across the global economy, but it must be built on the twin pillars of ethics and accountability. Organisations that proactively

embed these principles into their AI strategies will not only mitigate risk, but will also set themselves up for success well into the future.