The Trust Paradox: Why Explaining Your AI Might Be the Worst Thing You Can Do

Thursday morning, 9:47am, reviewing a vendor presentation and the slide deck promised "full transparency into our model's decision-making process." The room nodded approvingly. Then they showed the actual explanation. How the sausage was made.

We had asked for transparency and received it. And it had made everything worse.

Explainability can erode trust rather than build it. Understanding this paradox matters because organisations are pouring millions into explainable AI initiatives while regulators mandate ever-greater transparency. The benefits? Informed stakeholders, accountable systems, defensible decisions. The risks of getting it wrong? User abandonment, regulatory theatre, and the slow erosion of confidence in systems that actually work.

Unfortunately, most organisations treat explainability as a simple equation: more transparency equals more trust equals better outcomes. They invest in sophisticated tools and detailed audit trails without asking whether their audiences actually want to see how the sausage gets made.

If you don't grasp this, you'll spend significant budget on transparency initiatives that actively undermine the trust you're trying to build, while regulators tick boxes on systems nobody actually understands. Think European data legislation and cookies.

The unintuitive point at stake here is that humans don't want explanations. They want confidence. When those two things conflict, confidence wins every time. We prefer the doctor who says 'take this medication' to the one who walks us through probabilistic outcomes and confidence intervals. The paradox isn't that we're irrational. It's that we're rational about something other than accuracy.

I'm going to show you how I think about navigating this tension and propose a way to design explanations that build rather than destroy trust.

With that, let's dive in.

The Confidence We Gave Away 

For most of human history, expertise meant opacity. The village elder didn't explain their reasoning. The master craftsman didn't justify their techniques. Authority derived from demonstrated competence, not articulated logic. You trusted the doctor because they'd treated hundreds of patients, not because they could explain the biochemistry of your prescription.

This model worked because expertise genuinely accumulated. The person making decisions had seen more, processed more, developed better intuitions. Their confidence was earned.

Then algorithms arrived, and we demanded they justify themselves in ways we never required of human experts. The seminal research came from Wharton in 2014, when Berkeley Dietvorst and colleagues discovered something troubling: people who watched an algorithm perform became less likely to use it, even when they saw it outperform human forecasters. The mere act of observing the algorithm's reasoning process, including its inevitable errors, destroyed confidence in ways that watching humans err did not.

We hold algorithms to an impossible standard. We forgive human experts their mistakes because we understand human fallibility. But when an algorithm shows us its workings and we spot a flaw, we conclude the entire system is broken. This also explains in part the AI hate that emerges when the model can’t count the ‘Rs’ in Strawberry despite being able to solve complex coding and maths problems.

A 2024 study across 20 countries found something even more counterintuitive: statistical literacy was negatively associated with trust in algorithms for high-stakes decisions. The more people understood about how these systems actually worked, the less they trusted them. Explainability didn't appear to influence trust at all.

Think of it like watching your pilot check the instrumentation before takeoff. A brief, confident scan reassures you. A prolonged, detailed examination of every dial, accompanied by muttered calculations, does not. The explanation reveals the uncertainty that confidence conceals.

But this reveals a problem with where we are today. We've now mandated the prolonged examination.

The Regulatory Trap We've Built 

The EU AI Act, which entered into force in August 2024, requires high-risk AI systems to be transparent enough for users to interpret outputs appropriately. The UK's framework, while less prescriptive, expects firms to explain their AI use to regulators and demonstrate how they've assessed and managed associated risks. The intent is laudable: accountability, contestability, human oversight. As a risk manager, you won’t find me arguing about the intent.

The effect may be something else entirely.

We've created what one legal analysis calls "the explainability illusion": organisations generate explanations that satisfy legal requirements while operators admit they don't actually understand how their systems reach conclusions. Compliance without comprehension. Transparency without illumination. So despite the need for explainability and transparency, we shouldn't conflate this narrow, technical requirement with how we engage with end users. If we do we end up back in cookie approval territory.

People who know me know I like an aviation metaphor (see instrument scanning above) so I think of this rather like the safety demonstration on an aeroplane. Everyone knows how to use a seatbelt. Nobody is actually listening. But we perform the ritual because the ritual satisfies the requirement, even as it accomplishes nothing.

The risk is that we go down the path of mandating the ritual.

Matching Explanation to Audience

The solution isn't abandoning explainability. It's recognising that explanation is a design problem, not a disclosure exercise to end users. Research consistently shows that explanation effectiveness varies dramatically by audience expertise: experts value technical detail and reliability indicators (think model developers and regulators), while novices respond better to narrative and example-based explanations (think end-users).

The Confidence-Calibration Framework:

So here’s how I think we need to look at the problem.

First, identify your audience's baseline trust. Users who already trust the system need less explanation; detailed breakdowns satisfy curiosity but risk revealing uncertainty. Users who distrust the system need confidence signals first, explanations second.

Second, match explanation depth to decision stakes. For routine decisions, explanation overhead destroys efficiency without building trust. For consequential decisions, users want process transparency (how was this system built, tested, validated?) more than decision transparency (why did it make this specific choice?).

Third, communicate uncertainty honestly but strategically. Research on medical decision-making shows that patients prefer clinicians who acknowledge diagnostic uncertainty, but only when that acknowledgment comes with clear action recommendations. "I'm not certain, but here's what we should do" builds trust. "I'm not certain" alone destroys it.

The Bank of England's 2024 survey found that 38% of financial services firms consider appropriate transparency and explainability a "medium" constraint on AI adoption. Not because they can't explain their systems, but because they're uncertain what level of explanation actually serves their stakeholders.

In my experience (albeit a limited sample size at this point) the organisations navigating this best focus on demonstrating governance rather than explaining decisions. They show rigorous testing processes, continuous monitoring, clear escalation paths, human oversight mechanisms. They build confidence in the system rather than transparency into its reasoning.

The metaphor I use: you trust a bridge not because the engineer explained the load calculations, but because you know bridges go through rigorous certification processes and this one is still standing.

The Bottom Line

The old model assumed transparency and trust moved together: show people how decisions get made, and they'll trust those decisions more. This made intuitive sense. It aligned with democratic values. It satisfied our desire to believe that understanding leads to acceptance.

The new reality is messier. Humans are trust-seeking creatures first and truth-seeking creatures second. We want confidence, competence, and clear recommendations. Explanations that reveal uncertainty, complexity, or the mundane simplicity of algorithmic reasoning can actively undermine the trust they're meant to build.

Think Wizard of Oz once the curtain was pulled back.

This doesn't mean abandoning accountability. It means designing explanation systems that match audience sophistication, decision stakes, and baseline trust levels. It means recognising that regulatory compliance and genuine confidence-building may require different approaches. It means accepting that sometimes the most trustworthy thing you can do is project competence rather than expose process.

Until next time, you'll find me reviewing another explainability approach, wondering whether we've built a window or a mirror.