Explainability is Not Auditability.
Industry has sold reasoning traces as compliance solutions. This conflation is an existential risk for regulated enterprises when examinations happen.
Somewhere along the way, two very different questions got treated as one. These are orthogonal questions requiring fundamentally different artifacts, architectures, and investments.
Explainability
"What did it think?"
A narrative generated after the fact. It tells you what the system claims happened—but not what authorized it.
Auditability
"What authorized it?"
Documented evidence of governance. It proves which policy, which version, which criteria governed the decision.
A healthcare AI system approves a prior authorization. When questioned, it produces an impressive explanation:
Sounds compliant. Sounds defensible.
The auditor asks five questions:
Which policy version governed this decision?
Was that version effective on the decision date?
Did you evaluate ALL applicable criteria, or just the ones retrieved?
Who had the authority to approve this?
Can you reproduce this exact decision?
Silence.
The Hidden Gaps
Explainability exists on a spectrum. Each level tells you something—and leaves something unanswered.
| Level | What It Tells You | What It Can't Answer |
|---|---|---|
| Source attribution | This came from Document X | Was X the governing document? The current version? |
| Retrieval trace | I retrieved chunks A, B, C | Did I miss chunk D that contradicts? |
| Reasoning narrative | Here's my step-by-step logic | Is this reproducible? Who authorized this logic? |
| Feature attribution | These inputs mattered most | Did the right policy even get considered? |
Every level assumes retrieval was correct and complete. None of them prove governance.
The Real Picture
This is not a spectrum. It's a 2×2.
Most enterprise AI today sits in the bottom-left quadrant. Vendors positioned it as top-left.
Select a Quadrant
Click on the matrix to reveal specific architectural failure modes and defense strategies.
"Why did the agent approve this medical claim?"
System Warning: Narrative is synthetic / non-authoritative
The Retrieval Blind Spot
Explainability assumes retrieval was correct. But chunking destroys document structure. A "source attribution" only tells you where a piece of text came from—it doesn't tell you if that document currently governs.
Generated Reasoning
LLMs use pattern matching, not policy application. Generated narratives sound authoritative, but inconsistency is the hallmark of stochastic systems: different explanations for the same decision.
Required Artifacts
- Policy Provenance: Version, section, effective date.
- Criteria Mapping: Logic evaluated against discrete rules.
- Reproducibility: Deterministic, not stochastic results.
"When an auditor asks which policy version governed a decision made six months ago, and whether all applicable criteria were evaluated—can your system answer from records, or does it reconstruct?"
Auditability proves you governed.
Regulated industries need both—and they are not the same investment. If you are building on embeddings alone, you are architecting for a future audit failure.