A model that produces a correct answer with no visible reasoning is not usable in a regulated decision, because nobody can tell the difference between a correct answer for the right reasons and a correct answer that happened to land right this time. Explainability is what turns an output into something a compliance officer, auditor, or client can actually stand behind.
What "explainable" means in practice
It is not a single technique. In a multi-step agentic workflow it usually means each step logs what it found, what it decided, and why, so the full chain can be reconstructed after the fact rather than just the final output. In a single model call it can mean citing the source a claim was drawn from, or flagging confidence and uncertainty rather than presenting every answer with the same tone of certainty.
Where regulators draw the line
UK financial services regulators have been clear that AI-assisted decisions in regulated contexts need to be explainable and auditable, not just accurate. That requirement changes the architecture of a system, not just its documentation, which is why explainability has to be designed in from the start rather than added before an audit.