Explainable machine learning: what it is and what it is not

Two questions, two kinds of answer

People ask two different things of a model. Globally: what does this model rely on overall? Locally: why did it produce this result for this case? Different methods answer each, and a good workflow provides both.

Feature attribution methods such as SHAP estimate how much each input contributed to a prediction. Simpler models — a short decision tree, a linear model — can be read directly.

What explainability gives you

• A way to check that a model relies on sensible signals rather than an accident of the data.
• An answer for the person affected by a decision, and for the team accountable for it.
• Evidence for reviewers that goes beyond an accuracy score.
• An early warning when a model starts leaning on something it should not.

What it does not give you

An explanation describes the model, not the world. A model can be perfectly explained and still be wrong, because it learned a pattern that does not hold.

Explanations are also approximations with their own assumptions. Some methods do not apply to some models, and a chart that is produced anyway is worse than no chart. A trustworthy tool says when a method is unsupported.

Keep the evidence with the model

The practical failure is not a lack of explanation methods. It is that the explanation lives in a notebook while a slightly different model runs in production.

Treat the explanation as part of the model's record: the data version it was trained on, its performance, the features that drive it, its known limitations, and who approved it. That record is what makes a model usable in an organization that has to answer for its decisions.