Why your medical AI needs to explain itself

A consultant once told me something that reframed how I build models. I had shown him a classifier with strong validation numbers, and he asked one question: "If this disagrees with me, which of us is wrong?" I did not have an answer. Neither did the model.

That question is the whole problem with putting AI into medicine. Accuracy is table stakes. What a clinician actually needs is a reason they can weigh against their own judgement, because they are the one who signs the chart and they are the one accountable when it goes wrong.

Accuracy is not trust

A model that is right 90% of the time sounds excellent until you sit with what that means in practice. One in ten patients gets a wrong answer, and nothing in the output tells the clinician which one. Without a reason attached, every prediction has to be independently verified, at which point the model has saved nobody any time.

A prediction without a reason is not a second opinion. It is noise with a confidence score attached.

This is why I stopped treating explainability as something you bolt on at the end. When the explanation is an afterthought it usually explains the wrong thing: the model's internal representation rather than the clinical reality it is supposed to be reasoning about.

Surrogate models: borrowing accuracy, keeping transparency

The approach I ended up publishing pairs two models instead of compromising on one. A deep neural network does the predicting, because it captures the nonlinear feature interactions simpler models miss. An XGBoost surrogate then learns to mimic the network's behaviour, and because it is tree-based its logic can be read directly as rules.

The number that matters here is not accuracy, it is fidelity: how faithfully the surrogate reproduces the network's decisions. A beautifully interpretable surrogate that disagrees with the model it is explaining is worse than no explanation at all, because it confidently describes reasoning that never happened. In the FIRE framework the two agreed 99.61% of the time, and that is what makes the extracted rules usable as explanations.

Explanations have to speak the reader's language

Here is the detail that took longest to get right, and it has nothing to do with modelling. Models work in normalised feature space. Clinicians do not. A rule that reads if feature_3 > 0.72 is technically an explanation and practically useless.

So the rules get transformed back out of normalised space into the units the reader already thinks in: mg/dL, mmHg, kg/m². The moment we did that, the reaction changed completely. The same underlying logic went from being ignored to being argued with, which is exactly what you want. Being argued with means it was understood.

• Fidelity before elegance. Measure how well the explanation tracks the real model before worrying about how clean it looks.
• Explain in domain units. If the reader has to translate your explanation, it is not an explanation yet.
• Show the visual and the rule. A heatmap answers where; a rule answers why. Experts want both.
• Let people disagree. An explanation that cannot be challenged is marketing, not transparency.

What this looks like in a deployed system

In DermaAI the same principle shows up visually: alongside the classification, a Grad-CAM heatmap highlights the region that actually drove the prediction. When the heatmap sits on the lesion, confidence is earned. When it sits on a ruler or a shadow at the edge of the frame, and sometimes it does, the user has caught a failure that accuracy alone would never have surfaced.

That is the underrated benefit. Explainability is usually sold as a trust feature for users. In practice it has been just as valuable to me as a debugging tool. Half the modelling problems I have found were found by looking at an explanation that made no sense.

Where this goes next

Regulation is moving toward requiring this, but the incentive is simpler than compliance. Systems that can justify themselves get adopted. Systems that cannot get quietly switched off after the pilot ends, however good the validation numbers looked.

If you are working on something similar, or you disagree with any of this, get in touch. The full method is written up in the IEEE paper.