What a machine must never decide for a patient

The most common question I get about AI in medicine is a version of “how good is it”. It is the wrong first question. A model can be very good and still be the wrong thing to put in charge.

Here is where I draw the line. A machine may search, recall, summarise, draft, compare, flag, and remind. It may read everything that was ever written about a condition and hand the physician the three things that matter today. It may notice that a lab value drifted while everyone was looking elsewhere. These are enormous gifts and I want all of them in the room.

A machine must not decide what matters to this patient. What a good outcome is for a seventy year old who has already buried a spouse is not a fact in the record. Whether a risk is worth taking is not a property of the risk. Those are decisions that belong to a person who can be held to them, and who can be looked in the eye afterwards.

The reason is not that models are inaccurate. It is that accountability does not transmit over a wire. When a physician decides, there is a name attached, a licence, a conscience, and a conversation that can be reopened. When a model decides, there is a log. A log is not an answer to a grieving family.

Let the information flow in as wide and as fast as we can make it. But somewhere in the system something still has to open and close on its own judgement, and that something should be a person.

This is also why I am not afraid of the technology. A tool that knows its place makes the person holding it better at theirs. The whole history of medicine is the history of such tools.