How accurate are AI differential diagnosis tools? On hard published cases, often more accurate than doctors. Google’s AMIE included the right diagnosis in its top ten 59.1% of the time against 33.6% for unassisted clinicians. Yet in a 2024 randomised trial, giving physicians GPT-4 did not improve their diagnostic reasoning. Clinicians gain least where the tools could help most.
A family doctor sees her twenty-third patient of the day: a woman in her thirties, back for the third time with abdominal pain that comes and goes, now with tingling in her hands and a vague account of dark urine. Previous visits found nothing. The doctor has eight minutes. Her mind runs through the usual suspects, irritable bowel, anxiety, a urinary infection, and none quite fits. Somewhere on a list she has not looked at since medical school is a rare metabolic disorder that fits rather well.
This is the case AI diagnostic assistants were built for. Not the sore throat or the sprained ankle, where doctors are fast and right, but the atypical presentation that sends patients round the system for months or years. Large language models, trained on vast amounts of medical text, are now remarkably good at producing differential diagnoses for exactly these puzzles.
Here is the problem. The cases where these tools add most are the cases where clinicians trust them least. A suggestion that matches a doctor’s hunch feels useful and adds little. A suggestion that points somewhere unexpected is the valuable one, and it is also the one most likely to be dismissed as a hallucination or a zebra. The design challenge is not accuracy. It is getting the right suggestion believed at the right moment.
How accurate are AI differential diagnosis tools?
On difficult, published cases, the best models now outperform physicians working alone.
In July 2023, a research letter in JAMA by Zahir Kanjee, Byron Crowe and Adam Rodman tested GPT-4 on the New England Journal of Medicine’s clinicopathological conferences, famously hard diagnostic puzzles. The model included the final diagnosis in its differential in 64% of cases and named it as the top diagnosis in 39%.
A larger study from Google, published in Nature in April 2025 by Daniel McDuff and colleagues, evaluated AMIE, a language model optimised for diagnostic reasoning, on 302 challenging real-world case reports. Twenty clinicians gave unassisted differentials first, then worked with either standard search tools or AMIE. AMIE alone achieved top-10 accuracy of 59.1%, compared with 33.6% for unassisted clinicians.
In June 2025, Microsoft researchers posted a preprint describing MAI-DxO, a system that orchestrates language models to request tests and history step by step on 304 NEJM case records. They reported 85.5% accuracy in its most accurate configuration, against a 20% average for the generalist physicians they tested. That figure is striking, but it is a preprint, not yet peer reviewed, and the cases were chosen precisely because they are hard. A 20% physician score is a reminder of what kind of cases these are, not of what primary care looks like.
All of these benchmarks share a limitation. They use written case reports, often with the key clues already selected and neatly described. Real patients give messy, incomplete histories in eight-minute appointments.
Do AI diagnosis tools make doctors better?
This is where the evidence becomes more interesting and less flattering to the human-AI partnership.
In the Google study, clinicians assisted by AMIE reached top-10 accuracy of 51.7%, better than clinicians assisted by search alone at 44.4%. That is a real gain. But it is still below the 59.1% AMIE achieved on its own. The doctors, given a better list, did not fully use it.
A randomised trial published in JAMA Network Open in October 2024 by Ethan Goh and colleagues pushed further. Fifty physicians in family, internal or emergency medicine worked through clinical vignettes with either GPT-4 plus conventional resources or conventional resources alone. The median diagnostic reasoning score was 76% with the language model and 74% without, a difference that was not statistically significant. Time per case did not differ significantly either. Yet in an exploratory analysis, GPT-4 on its own scored 16 percentage points higher than the physicians using conventional resources.
Read those studies together and a pattern emerges. The model knows more than the doctor-with-the-model uses. Something in the handoff is losing value.
Why do clinicians distrust AI on the cases where it helps most?
Three reasons, and none of them is irrational.
Base rates. Primary care runs on common things being common. A clinician who chases every rare possibility will over-test, frighten patients and waste resources. Learning to discount unlikely diagnoses is a core skill, and an AI list with a rare condition near the top triggers exactly that discount.
Unexplained suggestions. When a tool suggests a diagnosis the clinician has not considered, the clinician needs to know why. A bare list gives no way to tell an insight from a confabulation. It is easier to dismiss the unfamiliar item than to investigate it.
Asymmetric accountability. If a doctor follows an AI suggestion and orders unnecessary tests, the doctor owns the cost. If she ignores it and misses a rare disease, she owns that too, but the miss may never be discovered. Ignoring an unexpected suggestion is rarely visible; acting on a wrong one usually is.
The result is a quiet selection effect. Clinicians accept AI suggestions that match their thinking, which adds confidence but not accuracy, and reject those that do not, which is where the accuracy was. We have seen the mirror image of this elsewhere, in the risk of the doctor who simply agrees with AI-drafted messages. In diagnosis, the risk runs the other way: the doctor who agrees only when the AI agrees with her.
A worked example: the third visit
Hypothetical, illustrative case. Return to the woman with intermittent abdominal pain, tingling hands and dark urine. Suppose the practice has an AI diagnostic assistant built into the record.
In version one, the assistant runs automatically on every visit and presents a list of ten possible diagnoses. For this patient, the list includes irritable bowel syndrome, anxiety, urinary infection, gallstones and, in seventh place, acute porphyria. The doctor has seen hundreds of these lists this month, mostly repeating what she already thought. She glances, sees the familiar diagnoses and closes it.
In version two, the assistant stays quiet on routine visits. It activates when a patient returns with an unresolved problem for the third time, or when the clinician asks for it. Instead of a ranked list, it highlights one or two possibilities that do not fit the working diagnosis, explains which features point to them, and suggests an inexpensive test that would help discriminate. For this patient, it might suggest a condition the doctor had not considered and point to a simple urine test.
Same model, same accuracy. Version one gives the right answer and is ignored. Version two gives the right answer when the doctor is stuck, explains itself, and lowers the cost of checking. The difference is design, not intelligence.
Won’t AI differentials just cause over-testing and anxiety?
The strongest counterargument is that making rare diagnoses easier to consider will do more harm than good in primary care. Models can hallucinate. Case reports used in evaluation are biased towards rare and interesting diseases, so a model tuned on them may over-suggest zebras. If AI pushes clinicians to test for rare conditions more often, the result will be false positives, incidental findings, specialist referrals and anxious patients, spread across the many people who have common conditions. Diagnostic caution, on this view, is not a bug to be engineered away.
That argument is correct about the danger and wrong about the conclusion. The answer is to target the tool, not to abandon it. Diagnostic error is a large and measurable harm. A study published in BMJ Quality & Safety in January 2024 by David Newman-Toker and colleagues estimated about 795,000 serious harms a year in the US from diagnostic error, including death and permanent disability. A tool that runs only when a case is unresolved, that favours cheap discriminating tests, and that explains its reasoning addresses the over-testing risk directly. What the evidence does not support is either extreme: ignoring AI because it sometimes suggests zebras, or accepting its lists wholesale.
What this means for primary care, health systems and developers
For primary care clinicians, the most useful habit is to use AI deliberately when stuck: at the second or third visit for an unexplained problem, or when a case does not fit. Ask the tool what else could explain the findings and what would distinguish the options, not simply what the diagnosis is. Treat an unexpected suggestion as a hypothesis worth one inexpensive check, not a verdict.
For health systems, deployment design matters more than model choice. Trigger rules, explanations and links to low-cost next steps will decide whether a tool reduces diagnostic error or becomes background noise. Measure it on outcomes such as time to diagnosis for patients with repeated unresolved visits, not on how often clinicians open it.
For medical educators, the Goh and McDuff findings suggest a new skill: working with a diagnostic partner that is sometimes right when you are wrong. That requires a strong independent foundation first, the argument we made in teach residents to work without the AI first.
For developers, the benchmark race on published puzzles is less important than evidence from real consultations. The questions that matter are when the tool speaks, how it explains itself and whether patients get diagnosed sooner.
For patients, AI does not replace the value of a clear history. If a problem keeps coming back without an explanation, it is reasonable to ask your clinician what else it could be and whether any test would help rule things in or out.
Frequently asked questions
Can AI diagnose diseases better than doctors?
On difficult, written case reports, the best models now outperform physicians working alone. Google’s AMIE reached 59.1% top-10 accuracy against 33.6% for unassisted clinicians. But these are selected puzzles, not real consultations, and the evidence that AI improves diagnosis in everyday practice, or patient outcomes, is still limited.
What is a differential diagnosis?
A differential diagnosis is a list of possible conditions that could explain a patient’s symptoms, usually ranked by likelihood and seriousness. Clinicians use it to decide which questions to ask, which examinations to perform and which tests to order, narrowing the list until one diagnosis is most likely.
Does using ChatGPT help doctors diagnose?
A 2024 randomised trial in JAMA Network Open found that giving physicians access to GPT-4 did not significantly improve their diagnostic reasoning scores, 76% versus 74%, although GPT-4 alone scored higher than physicians using conventional resources. How clinicians use the tool, and when, appears to matter as much as the tool itself.
How common is diagnostic error?
It is common enough to be a major safety problem. A 2024 study in BMJ Quality & Safety estimated about 795,000 Americans a year suffer death or permanent disability from diagnostic error, concentrated in vascular events, infections and cancers. Many errors involve common diseases presenting atypically, not only rare conditions.
Should patients use AI to get a second opinion on their symptoms?
AI chatbots can help patients think of questions to ask, but they cannot examine you, may be wrong and lack your full history. Any concerns they raise should be discussed with a clinician rather than acted on alone. Seek urgent care for severe or rapidly worsening symptoms, whatever a chatbot says.
The best diagnostic AI does not need to be smarter. It needs to speak up when the doctor is stuck, explain why, and make checking cheap. Accuracy on puzzles has been solved faster than trust at the bedside.
Related on the GHC blog: Paying for Outcomes Rents Behaviour. It Doesn’t Buy It. and What Health Ministries Get Wrong About Airline Safety Culture.
When an AI tool suggests a diagnosis you had not considered, what would make you take it seriously?
