In 2021, researchers at Michigan Medicine tested one of the most widely deployed clinical prediction tools in American hospitals: the Epic Sepsis Model. It was already running in hundreds of hospitals. Their results, published in JAMA Internal Medicine, were sobering. The model’s area under the curve was 0.63, well below what had been reported by the developer. It missed about two-thirds of the patients who developed sepsis, and it fired so often that clinicians would have had to evaluate 109 alerts to find a single patient who needed intervention.
That is not an obscure failure. It is the most important lesson in AI diagnostics, and it is still underlearned.
Here is the pattern that emerges when you line the evidence up: AI performs best where the question is narrow, the input is a standardised image, and the ground truth is clear. It performs worst where the task is diffuse, the input is messy EHR data, and the “truth” is itself a clinical judgement. Hype tends to cluster around the second category. Evidence clusters around the first.
Where the evidence is genuinely strong
Start with the successes, because they are real.
In 2018, the US Food and Drug Administration authorised IDx-DR, the first autonomous AI system allowed to make a diagnostic call without a clinician reading the image. It screens retinal photographs for more-than-mild diabetic retinopathy. In its pivotal trial in primary care clinics, published in npj Digital Medicine, it achieved a sensitivity of 87.2% and a specificity of 90.7% against a demanding reference standard. The task is tightly defined: one disease, one image type, one decision.
Breast screening is the other standout. Sweden’s MASAI trial is the first randomised controlled trial of AI in mammography screening, with more than 100,000 women. Early safety results in The Lancet Oncology in 2023 showed AI-supported screening cut radiologists’ screen-reading workload by 44%. A further analysis in The Lancet Digital Health in early 2025 reported a 29% increase in cancer detection without a rise in false positives.
Notice what these share. A randomised or prospective design. A single, specified task. A clear outcome. And, in the mammography case, a workflow where AI supports an existing double-reading system rather than replacing clinical judgement wholesale.
Where the evidence is thin
The FDA’s public list of AI-enabled medical devices now runs to well over a thousand entries, and the large majority are in radiology. That number is often quoted as a sign of maturity. It is better read as a sign of how the regulatory pathway works. Most of these devices were cleared through the 510(k) route, which requires showing substantial equivalence to an existing device, not proof that patients do better.
Very few AI tools have been tested in randomised trials with patient outcomes as the endpoint. Many are validated retrospectively on data from the same institutions that built them. Performance on a curated test set tells you whether the model learned the pattern. It does not tell you whether a clinician using it, on a Tuesday night with a full waiting room, will make better decisions.
The sepsis story shows what happens in the gap. Predicting deterioration from EHR data is a far harder problem than reading a retinal image. The inputs depend on what clinicians chose to order and when they documented it, which means a model can end up learning clinician suspicion rather than patient physiology. Epic later revised its approach and began recommending that hospitals train the model on local data before going live, which is a quiet admission that the original generalisation claim did not hold.
The incentive nobody wants to name
Why does hype pile up in the weakest area? Because that is where the money is.
A narrow tool like diabetic retinopathy screening has a clear, modest market. A tool that promises to predict sepsis, deterioration, readmission or “risk” across an entire hospital sells into every service line and every enterprise contract. Vendors are rewarded for breadth of claim. Health systems are rewarded for being seen to adopt AI. Neither party has a strong incentive to fund the external validation that might puncture the claim.
Clinicians end up absorbing the cost in alert fatigue. Every false alarm spends a little of the trust that the next, genuinely useful tool will need.
A more useful set of questions
Before adopting an AI diagnostic tool, a clinical leadership team could ask:
- How narrow is the task? One disease and one image type is a very different proposition from “predicts deterioration.”
- Was it validated outside the developer’s own data? Ideally prospectively, ideally in a population like yours.
- What is the positive predictive value at your prevalence? Sensitivity and AUC flatter tools used in low-prevalence settings.
- What happens to workload? A tool that finds three more cases but triples review time may not be a gain.
- Who monitors drift after go-live? Models degrade as practice patterns, coding and populations change.
The mechanism: why narrow tasks travel and broad ones don’t
There is a reason the evidence divides so cleanly, and it is worth spelling out, because it predicts which future tools are likely to disappoint.
A retinal photograph or a mammogram is a physical measurement. The pixels are produced by a device following a protocol. Two hospitals using similar cameras produce broadly similar images, and the disease leaves broadly similar marks in them. When a model learns to recognise those marks, what it has learned is largely about biology. That knowledge travels reasonably well from one hospital to another, although even here, differences in equipment and populations can erode performance.
EHR data is not a measurement in the same sense. It is a record of what clinicians did. Whether a lactate was ordered, when a vital sign was charted, which diagnosis code was entered: all of these reflect local practice, staffing, documentation habits and billing rules as much as the patient’s condition. A model trained on that data learns a mixture of physiology and institutional behaviour. When it moves to a hospital with different habits, the behavioural part of what it learned no longer applies.
The label is often a problem too. “Sepsis” in a retrospective dataset is usually defined by a combination of codes, cultures and antibiotic orders. That definition is itself influenced by clinicians’ suspicion. So a model can end up predicting when clinicians will suspect sepsis, which is not the same as predicting sepsis, and is least useful precisely in the cases clinicians would have missed.
This is not an argument that EHR-based prediction can never work. It is an argument that it needs local validation, local calibration and ongoing monitoring as a matter of course, and that claims of portability should be treated with suspicion.
A worked example: what an AUC hides
Consider a hospital with 1,000 admissions a month, of whom 30 develop the condition a prediction tool is designed to catch. That is a 3% prevalence, which is realistic for many inpatient deterioration problems.
Suppose the tool has a sensitivity of 80% and a specificity of 90%. Those numbers sound good and would support a respectable AUC. In practice, the tool correctly flags 24 of the 30 patients. It also flags 10% of the 970 who never develop the condition, which is 97 false alarms. So the ward receives 121 alerts a month, of which about one in five is real.
Now consider what happens on the ward. If each alert takes a nurse and a doctor ten minutes to assess, that is about 20 hours of clinical time a month, most of it spent confirming that nothing is wrong. After a few weeks of mostly false alarms, staff start to respond more slowly or not at all. The six patients the tool missed are still missed, and some of the 24 it caught may now be caught later than they would have been, because the alert has lost credibility.
None of this shows up in a validation paper that reports AUC. It shows up in the workflow. That is why positive predictive value at local prevalence, and the human cost of false alarms, should be central to any adoption decision.
The strongest counterargument
Enthusiasts have a serious reply. Clinicians are not perfect either. Human performance on many diagnostic tasks is variable, fatigued and unevenly distributed. Randomised trials are slow and expensive, and demanding them for every tool would mean waiting years while patients miss out on real benefits. Many useful clinical tools, from early warning scores to risk calculators, were adopted without outcome trials. Why hold AI to a higher standard?
This argument partly holds. A perfect-evidence standard would freeze innovation, and some tools with only accuracy data are probably helping patients today. It is also true that human baselines are rarely measured as carefully as the algorithms that are meant to replace or assist them.
But AI tools differ from a paper risk score in two ways that justify extra scrutiny. They are often opaque, so clinicians cannot easily see why a prediction was made or spot when it has gone wrong. And they can be deployed at enormous scale very quickly through a single software update, which means an error can spread across hundreds of hospitals before anyone notices. The right standard is proportionate: the broader the claim and the wider the deployment, the stronger the evidence should be.
What this means for different players
Hospital leaders should create a standing process for evaluating and monitoring AI tools, with clinical, informatics and equity input. That process should include a silent-mode trial, where the tool runs in the background and its predictions are compared with outcomes before anyone acts on them. It is one of the cheapest ways to find out whether a tool works locally.
Clinicians should ask for performance data in their own population and should feel entitled to report tools that generate unhelpful alerts. Alert fatigue is a safety issue, not a matter of individual resilience.
Regulators face a structural problem. Pathways designed for physical devices struggle with software that changes frequently and behaves differently in different settings. Requirements for post-market performance monitoring and transparent reporting of real-world results would close part of the gap.
Ministries in lower-income countries are often offered AI tools validated elsewhere. The portability problem is sharper there, because populations, equipment and disease patterns may differ more from the training data. Local validation should be a condition of adoption, and procurement contracts should require vendors to support it.
Funders could do more good by paying for rigorous external evaluations and pragmatic trials than by funding yet another model development project. The field has far more models than evidence about them.
What to watch next
Three signals will show whether the evidence base is catching up. First, the number of AI tools tested in randomised or prospective studies with patient outcomes, not just accuracy, as the endpoint. Second, whether large vendors publish local performance data by site and subgroup. Third, whether health systems begin publicly retiring tools that fail to perform, which would be a sign of a maturing market rather than a failing one.
The takeaway
AI diagnostics is not one field with one verdict. It is a handful of well-evidenced narrow tools surrounded by a much larger ring of plausible, lightly tested ones. The mistake is to let the credibility of the first group lend cover to the second.
The strongest evidence to date comes from tools that do one thing, on standardised inputs, tested prospectively. If a vendor’s pitch is broad, the evidence standard should rise to match it.
Has your organisation retired an AI tool after seeing how it performed locally? We would like to hear what you learned.
