AI sepsis prediction accuracy in real hospitals is usually modest. The most widely deployed model, Epic’s, scored an AUROC of 0.63 in an independent Michigan study and missed two in three sepsis cases while alerting on 18% of patients. Tools that have cut deaths, such as TREWS and COMPOSER, did so because clinicians acted on alerts fast.
A night nurse on a medical ward gets the same banner for the fourth time in one shift: possible sepsis, please assess. The first three were a post-operative patient with a fever everyone already knew about, a man whose heart rate was up because he had pulled out his cannula, and a woman already on antibiotics for pneumonia. The fourth might be real. By now she reads it the way most of us read a cookie banner.
That scene is the whole problem with AI sepsis alerts, and it is not the problem most vendors talk about. The industry argues about accuracy, usually summarised as a single number called the AUROC. The question that decides whether a sepsis model saves lives is different and harder: does the alert tell the team something they did not already know, early enough to change what they do, without firing so often that nobody listens?
Most sepsis models fail that second test. A few pass it, and the difference between them is less about the algorithm than about what happens in the three hours after the alert fires.
How accurate are AI sepsis prediction models in real hospitals?
The most cited answer comes from Michigan Medicine. In 2021, Andrew Wong and colleagues published an external validation of the Epic Sepsis Model in JAMA Internal Medicine. The model was proprietary, embedded in the electronic record used by hundreds of US hospitals, and had been switched on in many of them without independent testing.
The Michigan team looked at 38,455 hospitalisations among 27,697 patients. Sepsis occurred in 2,552 of them. The model’s AUROC was 0.63, well below the performance the developer had reported. At the alert threshold the hospital would actually use, it identified only a third of sepsis cases and missed 1,709 patients, or 67%. Meanwhile it generated alerts on 18% of all hospitalised patients. Clinicians would have had to evaluate eight patients to find one eventual sepsis case.
The finding that should have ended more conversations than it did was the last one. Of those 2,552 sepsis patients, the model flagged only 183, about 7%, who had not received timely antibiotics. In other words, most of the patients it caught were patients the clinical team had already caught.
A 2024 follow-up by the same Michigan group, published in NEJM AI, sharpened the point. Looking at 77,582 patients, it asked how the Epic model performed using only the information available before clinicians had started acting on a suspicion of sepsis. Before sepsis criteria were met, the AUROC was 0.62. Restricted to predictions made before the first signs of treatment, such as a blood culture order or antibiotics, it fell to 0.47. That is worse than a coin toss. Part of what the model had “learned” was to notice that clinicians were already worried.
Why does AUROC overstate a sepsis model’s value?
AUROC measures how well a model ranks patients: if you pick one patient who develops sepsis and one who does not, how often does the model give the septic patient a higher score? It is a reasonable way to compare models on a dataset. It is a poor way to predict what will happen on a ward, for three reasons.
First, it ignores lead time. A model that flags sepsis two hours after the doctor ordered blood cultures can score well on AUROC and add nothing at the bedside. What matters is the gap between the alert and the moment a competent clinician would have recognised the problem anyway. The Michigan NEJM AI paper is essentially a measurement of that gap, and it came out close to zero.
Second, it ignores alert burden. AUROC is threshold-free, but hospitals are not. Someone has to pick a score at which the banner appears, and that choice trades missed cases against false alarms. A model with a respectable AUROC can still produce eight or ten alerts for every true case at a threshold sensitive enough to be useful. Every one of those alerts costs a nurse’s attention.
Third, it ignores who the alert reaches. Sepsis is a condition where the right action, fluids, cultures, antibiotics, is well known. The bottleneck is recognition and response. An alert routed to a busy ward inbox is a different intervention from an alert routed to a rapid-response nurse who must review it within the hour.
This is not a quirk of sepsis. It is the general pattern in clinical prediction, the same one we described in why predicting hospital capacity is the easy part. A forecast has value only through the action it triggers.
Do AI sepsis alerts actually reduce deaths?
Some do, and the ones that do share a design choice: they treat the alert as the start of a workflow, not the end of a calculation.
The best-known example is TREWS, the Targeted Real-time Early Warning System developed at Johns Hopkins. In a prospective study across five hospitals published in Nature Medicine in 2022, Roy Adams, Suchi Saria and colleagues monitored 590,736 patients. The analysis focused on 6,877 patients with sepsis whom the alert identified before antibiotics had been started. When a provider evaluated and confirmed the alert within three hours, in-hospital mortality was 18.7% lower in relative terms after adjustment, with improvements in organ failure and length of stay as well. About 61% of alerts were confirmed within that window.
The second is COMPOSER, a deep-learning model deployed in two emergency departments at UC San Diego Health. In a before-and-after study of 6,217 septic patients published in npj Digital Medicine in January 2024, the team reported a 17% relative reduction in in-hospital sepsis mortality after the model went live. COMPOSER was built to say “I don’t know” when inputs looked unfamiliar, a small feature that reduces the number of confident false alarms.
The regulatory picture has also moved. In April 2024 the FDA granted de novo authorisation to Prenosis’s Sepsis ImmunoScore, the first AI tool authorised to help diagnose sepsis. It combines 22 parameters and is ordered like a lab test, which is itself a statement about workflow: a clinician asks for it, rather than being interrupted by it.
Notice what the TREWS result actually says. The benefit belongs to patients whose alerts were confirmed quickly. It is a result about a model plus a response, not a model alone.
What should hospitals measure before switching on a sepsis alert?
Three numbers matter more than the AUROC on a vendor slide: the proportion of true sepsis cases the model flags before clinicians have acted, the median lead time over clinician recognition, and the number of alerts per genuinely new catch.
A worked example (hypothetical, using the Michigan rates). Take a hospital with 1,500 adult admissions a month. Applying the rates from the Wong study, about 18% of admissions trigger an alert: roughly 270 alerts a month, or nine a day. Sepsis affects around 6.6% of admissions, so about 99 patients a month. The model flags a third of them, about 33, most of whom the team has already recognised. The group where the alert could plausibly change care, septic patients flagged who had not received timely antibiotics, was 7% of sepsis cases in Michigan. Here that is about seven patients a month.
So the real trade is 270 interruptions for seven chances to help. That may still be worth it. Sepsis kills, and seven patients a month is not nothing. But it is a very different business case from “the model is 80% accurate,” and it tells a hospital exactly where to work: raise the threshold, suppress alerts on patients already on antibiotics, route alerts to a person with protected time to respond, and measure time from alert to evaluation.
These are local numbers. A model that performs one way in Michigan can perform differently in a community hospital with a different case mix and different documentation habits, which is why local validation is not optional. The broader argument for that discipline is in our piece on bias in clinical algorithms.
Isn’t an over-sensitive sepsis alert better than a missed case?
This is the strongest counterargument, and it deserves a serious answer. Sepsis mortality climbs with every hour of delayed treatment. A missed case can mean a death; a false alarm usually means a few minutes of a nurse’s time. On that arithmetic, you should tolerate many false alarms to catch one more real case.
The argument holds for any single alert. It fails at the level of a ward over months. Attention is finite and it adapts. When staff learn that most alerts are noise, they slow their response to all of them, including the true ones. The Joint Commission issued a sentinel event alert on alarm fatigue in 2013 for exactly this reason. And the TREWS data point the same way: the mortality benefit showed up where alerts were confirmed within three hours. An alert that gets dismissed at hour five does not carry that benefit, however sensitive the model.
There is a fair methodological objection to the positive studies too. TREWS compared patients whose alerts were confirmed quickly with those whose alerts were not, and COMPOSER used a before-and-after design. Neither is a randomised trial, and teams that respond fast may differ in other ways. The honest reading is that well-designed sepsis alerts are promising, not proven, and that poorly designed ones are close to proven failures.
What this means for hospitals, clinicians and vendors
For hospital leaders, the purchase decision should rest on local silent-mode testing: run the model in the background for a few months, then count lead time, alerts per new catch and the share of alerts arriving after treatment has started. If the vendor cannot support that, treat it as the answer. The alert is also a staffing decision. Budget for who responds, because the benefit in the best evidence depended on response, and alerts landing on an already overloaded team add to the burden we described in the EHR as the place where everyone else’s demands land.
For clinicians, the useful habit is to treat an alert as a prompt for a structured bedside look, not a verdict. It helps to know what the model weighs heavily. If it leans on lactate orders or vital-sign trends, it will echo your own suspicion more often than it pre-empts it.
For vendors, the bar has moved. Publishing AUROC on retrospective data is no longer enough after Michigan. Buyers should ask for performance restricted to the period before clinical recognition, for alert rates at the recommended threshold, and for prospective outcome data. Vendors that report those numbers openly will earn trust faster than those that do not.
For regulators and payers, sepsis models sit awkwardly. Many are embedded in record systems as decision support and never passed through device review, while a tool like Sepsis ImmunoScore went through de novo authorisation. Consistent expectations on lead-time reporting would help buyers compare like with like.
Frequently asked questions
What is the Epic Sepsis Model?
It is a proprietary prediction model built into Epic’s electronic health record that scores hospitalised patients for sepsis risk and can trigger alerts. An independent 2021 JAMA Internal Medicine study at Michigan Medicine found an AUROC of 0.63, missing 67% of sepsis cases at a typical threshold. Epic has since released an updated version designed to be trained on local data.
What is a good AUROC for a sepsis prediction model?
There is no single cutoff that makes a model useful. Many published sepsis models report AUROCs between 0.7 and 0.9 on retrospective data, but those numbers often include predictions made after clinicians already suspected sepsis. A better test is performance using only data from before clinical recognition, plus alert rates and lead time at the threshold you will actually use.
Has any AI sepsis tool been approved by the FDA?
Yes. In April 2024 the FDA granted de novo marketing authorisation to Prenosis’s Sepsis ImmunoScore, the first AI tool authorised to help diagnose sepsis. It uses 22 clinical and laboratory parameters to produce a risk score and category, and clinicians order it within the health record much like a laboratory test.
Do sepsis alerts cause alert fatigue?
They can. At a common threshold, the Epic model in the Michigan study alerted on 18% of hospitalised patients, with eight patients evaluated for each eventual sepsis case. High false-alarm rates teach staff to discount alerts. Hospitals reduce this by raising thresholds, suppressing alerts for patients already on treatment and routing alerts to a dedicated responder.
Should patients ask whether their hospital uses a sepsis alert?
Patients and families can reasonably ask how a hospital screens for sepsis, but the more useful step is to speak up quickly about warning signs such as confusion, fast breathing, fever or feeling very unwell after an infection. Any concern about sepsis should go straight to the clinical team, whatever software is running in the background.
The takeaway is uncomfortable for anyone who has bought a sepsis model on the strength of its accuracy: the number that predicts benefit is not on the vendor slide. It is the number of new, early catches per hundred alerts, and the minutes it takes someone to answer them.
Related on the GHC blog: Why Universal Health Coverage Stalls After the Pilot Phase.
If your hospital runs a sepsis alert, do you know how many of its alerts arrive before the team has already acted?
