AI pilot purgatory is the state in which a health system runs successful AI pilots that never become standard practice. It happens because pilots are staffed, funded and chosen under unusually good conditions, while nobody owns the budget, integration, monitoring and workflow change required to run the tool everywhere. The fix is designing pilots to test scale, not just accuracy.
Walk into almost any large hospital and ask how many AI pilots it has running. The answer is usually a number that surprises the person saying it. Ask how many AI tools have been switched on across the whole organisation, funded from a permanent budget line, with someone responsible for monitoring them. The number drops sharply. Ask how many pilots were formally stopped, and there is often a pause.
This is pilot purgatory, and it is not unique to healthcare. MIT’s Project NANDA, in its 2025 report The GenAI Divide, drew on 52 executive interviews, a survey of 153 leaders and an analysis of 300 public AI deployments across industries, and concluded that about 95% of generative AI pilots delivered no measurable impact on profit and loss. The report attributed the failures less to model quality than to brittle workflows, tools that did not learn or adapt, and poor fit with daily operations. The exact figure has been debated, but its direction matches what hospital leaders describe.
The thesis here is one we have seen before in a completely different setting. In our analysis of why universal health coverage stalls after the pilot phase, the argument was that pilots are built to answer a question nobody at scale is asking, and the incentives around them reward the answer. Clinical AI is repeating that pattern almost exactly, on a shorter timeline and with a different vocabulary.
What is AI pilot purgatory?
It is the gap between “this worked” and “this is how we work now.” A pilot proves that a tool can perform under a particular set of conditions. Scaling requires the tool to perform under all the conditions the organisation actually has, including the units with high staff turnover, the hospitals on an older version of the EHR, and the clinicians who did not volunteer.
Purgatory is not failure. Failure would be informative. Purgatory is a pilot that neither dies nor grows. It keeps running in one department, on a temporary budget, supported by the enthusiasts who started it, until the grant ends, the champion leaves or the contract lapses.
Why do clinical AI pilots fail to scale?
The reasons map almost one for one onto why health financing pilots stall.
The site is chosen, not inherited. AI pilots go where there is an enthusiastic physician champion, a strong informatics team and a department head who wants to be first. Results from such a site tell you what the tool does under excellent management. They do not tell you what it does on a night shift in a unit that did not ask for it.
The pilot runs on parallel resources. Vendors often provide extra implementation support, data scientists and clinical liaisons free during a pilot. The hospital’s informatics team gives it priority. None of that is available at scale, so the pilot quietly measures the effect of the tool plus a support team that will not exist later.
The money changes shape. Pilots are funded from innovation budgets, grants or vendor concessions. Scaling requires a permanent operating line, competing with nurses, drugs and equipment. Nobody drew up that line during the pilot.
Integration is postponed. It is common to pilot a tool in a lightweight way, with a separate screen, a manual data feed or a limited interface, precisely to move fast. Scaling requires deep integration into the EHR across every site, security review and support. The part skipped to make the pilot quick is the part that makes scale slow.
Ownership is unclear. The pilot belonged to an innovation team. Scale belongs to operations. Operations did not choose the tool, was not involved in the design and has other priorities.
Success is defined by the wrong metric. Pilots often report accuracy, user satisfaction or anecdotes. A chief financial officer or chief operating officer deciding on system-wide rollout wants to know what changes in cost, capacity or outcomes, and the pilot was not designed to answer that.
What does real-world evidence say about AI at scale?
One of the most instructive cases is from Google’s diabetic retinopathy work in Thailand. The deep learning system had performed well in retrospective studies. When researchers deployed it in eleven primary care clinics and studied what happened, they published their findings at the CHI conference on human-computer interaction in 2020. The model’s strict image-quality thresholds clashed with the real conditions in clinics: variable lighting, rushed screening sessions, and connectivity problems. Nurses found a meaningful share of images rejected as ungradable, which meant patients were referred on or asked to return, adding burden. The accuracy the model showed in the lab was not the experience patients and nurses had in the clinic.
The important part is what came next. The work did not stop at the field study. A later prospective study of real-time AI screening within Thailand’s national diabetic retinopathy programme was published in The Lancet Digital Health in 2022. That sequence is the opposite of purgatory: a field study that surfaced what breaks in real clinics, followed by a larger prospective evaluation inside a national programme.
Ambient scribes show a softer version of the same issue. The Peterson Health Technology Institute found in 2025 that scribes reduce burnout, but that financial returns were not yet clear. That matters for scaling. A tool that clinicians love but finance cannot justify is likely to remain limited to the specialties or individuals who push hardest for it.
Federal data adds a structural layer. In 2024, according to the US health IT office, 86% of system-affiliated hospitals used predictive AI, but only 37% of independent hospitals did. Pilots that succeed in large systems do not automatically travel to smaller ones, because the smaller ones lack the informatics staff to integrate, validate and monitor.
A worked example: the sepsis pilot that never left the ICU
Imagine a hypothetical five-hospital system that pilots an AI sepsis alert in the medical ICU of its flagship hospital. The unit is well staffed, the lead intensivist champions the project, and the vendor sends a clinical specialist twice a week. Over six months, time to antibiotics for flagged patients falls, and the team presents the results at a conference.
The system then asks to roll it out to all general wards across five hospitals. Problems appear immediately. The wards have more patients per nurse and a different alert-response process. The rapid response team, which would receive the alerts, is already stretched. Two of the hospitals run a different EHR configuration, requiring new integration work. The vendor’s clinical specialist is not included in the enterprise price. Nobody has budgeted for the extra nursing time to act on alerts, or for a data analyst to monitor performance across sites.
The rollout stalls. The pilot keeps running in the ICU. Two years later the champion leaves, and the tool is switched off during an EHR upgrade without anyone noticing a difference.
None of the steps were irrational. The pilot answered “can this work in our best unit?” The organisation needed the answer to “what will it take to work in our worst one?”
The strongest case for small AI pilots
It is worth defending pilots properly. Healthcare is right to be cautious about deploying AI at scale without testing it first. A pilot in a controlled setting can catch safety problems, workflow clashes and unexpected failure modes before they harm many patients. Starting where champions exist is sensible: early adopters tolerate rough edges and give useful feedback. And given how fast AI tools are changing, a quick, cheap pilot can prevent a big investment in something that will be obsolete in a year.
All true. The problem is not the existence of pilots. It is that pilots are usually informative when they fail and much less informative when they succeed. A tool that fails even in the best unit should be stopped. A tool that succeeds in the best unit has shown its ceiling, not its floor. Treating that ceiling as a forecast is how purgatory begins.
How can health systems escape AI pilot purgatory?
By designing pilots as rehearsals for scale, not as demonstrations.
Choose a typical site, or at least include one. If the tool only works with a champion, you need to know that.
Run the pilot through the real system. Use standard integration, standard support and standard staffing. If the vendor provides extra help, record its cost and include it in the scale-up case.
Define the scale decision at the start. Agree which outcome, cost or capacity metric would justify system-wide rollout, what threshold counts, and who will decide. Put a date on it.
Name the operational owner from day one. The person who will run the tool at scale, and whose budget will pay for it, should be involved from the design phase.
Price the full rollout up front: licences across all sites, integration, monitoring, training and the clinical staff time needed to respond to the tool’s outputs.
Stop pilots on purpose. A formal stop decision, with the reasons written down, is a success of governance. Our playbook for scaling a health pilot sets out how to put a name against each of these steps, and the argument that startups die in procurement rather than in the pilot explains why vendors should welcome that clarity.
What this means for health leaders and vendors
For executives, count your pilots and your scaled tools, and look at the ratio. If it is high, the problem is not the technology. It is the absence of a pathway from innovation budgets to operating budgets.
For innovation teams, measure yourselves on tools scaled or stopped, not pilots launched.
For clinical champions, your enthusiasm is necessary and not sufficient. Enlist the operations and finance leaders early, or your project will stay yours.
For vendors, pilot pricing that hides the true cost of support creates your own purgatory. Better to show the enterprise cost and the enterprise evidence plan from the first meeting.
Frequently asked questions
What percentage of AI pilots in healthcare reach production?
There is no reliable healthcare-specific figure. Across industries, MIT’s Project NANDA reported in 2025 that about 95% of generative AI pilots showed no measurable profit-and-loss impact. Health system leaders commonly report many more pilots than scaled tools, and federal data shows large gaps in AI adoption between well-resourced systems and independent hospitals.
What is the difference between an AI pilot and a proof of concept?
A proof of concept tests whether a technology can do a task at all, often with retrospective data. A pilot puts the tool into real clinical use for a limited group to see how it performs in practice. Neither guarantees success at scale unless it was designed to test scale-up conditions.
Who should own AI implementation in a hospital?
Clinical and operational leaders who will run the tool at scale should own it, supported by informatics, IT and governance teams. Innovation teams can lead early testing, but the operational owner and budget holder should be named before the pilot starts, or the tool is likely to stall.
How long should a clinical AI pilot run?
Long enough to measure the outcome or operational metric that will justify scale, usually several months, and with a fixed decision date. Open-ended pilots are a hallmark of purgatory. A pilot without a pre-agreed decision point tends to continue indefinitely without either scaling or stopping.
Is stopping an AI pilot a failure?
No. Stopping a pilot that does not meet pre-agreed criteria is good governance and frees resources for better tools. The costlier failure is a pilot that neither scales nor stops, consuming staff time and attention while delivering little value.
A successful pilot is a question, not an answer. Health systems that keep treating it as an answer will keep filling their drawers with success stories, in AI just as they did in health financing.
How many AI pilots is your organisation running right now, and how many have become standard practice? We would like to hear the ratio.
