Clinical AI validation before deployment means testing a model on your own patients, data and workflows before clinicians rely on it. Vendor studies and regulatory clearance show a tool can work somewhere. Local validation shows whether it works in your hospital, for your population, at a threshold your staff can act on.
In 2018, a team led by John Zech published a study in PLOS Medicine on a deep learning model that detected pneumonia on chest X-rays. Trained and tested within the same hospital systems, it looked impressive. Tested on images from a different hospital, it did worse. Part of the reason was unsettling. The model could recognize which hospital system an image came from, and that signal correlated with pneumonia rates. It was partly predicting the hospital, not the disease.
That study is now eight years old, and the lesson has not changed: a model’s performance is a property of the model and the setting together. Move either one and you have a new result.
My argument is blunt. A vendor’s validation, however rigorous, answers a question you did not ask. It tells you how the tool performed on someone else’s patients, scanners, documentation habits and workflows. Before a hospital lets a model influence care, it needs to answer its own question, on its own data, with its own staff. That is not duplication. It is the only validation that applies to you.
Why is vendor validation not enough?
Vendor studies are usually honest. They are also structurally limited in four ways.
Different patients. Training and test data come from specific sites. If those sites are academic centres in one region, performance in a rural community hospital, a paediatric unit or a population with different disease prevalence is an extrapolation.
Different data plumbing. Models depend on inputs that are recorded differently in every hospital: which lab codes are used, how vital signs are charted, when notes are signed, which scanner protocol is standard. The same model can receive subtly different data at each site.
Different prevalence. Positive predictive value depends on how common the condition is. A model with the same sensitivity and specificity will produce far more false alarms per true case where the condition is rare.
Different workflow. A model validated retrospectively has never met a tired nurse at 3 a.m. Whether an alert arrives at a moment when someone can act is not something any vendor study can measure for you.
The best-known cautionary case remains the external validation of a widely used proprietary sepsis model, published in JAMA Internal Medicine in 2021 by Andrew Wong and colleagues at Michigan Medicine. Across more than 38,000 hospitalizations, its area under the curve was 0.63, well below the developer’s reported figures. It missed about two thirds of sepsis cases while alerting on nearly one in five inpatients. Hospitals that switched it on with default settings were relying on validation done somewhere else.
Regulatory clearance does not close this gap. Most AI devices in the US are cleared through the 510(k) pathway. A study in JAMA Health Forum in August 2025 found that AI devices with no reported clinical validation were more likely to be recalled. And many predictive models built into EHRs are not regulated devices at all.
What should a hospital test before deploying clinical AI?
A local validation does not need to be a research project. It needs to answer five practical questions.
- Does it discriminate here? On a representative local sample, how well does it separate patients with and without the condition? Report sensitivity and specificity at the threshold you plan to use, not just the area under the curve.
- Is it calibrated here? If the model says 20 percent risk, do about 20 percent of such patients have the outcome locally? Poor calibration can make a well-ranking model mislead clinicians about urgency.
- What will it do to workload? How many alerts per day, per unit, per clinician? What share will be true? A tool with good accuracy and an unmanageable alert volume will be ignored.
- Does it work for everyone? Break results down by age, sex, race and ethnicity, language, insurance and site. Look for groups where performance drops.
- Does it fit the workflow? Who sees the output, when, and what can they do about it? A radiology triage flag is only useful if the worklist actually reorders.
The Joint Commission and the Coalition for Health AI, in voluntary guidance released in September 2025, recommended that organizations ask vendors for testing, validation and bias information during procurement and then validate and monitor tools locally. That is the right sequence. Vendor evidence tells you whether a tool is worth testing. Local evidence tells you whether to use it.
How do you validate an AI model locally?
There are three common designs, in increasing order of realism and effort.
Retrospective validation. Run the model on historical local data and compare its outputs with known outcomes. This is fast and cheap. Its weakness is that historical data may differ from live data, because some inputs are recorded or corrected after the fact.
Silent or shadow mode. Run the model live on current patients, but hide its outputs from clinicians. Record what it would have said and compare with what happened. This tests the real data pipeline in real time without affecting care. It is the single most useful step many hospitals skip.
Limited live pilot. Show the output to a small group of clinicians on one unit, with close monitoring of alerts, actions, overrides and outcomes. This is the first test of whether the tool changes behaviour and whether that change helps.
Each design answers different questions. Retrospective validation checks accuracy. Silent mode checks the plumbing and the alert burden. A pilot checks the human side.
A worked example: an eight-week silent trial
Here is a hypothetical community hospital evaluating an AI tool that flags suspected pneumothorax on chest X-rays and moves those images up the radiologist’s worklist.
The vendor’s evidence shows high sensitivity and specificity in a multi-site study and an FDA clearance. The hospital’s radiology chair is keen.
Step one, retrospective. The informatics team runs the tool on a year of archived X-rays, enriched with confirmed pneumothorax cases from radiology reports. Sensitivity looks close to the vendor’s figure. Specificity is lower. On review, many false positives come from images with chest drains and skin folds, which are common in the hospital’s older inpatient population.
Step two, silent mode for eight weeks. The tool runs live, invisibly. The team discovers that portable X-rays from the ICU reach the tool several minutes late because of a routing rule in the image archive. For the sickest patients, the triage benefit would have been close to zero.
Step three, fix and decide. The routing rule is fixed. The radiology group agrees to a threshold that keeps false flags at a level they can tolerate, accepting slightly lower sensitivity. They document the tool’s weakness with drains and skin folds and brief radiologists on it. Then they start a limited live pilot on day shifts with monthly review.
None of this contradicted the vendor. It revealed what the vendor could not know: this hospital’s patients, this hospital’s routing rules, this hospital’s radiologists.
What is the strongest argument against local validation?
The strongest argument is feasibility. Local validation needs data engineers, clinical informaticists and time. Most small and rural hospitals have none to spare. If every hospital must validate every tool, only large academic centres will adopt AI, and the gap between well-resourced and under-resourced hospitals will widen. Critics also point out that small hospitals may not have enough cases to validate anything with statistical confidence. A hospital with forty pneumothoraxes a year cannot produce a tight confidence interval.
This is a real problem, and the answer cannot be “validate everything yourself.” It has to be shared infrastructure. Health system networks can pool validation across member hospitals. Vendors can supply local validation toolkits and silent-mode capability as standard. Independent assurance labs, an idea promoted by the Coalition for Health AI, may help if they test on data that resembles the hospitals using the tool. Small hospitals can still do the cheap, high-yield steps: check the data pipeline in silent mode, count alert volume and review a sample of false positives.
What they should not do is skip validation because the full version is too hard. The sepsis model case shows what defaults cost.
What this means for hospitals, vendors and funders
For hospitals, make local validation a condition of go-live, scaled to risk. Put it in the contract: the right to run the model in silent mode, access to model outputs and inputs, and vendor support to evaluate performance. Our checklist on buying digital health covers where these clauses belong. Record the results and the threshold you chose, and why.
For vendors, treat local validation as part of the product, not a sales obstacle. A vendor that offers a silent-mode period and a local performance report will win trust faster than one that offers a glossy multi-site study. As we argued in health startups don’t die in the pilot, they die in procurement, the buyer’s due diligence is where deals stall, so make it easy.
For funders and policymakers, the equity problem needs money, not exhortation. Funding shared validation capacity for community and rural hospitals would do more to spread safe AI than any new rule requiring it.
Frequently asked questions
What is local validation of a clinical AI model?
It is testing a model on a hospital’s own patients, data and workflows before relying on it, and usually again after go-live. It checks whether the model’s accuracy, calibration, alert volume and performance across patient groups hold up locally. It complements, rather than replaces, the developer’s validation studies and any regulatory review the tool has received.
What is silent mode testing for AI?
Silent mode, also called shadow mode, means running an AI model on live patient data without showing its outputs to clinicians. The team records what the model would have flagged and compares that with actual outcomes. It tests the real data feed, timing and alert burden without any effect on care, which makes it one of the most useful and least risky validation steps.
Does FDA clearance mean a hospital does not need to validate an AI tool?
No. FDA clearance means the agency judged the device reasonably safe and effective for its labelled use based on the evidence submitted, often from other sites. It does not show performance on your equipment, population or workflow. Many predictive models embedded in EHRs are not FDA-regulated at all. Local validation is still needed.
How long should local validation of a clinical AI tool take?
It depends on the tool’s risk and how common the condition is. A retrospective check can take weeks. A silent-mode period of one to three months is common, long enough to see enough cases and catch data pipeline problems. Rare conditions need longer periods or pooled data. The schedule should be set before testing starts, not after.
The takeaway
Every clinical AI model arrives with evidence from somewhere else. That evidence tells you whether the tool deserves a test, not whether it deserves your patients. The hospitals that get value from AI are the ones that insist on answering the local question first: in silent mode, on their own data, with a threshold their staff can live with. The ones that trust the brochure find out the answer from their incident reports.
What did your last local AI validation reveal that the vendor’s study did not? We’d like to hear specifics.
