AI biomarker discovery from real-world data means using machine learning on health records, claims, imaging and lab data to find measurable signals that predict disease or treatment response. The data are plentiful, and models find patterns easily. But most such patterns reflect how care is delivered and recorded rather than biology, so few survive independent validation.
In 2018, a Harvard team including Isaac Kohane and Griffin Weber published a study in The BMJ that should be required reading for anyone mining health records for biomarkers. They looked at 272 types of laboratory tests across more than 669,000 patients at two Boston hospitals. For 233 of those tests, whether the test had been ordered at all was significantly associated with patients’ three-year survival, regardless of the result. For 118 of 174 tests examined, the timing of when the test was ordered predicted survival more accurately than the value it returned.
Read that again. In routine health records, when a doctor ordered a test often told you more about a patient’s future than what the test found.
That is not a failure of the data. It is what real-world data are: a record of care, shaped by clinical suspicion, hospital routines, insurance rules and documentation habits. Machine learning is extraordinarily good at finding the strongest patterns in such records. The strongest patterns are often the ones that describe the health system, not the patient.
Here is the thesis. AI makes it cheap to find candidate biomarkers in real-world data, and that is exactly why so many fail to replicate. The limiting factor is not discovery. It is validation, and validation needs different data, designed for the purpose, that most organizations never budget for.
What is AI biomarker discovery from real-world data?
A biomarker is a measurable characteristic that indicates a biological state: a blood protein, a genetic variant, an imaging feature, a pattern in heart rhythm. Biomarkers are used to diagnose disease, predict outcomes, choose treatments and measure response.
Traditionally, biomarkers were found through hypothesis-driven laboratory research and tested in dedicated cohort studies. Real-world data offer a different route. Electronic health records, insurance claims, disease registries, imaging archives and linked genomic data contain information on millions of patients. Machine learning can scan these for patterns that predict outcomes, including combinations of variables no researcher would have thought to test.
The appeal is obvious. The data already exist. Patient numbers are huge. Discovery that once took years of cohort building can happen in weeks.
Why don’t AI-found biomarkers replicate?
The problem is old, and AI amplifies it. Writing in Nature in 2011, George Poste observed that some 150,000 papers had documented thousands of claimed biomarkers, but fewer than 100 had been validated for routine clinical practice. That gap predates modern machine learning. Real-world data and flexible models widen it, for three main reasons.
The data reflect care processes. As the BMJ study showed, the presence, timing and frequency of measurements carry information about clinicians’ concerns. A model may learn that patients with a certain lab ordered at night do badly, which is true, useful for some operational purposes and meaningless as biology. Move to a hospital with different ordering habits and the “biomarker” vanishes.
Models find shortcuts. In a 2018 PLOS Medicine study, John Zech and colleagues trained deep learning models to detect pneumonia on chest X-rays from three hospital systems. The models could identify which hospital an image came from with near-perfect accuracy, and performance dropped when tested on an outside hospital, from an area under the curve of 0.931 internally to 0.815 externally. Part of what the models learned was the hospital, not the disease. Imaging-derived biomarkers are especially vulnerable, because scanners, protocols and even labels on images differ between sites.
The search space is vast. When a model can test thousands of variables and their combinations, some will predict outcomes by chance in any single dataset. Without rigorous separation of discovery and validation data, and ideally validation in entirely different populations, false discoveries are almost guaranteed.
What does failure look like in practice?
The clearest public example comes from a prediction model rather than a single biomarker, but the lesson is the same. In 2021, Wong and colleagues at Michigan Medicine published in JAMA Internal Medicine an external validation of a widely deployed proprietary sepsis prediction model built into a major health records system. In over 38,000 hospitalizations, the model achieved an area under the curve of 0.63, well below the performance reported by its developer. It missed 67 percent of sepsis cases while generating alerts in 18 percent of all hospitalizations.
That model was trained on real-world data and deployed widely before independent evaluation. It is a warning for any signal found in records and assumed to transfer. We made a similar point in AI diagnostics: the evidence is strongest where the question is narrowest and the validation most rigorous.
A worked example: a biomarker that was really a clinic
Here is a hypothetical, with illustrative details.
A company analyzes records of 200,000 patients with a chronic inflammatory disease across a large health network. Its model finds that a particular pattern of routine blood counts, measured in the months before treatment starts, strongly predicts which patients respond to an expensive biologic drug. The signal is strong, statistically robust and holds in a held-out test set from the same network. The company announces a promising predictive biomarker.
A second analysis tells a different story. Patients with the favorable pattern disproportionately attended two specialist clinics that monitor blood counts more often, start biologics earlier in the disease course, and document response more consistently. Their blood count pattern reflected more frequent testing and earlier, milder disease. The “biomarker” was partly a marker of where patients were treated and how early.
In a prospective validation in another health system, with standardized blood testing and response assessment, the signal shrinks to almost nothing.
Nothing in this story required bad faith or poor modeling. The model found a real pattern. The mistake was assuming the pattern was biological because it was predictive.
What would have caught it earlier? Checking whether the signal survived adjustment for site, testing frequency and disease duration. Validating in a different health system before announcing. And, ideally, confirming in data collected under a protocol, where measurement is standardized and not driven by clinical suspicion.
The strongest case for AI biomarker discovery from real-world data
The counterargument deserves a fair hearing, and much of it is right.
Real-world data include patients that trials and cohort studies exclude: older people, people with multiple conditions, people from underrepresented groups. A biomarker found and validated in real-world data may apply better to actual patients than one found in a narrow research cohort.
The scale is also valuable for rare outcomes. Some adverse effects, rare disease subtypes and unusual responses are almost impossible to study in purpose-built cohorts. Only large real-world datasets contain enough cases.
And there are successes. Real-world data are increasingly used to support regulatory decisions, as discussed in real-world evidence is winning the easy cases. Linked clinical and genomic datasets have helped characterize how patients with specific tumor mutations fare on particular treatments.
All of that is true. The point is not that real-world data are useless for biomarker discovery. It is that they are excellent for generating hypotheses and weak for confirming them. The discipline required is to treat every AI-found signal as a hypothesis until it survives testing in different data, ideally collected under a protocol.
What this means for researchers, companies and health systems
For researchers, the practical checklist is not new, but it is more urgent. Separate discovery and validation data strictly. Validate in a different institution, not just a held-out sample. Test whether signals survive adjustment for site, measurement frequency and timing. Report negative replication results. Explore what the model is actually using before claiming biology.
For diagnostics and biotech companies, budget for validation from the start. The cheapest part of biomarker development is now discovery. The expensive part is prospective validation, analytical validation of the assay and, for clinical use, evidence that using the biomarker improves decisions. A company that raises money on discovery and has nothing left for validation is producing press releases.
For health systems and data custodians, the most useful contribution is standardization. Consistent data definitions, documentation of how and why tests are ordered, and participation in multi-site networks make external validation possible. Health systems should also be cautious about deploying models validated only on their own data, or only on the vendor’s.
For regulators and payers, the useful stance is to judge an AI-derived biomarker by the same standard as any other: analytical validity, clinical validity in independent populations and evidence that using it changes outcomes. The source of the discovery should earn it neither extra suspicion nor a shortcut. Payers in particular should ask whether a test’s performance was established in data resembling their own members, because a biomarker that works in one network’s records may not work in another’s.
For clinicians and patients, a new AI-derived biomarker reported from records data is an early signal, not established practice. Before a test changes treatment, it should have evidence from independent validation, and clinical guidelines should support its use.
Frequently asked questions
What is a real-world data biomarker?
It is a measurable characteristic, such as a lab pattern, imaging feature or combination of clinical variables, identified by analyzing routinely collected health data like electronic records, claims or registries rather than a purpose-built research study. Such biomarkers can be identified quickly at large scale, but they must be validated in independent data before they can be trusted for clinical decisions.
Why do most biomarker discoveries fail to replicate?
Common reasons include chance findings from testing many variables, models learning site-specific or process-related patterns instead of biology, and differences between the discovery population and later populations. In health records, the timing and frequency of tests themselves carry information about clinicians’ concerns, as a 2018 BMJ study showed, which can create signals that do not generalize.
How should an AI-discovered biomarker be validated?
At minimum, it should be tested in data from a different health system or population, with the model fixed before testing. Stronger validation uses data collected prospectively under a standardized protocol. For clinical use, the assay must be analytically reliable, and there should be evidence that acting on the biomarker improves patient decisions or outcomes, not only that it predicts them.
Can AI find biomarkers that humans miss?
Yes. Machine learning can detect complex combinations of variables and subtle patterns in images or signals that human analysts would not think to test. That is its real strength in discovery. The same power makes it prone to finding spurious or process-driven patterns, which is why independent validation is essential before any AI-found biomarker is used in care.
The takeaway
Real-world data have made biomarker discovery easy. That is the problem. When finding a signal costs almost nothing, the value lies entirely in knowing which signals are real, and that knowledge still comes from slow, expensive, well-designed validation in different data.
The organizations that win will not be the ones with the biggest datasets or the most sophisticated models. They will be the ones that spend as much on proving their findings as they do on finding them.
What’s the most convincing real-world biomarker you’ve seen that later failed validation? Share the story in the comments.
