In April 2019, the US Food and Drug Administration expanded the approval of palbociclib, sold by Pfizer as Ibrance, to include men with hormone receptor-positive, HER2-negative advanced or metastatic breast cancer. The original trials had not enrolled men. Male breast cancer accounts for under 1 percent of cases, which makes a dedicated randomized trial hard to run. Instead, the supplemental approval leaned on real-world data: electronic health records and post-marketing reports drawn from an IQVIA insurance database, the Flatiron Health breast cancer database and Pfizer’s own global safety database.
It was a landmark, and it is still cited as one of the cleanest examples of real-world evidence informing a regulatory decision.
Look closely at why it worked, though. The drug’s efficacy had already been established in large randomized trials in women. The biology of the disease in men was similar. The real-world data were being asked a narrow question: is there any sign that this drug behaves differently in a small subgroup the trials missed?
That is the thesis of this piece. Real-world evidence is succeeding in regulation mainly in the cases where the stakes of being wrong are lowest. That is not a failure. It is the correct use of the tool. The danger is the growing pressure to use it for the hard cases, where it is weakest.
Why regulators opened the door
The push for real-world evidence has a legal foundation in the United States. The 21st Century Cures Act of 2016 required the FDA to develop a framework for evaluating real-world evidence, and the agency published that framework in December 2018. Europe’s regulators have been building similar capacity, including a network for analysing real-world data on behalf of the European Medicines Agency.
The motives are reasonable. Randomized trials are expensive and slow. They often exclude older adults, people with multiple conditions, pregnant women and rare subgroups. Once a drug is on the market, millions of people take it, and their records hold information that no trial could collect.
So the question is not whether regulators should use real-world data. They already do, heavily, for safety monitoring. The question is how far it can stand in for randomization when deciding whether a drug works.
The problem randomization solves
Randomization exists to deal with one problem: the people who get a treatment in real life differ from the people who do not, in ways that affect outcomes.
Doctors prescribe new drugs to patients they think will benefit, or tolerate them, or can afford them. Sicker patients may be steered away, or toward. Patients who take their medicines faithfully also tend to exercise, attend follow-up and have better outcomes for reasons unrelated to the drug. Statistical methods can adjust for the factors that are recorded. They cannot adjust for the ones that are not.
Electronic health records add their own complications. Outcomes are recorded for billing and clinical care, not research. A progression date in an oncology record depends on when the scan happened to be ordered. Missing data are not random: patients who move, switch insurers or stop attending simply disappear.
This is why the Ibrance case was comfortable. Confounding matters less when the question is “is this consistent with what the trials showed?” than when it is “does this work at all?”
A back-of-envelope look at why size does not save you
A common reply to all this is that real-world datasets are enormous, so the noise washes out. It does not, and a simple illustration shows why.
Imagine a new drug with no real effect on one-year mortality. Suppose that among the patients who receive it, doctors have quietly favoured the ones who look robust: better kidney function, fewer recent admissions, more support at home. Say those unrecorded advantages lower death risk by a tenth. The treated group’s mortality comes out around 10 percent lower than the untreated group’s, and the drug gets the credit.
Now scale up. With a few thousand patients, that 10 percent difference might sit inside a wide confidence interval and be dismissed as noise. With a few hundred thousand, the interval shrinks to a sliver around the same biased estimate. More data does not remove the bias. It makes the wrong answer look more certain. Randomization is the only design that breaks the link between who gets the drug and how they would have fared anyway, and no quantity of records substitutes for it.
The numbers here are invented for illustration, but the arithmetic is general. Any systematic error in who gets treated survives scale untouched, while random error shrinks. Big observational datasets are therefore most dangerous precisely where they feel most persuasive.
Where the incentives pull
Here is the uncomfortable part. For a company, real-world evidence is attractive precisely because it is cheaper and faster than a trial. For a regulator under pressure to speed access, it is attractive for the same reason. For patient groups waiting on a therapy, any path that shortens the wait looks good.
Those incentives all point in one direction: toward accepting real-world evidence in more situations. Very few actors have a strong incentive to say “this question needs a trial.” Regulators are left holding that line, often in public, often accused of bureaucratic caution.
There is also a subtler incentive. Real-world analyses involve many choices: which database, which patients, which comparison group, which follow-up window. When different choices give different answers, a sponsor can explore quietly and present the most favourable. Pre-registration of real-world study protocols, with the analysis plan fixed before the data are examined, is the best defence, and it is still not universal.
Data vendors add another layer. Companies that curate and sell real-world datasets are paid by the sponsors whose products they help evaluate. That is not corruption, and the better firms invest heavily in data quality. But it means the commercial pull on the whole supply chain runs toward usable, publishable answers rather than toward the awkward conclusion that a dataset cannot answer the question put to it.
The strongest counterargument
The best case for pushing real-world evidence further deserves a fair hearing. It goes like this: randomized trials are not neutral either. They enrol a narrow, healthier, more closely monitored population, often at academic centres, and then the drug is used in the elderly, the frail and the poorly followed. A trial can be internally rigorous and still tell you little about the patient in front of you. Meanwhile, insisting on a trial for every question means some questions are never answered, because nobody will fund them. For rare diseases, for off-patent drugs and for comparisons between two existing treatments, “wait for the trial” often means “never know.”
That argument partly holds. The external validity problem with trials is real, and the choice is frequently not between a trial and an observational study but between an observational study and nothing. In those cases, a carefully designed analysis with a locked protocol is better than a shrug.
Where it breaks down is in the leap from “trials have limits” to “observational data can establish that a new drug works.” The first is a question about generalizing a known effect. The second is a question about whether there is an effect at all. Real-world data are good at the first and unreliable at the second, for the arithmetic reasons above. The counterargument is strongest for the modest questions, which is exactly where this piece says real-world evidence already belongs.
Where real-world evidence genuinely shines
None of this argues for less use. It argues for better-matched use. Real-world evidence is strongest when the question is safety at scale, because rare adverse events that no trial could detect show up in large populations. It is credible when the effect is large and the natural history well known, as in some rare diseases where an untreated course is so predictable that a dramatic response stands out against historical data. It works when it extends, rather than establishes, efficacy, as with Ibrance in men. And it is at its best when combined with randomization.
That last category deserves far more attention than it gets. Pragmatic trials randomize patients but collect outcomes from routine records, keeping the protection of randomization while cutting the cost of dedicated trial visits and case report forms. The false choice between a gold-plated trial and an observational study leaves out the most promising middle ground.
Second-order effects worth anticipating
If real-world evidence keeps winning the easy cases and gets stretched into harder ones, a few knock-on effects follow.
The first is on trust in labels. Clinicians read a label change as an endorsement of evidence quality. If some indications rest on randomized trials and others on observational analyses, but the label looks the same, prescribers lose the ability to weigh them differently. The difference needs to be legible, not buried in a review document few people open.
The second is on data infrastructure. Once regulators accept routine records for decisions, the quality of coding, the completeness of outcome recording and the linkage between hospital, pharmacy and death records become regulatory assets. Health systems with clean, linked data gain leverage with sponsors; those without it get left out of studies, and their populations stay invisible.
The third is on trials themselves. If a sponsor can get a subgroup extension through records, the incentive to enrol diverse populations in the original trial weakens. The very success of the Ibrance approach could, over time, reduce the pressure to include men, older adults or other groups at the start. That would be a perverse outcome from a well-intentioned tool.
What different actors should do
For regulators and ministries, the priority is transparency about evidence type. Every approval or label change that relies materially on real-world data should say so plainly, state the question the data were asked to answer, and link to a pre-registered protocol. Funding public data infrastructure is also a regulatory task now, not just a research one.
For hospital CEOs and health system leaders, data quality is becoming a strategic asset. Investing in structured outcome capture, such as consistent recording of progression, hospitalisation and cause of death, makes a system a more valuable research partner and a better-run hospital at the same time.
For payers, the lesson is to price evidence strength. A drug whose benefit in a population rests on observational extension might reasonably come with outcome-based agreements or a requirement for follow-up data, rather than the same terms as one backed by randomized evidence.
For funders and founders in the real-world data business, the durable advantage will not be the size of a dataset but its auditability: provenance, completeness and the ability to show a regulator exactly how a variable was derived. Firms that make their methods inspectable will outlast those selling scale.
For clinicians, the practical habit is simple. When a label changes, ask what question the data were asked, whether the analysis plan was fixed in advance, and whether the result extends a known effect or claims a new one.
What to watch over the next two years
Watch whether regulators publish clearer statements, in approval documents, of when real-world data were decisive rather than supportive. Watch whether pragmatic randomized trials embedded in routine care become a normal part of post-approval commitments rather than an academic curiosity. Watch for the first contested case where an efficacy claim built mainly on observational data is later challenged by a randomized trial; how regulators handle that reversal will set the tone for a decade. And watch whether pre-registration of real-world study protocols becomes an expectation rather than a courtesy.
The takeaway
Real-world evidence has earned its seat in regulatory decisions, and it will keep growing. But its credibility rests on a small number of clean successes in cases where it was answering modest questions. Spending that credibility on big, contested questions of efficacy, without randomization and without locked analysis plans, is the fastest way to lose it. Health professionals reading a label change should ask what kind of question the real-world data were asked to answer, not just whether they were used.
Where do you think the line should sit: should real-world evidence ever support a first approval for a common disease? We’d like to hear your view.
