In January 2023, US kidney transplant programmes were handed an unusual instruction. They had to go back through their waiting lists, find Black patients whose eligibility had been judged using a race-adjusted kidney function equation, recalculate it without the race term, and give those patients back the waiting time they should have been accruing all along.
Within six months, according to the Organ Procurement and Transplantation Network, more than 6,100 Black candidates had their waiting times modified. The median credit was 1.7 years.
That is the most important case study in algorithmic bias that most people have never heard of. Not because it exposed the bias, which was already known, but because it did something almost no other correction has done. It tried to repair the harm to the people already affected.
Three cases, one pattern
The best-known examples of clinical algorithm bias look different on the surface. Underneath, they share a feature: the bias sat in what the algorithm was measuring, not in the mathematics.
The cost proxy. In 2019, Ziad Obermeyer and colleagues published a study in Science of a commercial algorithm used by health systems to decide which patients should get extra care management. It predicted future health care costs and used that as a stand-in for health need. Because less money was spent on Black patients with the same level of illness, often because of unequal access to care, the algorithm systematically rated them as healthier than they were. At a given risk score, Black patients were considerably sicker. The authors showed that correcting the problem would raise the share of Black patients flagged for additional help from 17.7% to 46.5%.
The race coefficient. For years, the standard equations for estimating kidney function from blood creatinine included a multiplier that raised the estimate for Black patients. The effect was to make their kidneys look healthier than they were. That could delay referral to specialists and, crucially, delay the point at which a patient qualified to begin accumulating time on the transplant list. In 2021, a joint task force of the National Kidney Foundation and the American Society of Nephrology recommended a new equation without the race term.
The device. In December 2020, Michael Sjoding and colleagues reported in the New England Journal of Medicine that pulse oximeters more often overestimated oxygen saturation in Black patients. In one cohort, occult hypoxaemia, a dangerously low arterial oxygen level that the oximeter missed, occurred in 11.7% of Black patients compared with 3.6% of White patients. Many clinical algorithms and triage rules take pulse oximetry readings as inputs, so the error travels downstream into every system built on top.
In none of these cases was the core problem a badly trained neural network. It was a proxy, a coefficient or a sensor that encoded an unequal world and then treated it as ground truth.
The conventional fix is necessary but incomplete
The standard response to a biased algorithm has three steps: identify the bias, fix the formula, deploy the new version. All three matter. But they share a blind spot. They improve decisions for future patients and do nothing for past ones.
This is the uncomfortable part. Correcting a biased algorithm has a clear cost for the institution: the audit, the software change and the retraining. Repairing the harm it already caused has a much larger and messier cost. It means finding affected patients, reopening decisions, and in some cases admitting that care was worse than it should have been. There is very little institutional incentive to do that, and some legal incentive not to.
The kidney transplant correction happened because the harm was unusually easy to quantify. Waiting time is a number, and the policy change could restore it directly. Most algorithmic harms are not like that. A patient who was never offered care management, or whose low oxygen went unnoticed, cannot be given back a missed opportunity so neatly. That is precisely why the question tends to be avoided.
What real fixes look like
Looking across these cases, a few principles emerge for anyone deploying or buying clinical algorithms:
- Interrogate the label. Ask what the model is actually predicting. Costs, utilisation and past diagnoses all carry the fingerprints of unequal access.
- Audit inputs, not only outputs. A fair model fed by a biased device will still produce biased decisions.
- Check performance by subgroup before and after deployment. Averages hide the patients most at risk.
- Plan for look-back. When a bias is found, ask who was affected and whether anything can be restored, whether that is waiting time, a follow-up appointment or a re-review of past results.
- Be cautious with simple removal. Deleting a race variable can be the right call, as with kidney function, but it is not automatically neutral. The question is always whether the resulting estimates are more accurate for everyone.
The mechanism: how bias hides in plain sight
Each of these cases went undetected for years, and the reasons are worth understanding because they will recur.
First, the biased tools often performed well on standard metrics. The cost-prediction algorithm was good at predicting costs. That was what it was built and evaluated to do. Its accuracy on the task it was given was never the problem. The problem was that the task was a poor stand-in for the thing users actually cared about. No validation study that measured accuracy on the stated target would have caught it.
Second, the harm was diffuse. A single delayed referral for kidney care, a single missed case of low oxygen or a single patient not offered care management looks like ordinary clinical variation. It is only when outcomes are aggregated by group, and compared with what the patients actually needed, that a pattern appears. Most health systems do not routinely perform that kind of analysis.
Third, the biased elements were often embedded as settled science. The race coefficient in kidney equations was published in respected journals and incorporated into laboratory software. Clinicians did not choose to apply it. It was simply part of the number that came back from the lab. Pulse oximeters were approved medical devices. Challenging them meant questioning infrastructure, not an individual decision.
This combination, good performance on the wrong target, diffuse harm and institutional embedding, is exactly what makes bias in newer AI tools hard to detect. The same questions apply to any model trained on historical data about who received care, how much it cost and what diagnoses were recorded.
A worked scenario: a look-back that nobody planned
Imagine a health system that discovers that a sepsis alert, used for three years, performed noticeably worse for patients whose first language is not the local one, because the tool relied on nursing notes that were less complete for them.
The forward fix is straightforward. The tool is recalibrated or replaced, and performance is checked by language group before it goes live again.
The look-back is harder. The system has to decide whether to identify patients who were admitted during those three years, fell into the affected group and had poor outcomes that an earlier alert might have prevented. It has to decide whether to tell them, and what to say. It has to consider the legal exposure of doing so and the ethical exposure of not doing so. And it has to find the staff time to review records, when those same staff are already stretched.
Most organisations would, quietly, do only the forward fix. That is understandable. It is also the reason why the kidney transplant case stands out. It shows that look-back is possible when there is a clear, restorable harm and a regulator willing to require it.
The strongest counterargument
A fair objection is that the look-back standard is unrealistic. Almost every clinical tool is imperfect for someone. If every discovered bias required a retrospective review of all affected patients, health systems might become reluctant to audit their tools at all, since finding a problem would trigger an expensive obligation. Better, on this view, to focus resources on fixing tools quickly and improving care for future patients.
This argument partly holds. A blanket obligation to review every patient ever affected by any imperfect tool would be impractical and could discourage the very audits we need. Resources spent on retrospective reviews are resources not spent on current care.
But the objection proves less than it seems. The look-back need not be universal to be meaningful. It can be proportionate: triggered where the harm was significant, where affected patients can be identified, and where something can still be done for them. Waiting time, missed screening, delayed referral and unrecognised chronic disease often meet those tests. The kidney case shows that a targeted look-back is achievable at national scale. The question is whether health systems will adopt it as a norm or treat that case as a one-off.
Second-order effects to watch
Overcorrection. Removing a variable from an equation, or adjusting thresholds for one group, can have unintended consequences for others. Changes need their own evaluation rather than being assumed to be neutral.
Measurement gaps. Many health systems record race, ethnicity, language and disability inconsistently. Without reliable data on who patients are, subgroup audits are impossible. Improving that data raises its own privacy and trust questions.
Legal and regulatory pressure. As regulators and courts pay more attention to algorithmic discrimination, organisations may respond by auditing more, or by documenting less. The direction will depend on whether audits are protected and encouraged or treated mainly as evidence of liability.
Device standards. The pulse oximetry evidence has pushed regulators to reconsider how devices are tested across skin tones. How quickly updated standards translate into devices on hospital wards is a test of whether the system can learn.
What this means for different players
Health system leaders should maintain an inventory of algorithms and scoring tools in clinical use, including those embedded in laboratory and device software, and require subgroup performance checks before and after deployment. They should also decide in advance what triggers a look-back.
Clinicians should be encouraged, and protected, when they raise concerns that a tool behaves differently for some patients. Frontline observations were part of how several of these biases came to light. A clinician who notices that a reading does not match the patient in front of them is running an audit, whether or not anyone calls it that.
Regulators can require subgroup performance reporting for clinical algorithms and devices, and can follow the kidney example by making remediation, not just correction, part of the expected response to a discovered bias.
Vendors should disclose what their models predict, what data they were trained on, and how performance varies across groups. Proxy labels, such as cost, should be named plainly.
Funders and researchers should support methods for fairness audits that work with the imperfect data real health systems have, and studies that follow up patients affected by biased tools.
The takeaway
Bias in clinical algorithms is not mainly a problem of rogue code. It usually starts with a proxy or a measurement that quietly reflects who has historically received care. Fixing the formula is the easy part.
The harder test, and the one that separates genuine accountability from a software update, is whether a health system is willing to go back and find the patients the old version got wrong.
Has your organisation ever gone back to review patients affected by a biased tool or measurement? We’d like to hear how it was handled.
