Automation bias in healthcare AI is the tendency of clinicians to accept a machine’s output even when it is wrong, or to miss problems the machine did not flag. Controlled studies show it affects experienced clinicians too. Training alone does not fix it; interface and workflow design, such as forming an independent judgment first, reduce it.
In 2023 the journal Radiology published an experiment that should be required reading for anyone buying clinical AI. Twenty-seven radiologists read mammograms with the help of what they were told was an AI system suggesting a BI-RADS category, the standard scale for how suspicious a breast finding is. Some suggestions were correct. Some were deliberately wrong.
When the AI was right, inexperienced readers classified about 80% of mammograms correctly. When it was wrong, their accuracy fell below 20%. Experience helped, but not as much as anyone would hope: radiologists with more than 15 years of practice fell from 82% accuracy to 45.5% when the suggestion was wrong.
These were trained specialists, working in their own field, on a task they do every day. A confident label on the screen pulled them away from what they would otherwise have seen.
This piece makes a design argument rather than a training one. Automation bias is not a character flaw that education removes. It is a predictable property of how people interact with confident machines under time pressure. The fix therefore sits mostly with the people who design the interface and the workflow, and with the organizations that buy it.
What is automation bias in healthcare?
Automation bias is the tendency to favor suggestions from automated systems and to under-weight contradictory information, including one’s own judgment. It produces two kinds of error.
Commission errors happen when a clinician follows an incorrect recommendation: accepting a wrong dose, a wrong diagnosis or a wrong risk category because the system suggested it.
Omission errors happen when a clinician misses a problem because the system did not flag it: assuming a scan is normal because no box was drawn, or a drug combination is safe because no alert fired.
The phenomenon is well studied. A 2012 systematic review in the Journal of the American Medical Informatics Association by Kate Goddard and colleagues found automation bias across clinical decision support studies, and identified factors that make it worse, including workload, task complexity and trust in the system, as well as some that reduce it.
It is distinct from deskilling, which is the erosion of skill over time through reliance. Automation bias happens in the moment, even in clinicians whose skills are intact.
Does explaining the AI’s reasoning prevent automation bias?
This is the most common answer offered by vendors and policymakers, and the evidence is discouraging.
In December 2023, JAMA published a randomized study led by Sarah Jabbour at the University of Michigan. Hospital clinicians diagnosed patients with acute respiratory failure from clinical vignettes, some with help from an AI model. A standard model improved their accuracy modestly. A model deliberately built to be systematically biased reduced their accuracy by 11.3 percentage points compared with baseline.
The researchers then added explanations: images showing which parts of the chest X-ray the model was focusing on. In the biased cases, the explanations highlighted regions unrelated to the condition, which in principle should have tipped clinicians off. They barely helped. The drop in accuracy improved by a non-significant 2.3 percentage points.
This matters because explainability is often treated as the safety feature for clinical AI. The Jabbour study suggests that clinicians under realistic conditions do not interrogate explanations closely enough for them to act as a reliable brake. An explanation is information. It is not, by itself, a design that changes behavior.
Why do experienced clinicians still fall for it?
Several forces push even experts toward over-trust.
Time pressure is the most important. Checking a machine’s output independently takes time, and clinical work rarely has spare time. Accepting a plausible suggestion is the fastest path, and most of the time it is correct, so the habit is rewarded.
Base rates matter too. A tool that is right 95% of the time teaches its users, correctly, that it is usually right. The 5% of errors arrive without warning, looking identical to the correct outputs.
Anchoring compounds the problem. Once a clinician has seen a suggested answer, their own reasoning tends to start from it rather than from the raw evidence. In the mammography study, the suggestion arrived with the image, so the anchor was set before the radiologist looked.
Finally, accountability can become blurred. If an organization has bought and endorsed a tool, disagreeing with it can feel like an extra burden: the clinician must justify overriding it. We made a similar point about the doctor who agrees with an AI-drafted patient message: the risk sits with the human who approves without really checking.
How can interface design reduce automation bias?
If training and explanations are weak defenses, design has to do more of the work. Several approaches are supported by research or by practical experience.
Independent first, AI second. Where the workflow allows, the clinician forms and records their own judgment before the AI output is revealed. This removes the anchor. It costs a little time, which is why it is rarely the default, but it directly targets the mechanism in the mammography study.
Use AI as a triage or second reader rather than a first answer. In the Swedish MASAI breast screening trial, AI sorted mammograms by risk to decide how many radiologists read each one, rather than showing each radiologist a suggested verdict. The full results, published in The Lancet in January 2026, found more cancers detected at screening and fewer interval cancers, with at least one human reading every screen.
Show uncertainty honestly. The authors of the mammography study recommended displaying the confidence of each output. A tool that says “low confidence” on the cases where it is least reliable gives clinicians a cue to look harder.
Make disagreement easy and valued. Overriding the tool should take one click and a short reason, not a form. Organizations should review overrides as a source of learning, not treat them as non-compliance.
Monitor the humans as well as the model. Track how often clinicians accept outputs and whether acceptance rises over time. A near-100% acceptance rate is not proof that the tool is perfect. It may be proof that nobody is checking.
A worked example: two ways to deploy the same chest X-ray tool
Consider a hypothetical emergency department deploying an AI that flags suspected pneumothorax and other urgent findings on chest X-rays.
Design A shows the AI’s verdict and a heat map as soon as the image opens. Busy emergency physicians glance at the heat map and move on. When the AI marks an image normal, reviewers later find that some subtle findings were missed that the same physicians would probably have caught unassisted. When the AI flags a finding incorrectly, some patients receive unnecessary further imaging.
Design B uses the AI to reorder the radiology reading queue, so suspected urgent findings are reported first, and alerts the emergency physician only for high-confidence urgent findings. The physician reviewing an image sees their own read first; the AI’s output appears after they record an impression. Disagreements are logged and reviewed monthly.
Design B is slower per image for the physician and requires more governance. It also turns the AI into a safety net rather than an anchor. The model, the accuracy and the price are identical. The error profile is not.
What is the strongest argument against designing for skepticism?
The counterargument is that friction has costs. Independent-first workflows slow clinicians down, and in some tasks the AI is genuinely more accurate than the average user. If you force a clinician to form a view first, you may get more errors on cases where the machine was right and the human was not, and you lose some of the efficiency that justified the purchase.
This is a fair point, and it means design should vary with the stakes and the evidence. For low-risk, high-volume tasks where the tool is thoroughly validated, a lighter-touch design may be right. For high-stakes decisions, or tools with limited local validation, protecting independent judgment is worth the time.
What is not defensible is choosing the most frictionless design by default, with no analysis, because it demos well. That is how organizations end up with clinicians who approve whatever the screen says.
What this means for hospitals, vendors and clinicians
For hospital leaders and AI governance committees, automation bias belongs in procurement. Ask vendors how outputs are presented, whether the display can be configured to reveal AI output after the clinician’s own read, how uncertainty is shown, and how overrides are captured. A safety culture that invites challenge, the subject of our piece on psychological safety in surgery, applies to challenging machines as much as challenging senior colleagues.
For vendors, the market is moving toward accountability for human-AI performance, not just model accuracy. Products that can show how clinicians perform with the tool in practice, including when it is wrong, will be easier to defend.
For clinicians, the most effective personal habit is to commit to your own view before you look, where you can, and to treat unusually confident outputs on unusual cases as a reason to slow down.
For regulators, pre-market studies should test the combined human-AI system, including its behavior when the model is wrong, rather than relying on standalone model accuracy.
Frequently asked questions
What is an example of automation bias in healthcare?
In a 2023 Radiology study, radiologists reading mammograms with deliberately incorrect AI suggestions saw their accuracy drop sharply. Inexperienced readers fell from about 80% to under 20%, and very experienced readers from 82% to 45.5%. They followed the wrong suggestion even though they could have classified many of those cases correctly without it.
Does training clinicians reduce automation bias?
Training helps clinicians understand a tool’s limits, but evidence suggests it is not enough on its own. Time pressure and anchoring affect experienced and well-informed users. More effective defenses are built into the workflow, such as forming an independent judgment before viewing AI output, showing uncertainty, and making overrides easy.
Do AI explanations stop clinicians from following wrong advice?
Not reliably. A 2023 JAMA randomized study found a systematically biased AI model cut clinicians’ diagnostic accuracy by 11.3 percentage points, and adding visual explanations improved this by only a non-significant 2.3 points. Explanations can support scrutiny, but they do not guarantee clinicians will use them.
What is the difference between automation bias and alert fatigue?
Automation bias is over-trust: accepting a machine’s suggestion or assuming that no alert means no problem. Alert fatigue is under-trust caused by excessive, often irrelevant alerts, leading clinicians to ignore warnings. Poorly designed systems can produce both at once, and good design aims for fewer, more reliable signals.
Should patients worry about automation bias?
Patients benefit when clinicians use well-validated AI carefully, and automation bias is a design and governance problem for organizations to manage. Patients who want to know how AI contributes to their diagnosis or treatment can ask their clinician directly, and should consult a clinician rather than rely on an AI tool alone.
The dangerous AI is not the one that is often wrong. It is the one that is usually right, looks certain, and appears on the screen before the clinician has had a chance to think.
In your organization, does the AI’s answer appear before or after the clinician’s own judgment, and did anyone choose that on purpose?
