A patient on radiation treatment sends a portal message at 9 p.m. Her mouth sores are worse and she hasn’t been able to drink much today. By morning, a draft reply is waiting in her oncologist’s inbox, written by a large language model. It is warm, fluent and thorough. It explains mucositis, suggests salt-water rinses and lists when to call the clinic.
The oncologist reads it, tweaks a sentence and clicks send.
Whether that was good medicine depends on one thing the draft may have gotten wrong: how urgent the situation actually was. And the evidence so far suggests the physician is less likely to catch that than we assume.
Most of the debate about language models in patient communication is about whether the machine is accurate. That is the wrong place to look. The guardrail that matters most is not on the model. It is on the human who is supposed to be checking it.
The machine is already good at the easy part
The case for AI-drafted messages rests on a real finding. In 2023, a study in JAMA Internal Medicine compared physician answers to patient questions posted on Reddit’s r/AskDocs forum with answers generated by ChatGPT. Licensed health professionals, blinded to the source, preferred the chatbot’s response in 78.6% of 585 evaluations, and rated it higher on both quality and empathy.
That result was widely read as “AI is better at talking to patients.” A more careful reading is that doctors answering strangers online in their spare time write short, brisk replies, and a model with unlimited patience writes long, kind ones. Length and warmth are exactly what raters reward.
Nobody should be surprised that a language model is good at sounding caring. The harder question is whether it is good at triage, and whether the clinician in the loop will notice when it isn’t.
What the Boston study actually found
The most useful evidence on this comes from a small study published in The Lancet Digital Health in 2024 by researchers at Mass General Brigham. Six radiation oncologists answered simulated cancer-patient messages twice: once on their own, and once by editing a GPT-4 draft.
Three findings stand out.
First, the drafts were mostly fine but not always. Assessing physicians judged that unedited drafts posed a risk of severe harm in 11 of 156 survey responses (7.1%), and a risk of death in one. The researchers noted that most harmful drafts failed not on medical facts but on recognising or conveying how urgent the situation was.
Second, doctors felt more efficient. The draft improved subjective efficiency in about 77% of cases.
Third, and most important, the doctors’ answers changed. Their own unassisted replies were short, averaging 34 words, and varied widely between physicians. With the draft in front of them, their replies grew to about 160 words and became much more alike. The content of the edited responses looked more like the machine’s draft than like what the same physicians had written unaided. The drafts were less likely to tell patients to come in, and more likely to offer education and self-care advice.
The authors called this what it looks like: anchoring and automation bias. The physicians were not just using the model to phrase their own judgement. To a measurable degree, they were adopting its judgement.
Why “human in the loop” is a weaker guardrail than it sounds
Almost every health system deploying drafted replies relies on the same safety argument: a clinician reviews every message before it goes out. It is a reassuring sentence. It is also the guardrail most likely to erode over time.
Review works when the reviewer is sceptical. But drafts are designed to be accepted. They arrive complete, well formatted and plausible. The busier the clinician, the more the draft becomes the default, and the whole point of the tool is to serve busy clinicians. The incentive and the safeguard pull in opposite directions.
There is a second trap. A draft that errs toward reassurance reads like a good message. A patient told to use salt-water rinses feels cared for. A patient told to come in tonight may feel alarmed. If the model’s tilt is toward education over escalation, the errors it makes will be the ones that look most like good bedside manner.
Guardrails worth building
If the risk sits in the human-machine interaction, then the guardrails have to sit there too. Some that deserve more attention than they get:
- Separate triage from drafting. Let a narrow, auditable process flag acuity first, whether a nurse protocol or a validated rule set, and only then generate text. Do not ask one fluent model to decide urgency and write the reply in the same breath.
- Make the draft show its reasoning on acuity. A single line such as “assessed as non-urgent because…” gives the reviewer something specific to disagree with, instead of a polished paragraph to skim.
- Measure edit rates, not just satisfaction. If clinicians send most drafts almost unchanged, that is not proof of quality. It may be proof of drift.
- Audit the escalations that didn’t happen. Sample drafted replies to messages that later led to an ED visit or admission. That is where the harm will hide.
- Tell patients. Disclosure is not only an ethics requirement. A patient who knows a reply was machine-drafted may push back when it doesn’t match what they feel.
How the drift happens, week by week
It helps to picture how this plays out in an ordinary clinic, because the risk does not arrive as a single bad message.
In the first month after go-live, clinicians read every draft closely. They are curious and a little suspicious. They rewrite openings, delete paragraphs of generic advice and add their own instructions. Edit rates are high, and the tool saves less time than promised.
By month three, the drafts have been right often enough that trust has formed. Reviewers start reading the first and last lines and skimming the middle. The edits get smaller. This is exactly what the tool’s designers hoped for, and it shows up on the dashboard as success: time per message falls, satisfaction rises.
By month six, a new resident joins who has never answered the inbox without drafts. For her, the draft is not an aid to her judgement. It is the starting point of her judgement. She has no unassisted baseline to compare it against.
Nothing in that sequence involves a malfunction. The model has not changed. What changed is the human. And the Boston study hints at the direction of the change: toward longer, more educational, more uniform replies that are less likely to tell a patient to come in.
That last shift matters because of how harm shows up in messaging. A reply that over-escalates produces a visible cost: an unnecessary visit, an annoyed patient, a crowded urgent care slot. A reply that under-escalates produces an invisible one. The patient stays home. Most of the time, nothing happens. Occasionally, something does, and it may surface days later in an emergency department with no obvious link back to the portal message.
Systems learn from the costs they can see. So there is a real danger that organisations will tune drafting tools, deliberately or not, toward the error that is cheaper to observe.
The strongest counterargument
The best case against this worry deserves a fair hearing, because it is substantial.
Unassisted physicians are not a gold standard. In the same Boston study, agreement between physicians on the clinical content of their manual replies was poor, with a mean Cohen’s kappa of 0.10. Six experienced specialists, answering the same messages, often said quite different things. With the draft, agreement rose to 0.52.
One reading of that is anchoring. Another is that the draft pulled everyone toward a more complete, more consistent answer, and that variation between doctors was itself a quality problem. A patient’s advice should not depend on which physician happens to be on inbox duty.
The counterargument also points to the baseline. A 34-word reply written at the end of a long day by a tired clinician is not a safe default either. If drafts make replies more thorough, and a reviewer catches most of the acuity errors, the net effect on patients could be positive.
This argument partly holds. Consistency is valuable, and the manual baseline is weaker than defenders of “the doctor’s judgement” like to admit. But it doesn’t dissolve the concern. Consistency around a model’s tendencies is only good if those tendencies are right, and the study’s own finding was that the model’s characteristic mistake was misjudging urgency. Uniformly under-triaged replies are consistent. They are not safe.
The honest conclusion is that drafting could make patient communication better on average and worse at the tail. Both can be true at once. The job of governance is to capture the first without accepting the second.
Second-order effects worth watching
A few consequences are likely to show up only after deployment has been running for a while.
Patients will write differently. When replies become longer and more educational, patients may learn that the portal is a good place for complex clinical questions. Message volume could rise, which would erode the time savings the tool was bought to produce.
Deskilling of triage. If trainees learn messaging with drafts from the start, the skill of reading a short patient message and sensing that something is wrong may not develop in the same way. That skill matters beyond the inbox, on the phone and at the bedside.
Liability will get murky. When a drafted reply contributes to harm, it will be hard to say whether the error was the model’s, the reviewer’s or the organisation’s for configuring the workflow. Until that is settled, clinicians carry the risk for a decision they increasingly share with software.
Vendor prompts become clinical policy. The Boston authors called for more transparency from electronic record vendors and institutions about how prompts are designed. That is not a technical footnote. The prompt that shapes every draft is effectively a triage protocol, and it should be governed like one.
What this means for the people deciding
For health system leaders, the practical step is to put drafted messaging under the same governance as clinical decision support, not under IT productivity projects. That means a named clinical owner, a safety review of the prompt, and routine sampling of sent messages against later outcomes.
For clinicians, it means reading the acuity judgement first. Before looking at the tone or the wording, ask one question of every draft: does this reply tell the patient to do the right thing, at the right speed? Everything else is polish.
For regulators and professional bodies, it means deciding whether a tool that influences what patients are told to do about symptoms is merely an administrative aid. The Boston authors argued these systems deserve the same rigour as any other software used as a medical device. That argument will only get stronger as drafts become the default.
For patients, the ask is simpler. If a reply doesn’t match how unwell you feel, say so, or call. A well-written message is not the same as a correct one.
What to watch next
The evidence base here is still thin. Most published studies, including the Boston one, are simulations or small evaluations. The studies that matter next are real-world ones that link drafted replies to downstream events: emergency visits, admissions, delayed diagnoses. Watch for whether health systems publish edit rates and escalation rates over time, not just clinician satisfaction. The first organisation to report that its escalation rate quietly fell after go-live, and to ask why, will have done the field a service.
The trade-off nobody wants to name
Inbox burden is real, and it drives burnout. A tool that saves clinicians time on routine messages is valuable, and pretending otherwise helps no one.
But the economics are lopsided. The time savings show up immediately and are easy to count. The cost, a slow shift in clinical judgement toward whatever the model tends to say, is diffuse, delayed and very hard to measure. Organisations tend to optimise for what they can count.
The Boston study was a simulation with six physicians. It proves nothing about any specific deployment. What it does show is that the right question is not “is the model accurate?” but “what happens to the doctor’s judgement when the model is in the room?”
Health systems that treat drafted messages as a typing aid will miss that. The ones that treat them as a form of decision support, with the monitoring that implies, have a chance of catching it.
Has AI drafting changed how your team answers patient messages, and would you know if it had? Tell us in the comments.
