A generative AI tool can be used with patient data under HIPAA only if the vendor signs a business associate agreement (BAA) and the tool is configured and used within that agreement. Free consumer chatbots generally come with no BAA. No product is “HIPAA certified.” Compliance depends on the contract, the configuration and how staff actually use it.
It is 6:40 p.m. on a busy medical ward. A resident has four discharge summaries left and a handoff in twenty minutes. She copies a patient’s hospital course from the EHR, opens a free chatbot on her phone, pastes it in and asks for a clean summary in plain language. It takes eleven seconds. The result is good. She pastes it back, edits a line and moves on.
Nothing alarms. No firewall logs it as an incident. Nobody in compliance will ever know.
That is the HIPAA problem with generative AI in 2026. Health systems are spending real money on approved, contracted AI tools, and they should. But the more common exposure is not the enterprise product with a signed agreement. It is the free tab in the browser. My argument is that most hospitals are auditing the wrong layer. They review vendors carefully and barely measure what staff actually do.
This article is general information, not legal advice. It reflects HIPAA rules and vendor offerings as of May 2026. Check current vendor terms and your own counsel before relying on any product.
Is ChatGPT HIPAA compliant?
The question is common and slightly wrong. HIPAA does not certify products. It regulates covered entities, such as providers, health plans and clearinghouses, and their business associates, meaning vendors that create, receive, maintain or transmit protected health information (PHI) on their behalf.
When a hospital lets a vendor handle PHI, HIPAA requires a business associate agreement. The BAA commits the vendor to safeguard the data, limit its use, report breaches and pass the same duties on to its subcontractors. Without a BAA, disclosing PHI to that vendor is generally an impermissible disclosure, however secure the vendor’s systems are.
So the useful questions are these. Will the vendor sign a BAA? For which products and configurations? And is your staff actually using that product, or a different one with the same logo?
As of this writing, the major model providers offer BAAs only for specific business products. OpenAI launched ChatGPT for Healthcare in January 2026 as an enterprise workspace for health systems, available under a BAA, and has offered BAAs for some enterprise and API uses. Anthropic offers BAAs for its API and for HIPAA-ready enterprise plans. Microsoft, Google and Amazon offer BAAs covering certain AI services in their clouds. In each case, the free and consumer tiers are outside the agreement.
That distinction is where most real-world risk lives. The resident on the ward did not use a contracted enterprise tool. She used the free app, which carries no BAA.
How common is “shadow AI” in hospitals?
More common than most compliance teams assume. In a survey of more than 500 US healthcare workers conducted in December 2025 and published by Wolters Kluwer Health, about 40 percent said they had come across unauthorized AI tools in their organization, and nearly one in five said they had used one. The main reason given was speed. About a third pointed to a lack of approved tools, or approved tools that lacked the features they needed.
The finding matters less as a number than as a pattern. Staff are not using shadow AI to be reckless. They are using it because documentation burden is crushing and the approved alternative is either missing or worse. As we argued in the EHR is not the burden, people route around the systems that slow them down.
Why is pasting patient notes into a chatbot a HIPAA problem?
Three reasons, in rising order of importance.
It is a disclosure without a BAA. A clinical note is PHI. Sending it to a service with no business associate agreement is, in most cases, a disclosure HIPAA does not permit. Removing a name is not enough. HIPAA’s Safe Harbor de-identification standard requires stripping 18 kinds of identifiers, including dates more specific than a year, small geographic units, medical record numbers and any unique identifying characteristic. A hospital course usually contains several.
The organization loses control of the data. Consumer terms differ by product and change over time. Some consumer services may retain conversations or use them to improve models unless users opt out. The hospital has no contract governing retention, deletion, access or breach notification.
Nobody can reconstruct what happened. This is the part that makes the problem invisible. Enterprise tools produce logs. Personal accounts on personal phones do not produce logs the hospital can see. If a patient complains or a regulator asks, the organization cannot say what was shared, when or with whom. Under HIPAA’s breach rules, an impermissible disclosure is presumed to be a breach unless a risk assessment shows a low probability that the data was compromised. It is hard to do that assessment on events you cannot see.
What makes an AI tool “HIPAA compliant” in practice?
A signed BAA is necessary. It is not sufficient. A reasonable checklist for any generative AI tool that will touch PHI looks like this:
- A BAA covering the specific product and configuration you will use, not just the vendor in general.
- Clear data terms: no training on your data without permission, defined retention periods, and deletion on request.
- Access controls tied to your identity system, so only authorized staff use the tool and access ends when employment does.
- Audit logs you can see and export.
- A security risk analysis that includes the tool. The HIPAA Security Rule already requires an accurate and thorough risk analysis of risks to electronic PHI. An AI tool that holds PHI belongs in it.
- Minimum necessary use. Staff should share only what the task needs. A discharge summary tool does not need the social security number.
- Clinical review rules. HIPAA does not make an output accurate. Policies should say who checks AI-generated content before it enters the record or reaches a patient.
On the regulatory horizon, HHS proposed a major update to the HIPAA Security Rule in January 2025. It would, among other things, require a written inventory of technology assets and a network map showing how electronic PHI moves through systems. As of this writing it remains a proposal, and the current Security Rule still applies. But an organization that cannot list the AI tools touching PHI would struggle under either version.
A worked example: fixing the ward, not just the contract
Here is a hypothetical 400-bed hospital that discovers shadow AI use.
A privacy officer notices a spike in traffic to a consumer chatbot domain from the hospital’s guest Wi-Fi during evening shifts. A short anonymous survey of residents and nurses confirms it: many use free chatbots for discharge summaries, patient instructions and translation.
The wrong response is to block the domain and send a stern email. Traffic moves to personal phones on cellular data, and the hospital loses what little visibility it had.
The better response has four parts. First, the hospital licenses an enterprise generative AI tool under a BAA, integrated with its identity system and, ideally, the EHR. Second, it makes that tool easier to use than the free one, with templates for exactly the tasks staff were already doing. Third, it writes a one-page policy that says plainly what can and cannot be pasted where, with examples. Fourth, it keeps measuring, through periodic anonymous surveys and network monitoring, because behavior is the thing being managed.
Six months later, the hospital cannot claim zero shadow AI. It can show that most of the work moved inside a governed tool, which is a defensible position. The earlier state, where nobody knew, was not.
What is the strongest counterargument?
The strongest counterargument is that this is overblown. A de-identified snippet pasted into a chatbot is a small disclosure compared with the ransomware attacks that expose millions of records, a problem we covered in hospital cyberattacks are a patient-safety problem. Regulators have finite resources and have not focused on individual chatbot use. Meanwhile, staff using AI to finish documentation faster may be reducing burnout and errors. Heavy-handed enforcement could push the behavior further underground.
Some of that is true. The individual risk per paste is small, and there is no public record of enforcement built around a single clinician’s chatbot use. But the argument misreads what compliance is for. HIPAA violations are often discovered through something else: a complaint, a breach investigation, a lawsuit. When that happens, an organization that tolerated widespread undocumented disclosures looks negligent, not unlucky. And the aggregate matters. Thousands of small disclosures to services with no contractual duties is a larger exposure than any single one suggests. The fix is not punishment. It is supply: an approved tool good enough that people stop needing the free one.
What this means for hospitals, clinicians and vendors
For hospitals, measure behavior, not just contracts. Survey staff anonymously about AI use. Watch network traffic to consumer AI services. Then supply a governed alternative that meets the need. Add every AI tool that touches PHI to your security risk analysis.
For clinicians, the rule is short. Do not put patient information into any AI tool your organization has not approved for PHI. If the approved tool does not do what you need, say so. Your feedback is how the gap gets fixed.
For vendors, “HIPAA compliant” in marketing copy is not a feature. A BAA that covers the actual product, clear retention and training terms, and exportable audit logs are. As we noted in your health app isn’t covered by HIPAA, the gaps usually hide in the parts of the stack nobody reads.
Frequently asked questions
Is there such a thing as a HIPAA-certified AI tool?
No. HHS does not certify products as HIPAA compliant, and there is no official HIPAA certification. Vendors may have third-party security attestations, such as SOC 2 or HITRUST, which are useful evidence. But HIPAA compliance depends on a signed business associate agreement and on how the covered entity configures and uses the tool.
Can doctors use ChatGPT with patient information?
Only through an offering covered by a business associate agreement with the covered entity, such as an enterprise product the organization has contracted for, and within its policies. Using a free or personal account with identifiable patient information is generally an impermissible disclosure. Individual clinicians cannot fix that by opting out of model training.
Does removing the patient’s name make it safe to paste notes into AI?
Usually not. HIPAA’s Safe Harbor method requires removing 18 categories of identifiers, including dates more specific than the year, small geographic subdivisions, record numbers and unique characteristics. Clinical notes often contain many of these, as well as rare diagnoses or events that could identify someone. Properly de-identified data is not PHI, but casual redaction rarely meets the standard.
What happens if staff put PHI into an unapproved AI tool?
The organization should treat it as a possible breach. HIPAA presumes an impermissible disclosure is a breach unless a documented risk assessment shows a low probability that PHI was compromised. If it is a breach, affected individuals must be notified without unreasonable delay and within 60 days of discovery. Breaches affecting 500 or more people also require notice to HHS and, in some cases, the media.
Does the proposed HIPAA Security Rule update cover AI?
The January 2025 proposal does not single out AI, but its requirements, such as a written technology asset inventory, network mapping and more specific risk analysis, would apply to AI systems that handle electronic PHI. As of May 2026 it has not been finalized. Organizations should already include AI tools in their existing risk analysis.
The takeaway
Buying an approved AI tool with a signed BAA is the start of HIPAA compliance for generative AI, not the end. The quiet breach is happening in browser tabs and on personal phones, driven by overworked staff with a real need. Hospitals that only audit contracts will keep missing it. The ones that measure behavior and offer a better tool will not eliminate it, but they will know where they stand.
Does your organization know how much patient data goes into unapproved AI tools? How did you find out, or how would you?
