A hospital AI governance committee decides which AI tools the organization uses, sets how they are validated and monitored, and can pause or retire a tool that causes harm. To work, it needs three things: real authority over purchasing and deployment, a complete inventory of AI in use, and a tested process to switch any model off.
In April 2020, a clinical AI governance committee at Michigan Medicine made a decision most hospital committees have never made. It switched off a widely used sepsis alert. The model had started firing spuriously as COVID-19 changed who was in the hospital. The committee saw the problem, had the authority to act and acted. Researchers later described the episode in the New England Journal of Medicine as a case study in dataset shift.
The notable part is not the model’s failure. Models fail. It is that someone was watching, knew the model existed, and could turn it off.
Most hospitals now have something called an AI governance committee. According to a federal data brief from ASTP/ONC published in September 2025, about two thirds of US hospitals that used predictive AI in 2024 said a specific committee or task force was responsible for evaluating it. That sounds reassuring until you ask what those committees can actually do. My argument is that an AI governance committee is only as real as three capabilities: authority to say no, an inventory of what it governs and a kill switch someone has tested. Without them, it is a discussion group with minutes.
Why do hospitals need an AI governance committee?
Because AI is arriving through too many doors for any single department to see.
Some tools come through formal purchasing, such as a radiology triage product. Many arrive inside software the hospital already owns. The same ASTP/ONC brief found that 80 percent of hospitals using predictive AI used models supplied by their EHR developer, often available with a configuration change rather than a contract. Others are built in-house by research teams. And some arrive through staff themselves. In a Wolters Kluwer Health survey of more than 500 healthcare workers in December 2025, about 40 percent said they had encountered unauthorized AI tools in their organization.
Regulators and accreditors are also raising expectations. Since May 2025, federal rules under Section 1557 of the Affordable Care Act have required covered entities to make reasonable efforts to identify and mitigate the risk of discrimination from patient care decision support tools they use. The Joint Commission and the Coalition for Health AI released voluntary guidance in September 2025 whose first recommendation is formal AI policies and a governance structure. In Europe, hospitals that use high-risk AI systems will have legal duties as “deployers” under the AI Act.
No existing committee was built for this. Pharmacy and therapeutics committees know drugs. Value analysis committees know devices and cost. IT steering committees know infrastructure. AI cuts across all three.
What authority should an AI governance committee have?
The committee’s authority is the first test of whether it is real. There are three levels, and many committees sit at the lowest.
Advisory. The committee reviews tools and makes recommendations. Departments can proceed regardless. This is where most committees start, and where many stay.
Gatekeeping. No AI tool that affects patient care or uses patient data can go live without committee approval. Purchasing will not issue a contract, and IT will not switch on a feature, without a committee sign-off. This is the minimum for governance to mean anything.
Operational. The committee can also require monitoring, set conditions on use and pause or retire a tool already in production. This is where the kill switch lives.
Authority is not a line in a charter. It is whether the chief financial officer’s team will hold a contract that lacks committee approval, and whether a department chair who wants a tool will accept a “not yet.” Boards should make this explicit. As we argued in the translation problem in the hospital boardroom, boards often approve structures without asking what they can actually stop.
Membership follows from the job. The Joint Commission and CHAI guidance suggests people from compliance, IT, clinical programs, operations and data privacy. Add a data scientist who can read a validation report, frontline clinicians who will use the tools, and someone who represents patients. Keep it small enough to decide.
What should be in an AI inventory?
You cannot govern what you cannot list. An AI inventory is the unglamorous core of the whole effort, and it is usually incomplete.
For each tool, a useful inventory records:
- What it does and who uses it, in plain language.
- Where it came from: EHR vendor, third-party product or internal build.
- Its risk tier: whether it affects diagnosis or treatment, communicates with patients, or supports operations only.
- Its regulatory status: FDA-cleared device, exempt decision support or neither.
- Validation evidence: vendor studies, local validation results and the chosen threshold.
- A named owner: a clinical leader accountable for its use.
- Its monitoring plan: what is measured, how often and by whom.
- Its data flows: what patient data it uses and whether a vendor holds it under a business associate agreement.
- How to switch it off.
Building this means looking beyond purchasing records. EHR configurations, research projects and departmental software subscriptions all hide AI. The ASTP/ONC data suggest accountability is often spread across many hands: 74 percent of hospitals using predictive AI reported multiple entities responsible for evaluation, and a quarter reported four or more. An inventory is how scattered accountability becomes visible.
What does an AI kill switch look like in practice?
A kill switch is a documented, tested process to stop a tool quickly and safely. It has four parts.
Triggers. Written conditions that prompt review or suspension: a safety event linked to the tool, a drop in a monitored metric below a set threshold, a vendor notice of a defect, or a significant change in the data the model depends on.
Authority. A named person, such as the chief medical information officer, who can suspend a tool immediately without waiting for the next committee meeting, with the committee reviewing the decision afterward.
Mechanics. IT knows exactly how to turn it off, whether that means a configuration change, disabling an interface or removing an alert, and has done it at least once in testing.
Fallback. Clinicians know what to do instead. If a sepsis alert disappears, the old screening process has to be ready. A tool that cannot be switched off without disrupting care is a tool the organization no longer controls.
This is where most committees are weakest. They spend their energy on approval and very little on retirement. We made a related argument about drug regulation in fast approval isn’t the danger, slow withdrawal is. The same logic holds inside a hospital.
A worked example: governance in one quarter
Here is a hypothetical regional health system with four hospitals.
In January, the new AI governance committee starts an inventory. It expects about 15 tools. It finds 41, including 12 predictive models switched on inside the EHR, an ambient scribe pilot in two clinics, a readmission model built by the research team and several departmental subscriptions to AI writing tools.
It tiers them. Six affect diagnosis or treatment directly. For those, it names clinical owners and asks for validation evidence. Two have none locally. One, a deterioration model, is moved to silent mode for eight weeks while the informatics team checks performance.
In March, monitoring shows that an EHR-supplied no-show prediction model, used to overbook clinic slots, performs worse for patients whose preferred language is not English. The model’s owner and the committee agree to suspend its use for overbooking pending a fix. IT switches it off in an afternoon because the process had been written down in February.
None of this required new technology. It required a list, a set of owners and an agreed way to say stop.
Is an AI governance committee just more bureaucracy?
The strongest objection is that committees slow things down and diffuse responsibility. Innovation teams complain that approval takes months. Clinicians see another form to fill. And a committee can become a place where risk is discussed rather than managed, where approving a tool feels like making it safe.
There is truth in this. A committee that reviews a scheduling assistant with the same rigor as a diagnostic model will drown. A committee that meets monthly cannot act on an urgent safety problem. And a committee is not a substitute for accountable owners.
The answer is proportion and speed, not abolition. Tier the review: light for low-risk operational tools, full for anything that touches diagnosis, treatment or patient communication. Give a named executive emergency authority. Set target times for decisions. Fold AI governance into existing quality and safety structures where possible rather than building a parallel bureaucracy. The alternative, no central view at all, is not faster. It just moves the delay to the incident review.
What this means for boards, executives and clinicians
For boards, ask three questions at the next meeting. How many AI tools do we use? Who can switch one off? When did we last do it? If nobody can answer, the governance is not yet real.
For executives, give the committee gatekeeping authority in purchasing and IT change control, fund the inventory and monitoring work, and name a person with emergency suspension power. Our piece on buying digital health covers the contract terms that make governance enforceable.
For clinicians, report AI problems the way you would report a medication error. The committee can only act on what it hears about. And if you are using a tool the committee does not know about, tell them.
Frequently asked questions
Who should sit on a hospital AI governance committee?
A practical committee includes clinical leaders, nursing, the chief medical information officer or equivalent, IT and security, privacy and compliance, legal, quality and patient safety, and a data scientist who can assess validation evidence. Frontline users and a patient representative add perspective. The Joint Commission and CHAI guidance suggests compliance, IT, clinical, operations and privacy as a core.
How is AI governance different from IT governance?
IT governance focuses on infrastructure, security, cost and system reliability. AI governance adds clinical questions: whether a model is accurate for local patients, whether it performs fairly across groups, how it changes clinical decisions and when it should be retired. It also needs ongoing monitoring of performance, which ordinary software governance rarely requires.
Do small hospitals need an AI governance committee?
They need the functions, even if not a separate committee. A small hospital can assign AI oversight to an existing quality or medical executive committee, keep a simple inventory, name an owner for each tool and write down how to switch each one off. Many small hospitals rely on EHR-supplied models, which still need local owners and monitoring.
What is the Joint Commission’s guidance on AI governance?
In September 2025, the Joint Commission and the Coalition for Health AI released voluntary guidance on the responsible use of AI in healthcare. It sets out seven elements, starting with AI policies and governance structures and including privacy and transparency, data security, quality monitoring, safety event reporting, bias assessment and staff training. It does not currently affect accreditation.
The takeaway
An AI governance committee is easy to create and easy to hollow out. The test is not its charter or its membership list. It is whether it knows every AI tool in the building, whether a contract can be stopped without its approval, and whether someone can switch off a model this afternoon without a crisis. Hospitals that can answer yes to all three have governance. The rest have a meeting.
Does your AI governance committee have the authority to turn a model off? Has it ever used it?
