Board Case Study · Healthcare (Provider System)
The diagnostic AI called the scan clear. The radiologist overrode it and was right. The data shows clinicians now defer to the model 94% of the time. Who's accountable when it's usually right?
The situation
A nonprofit hospital system deployed an FDA-cleared diagnostic-imaging AI across its radiology network to manage rising volume and a radiologist shortage. In a recent case the model marked a scan as clear; an experienced radiologist overrode it and caught an early malignancy. The save was celebrated, until the analytics surfaced the pattern beneath it.
Clinicians now defer to the model's read 94% of the time, and override rates fall the longer the tool is in use. Automation bias is measurable. The model's performance also degrades on patient populations underrepresented in its training data, and the system serves a diverse community.
The board's quality committee must weigh patient safety, malpractice and corporate-liability exposure, informed-consent questions about AI in the diagnostic path, and the reality that the tool genuinely improves throughput. Removing it isn't obviously safer; trusting it blindly clearly isn't either.
“The model is right more often than any one of us. So when a clinician overrides it, or doesn't, who is accountable for the outcome?”
Chair, Quality & Safety Committee
The decision on the table
The full case, scenario architecture, board materials, and facilitation notes, is shared with boards and partners on request.