The three risks holding back healthcare AI | Viewpoint

Healthcare has an AI problem.

It isn’t that doctors aren’t implementing it. Around 80% of physicians say that they use AI tools professionally. And it isn’t that the technology doesn’t work. In fact, AI is showing tremendous promise in radiology, cancer detection, and drug development, among other disciplines.

The real issue is that AI is not being designed for healthcare; we are trying to redesign healthcare for AI. Organizations often adopt AI without the governance, structure, or workflow alignment needed to use it well. Perhaps even worse, many AI tools are being generated and implemented simply for the sake of having the latest AI features with the trendiest buzz words. There is no clear intention or goal as to the desired outcome, just vague promises of productivity or unproven cost savings.

If AI is going to fulfill its promise, it cannot be treated as an add-on or an experiment. It has to be implemented with the same rigor as any clinical intervention. That means proactive planning, clear accountability, and a defined purpose tied to patient outcomes. Without that foundation, even the most advanced tools will fall short.

There are three areas most likely to break down in AI implementation: governance, workflow misalignment and weak data. It’s true that each one represents a real risk and even potential failure point, but they are also where we need to focus if we want to get AI right.

Risk one: Governance gaps

AI tools are being introduced into clinical environments at such a rapid pace that ownership is often unclear. When something goes wrong, there is no single point of responsibility, only a series of assumptions across teams. In a clinical setting, any lack of clarity can be an urgent patient safety risk.

Unlike drugs or medical devices, which go through rigorous standardized approval processes, many AI tools are tested primarily by the companies that develop them. There is no universally accepted framework for validation, no shared definition of readiness, and often no formal oversight once tools are deployed.

Mistakes are always possible, and, without clear governance, those missteps are more likely to be repeated. When no one is explicitly responsible for monitoring performance or making corrections, issues persist and lessons go unlearned. Some organizations are trying to bridge this gap by adding new roles, like Chief AI Officer or Clinical AI Lead. This is progress, but fancy titles are not enough. Governance has to be truly integrated across IT, legal, clinical, and executive leadership, with shared accountability for how AI is implemented and maintained. In some cases, that means rethinking how teams are structured to support AI as a core capability, not an isolated initiative.

Done well, governance is not a barrier to innovation. Instead, it allows AI to be used safely, improved consistently, and trusted by providers, staff, payers, and patients.

Risk Two: Workflow misalignment

It is easy for AI tools to look great in demos or pilots, where all of the variables are controlled. But healthcare is far from a controlled environment. It’s not even a generic business environment. Healthcare has rules, red tape, and very specific processes. A pilot that doesn’t reflect real workflows, clinical procedures and time pressures is not a true test, and it won’t hold up in practice.

Organizations often invest significant time and resources into tools that quickly fail. Patched-on fixes and workarounds can’t hide the fact that they were never going to work in healthcare. They just weren’t built for the way care is actually delivered.

Leading organizations like Mayo Clinic take a different approach. They run structured pilots in controlled, but realistic environments, involve frontline clinicians early, and iterate before scaling. The goal is to prove that these tools work alongside doctors, for a specific purpose, within the constraints of real care delivery.

That distinction matters. Clinicians and end users cannot be an afterthought. They need to be involved from the beginning so tools are shaped around the way they work, not forced on them later.

Risk three: Weak data integrity

We have all heard it before. AI is only as good as the data behind it. AI can be a powerful engine, piecing together patterns and insights to support faster, more informed decision-making. But because it lacks the true critical thinking skills of a human, AI cannot create context or challenge what it is given, it can only reflect it. If the underlying data is flawed, incomplete, or misleading, the output will be too, no matter how advanced the model is.

Weak data integrity can lead to serious harm. In fact, nearly two-thirds of healthcare organizations say that inaccurate data impacts clinical decision-making. AI-driven patient identity verification systems might incorrectly validate patient ages, predictive models might contain racial bias, or medication management tools might miss critical allergy data. Small breakdowns in accuracy can have devastating consequences.

Healthcare data must be real, accurate, secure, and usable across systems. That means not only ensuring quality at the point of input, but also maintaining integrity as data moves between platforms, organizations, and use cases. Without that, AI systems are operating on unstable ground.

How we get it right

AI isn’t going anywhere, but it also won’t move healthcare forward in the dramatic ways everyone is hoping for without a change in both leadership and culture. Organizations are most likely to succeed with clinician buy in. People have to feel comfortable raising concerns and offering feedback as part of the process.

Success also means bringing in qualified, vetted external expertise to inform insights and decision-making. The pace of emerging technology in digital health is simply too fast for most organizations to stay current on regulatory shifts, liability frameworks, and clinical safety standards simultaneously. The organizations succeeding right now are the ones honest enough to name that gap and fill it strategically.

In healthcare, AI is being treated as a passing trend instead of the structural, world-changing technology it is meant to be. Until we integrate AI with full governance, for real workflows, with open and complete data behind it, it will continue to be a problem in healthcare environments. Once we move past those risks, AI will finally be an advantage.

By Erkeda DeRouen, MD, CEO and Principal Consultant, Digital Risk Compliance Solutions