AI healthcare tools with bias need to be pulled

Avishkar Sharma, director of artificial intelligence at Jefferson Health and an AI researcher at Stanford University, said he sees a “clinical AI renaissance” in the future.

In a presentation at the HIMSS Global Health Conference & Exhibition, Sharma talked about the potential of AI and the need to closely examine AI tools.

But he also offered words of caution about incorporating AI into clinical practice ethically and safely.

“Be the chief ethical officer of your data,” Sharma said in a presentation at the conference. “Do not wait for a federal mandate to tell you how to audit your algorithm for bias. Establish your own cross functional community today.”

If close scrutiny reveals an AI algorithm is biased, he suggested that it should be treated like a contaminated batch of propofol, a widely used anesthetic.

“You would pull it immediately,” he said. He said the source of the contamination would be investigated, and health systems need to take similar steps if they find bias in their AI tools.

“If an AI system isn't equitable, it is not clinical,” Sharma said. “Ethical AI is not a global checkbox. It is a patient safety obligation.”

Concerns about inequities and racial bias in medicine are well-founded.

The Organ Procurement and Transplantation Network prohibited the use of a biased test of kidney function in determining eligibility for transplants. Researchers found the tests improperly measured kidney function in Black patients, which led to delays in transplants. This week, researchers found efforts to help Black patients gain credit for lost time on the transplant list are having success, according to data published in JAMA Internal Medicine.

Healthcare leaders have also warned about the potential for AI algorithms and tools such as chatbots to reflect and even expand racial bias.

ECRI, a nonprofit organization focused on patient safety, publishes an annual list of the top 10 healthcare technology hazards, and the misuse of chatbots in medicine placed at the top of this year’s list.

Dr. Marcus Schabacker, president and CEO of ECRI, told Chief Healthcare Executive® in an interview last month that he’s worried about chatbots amplifying bias in medicine.“If there's bias introduced somewhere in this process, then that bias will be exaggerated the more the chatbot is used,” Schabaker says.

Many medical studies are based largely on participants who are young, white men. If clinicians use chatbots for research for issues affecting women, children, or seniors, and the chatbot is looking at data from a study limited to white men, “then automatically you have a bias,” he says.

“You're going to get a wrong answer,” Schabacker says. “And that bias, depending on how you use it … might be perpetuated through the use and it confirms your own bias.”

Schabacker has also expressed concerns about insufficient governance of AI in large health systems. In 2025, ECRI listed inadequate governance of AI solutions as a top threat to patient safety.

“We continue to have concerns because, particularly the larger healthcare providers … they don't have a good governance structure to oversee the utilization of AI,” he told Chief Healthcare Executive in a 2025 interview. “And you don't have to stretch your imagination too far to think that if you do not have good governance in place, these new tools can be abused or used in the wrong context.”

Schabacker said in 2025 that if technologies haven’t been thoroughly tested, doctors could use solutions that could provide “a very wrong recommendation.”

“When you tend to rely on those recommendations, they become very quickly decision-making tools, and that's what really concerns us,” he says. “And there's very, very little oversight.”

Researchers with the Stanford School of Medicine tested large language models to see if they were providing biased information. The findings, published in a Digital Medicine paper in 2023, indicated that chatbots provided information reflecting racial bias. “Every LLM model had instances of promoting race-based medicine/racist tropes or repeating unsubstantiated claims around race,” the authors wrote.

The New York City Health Department established a Coalition to End Racism in Clinical Algorithms. The city health department has worked with New York health systems to revise algorithms to remove bias.

The New York City health department teamed with The Digital Medicine Society and The SCAN Foundation to produce a toolkit to reduce algorithmic bias in health care.