Why hospitals must avoid ‘pilot purgatory’

With hospitals looking to add new technology, including AI, health systems routinely seek partners to find innovative solutions.

Commonly, some health systems will undertake pilot programs to test a digital tool and see if it can help improve efficiency or patient care. But pilot programs can crash without the proper design and direction.

Some healthcare leaders argue against pilot programs, saying it’s better to simply commit to a new solution and assess quickly if it’s useful for the system’s needs.

Andy Miller, AARP’s senior vice president of innovation and product development, works with startups and larger companies in the AARP’S AgeTech Collaborative, which aims to develop new technologies to improve the lives of older Americans.

In a recent interview with Chief Healthcare Executive®, Miller talks about the problems when pilot programs aren’t designed properly.

“What I'm seeing is again good intentions around pilots,” Miller says. “We call this pilot purgatory, and what we always coach our startups on is that yes, we love pilots, but a pilot that doesn't have metrics or KPIs and trigger events before you start the pilot is a waste.”

Often, healthcare organizations are working with good intentions, but they are viewing the pilot program as “more of a learning journey,” Miller says. In some cases, organizations aren’t going to do anything with the startup company, he says.

If health organizations are going to do a pilot program, Miller says it’s better when there are goals that are clearly defined, even if they expect to see a return on investment quickly.

Miller says pilots are more likely to be successful if organizations identify several key metrics before moving forward. Miller says that a clear plan is beneficial to both the healthcare organization and the aspiring vendor hoping to land a long-term commercial deal.

Otherwise, health organizations may see little chance of returns, and tech companies aren’t likely to secure an ongoing relationship. And that can lead to frustration on both sides.

“The pilot purgatory thing is real,” Miller says.

Saying no to pilots

John Couris, president and CEO of Tampa General Hospital, said in a discussion onstage at the ViVE digital health conference in February that his health system doesn’t engage in pilot programs.

“We don't believe in pilots,” Couris said. “We believe in advanced work. We will not do pilots in our institution. We do advance work quickly. If we cannot scale it, we dump it, and if we can't move fast, we dump it. We in health care, when you sit on the provider side, we don't have the time to admire problems for extended amounts of time, and I certainly don't want to torture our partners with a ‘pilot.’”

In an interview with Chief Healthcare Executive after the discussion onstage, Couris elaborated on why his health system doesn’t engage in pilot programs.

“Pilots are just a way to say no to people without saying it,” Couris says. “I mean, honestly, people pilot things for years in our industry, and everybody in the audience was smiling and nodding their heads because they know exactly what I'm saying. It's a lived experience for probably everyone in that audience. No one likes it. We don't like it. They don't like it. It's frustrating.”

Tampa General Hospital has been working with fewer partners, but seeking deeper relationships with those partners, Couris says.

Some companies tout digital solutions that aren’t designed to address problems in health systems, he says, echoing a common frustration among hospital leaders.

“We want to work with people that fall in love with the problem and then build a solution,” Couris said. “A lot of times, when you sit in our seats, you meet a lot of people that have wonderful solutions, looking for a problem. But you have very few people who actually fall in love with the problem and then build the solution to actually solve the problem. So that's a big strategy of ours and a big philosophy of ours, and it's really worked quite well at sort of navigating through some of the challenges that the industry faces.”

Dr. Peter Slavin, the president and CEO of Cedars-Sinai, appeared with Couris on the ViVE stage. Cedars-Sinai engages in pilot programs, Slavin says.

“Before we launch a program on the full scale, we want to pilot it and test it and make sure that it's working properly,” Slavin said on the ViVE stage.

But he also echoed some of Couris’ sentiments.

“I guess at times we probably want to be polite to vendors and brush them off for a pilot,” Slavin said.

But he added, “There are also times where we're quite interested in a product, we want to see it on a small scale before we're willing to deploy it on a much larger scale.”

The right teams for pilots

Courtney Holladay, the vice president and chief learning officer of the University of Texas MD Anderson Cancer Center, offered some useful perspective on the system’s approach to pilot programs.

MD Anderson looks carefully at how the system’s teams are handling the rapid changes in the industry. In weighing a pilot program, Holladay told Chief Healthcare Executive® that the system will assess if a certain department is already adopting some new solutions before asking that team to take on another pilot.

“When we're looking to maybe introduce something new, and we need a pilot area, this is not the group to go ask to pilot,” Holladay said. “We need to look over here if that (team) has a little more capacity for that. So it's been helpful, especially when we're looking to pilot something.”

If health systems are going to try and undertake new initiatives, they can sometimes struggle to build support within the organization if the effort is designated a pilot program.

Shiv Sutaria, assistant chief medical information officer for Mass General Brigham Healthcare at Home, says he sees the potential for the development of more home hospital programs nationwide. Mass General Brigham has been an earlier pioneer in hospital-at-home programs.

In a recent interview with Chief Healthcare Executive®, Sutaria said that health systems that are considering launching home hospital programs shouldn’t develop them as pilot programs.

“Any new health systems out there who are wanting to do this program, they need to have the executive level buy-in to be successful. If you are going to try to pilot a program, it's not going to be successful,” Sutaria said.“It's good to do pilots for a lot of things, but you need to be committed,” he added. “This is not, like, ‘We will do this for six months and see what happens.’ This needs to be a strategic initiative so you could get the right resources, and you could set up your team on the ground for success.”

For tech companies, particularly startups looking to build relationships with health systems, Miller of the AARP says that organizations should be seeking key metrics for success in evaluating new solutions. Miller says some health organizations are clearly seeking returns in a short amount of time, or they are going to move on.

“They don't want to waste their time on pilots because they know that it's got to be leading to something, some material benefit, for them and their audience,” Miller says. “And so we love those folks, because they are going to put metrics in place up front.”