John Halamka, M.D., M.S.

home

authors

john-halamka-md-ms

, Siddhartha Mukherjee, M.D., called cancer the “Emperor of All Maladies.” The term is spot on. The disease commands the attention of all who come in contact with it, whether they be patients, clinicians, or technologists, claiming the lives of more than 8 million worldwide and disrupting the lives of millions more. That’s not to suggest that we aren’t making any progress. The latest statistics from the 

 state that the U.S. cancer death rate has dropped by 27% in 25 years. But that still means about 1.7 million new cancer cases and over 606,000 deaths are expected this year alone. Many stakeholders are hoping that advances in artificial intelligence and machine learning will make a significant dent in these numbers.

, we discussed the value of machine-learning algorithms to assist in the diagnosis of melanoma and to improve colorectal cancer screening. The software used to help detect skin cancer takes advantage of neural networks while the colorectal screening system, commercially available as ColonFlag in Europe and LGI Flag in the United States, uses a machine-learning tool called extreme gradient boosting. There have been similar advances in several other specialties, including breast and lung cancers.

To make sense of the role of ML in breast cancer risk assessment, it helps to understand how breast density fits into the picture. Women with denser breasts, which contain more fibrous and glandular tissue, are at greater risk of the malignancy than women with fatty breasts. Traditionally, there have been tewo risk models to help clinicians in this area: the Gail model and, more recently, the Tyrer-Cuzick (TC) model. The latter has been incorporated into the Gail model to take into consideration a woman’s breast density. The TC model is now considered the clinical standard. However, because assessing a patient’s breast density can be subjective, it may be possible to improve the assessment using deep learning to analyze mammograms for subtle differences not detectable by the naked eye. 

, a Ph.D. candidate from the Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, and his colleagues have developed an algorithm that looks at full-field mammography images to assist clinicians in 

Yala and his colleagues used data from patients’ EHRs and questionnaires to evaluate their likelihood of developing breast cancer using 3 different models. One used traditional risk factors, including the TC model and logistic regression, the second used a convolutional neural network (CNN) to analyze mammogram images, and the third combined both approaches. The analysis found that the CNN was much better than the TC model for predicting breast cancer; combining CNN with TC was even better. One of the surprising findings was that women who did not have dense breasts but were considered at high risk based on this more sophisticated risk assessment model were almost four times as likely to have developed cancer, when compared to those with dense breasts and model-assessed low risk.

Machine learning is also helping pathologists as they try to interpret biopsy slides and determine whether or not a patient’s breast cancer has metastasized to nearby lymph nodes. 

 from Google AI Healthcare, and associates applied an algorithm called Lymph Node Assistant (LYNA) to two separate data sets of pathology slide images. The first set of slides was also reviewed by one of two pathologists while slides from the second data set were interpreted by two pathologists. Comparing LYNA to the skills of a practicing pathologist, investigators found an area under the curve (AUC) of 99.3% for nodal metastasis present or absent for the first data set. (AUC, also called receiver operating characteristic (ROC) is a metric that helps to distinguish between two diagnostic groups, in this case slides that indicate metastasis vs no metastasis). Pathologists returned an AUC of 96.6% when they were allowed to take as much time as they needed to arrive at a diagnosis. That AUC dropped to 81% when a real-world scenario was put in place: They were only allowed one minute per slide. Liu and associates concluded that: “Artificial intelligence algorithms can exhaustively evaluate every tissue patch on a slide, achieving higher tumor-level sensitivity than, and comparable slide level performance to, pathologists. These techniques may improve the pathologist’s productivity and reduce the number of false negatives associated with morphologic detection of tumor cells.” In a separate study that used LYNA, 

 demonstrated that combining the algorithm with the services of a human pathologist resulted in more accurate detection of micrometastases than either the pathologists on their own or LYNA on its own.

One of the challenges facing pathologists is their lack of consistency in making a cancer diagnosis. ML—enhanced algorithms may eventually solve this problem by supplementing pathologists’ judgment. 

 addressed this challenge using a convolutional neural network to improve the diagnosis of invasive breast cancer in whole slide images. Invasive disease was defined as the spread of the tumor beyond the breast’s milk ducts or lobules. The researchers used a ConvNet classifier and validated it with images from three different institutions. They then evaluated it using pathologic and normal cases from the Cancer Genome Atlas and University Hospitals Case Medical Center in Cleveland. When compared to pathologists’ findings, the classifier generated a positive predictive value of 71.6% and a negative predictive value of 96.8%. One of the criticisms of ML—based diagnostic tools is that they are often trained on a single data set and may not be generalizable to the larger patient population. The research team addressed this concern by using 2 different data sets. Comparing the two resulted in highly correlated performance measures (r >/–0.8).

Technologists are hoping that AI-based diagnostic tools like this will help address the epidemic of diagnostic errors we face in clinical medicine. It’s estimated that a delayed diagnosis is one of the four common causes of diagnostic errors, which also include missed diagnosis, misdiagnosis, i.e. incorrectly diagnosed disease, and overdiagnosis. Unfortunately, technology can’t solve all these problems. A report from the Institute for Health Improvement and CRICO, the risk management foundation of the Harvard Medical Institutions, points out that there are over 100 million outpatient referrals to specialists annually, but up to half of these referrals are never completed. A single missed referral can easily lead to a delayed diagnosis and a malpractice claim. 

: “Of malpractice claims related to missed or delayed diagnosis in the ambulatory setting, almost half involve failure to follow up, many of which involve problems with specialist referrals.”

Luke Sato, M.D., chief medical officer at CRICO, explains that the diagnostic errors that lead to malpractice claims fall into two broad categories: cognitive errors and system-based errors. Although diagnostic errors may result from the failure of a physician to order a colonoscopy, for instance, or refer a patient to the appropriate specialist, CRICO has found that errors also result from failing to “close the loop,” for example, a fault in the healthcare system that somehow interferes with the referral process. How serious is this problem? Sato points out that although 

 say that they include the patient’s history and the reason for referring patients to a specialist “always” or “most of the time,” fewer than 35% of specialists say they receive this information. In some cases, the technology is unavailable within the EHR to allow clinical or administrative staff to properly close the loop. In other cases, it is available but not well utilized. The IHI/CRICO white paper discusses a long list of communication breakdowns responsible for the problem, but part of the problem is a lack of accountability. In the U.S. healthcare ecosystem, there is no one consistently held responsible for scheduling the process. Equally troubling is the fact that no one is consistently in charge of knowledge management.

Tackling Cervical Cancer with Deep Learning

While diagnostic errors remain a stubborn, expensive problem, there are many success stories worth telling. The prevention of cervical cancer in the United States is one of the genuine triumphs in oncology.  In the 1940s, the disease was a major cause of death among women of childbearing age in the U.S., but since the Pap smear was introduced in the 1950s, invasive cervical cancer has dropped dramatically: From 1955 to 1992, its incidence and death rate dropped by more than 60%, Unfortunately, the disease continues to be a major cause of death and suffering in poorer countries that can’t afford routine Pap smear screening. Worldwide, there are about 500,000 cases, and 

 occur in low and middle income countries. A recent data analysis suggests that an automated visual evaluation of cervical images, developed with the help of a convolutional neural network, can have a significant impact on this worldwide problem. (Two video tutorials are available that provide a basic understanding of how neural networks work. One was created for an episode of PBS’s 

Currently, clinicians in these poorer countries use an acetic acid test to estimate if a woman has precancer of the cervix. The test involves applying acetic acid to the cervix to see if the tissues turn white, which suggests precancer or cancer. While the test is inexpensive, its interpretation is subjective, and isn’t very accurate in differentiating between precancer and more common minor abnormalities. That in turn results in overtreatment and undertreatment. 

 Ph.D., with the Intellectual Ventures Global Good Fund, Bellevue, Washington, and his colleagues, analyzed a data set of cervical images from over 9,000 women in Costa Rica who had been screened for the malignancy and followed for seven years.

The automated visual evaluation algorithm was designed to accomplish two tasks:  It located the cervix on the camera image, and it predicted the likelihood of the patient having cervical intraepithelial neoplasia (CIN2+), a group of precancerous lesions. Hu and his colleagues found that the machine-learning enabled algorithm was more accurate than manually performed visual inspection of the cervix and more accurate than Pap smears.

Radiologists have two invaluable tools to help them detect diagnostic clues in the images they review: Their eyes and their clinical training.  But the human eye obviously has its limitations.  There are millions of pixels in a typical X-ray, making it virtually impossible to notice subtle abnormalities that can be easily detected by a machine-learning based algorithm. The computer’s advantage has been observed in numerous diagnostic settings, including lung cancer.

Although screening of high-risk patients with computed tomography (CT scans) has been shown to reduce the death rate in lung cancer patients, many cancers are still missed due to misjudgements in reading and interpreting the scans. Several software systems have been used to help address this problem, including those that rely on deep learning neural networks. 

, a Ph.D. candidate with the Department of Systems & Technology, Auburn University in Alabama, and his colleagues, have created a two-tier system to analyze CT scans, looking for suspicious pulmonary nodules that suggest lung cancer. One neural network was used to locate nodule candidates worth investigating and the second network helped eliminate false positive readings of the CT scans. Gruetzemacher and his associates used a data set derived from National Cancer Institute’s Cancer Imaging Archive. They explain in their report: “Deep learning offers unique advantages over traditional methods for nodule detection. Rather than relying on mathematical techniques or hand-crafted features, deep learning enables learning internal feature representations directly from the input data.” Put another way, neural networks teach themselves how to recognize nodules by evaluating thousands of images. Their two-tiered system worked with a detection rate of 89.3%, with only 1.7 false positives per scan. 

Machine learning will never replace an experienced clinician’s diagnostic skills, but as these research projects demonstrate, it is slowly emerging as a valuable complement that will likely save lives.

Paul Cerrato has more than 30 years of experience working in healthcare as a clinician, educator, and medical editor. He has written extensively on clinical medicine, electronic health records, protected health information security, practice management, and clinical decision support. He has served as editor of Information Week Healthcare, executive editor of Contemporary OB/GYN, senior editor of RN Magazine, and contributing writer/editor for the Yale University School of Medicine, the American Academy of Pediatrics, Information Week, Medscape, Healthcare Finance News, IMedicalapps.com, and Medpage Today. HIMSS has listed Mr. Cerrato as one of the most influential columnists in healthcare IT.

John D. Halamka, M.D., leads innovation for Beth Israel Lahey Health. Previously, he served for over 20 years as the chief information officer (CIO) at the Beth Israel Deaconess Healthcare System. He is chairman of the New England Healthcare Exchange Network (NEHEN) and a practicing emergency physician. He is also the International Healthcare Innovation professor at Harvard Medical School. As a Harvard professor, he has served the George W. Bush administration, the Obama administration and national governments throughout the world, planning their healthcare IT strategies. In his role at BIDMC, Dr. Halamka was responsible for all clinical, financial, administrative and academic information technology, serving 3,000 doctors, 12,000 employees, and 1 million patients.

 Get the best insights in digital health 

Replacing the Old School with New-School AI in Medicine

Mental Health Apps: Do They Work? Are They Safe?

Tackling the Misdiagnosis Epidemic with Human and Artificial Intelligence

Articles

Stakeholders hope AI, machine learning will significantly decrease death toll.

Will AI Improve Cancer Diagnosis and Treatment?

Software as a medical device (SaMD) and U.S. Food and Drug Administration (FDA) 510(k) clearance might not be the most familiar terms to all health system leaders, physicians and nurses who use computers, but if they hope to take advantage of the latest artificial intelligence (AI)-enabled programs, they will need to keep up. The most recent FDA proposed regulatory framework on SaMD is likely to transform patient care in the next few years. Understanding how the guidelines fit into the picture will enable clinicians and their organizations to take full advantage of the machine learning-based diagnostic and treatment algorithms now being introduced.

To make sense of SaMD, it helps to understand the FDA’s expanded definition of a medical device. In the not-too-distant past, a medical device was a physical object — an EKG machine or a blood pressure cuff, for example. But the term now encompasses software used to assist in the diagnosis, treatment, mitigation or prevention of disease. As we explained in 

The Transformative Power of Mobile Medicine

, the FDA believes that most mobile apps intended to diagnose or treat disease are “medical devices under federal statute.” On the other hand, apps or other types of software that are either designed to provide wellness services or that receive, store, transmit and offer more general health information are not considered medical devices.

Software that the agency says falls into the SaMD category includes:

Mobile apps that transform a mobile platform into a regulated medical device and therefore are mobile medical apps: These mobile apps use a mobile platform’s built-in features such as light, vibrations, camera or other similar sources to perform medical device functions (e.g., mobile medical apps that are used by a licensed practitioner to diagnose or treat a disease).

Mobile apps that connect to an existing device type for purposes of controlling its operation, function or energy source and therefore are mobile medical apps: These mobile apps are those that control the operation or function (e.g., change settings) of an implantable or body worn medical device.

Mobile apps that are used in active patient monitoring or analyzing patient-specific medical device data from a connected device and therefore are mobile medical apps.

Although the FDA’s position on SaMD was designed to protect the public from unscrupulous vendors who create ineffective or unsafe digital tools, since publishing these regulations, the agency has come to realize that they might be too restrictive in some situations and could actually damper healthcare innovation. That’s especially the case when it comes to machine learning-based software. A closer look at the FDA clearance/approval process helps explain why this problem has arisen.

Medical device applications are handled with three processes:

 seeks to gain clearance of a device that is at least as safe and effective as a legally marketed device that’s already available to the public. 

, on the other hand, is a more stringent review process required of Class III medical devices, which support or sustain human life or “which present a potential, unreasonable risk of illness or injury.” De Novo classification applies to novel medical devices “for which general controls alone, or general and special controls, provide reasonable assurance of safety and effectiveness for the intended use, but for which there is no legally marketed predicate device.” Unfortunately, AI and machine-learning algorithms do not easily fit into any of these 3 classifications, which has forced FDA to develop a new approach, which it has published as 

Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)

The need for new regulations can be summed up in two words: adaptive technology. The current regulations were designed to handle “locked” technology, such as software whose functioning does not change. Because many machine-learning systems are self-learning, by definition the software does change, adapting to new input from training and test data sets. In 

, these software systems “have the potential to adapt and optimize device performance in real-time to continuously improve healthcare for patients.”

To cope with this new situation, the agency is proposing what it calls a “predetermined change control plan” for device developers who are seeking premarket clearance. Submitters would have to provide FDA with a plan that spells out the types of anticipated modifications they see coming down the road, as well as the methodology they plan to use to implement these changes in a controlled manner. The ultimate goal of such a plan is to manage patients’ risk from the software. Vendors who agree to this proposed arrangement would be expected to commit to transparent, real-world performance monitoring of their AI and ML software. They would also be willing to provide periodic updates to the agency. According to FDA, this new approach to SaMD would allow the regulator to embrace the iterative improvement power of AI- and machine learning-based software as a medical device, while assuring patient safety.

To understand how the proposed regulatory framework world work in the real world, one first needs to grasp two rather obscure terms that the agency has introduced: SaMD pre-specifications (SPS) and algorithm change protocol (ACP). SPS refers to the 

 that a vendor plans to implement in its software. In other words, the company has to spell out what its algorithm is supposed to become as it learns from its new inputs, namely the new training and testing data that it encounters. ACP describes the 

 the vendor plans to use to make sure the revised algorithm performs as expected. The guidelines explain: “The ACP is a step-by-step delineation of the data and procedures to be followed so that the modification achieves its goals and the device remains safe and effective after the modification.” The agency envisions four components to this change protocol, as outlined in the Figure.

These proposals might sound rather vague without a real-world example of how they might apply to an actual AI/ML system. Imagine a mobile app that allows a smartphone to take a photo of a skin lesion so that a physician can make a clinical judgement about whether it’s a normal mole or skin cancer. Since the app drives clinical management of a serious healthcare problem, it’s considered a medical device that falls within the FDA’s jurisdiction. If a vendor decides it wants to modify the app, it first has to explain what changes it plans to make — the SPS portion of the review process. The FDA framework document gives possible modifications. For instance, the algorithm may be exposed to a new data set with images of skin lesions with the goal of improving its sensitivity and specificity. A more sensitive algorithm is more likely to detect true positive findings, such as how the number of photos identified as skin cancer that really 

 skin cancer would increase as a result of the new training. The software would be more specific if it succeeded in identifying more true negatives, such as how the number of photos labeled as not skin cancer that really aren’t skin cancer would increase.

The change protocol or ACP for this dermatologic app would involve the manufacturer submitting to FDA its method for making these improvements. The developer would have to explain how it generated the data set, discuss reference standard labeling and provide a comparative analysis and performance requirements, including sensitivity and specificity. If the vendor could collect the needed data and demonstrate its improved sensitivity and specificity, and it updated its labeling and communicated the improvements to its audience, “the modified algorithm that ‘learned’ on real-world data can be marketed without additional FDA review.”

On the other hand, if this same hypothetical app manufacturer wanted to sell this mobile app to patients to use on their smartphones, it would face a different regulatory scenario. The vendor would direct patients to follow up with a dermatologist after receiving a preliminary analysis of a malignancy. And most important, because the app now introduces new risks that were not addressed in the current SPS or ACP, the vendor might have to obtain a new premarket submission or application and submit an updated SPS and ACP.

The agency hopes that by outlining these two scenarios — one in which the company requires additional FDA clearance or approval and one in which it does not &mdash; it will convince developers that it’s becoming more flexible and is fostering innovation. The proposed framework was released in April 2019 for public comments; the comment period closed on June 3. 

medical devices, not just software that acts as a medical device. Others asked for better definitions of machine learning and more clarity on terms like “locked down” and “continuous learning” algorithms. Many wanted the agency to provide more examples of how changes in AI/ML software might trigger an expectation of premarket review. Allison Fulton, a lawyer with the Life Sciences and FDA team at SheppardMullin, believes that, “While the industry largely encourages FDA to react quickly to evolving technology so as not to stifle innovation, the agency will have to take time to more clearly define foundational terms in this complex area and clarify its expectations for software developers and medical device companies.”

It’s impossible to predict how the FDA will respond to all these comments. And the departure of Scott Gottlieb, M.D., as commissioner adds to this uncertainty. But it seems the policies and philosophy put in place by Gottlieb may well continue to steer the ship. According to 

Aaron Josephson, who spent a decade at FDA

, “… I do not expect the tone and agenda to change with whoever is named acting commissioner or is ultimately confirmed to replace Dr. Gottlieb. [Ned Sharpless, M.D., is now acting Commissioner.] During my time at the FDA, I worked with the team of staff who helped Dr. Robert Califf transition into his role as commissioner. Similarly, I was on the team of staff who aided the Trump administration's early FDA appointees (before Dr. Gottlieb was confirmed) in understanding the agency and prepared Dr. Gottlieb for his return to the agency. During both transition periods, the agency's core business — reviewing applications for new medical products and protecting marketed medical products and food &mdash; continued without issue thanks to the efforts of the FDA’s talented career staff and the statutory mandates FDA is required to satisfy.”

One reason to be optimistic about the FDA’s position on SaMD is the fact that much of SaMD’s future has already been decided by Congress with the passage of the 21st Century Cures Act, passed in December 2017 to encourage innovation. Section 3060, which deals specifically with regulation of medical devices, “expressly excluded from the definition of medical device five distinct categories of 

,” according to one expert. After the Cures Act was passed, the FDA released several guidelines to help developers and clinicians understand its implications, including a set of guidelines on SaMD.

Attorneys with Arnold & Porter explain that the guidelines established a three-step clinical evaluation process that “emphasizes that the level of evaluation and independent review should be commensurate with the risk posed and encourages manufacturers to use continuous monitoring to understand and modify software based on real-world performance.” The key word here is risk. While entrepreneurs planning to invest in SaMD would be wise to seek the advice of lawyers with expertise in the intricacies of FDA regulations, the agency is making it clear that it wants to give developers more room for innovation, especially if their products pose little risk of harming patients.

Replacing Old-School Algorithms with New-School AI in Medicine

Paul Cerrato, M.A., and John Halamka, M.D., M.S., analyze the future of software as a medical device.

How Will New FDA Guidelines Affect Digital Health?

The toll that depression, anxiety, psychosis and related disorders have taken on society is immeasurable. One can, of course, quote 

: About 46 million U.S. adults experience a mental disorder in any given year; 18% of adults experience an anxiety disorder; and 7% have had at least one major depressive episode in the past year.

While these figures are disturbing, the full impact of mental disease really strikes an emotional chord when you know friends, family or business associates who battle these problems. These disorders wreck lives by destroying promising careers, ending marriages, threatening domestic abuse and much more. Unfortunately, with so many individuals suffering from psychiatric problems, it is virtually impossible for every affected patient to receive the one-on-one professional care they need. There just aren’t enough mental health professionals available.

With that in mind, many patients are turning to online and mobile health (mHealth) resources for help. Similarly, several psychiatric experts are taking a closer look at these digital tools to evaluate their potential value, as well as any possible adverse effects.

A Closer Look at Mental Health App Evidence

The evidence on mental health apps and chatbots is mixed. 

, Ph.D., with the Department of Psychiatry and Behavioral Sciences at the Stanford School of Medicine, and her associates have studied 

, a conversational, text-based agent that uses the principles of cognitive behavioral therapy (CBT) to assist users experiencing depressive symptoms. They performed a randomized controlled trial (RCT) to compare the chatbot to a control group who had access to an ebook on mental health.

They found that the program delivered measurable improvements on the nine-item Patient Health Questionnaire, the seven-item Generalized Anxiety Disorder scale and the Positive and Negative Affect Scale among college students. It is important to note, however, that these students had not been officially diagnosed with depression or anxiety.

 uses natural language processing to create a conversational “counselor” that makes users comfortable revealing some of their most personal thoughts. Many patients with depressive symptoms apparently like this approach because they are too embarrassed discussing these issues with a real human being. It is estimated that as many as three out of four college students do not get the medical services they need, and it is not because the services are unavailable or expensive. 

 approximately 70% of teenagers are interested in mobile apps to help them monitor and manage psychiatric problems.

 on mental health chatbots evaluated 10 studies and concluded: “Overall, potential for conversational agents in psychiatric use was reported to be high across all studies. In particular, conversational agents showed potential for benefit in psychoeducation and self-adherence. In addition, satisfaction rating of chatbots was high across all studies, suggesting that they would be an effective and enjoyable tool in psychiatric treatment.” But on a more cautionary note, the analysis also found that, “given the heterogeneity of the reviewed studies, further research with standardized outcomes reporting is required to more thoroughly examine the effectiveness of conversational agents.”

One of the challenges for clinicians, patients, and consumers seeking mobile apps that address mental health is that many rely on unproven therapy techniques. Woebot is based on the principles of CBT, one of the few evidence-based approaches to mental health that has actually generated verifiable results. However, a 

 that are intended for patients with clinical depression is less than encouraging. It reviewed 117 programs that focused on depression and found only 12 of them (10.3%) offered CBT or behavioral activation, another evidence-based treatment approach. The analysis found that those apps that focused on CBT concentrated on education about depression; an explanation of how the CBT approach works; depression rating; monitoring cognitions; emotions, physical sensations and behaviors; conceptualization; and behavioral and cognitive techniques. The core ingredients of behavioral activation also include education about depression and explanation of the model, as well as “depression rating, activity monitoring, giving each activity a rating for pleasure, giving each activity a rating for mastery, activity scheduling of pleasant behaviors and activity scheduling of avoided behaviors.”

Among the many mobile apps targeting the mental health community, Deprexis stands out. It too uses CBT, as well as other well-documented approaches to mental health. It generates simulated conversations that last up to an hour.

 found that this digital “therapist” was moderately effective in relieving depressive symptoms. The evidence supporting Deprexis was strong enough to prompt Great Britain’s relatively 

conservative National Institute for Health and Care Excellence

 (NICE) to recommend that the National Health Service conduct an in-depth evaluation of the app and online program because it may serve as an “effective alternative therapy for adults with mild to moderate depression.”

 performed by Mary Rodgers, Ph.D., M.S. and her colleagues at the University of Michigan’s Department of Internal Medicine found that for every four depressive persons using Deprexis, one recovered, when compared to those who didn’t use the program. Critics still believe these findings should be duplicated in real-world clinical settings, as opposed to those found in a research protocol.

While Deprexis has attracted much attention among mental health professionals, it’s by no means the only mobile app that holds promise. 

Joseph Firth, Ph.D., an Australian investigator

 with Western Sydney University, Campbelltown, and his associates analyzed 18 RCTs and 22 mobile mental health apps and found overall, the apps significantly reduced depressive symptoms with a moderate positive effect size when compared to inactive control participants. An inactive participant refers to someone who didn’t receive any treatment during a given study, compared to an active control, who either used an app that was not designed to treat depression, was treated in person or was engaged in other activities. Firth and his colleagues found less of a difference between patients using a mental health app and someone in an active control group.

A closer look at this meta-analysis, including several subgroup analyses, reveals important insights. The RCTs that looked at smartphone apps lasted from four to 24 weeks, and depressive symptoms were measured using a variety of well-documented tools, including the Depression Anxiety Stress Scale, the Patient Health Questionnaire (PHQ-9) and the Beck Depression Inventory II scale. The meta-analysis also found that the mobile apps were only effective in users who had self-reported mild-to-moderate depression. They had no significant impact on patients with major depression, bipolar disorder or anxiety disorders.

Some of the subgroup analyses were unexpected. Apps that did not involve any in-person feedback generated statistically significant moderately positive effects, while those that did include human feedback did not. Apps that delivered their content entirely through a mobile device appear to have been more effective than those that were not self-contained, though the difference was just short of reaching statistical significance. Finally, those apps that offered cognitive training had less of an impact on users than those that focused more generally on

, conducted by Jesse Wright, M.D., Ph.D., with the University of Louisville School of Medicine in Kentucky, and his associates specifically evaluated computer-assisted CBT. They looked at 40 RCTs and found a moderately large treatment effect. They also noted that apps in which patients were supported by a clinician or other professional were more effective than apps that were used by patients without any outside help.

 that looked at the value of mental health apps, performed by German investigators, included 12 studies; it too found that the programs improved depressive symptoms. It’s important to point out, however, that none of the clinical trials that were evaluated lasted for more than 12 weeks, so it’s not possible to know whether these apps have any long-term impact. One unique aspect of this meta-analysis is that it also looked at health professionals’ attitudes toward mental health apps. The survey of 72 clinicians found that: “Significant differences were found between the level of technology experience and how much the healthcare professional would consider the use of mobile applications in clinical practice. Survey participants reported openness towards therapeutic app use but very little knowledge and experience in the field.”

Although RCTs are considered the gold standard to determine the effectiveness of any medical treatment, that doesn’t mean we should disregard less rigorous types of evidence. A mobile app called Mobilyze, for instance, which was developed by the Center for Behavioral Intervention Technologies at Northwestern Medicine, may have merit according to a 

 in which seven patients with major depression saw significant improvements in depressive and anxiety symptoms at the end of eight weeks. Also encouraging was the fact that the patients no longer met the criteria for depression spelled out in the Patient Health Questionnaire (PHQ-9).

Don’t Ignore the Therapeutic Relationship

There may be significant research support for mental health apps, but clinicians and the public need to keep in mind that a mobile app can never replace the therapeutic relationship that develops between a mental health professional and their patients. While standalone apps may benefit the casual user experiencing mild depressive symptoms, it is unlikely it will be a substitute for the kind of trusting, respective alliance that occurs in a doctor/patient relationship.

, an international authority on digital psychiatry and director of the digital psychiatry division, in the Department of Psychiatry at Beth Israel Deaconess Medical Center in Boston, points out: “The digital therapeutic relationship for mobile health has been ignored because it is often invisible to those building apps. App developers create impressive apps that are marketed to clinicians or patients as a discrete tool….While there isn’t a consensus definition of the therapeutic relationship, elements include mutual trust, alliance, respect, empathy and positive regard between the patient and clinician. The current therapeutic relationship is developed in the brick-and-mortar clinic and strengthened through future in-person clinical visits. The nature of the clinical workflow, appointments, electronic medical records, liability and billing reinforces the predominance of face-to-face therapeutic relationships.”

Torous emphasizes the importance of reframing mental health apps so that they can serve as complementary tools to strengthen the bond between therapist and patient rather than destroy it by encouraging patients to go it alone with an independent computer program — no matter how lifelike and empathetic its conversational interface may seem.

Professional Criteria Can Help Evaluate Apps

Clinicians in need of assistance as they sift through the many mental health apps can turn to the American Psychiatric Association for a 

. APA has published a list of five criteria to help users choose an app that fits their needs and those of their patient population.

While APA does not recommend specific phone apps, its five-step evaluation process can help narrow down one’s choices. The association recommends first collecting background information and then assessing the app’s risks, including the program’s potential to compromise patient privacy. Then review the evidence on effectiveness, evaluate its ease of use and finally evaluate its clinical integration — including interoperability &mdash; and how well it strengthens the therapist/patient relationship.

APA explains, “the last step in the model is interoperability. This is the topmost level, as the ability to share data only matters if this is an app that you and the patient want to use… if it is safe and secure… has some evidence base … and is easy to use…. The reason why interoperability becomes important in this model is because apps should not fragment care and the patient and psychiatrist should be able to share and discuss data or feedback from the app as appropriate. In some cases, the ability for apps to share data may not be relevant. For other apps, however (e.g., mood trackers and medication management), ensuring that such data can be easily shared and accessed by those who need to see it is an important factor to consider.”

Paul Cerrato has more than 30 years of experience working in healthcare as a clinician, educator and medical editor. He has written extensively on clinical medicine, electronic health records, protected health information security, practice management and clinical decision support. He has served as editor of Information Week Healthcare, executive editor of Contemporary OB/GYN, senior editor of RN Magazine, and contributing writer/editor for the Yale University School of Medicine, the American Academy of Pediatrics, Information Week, Medscape, Healthcare Finance News, IMedicalapps.com and Medpage Today. The Healthcare Information and Management Systems Society (HIMSS) has listed Mr. Cerrato as one of the most influential columnists in healthcare IT.

John D. Halamka, M.D., leads innovation for Beth Israel Lahey Health. Previously, he served for over 20 years as the chief information officer (CIO) at the Beth Israel Deaconess Healthcare System. He is chairman of the New England Healthcare Exchange Network (NEHEN) and a practicing emergency physician. He is also the International Healthcare Innovation professor at Harvard Medical School. As a Harvard professor, he has served the George W. Bush administration, the Obama administration and national governments throughout the world, planning their healthcare IT strategies. In his role at BIDMC, Dr. Halamka was responsible for all clinical, financial, administrative and academic information technology, serving 3,000 doctors, 12,000 employees and 1 million patients.

AbleTo Is Changing How We Use Behavioral Health Data

Finding mHealth Apps that Doctors Can Trust

An analysis of the evidence for these increasingly popular mHealth apps.

Can mHealth apps change healthcare? Possibly, but physicians must be prepared.

It may be hard to imagine a time when prescribing a mobile health (mHealth) app for patients is as common as writing a script for an antibiotic or statin, but if the quality of mHealth apps continues to improve, that future might not be too far away.

But for now, many physicians and nurses have serious doubts about the trustworthiness of mHealth apps. They question whether the apps are scientifically accurate, whether they are more effective than printed patient education materials and whether patients will use mHealth apps once the novelty wears off. On the other hand, clinicians are more willing to use medical apps that directly meet their professional needs, such as digital drug reference guides and clinical decision support systems.

READ: mHealth Apps: Trust the Studies, Not the Stars

Physicians and patients seem to view mHealth apps quite differently. One survey 

 that only one in three physicians recommends wellness apps to their patients — but 58 percent of consumers who use smartphones have downloaded at least one health app, and 41 percent have downloaded more than five.

Why the disparity? Might it be a function of doctors’ training or exposure to medical literature? A case in point is the Owlet Smart Sock, which new parents can put on a newborn to help monitor a child’s heart rate and oxygen saturation. The device sends readings to an mHealth app that lets parents track the child’s vital signs. Most pediatricians are quick to point at that this type of monitoring is completely unnecessary for a healthy infant. And the American Academy of Pediatrics’ policy statement on tracking this type of data for a healthy infant 

, “Do not use home cardiorespiratory monitors as a strategy to reduce the risk of SIDS.”

But the reluctance of health professionals to recommend mHealth apps is also prompted by their skepticism about the effects of mHealth apps on outcomes and their concerns that the apps may be unsafe or inconsistent with evidence-based clinical guidelines. Such skepticism has 

 some medical experts to refer to mHealth apps as “digital snake oil.” That extreme view isn’t justified, at least for many apps that have been tested in controlled clinical studies.

A review of mobile apps that addresses the needs of patients with clinical depression, for instance, 

 12 of the apps delivered cognitive behavioral therapy (CBT) or behavioral activation (BA), both of which are supported by scientific evidence. Anna Huguet, with the Center for Research in Family Health in Halifax Nova Scotia, and her colleagues found that two apps scored the highest in adhering to CBT or BA principles, eCBT Mood and Depression CBT Self-Help Guide. In a separate randomized clinical trial, a chatbot called WoeBot, which uses a text-based conversational agent to work with students in need of mental health services, has proven effective. Kathleen Fitzpatrick, Ph.D., and associates with Stanford School of Medicine, divided college students into two groups, one of which had access to the chatbot and the other acting as a control. After two to three weeks, students using WoeBot 

 significant improvement relative to the control group, who only had access to an e-book on depression in college students.

READ: mHealth App No Better Than Usual Care in Managing Chronic Musculoskeletal Pain

The IQVIA Institute for Human Data Science, a research organization that serves the healthcare IT industry, has also 

 studies that support mHealth apps. It used an evidence rating system that included multiple meta-analyses, followed by a single meta-analysis, multiple randomized controlled trials, a single RCT and observational studies, in descending order of strength. Among the mHealth apps it considered potentially disappointing were those designed to address exercise, pain management, dermatology, autism, schizophrenia, multiple sclerosis and autism. But IQVIA considered candidates for clinical adoption mHealth apps for weight management, asthma, COPD, congestive heart failure, stroke, arthritis, cancer, PTSD, insomnia, smoking cessation, stress management, cardiac rehabilitation and hypertension. The analysis also concluded that there were apps for diabetes, depression and anxiety that would be worth including in clinical guidelines. Finally, IQVIA published a list of top-rated mobile health apps, which included mySugr for diabetes management, SmartBP for hypertension, Headspace for stress management and KWIT for smoking cessation.

Of course, efficacy should not be the only metric used when considering an mHealth app. A recent study from 

 the value of usability ratings and user testing: “Typically, usability is measured across dimensions such as user ratings of app flexibility, operability, understandability, learnability, efficiency, satisfaction, attractiveness, consistency and error rates.” Many of these metrics can help developers create a new mHealth app that appeals to patients. So can relevant standards spelled out by the International Organization for Standardization.

When choosing an mHealth app to recommend to patients, clinicians need to be assured that it is not only effective and easy to use but 

That is not always the case. For example, if an app is designed to collect blood glucose levels, does it alert the patient if the readings are dangerously low or high? Similarly, if the app is connected to a wireless blood pressure cuff, will it tell users to see their provider when it registers very high or low readings? Karandeep Singh, M.D., of the University of Michigan Medical School, and his associates analyzed 121 mHealth apps that let patients record health-related data, 

 only 23 percent informed users that the data they entered were dangerous when they veered widely from normal values. Among the signposts that these mHealth apps ignored was a user’s suicidal mood or ideation.

Of equal concern was that in many disease categories, an mHealth app’s reactions to indications of a health danger were inadequate. For instance, researchers found that the percentage of danger alerts considered appropriate fell below 50 percent for apps used to manage cancer, dementia, hypertension, coronary heart disease, congestive heart failure, liver disease, chronic kidney disease and diabetes. Singh and his colleagues wrote, “While the app industry needs to do further work to meet basic safety and privacy standards, a subset of apps already conform to these standards. Policymakers need to consider how to encourage app developers to build apps that respond appropriately to dangerous information entered by users.”

Finally, as clinicians evaluate mHealth apps, they may want to consult respected professional organizations. For instance, the AMA has partnered with the American Heart Association and the Healthcare Information and Management Systems Society (HIMSS) to create an alliance — Xcertia &mdash; that sets guidelines to evaluate the quality, safety and effectiveness of mHealth apps.

READ: Why the Future of Digital Health Must Be Simple

The American Psychiatric Association has also taken a position on mHealth apps that can help clinicians make more informed choices. Like the AMA and its partners, APA is not rating individual apps but offering providers guidelines to help them make their own decisions. The association 

 a five-step evaluation process and recommended clinicians collect background information, take a close look at the app’s security safeguards, review the evidence, evaluate ease of use and analyze the interoperability of the product.

Enthusiasm for Doctor-Facing mHealth Apps

The list of mobile apps that physicians and nurses find useful in their professional lives is long and continues to grow. In a recent article 

™, we discussed mobile versions of UpToDate and ClinicalKey, two clinical decision support tools from Wolters Kluwer and Elsevier. AirStrip Technologies has also made a strong impression among clinicians with its remote patient monitoring application, which gives users near real-time access to bedside ECG readings, fetal monitoring data, multiple electronic health record (EHR) systems and more. They can tap into these resources from a hospital or community-based care setting, accessing medical devices, secure message systems, lab results and radiographic images as well.

Similarly, Isabel Healthcare has gained favor among physicians who realize the need for computer-based diagnostic and treatment aids. It includes a patient-facing symptom checker and several tools to help clinicians interpret diagnostic clues. Isabel Pro DDX Generator uses natural language processing to access a database of disease presentations. The clinician inputs lab readings, vital signs, co-morbidities, age, gender and other parameters in free text or from an EHR system, and the company’s algorithms use the data to suggest possible diagnoses.

Is the FDA “Seal of Approval” Enough?

In recent years, the U.S. Food and Drug Administration (FDA) has taken a stand on mHealth apps, publishing regulations to guide developers as they design them and to help clinicians as they evaluate their worth. But these regulations don’t guarantee effectiveness or safety. The agency has made clear that it will not regulate all mHealth apps. Instead, it is 

 on “most mobile apps that are intended to treat, diagnose, cure, mitigate or prevent disease or other conditions as medical devices under federal statute.” The FDA refers to these as “mobile medical apps.” On the other hand, apps that make wellness claims or apps that are designed to receive, transmit, store and provide information displays are receiving less attention from the agency. In fact, the latest FDA guidelines state that these apps will not require premarket review.

list of mobile medical apps that it has cleared

, including diagnostic spirometer GoSpiro, continuous glucose monitoring system Dexcom and drug dose calculator MyDose Coach. Apps that the FDA does not consider medical devices and therefore do not need premarket approval include medical dictionaries, first aid encyclopedias and surgical training videos. A third category, for which the agency says it will “exercise enforcement discretion”, includes patient portal apps and those “intended for individuals to log, record, track, evaluate or make decisions or behavioral suggestions related to developing or maintaining general fitness, health or wellness.”

That more lenient approach to mHealth app regulation doesn’t mean these software platforms are free from problems and scrutiny. For example, in 2017, the New York Attorney General accused vendors of three mHealth apps of misleading consumers and not adequately protecting their data. Adidas’ Runtastic, MIT Media Lab spinoff Cardiio and Matis’ My Baby’s Beat claimed to monitor heartbeat using a smartphone camera or microphone and algorithms. But all three faced legal action, 

 a combined $30,000 in penalties and agreeing to change their advertising and update their data privacy policies.

John Halamka, M.D., M.S., is the international healthcare innovation professor at Harvard Medical School, chief information officer of the Beth Israel Deaconess System, and a practicing emergency physician. He strives to improve healthcare quality, safety, and efficiency for patients, providers, and payers throughout the world using information technology. He has written 5 books, several hundred articles, and the popular Geekdoctor blog.

Get the best insights in healthcare analytics 

Podcast: Wearables Are Saving Human Lives. Can They Save Hospitals Too?

mHealth Is Powered by Potential but Dogged by Dubious Studies

Apple Unveils Health-Heavy Smartwatch with EKG Reader

Now that mHealth apps are flooding the market, clinicians and health systems must understand their benefits and detriments.